2024-04-25T15:18:47.697 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T15:18:47.698 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota} --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213 --verbose --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/orig.config.yaml 2024-04-25T15:18:47.745 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T15:18:47.833 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota} email: pdonnell@redhat.com first_in_suite: false job_id: '7673213' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGF+6zczyNRzUwUq4mHVemqqq+3Eg6daGTgqLqwuCskvRzMbWi9rZ0yYZq5Vc/jIzXC0M2JXZdKhRjZjbI0AgDQ= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCIapig+HZqUrvPepgPs4Q3bSMMG1nHS8RdgRYWO3o9F/hA/qS8LyrkryWyh3jsIsCNM79MRqkO9ImLP90vm1Yw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_quota teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T15:18:47.834 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T15:18:47.834 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T15:18:47.835 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T15:18:47.836 INFO:teuthology.task.internal:Checking packages... 2024-04-25T15:18:47.860 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T15:18:47.860 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T15:18:47.860 INFO:teuthology.packaging:ref: None 2024-04-25T15:18:47.860 INFO:teuthology.packaging:tag: None 2024-04-25T15:18:47.861 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:18:47.861 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:18:47.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:18:48.082 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T15:18:48.083 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T15:18:48.092 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T15:18:48.093 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T15:18:48.120 INFO:teuthology.task.internal:Saving configuration 2024-04-25T15:18:48.138 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T15:18:48.149 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T15:18:48.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:13:53.979530', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGF+6zczyNRzUwUq4mHVemqqq+3Eg6daGTgqLqwuCskvRzMbWi9rZ0yYZq5Vc/jIzXC0M2JXZdKhRjZjbI0AgDQ='} 2024-04-25T15:18:48.205 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:13:53.976999', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCIapig+HZqUrvPepgPs4Q3bSMMG1nHS8RdgRYWO3o9F/hA/qS8LyrkryWyh3jsIsCNM79MRqkO9ImLP90vm1Yw='} 2024-04-25T15:18:48.205 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T15:18:48.216 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-25T15:18:48.216 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-25T15:18:48.216 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T15:18:48.294 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1fc50d3f70>, signals=[15]) 2024-04-25T15:18:48.294 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T15:18:48.306 INFO:teuthology.task.internal:Opening connections... 2024-04-25T15:18:48.306 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:18:48.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:18:48.389 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:18:48.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:18:48.463 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T15:18:48.475 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-25T15:18:48.493 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-25T15:18:48.493 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-25T15:18:48.551 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-04-25T15:18:48.551 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-04-25T15:18:48.551 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T15:18:48.552 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T15:18:48.553 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-25T15:18:48.578 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-04-25T15:18:48.596 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-04-25T15:18:48.596 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-04-25T15:18:48.653 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-04-25T15:18:48.653 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T15:18:48.654 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T15:18:48.655 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-04-25T15:18:48.678 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T15:18:48.693 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T15:18:48.705 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T15:18:48.705 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:18:48.708 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:18:48.724 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T15:18:48.735 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T15:18:48.735 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:18:48.765 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:18:48.877 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T15:18:48.929 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T15:18:48.948 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T15:18:48.949 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-04-25T15:18:48.949 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-04-25T15:18:48.949 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:18:48.949 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:18:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:18:48.966 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-04-25T15:18:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:18:48.968 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-25T15:18:49.024 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-437.el9.x86_64 2024-04-25T15:18:49.024 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-437.el9.x86_64 2024-04-25T15:18:49.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-04-25T15:18:49.025 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-437.el9.x86_64 2024-04-25T15:18:49.025 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-437.el9.x86_64 2024-04-25T15:18:49.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-04-25T15:18:51.325 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-25T15:18:51.581 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-04-25T15:18:53.452 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-25T15:18:53.483 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T15:18:57.599 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-25T15:18:57.686 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T15:18:59.209 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-04-25T15:18:59.381 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-04-25T15:19:00.627 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-25T15:19:04.357 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 21 MB 00:04 2024-04-25T15:19:05.902 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-25T15:19:07.892 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T15:19:07.892 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T15:19:07.983 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-25T15:19:07.984 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-25T15:19:07.984 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-25T15:19:08.094 DEBUG:teuthology.orchestra.run.smithi067:> echo no | sudo yum reinstall kernel || true 2024-04-25T15:19:08.619 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:19:05 PM UTC. 2024-04-25T15:19:08.747 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-04-25T15:19:08.762 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-04-25T15:19:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:Operation aborted. 2024-04-25T15:19:08.823 INFO:teuthology.orchestra.run.smithi067.stdout:Is this ok [y/N]: 2024-04-25T15:19:08.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum reinstall -y kernel || true 2024-04-25T15:19:09.340 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 03:19:05 PM UTC. 2024-04-25T15:19:09.440 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-25T15:19:09.461 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:19:09.462 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-04-25T15:19:09.463 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-04-25T15:19:09.463 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-25T15:19:09.654 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64.rpm 2.0 MB/s | 255 kB 00:00 2024-04-25T15:19:09.655 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:19:09.655 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.3 MB/s | 255 kB 00:00 2024-04-25T15:19:09.676 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-25T15:19:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-25T15:19:09.864 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-25T15:19:09.869 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-25T15:19:09.870 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-25T15:19:09.994 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-25T15:19:10.072 INFO:teuthology.orchestra.run.smithi067.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:19:10.072 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:19:10.804 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:19:10.804 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:19:11.373 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T15:19:11.373 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T15:19:11.463 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-04-25T15:19:11.464 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-04-25T15:19:11.464 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-04-25T15:19:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:19:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:19:11.520 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalled: 2024-04-25T15:19:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T15:19:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:19:11.520 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-25T15:19:11.577 DEBUG:teuthology.orchestra.run.smithi176:> echo no | sudo yum reinstall kernel || true 2024-04-25T15:19:11.676 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T15:19:11.724 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T15:19:11.725 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T15:19:11.725 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T15:19:11.725 DEBUG:teuthology.parallel:result is None 2024-04-25T15:19:12.115 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:19:09 PM UTC. 2024-04-25T15:19:12.244 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-04-25T15:19:12.252 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout:Reinstalling: 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 255 k 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 0 2024-04-25T15:19:12.253 INFO:teuthology.orchestra.run.smithi176.stderr:Operation aborted. 2024-04-25T15:19:12.318 INFO:teuthology.orchestra.run.smithi176.stdout:Is this ok [y/N]: 2024-04-25T15:19:12.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum reinstall -y kernel || true 2024-04-25T15:19:12.814 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:19:09 PM UTC. 2024-04-25T15:19:12.931 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:Reinstalling: 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 255 k 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 0 2024-04-25T15:19:12.932 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-04-25T15:19:13.183 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.8 MB/s | 255 kB 00:00 2024-04-25T15:19:13.184 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:19:13.184 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.0 MB/s | 255 kB 00:00 2024-04-25T15:19:13.199 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-04-25T15:19:13.389 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-04-25T15:19:13.389 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-04-25T15:19:13.392 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-04-25T15:19:13.392 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-04-25T15:19:13.522 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-04-25T15:19:13.584 INFO:teuthology.orchestra.run.smithi176.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:19:13.585 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:19:14.319 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:19:14.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:19:15.033 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:19:15.033 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:19:15.033 INFO:teuthology.orchestra.run.smithi176.stdout:Reinstalled: 2024-04-25T15:19:15.033 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T15:19:15.033 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:19:15.033 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-04-25T15:19:15.216 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T15:19:15.263 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T15:19:15.263 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T15:19:15.263 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T15:19:15.263 DEBUG:teuthology.parallel:result is None 2024-04-25T15:19:15.263 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T15:19:15.271 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T15:19:15.272 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:19:15.274 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:19:15.291 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T15:19:15.298 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T15:19:15.307 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T15:19:15.307 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:19:15.334 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:19:15.365 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T15:19:15.372 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T15:19:15.372 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:19:15.402 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:19:15.434 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:19:15.434 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:19:15.444 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:19:15.448 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:19:15.450 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T15:19:15.457 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T15:19:15.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:19:15.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:19:15.518 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T15:19:15.527 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T15:19:15.527 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:19:15.561 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:19:15.575 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:19:15.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:19:15.708 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:19:15.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:19:15.832 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:19:15.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:19:15.897 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:19:15.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:19:15.989 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:19:16.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:19:16.111 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:19:16.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:19:16.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-25T15:19:16.177 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-04-25T15:19:16.213 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:19:16.246 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:19:16.748 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T15:19:16.756 INFO:teuthology.task.internal:Starting timer... 2024-04-25T15:19:16.756 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T15:19:16.768 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T15:19:16.781 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-25T15:19:16.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-04-25T15:19:16.843 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-04-25T15:19:16.845 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-04-25T15:19:16.903 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-04-25T15:19:16.905 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T15:19:16.905 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-04-25T15:19:16.929 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-04-25T15:19:16.930 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-04-25T15:19:16.947 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-04-25T15:19:16.947 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-04-25T15:19:16.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:19:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:19:16.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:19:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:19:17.018 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T15:19:17.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-04-25T15:19:17.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-04-25T15:19:17.083 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T15:19:17.092 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T15:19:17.100 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T15:19:17.227 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T15:19:17.242 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T15:19:17.243 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T15:22:48.563 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-04-25T15:22:48.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:22:48.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:22:48.641 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-25T15:22:48.685 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:22:48.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:22:48.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:22:48.757 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-04-25T15:22:48.792 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:22:48.793 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T15:22:48.804 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T15:22:48.804 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:22:48.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:22:48.807 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:22:48.807 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:22:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T15:22:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T15:22:48.860 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T15:22:48.869 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-04-25T15:22:48.873 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T15:22:48.880 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-04-25T15:22:48.894 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T15:22:48.895 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-04-25T15:22:48.905 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-04-25T15:22:48.907 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T15:22:48.918 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T15:22:48.931 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T15:22:48.962 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:22:48.966 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:22:48.966 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-25T15:22:48.966 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.966 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.966 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.966 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.984 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:22:48.988 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:22:48.988 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-04-25T15:22:48.988 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.988 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.988 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.988 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:22:48.988 INFO:teuthology.run_tasks:Running task install... 2024-04-25T15:22:48.996 DEBUG:teuthology.task.install:project ceph 2024-04-25T15:22:48.997 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T15:22:48.997 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T15:22:48.997 INFO:teuthology.task.install:Using flavor: default 2024-04-25T15:22:49.003 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T15:22:49.003 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T15:22:49.003 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T15:22:49.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:22:49.006 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T15:22:49.006 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:22:49.152 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T15:22:49.152 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T15:22:49.224 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T15:22:49.224 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T15:22:49.263 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T15:22:49.264 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:22:49.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T15:22:49.304 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T15:22:49.304 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T15:22:49.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T15:22:49.402 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T15:22:49.402 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:22:49.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T15:22:49.412 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T15:22:49.429 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T15:22:49.430 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T15:22:49.479 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-04-25T15:22:49.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-04-25T15:22:49.504 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T15:22:49.581 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T15:22:49.655 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-04-25T15:22:49.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-04-25T15:22:49.725 INFO:teuthology.orchestra.run.smithi067.stdout:48 files removed 2024-04-25T15:22:49.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T15:22:49.886 INFO:teuthology.orchestra.run.smithi176.stdout:48 files removed 2024-04-25T15:22:49.913 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T15:22:50.452 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 196 kB/s | 84 kB 00:00 2024-04-25T15:22:50.491 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 263 kB/s | 84 kB 00:00 2024-04-25T15:22:50.844 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-04-25T15:22:50.933 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-25T15:22:51.203 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-25T15:22:51.292 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-25T15:22:51.822 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-25T15:22:51.911 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-25T15:22:53.799 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T15:22:54.012 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-25T15:22:58.208 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.2 MB 00:00 2024-04-25T15:22:58.355 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.2 MB 00:00 2024-04-25T15:22:59.723 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-04-25T15:22:59.953 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-25T15:23:02.110 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-04-25T15:23:02.207 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-04-25T15:23:07.271 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-25T15:23:07.524 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-04-25T15:23:09.422 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:23:09.422 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:23:09.433 INFO:teuthology.orchestra.run.smithi176.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T15:23:09.433 INFO:teuthology.orchestra.run.smithi176.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T15:23:09.434 INFO:teuthology.orchestra.run.smithi176.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T15:23:09.435 INFO:teuthology.orchestra.run.smithi176.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T15:23:09.437 INFO:teuthology.orchestra.run.smithi176.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T15:23:09.439 INFO:teuthology.orchestra.run.smithi176.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T15:23:09.440 INFO:teuthology.orchestra.run.smithi176.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T15:23:09.501 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-04-25T15:23:09.509 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T15:23:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T15:23:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T15:23:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T15:23:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T15:23:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T15:23:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T15:23:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T15:23:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T15:23:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T15:23:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T15:23:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================= 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout:Install 148 Packages 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 4 Packages 2024-04-25T15:23:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:23:09.522 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 214 M 2024-04-25T15:23:09.522 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-04-25T15:23:09.743 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:23:09.744 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:23:09.755 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T15:23:09.756 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T15:23:09.756 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T15:23:09.757 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T15:23:09.759 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T15:23:09.761 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T15:23:09.762 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T15:23:09.826 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-25T15:23:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-04-25T15:23:09.835 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-25T15:23:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-04-25T15:23:09.835 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-25T15:23:09.835 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T15:23:09.836 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T15:23:09.837 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-25T15:23:09.838 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T15:23:09.839 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T15:23:09.840 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T15:23:09.841 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T15:23:09.842 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T15:23:09.842 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T15:23:09.842 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T15:23:09.842 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T15:23:09.842 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T15:23:09.842 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T15:23:09.843 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T15:23:09.844 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T15:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T15:23:09.846 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:Install 148 Packages 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 4 Packages 2024-04-25T15:23:09.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:23:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 214 M 2024-04-25T15:23:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-25T15:23:10.543 INFO:teuthology.orchestra.run.smithi067.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-04-25T15:23:10.910 INFO:teuthology.orchestra.run.smithi067.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 2.3 MB/s | 863 kB 00:00 2024-04-25T15:23:10.993 INFO:teuthology.orchestra.run.smithi067.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 1.7 MB/s | 147 kB 00:00 2024-04-25T15:23:11.022 INFO:teuthology.orchestra.run.smithi176.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-25T15:23:11.710 INFO:teuthology.orchestra.run.smithi067.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 3.9 MB/s | 5.4 MB 00:01 2024-04-25T15:23:11.760 INFO:teuthology.orchestra.run.smithi067.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 2.9 MB/s | 2.3 MB 00:00 2024-04-25T15:23:11.989 INFO:teuthology.orchestra.run.smithi176.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 893 kB/s | 863 kB 00:00 2024-04-25T15:23:12.172 INFO:teuthology.orchestra.run.smithi176.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 802 kB/s | 147 kB 00:00 2024-04-25T15:23:12.247 INFO:teuthology.orchestra.run.smithi176.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 3.5 MB/s | 5.4 MB 00:01 2024-04-25T15:23:12.569 INFO:teuthology.orchestra.run.smithi067.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-04-25T15:23:12.589 INFO:teuthology.orchestra.run.smithi176.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 4.4 MB/s | 1.5 MB 00:00 2024-04-25T15:23:13.244 INFO:teuthology.orchestra.run.smithi067.stdout:(7/152): ceph-common-19.0.0-3291.g994e2f20.el9. 6.7 MB/s | 20 MB 00:02 2024-04-25T15:23:13.602 INFO:teuthology.orchestra.run.smithi067.stdout:(8/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 2.6 MB/s | 4.9 MB 00:01 2024-04-25T15:23:13.719 INFO:teuthology.orchestra.run.smithi067.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 216 kB/s | 25 kB 00:00 2024-04-25T15:23:13.981 INFO:teuthology.orchestra.run.smithi176.stdout:(6/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 1.2 MB/s | 2.3 MB 00:01 2024-04-25T15:23:14.061 INFO:teuthology.orchestra.run.smithi067.stdout:(10/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 12 MB/s | 10 MB 00:00 2024-04-25T15:23:14.131 INFO:teuthology.orchestra.run.smithi176.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 3.2 MB/s | 4.9 MB 00:01 2024-04-25T15:23:14.136 INFO:teuthology.orchestra.run.smithi067.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.9 MB/s | 224 kB 00:00 2024-04-25T15:23:14.211 INFO:teuthology.orchestra.run.smithi067.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 361 kB/s | 27 kB 00:00 2024-04-25T15:23:14.279 INFO:teuthology.orchestra.run.smithi067.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 475 kB/s | 32 kB 00:00 2024-04-25T15:23:14.387 INFO:teuthology.orchestra.run.smithi067.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 6.8 MB/s | 756 kB 00:00 2024-04-25T15:23:14.462 INFO:teuthology.orchestra.run.smithi067.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.1 MB/s | 158 kB 00:00 2024-04-25T15:23:14.538 INFO:teuthology.orchestra.run.smithi067.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.7 MB/s | 127 kB 00:00 2024-04-25T15:23:14.630 INFO:teuthology.orchestra.run.smithi067.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 5.3 MB/s | 497 kB 00:00 2024-04-25T15:23:15.230 INFO:teuthology.orchestra.run.smithi067.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 8.6 MB/s | 5.2 MB 00:00 2024-04-25T15:23:15.297 INFO:teuthology.orchestra.run.smithi067.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 679 kB/s | 45 kB 00:00 2024-04-25T15:23:15.372 INFO:teuthology.orchestra.run.smithi067.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.8 MB/s | 135 kB 00:00 2024-04-25T15:23:15.447 INFO:teuthology.orchestra.run.smithi067.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.1 MB/s | 161 kB 00:00 2024-04-25T15:23:15.531 INFO:teuthology.orchestra.run.smithi067.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 3.8 MB/s | 321 kB 00:00 2024-04-25T15:23:15.606 INFO:teuthology.orchestra.run.smithi067.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 3.9 MB/s | 298 kB 00:00 2024-04-25T15:23:15.682 INFO:teuthology.orchestra.run.smithi176.stdout:(8/152): ceph-common-19.0.0-3291.g994e2f20.el9. 3.9 MB/s | 20 MB 00:04 2024-04-25T15:23:15.682 INFO:teuthology.orchestra.run.smithi067.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.3 MB/s | 100 kB 00:00 2024-04-25T15:23:15.732 INFO:teuthology.orchestra.run.smithi176.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 508 kB/s | 25 kB 00:00 2024-04-25T15:23:15.757 INFO:teuthology.orchestra.run.smithi067.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.1 MB/s | 86 kB 00:00 2024-04-25T15:23:16.207 INFO:teuthology.orchestra.run.smithi067.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 6.8 MB/s | 3.0 MB 00:00 2024-04-25T15:23:16.232 INFO:teuthology.orchestra.run.smithi176.stdout:(10/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 4.8 MB/s | 10 MB 00:02 2024-04-25T15:23:16.298 INFO:teuthology.orchestra.run.smithi176.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 3.3 MB/s | 224 kB 00:00 2024-04-25T15:23:16.308 INFO:teuthology.orchestra.run.smithi067.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 1.6 MB/s | 169 kB 00:00 2024-04-25T15:23:16.340 INFO:teuthology.orchestra.run.smithi176.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 650 kB/s | 27 kB 00:00 2024-04-25T15:23:16.382 INFO:teuthology.orchestra.run.smithi176.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 766 kB/s | 32 kB 00:00 2024-04-25T15:23:16.383 INFO:teuthology.orchestra.run.smithi067.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 457 kB/s | 34 kB 00:00 2024-04-25T15:23:16.467 INFO:teuthology.orchestra.run.smithi067.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.7 MB/s | 142 kB 00:00 2024-04-25T15:23:16.710 INFO:teuthology.orchestra.run.smithi176.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 2.3 MB/s | 756 kB 00:00 2024-04-25T15:23:16.783 INFO:teuthology.orchestra.run.smithi176.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.1 MB/s | 158 kB 00:00 2024-04-25T15:23:16.833 INFO:teuthology.orchestra.run.smithi176.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 2.5 MB/s | 127 kB 00:00 2024-04-25T15:23:16.834 INFO:teuthology.orchestra.run.smithi067.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 9.7 MB/s | 3.5 MB 00:00 2024-04-25T15:23:17.008 INFO:teuthology.orchestra.run.smithi176.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 2.8 MB/s | 497 kB 00:00 2024-04-25T15:23:17.276 INFO:teuthology.orchestra.run.smithi067.stdout:(31/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 3.8 MB/s | 18 MB 00:04 2024-04-25T15:23:17.384 INFO:teuthology.orchestra.run.smithi067.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.3 MB/s | 249 kB 00:00 2024-04-25T15:23:17.493 INFO:teuthology.orchestra.run.smithi067.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 455 kB/s | 49 kB 00:00 2024-04-25T15:23:17.593 INFO:teuthology.orchestra.run.smithi067.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 164 kB/s | 16 kB 00:00 2024-04-25T15:23:17.710 INFO:teuthology.orchestra.run.smithi067.stdout:(35/152): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-04-25T15:23:17.777 INFO:teuthology.orchestra.run.smithi067.stdout:(36/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.5 MB/s | 273 kB 00:00 2024-04-25T15:23:17.886 INFO:teuthology.orchestra.run.smithi067.stdout:(37/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 4.3 MB/s | 762 kB 00:00 2024-04-25T15:23:17.910 INFO:teuthology.orchestra.run.smithi067.stdout:(38/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 137 kB/s | 18 kB 00:00 2024-04-25T15:23:18.028 INFO:teuthology.orchestra.run.smithi067.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 80 kB 00:00 2024-04-25T15:23:18.086 INFO:teuthology.orchestra.run.smithi067.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-25T15:23:18.111 INFO:teuthology.orchestra.run.smithi067.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-25T15:23:18.137 INFO:teuthology.orchestra.run.smithi067.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-25T15:23:18.187 INFO:teuthology.orchestra.run.smithi067.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-04-25T15:23:18.212 INFO:teuthology.orchestra.run.smithi067.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-25T15:23:18.229 INFO:teuthology.orchestra.run.smithi067.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T15:23:18.255 INFO:teuthology.orchestra.run.smithi067.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T15:23:18.258 INFO:teuthology.orchestra.run.smithi176.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 4.1 MB/s | 5.2 MB 00:01 2024-04-25T15:23:18.280 INFO:teuthology.orchestra.run.smithi067.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T15:23:18.313 INFO:teuthology.orchestra.run.smithi067.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-25T15:23:18.325 INFO:teuthology.orchestra.run.smithi176.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 677 kB/s | 45 kB 00:00 2024-04-25T15:23:18.347 INFO:teuthology.orchestra.run.smithi067.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-25T15:23:18.372 INFO:teuthology.orchestra.run.smithi067.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T15:23:18.398 INFO:teuthology.orchestra.run.smithi067.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T15:23:18.409 INFO:teuthology.orchestra.run.smithi176.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.6 MB/s | 135 kB 00:00 2024-04-25T15:23:18.423 INFO:teuthology.orchestra.run.smithi067.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T15:23:18.440 INFO:teuthology.orchestra.run.smithi067.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 797 kB/s | 33 kB 00:00 2024-04-25T15:23:18.465 INFO:teuthology.orchestra.run.smithi067.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 743 kB/s | 18 kB 00:00 2024-04-25T15:23:18.501 INFO:teuthology.orchestra.run.smithi176.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.7 MB/s | 161 kB 00:00 2024-04-25T15:23:18.540 INFO:teuthology.orchestra.run.smithi067.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-04-25T15:23:18.566 INFO:teuthology.orchestra.run.smithi067.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 112 kB/s | 11 kB 00:00 2024-04-25T15:23:18.583 INFO:teuthology.orchestra.run.smithi067.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-04-25T15:23:18.608 INFO:teuthology.orchestra.run.smithi067.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-25T15:23:18.633 INFO:teuthology.orchestra.run.smithi067.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T15:23:18.651 INFO:teuthology.orchestra.run.smithi176.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 2.1 MB/s | 321 kB 00:00 2024-04-25T15:23:18.667 INFO:teuthology.orchestra.run.smithi067.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 582 kB 00:00 2024-04-25T15:23:18.693 INFO:teuthology.orchestra.run.smithi067.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 444 kB/s | 25 kB 00:00 2024-04-25T15:23:18.718 INFO:teuthology.orchestra.run.smithi067.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-25T15:23:18.743 INFO:teuthology.orchestra.run.smithi067.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T15:23:18.768 INFO:teuthology.orchestra.run.smithi067.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T15:23:18.794 INFO:teuthology.orchestra.run.smithi067.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 707 kB/s | 35 kB 00:00 2024-04-25T15:23:18.818 INFO:teuthology.orchestra.run.smithi176.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 1.7 MB/s | 298 kB 00:00 2024-04-25T15:23:18.919 INFO:teuthology.orchestra.run.smithi176.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.0 MB/s | 100 kB 00:00 2024-04-25T15:23:18.919 INFO:teuthology.orchestra.run.smithi067.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 36 MB/s | 4.4 MB 00:00 2024-04-25T15:23:18.986 INFO:teuthology.orchestra.run.smithi067.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-04-25T15:23:19.019 INFO:teuthology.orchestra.run.smithi176.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 863 kB/s | 86 kB 00:00 2024-04-25T15:23:19.028 INFO:teuthology.orchestra.run.smithi067.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-25T15:23:19.061 INFO:teuthology.orchestra.run.smithi067.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-25T15:23:19.087 INFO:teuthology.orchestra.run.smithi067.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-25T15:23:19.179 INFO:teuthology.orchestra.run.smithi067.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-25T15:23:19.254 INFO:teuthology.orchestra.run.smithi067.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 30 MB/s | 5.8 MB 00:00 2024-04-25T15:23:19.279 INFO:teuthology.orchestra.run.smithi067.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 896 kB/s | 87 kB 00:00 2024-04-25T15:23:19.321 INFO:teuthology.orchestra.run.smithi067.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 6.0 MB/s | 244 kB 00:00 2024-04-25T15:23:19.363 INFO:teuthology.orchestra.run.smithi067.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 6.2 MB/s | 248 kB 00:00 2024-04-25T15:23:19.455 INFO:teuthology.orchestra.run.smithi067.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 6.0 MB 00:00 2024-04-25T15:23:19.480 INFO:teuthology.orchestra.run.smithi067.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 224 kB/s | 26 kB 00:00 2024-04-25T15:23:19.506 INFO:teuthology.orchestra.run.smithi067.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-25T15:23:19.531 INFO:teuthology.orchestra.run.smithi067.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-25T15:23:19.572 INFO:teuthology.orchestra.run.smithi067.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 523 kB/s | 35 kB 00:00 2024-04-25T15:23:19.619 INFO:teuthology.orchestra.run.smithi176.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 5.1 MB/s | 3.0 MB 00:00 2024-04-25T15:23:19.631 INFO:teuthology.orchestra.run.smithi067.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.4 MB/s | 250 kB 00:00 2024-04-25T15:23:19.686 INFO:teuthology.orchestra.run.smithi176.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.5 MB/s | 169 kB 00:00 2024-04-25T15:23:19.706 INFO:teuthology.orchestra.run.smithi067.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-04-25T15:23:19.723 INFO:teuthology.orchestra.run.smithi067.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-25T15:23:19.745 INFO:teuthology.orchestra.run.smithi176.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 583 kB/s | 34 kB 00:00 2024-04-25T15:23:19.749 INFO:teuthology.orchestra.run.smithi067.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-04-25T15:23:19.774 INFO:teuthology.orchestra.run.smithi067.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 837 kB/s | 42 kB 00:00 2024-04-25T15:23:19.800 INFO:teuthology.orchestra.run.smithi067.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T15:23:19.812 INFO:teuthology.orchestra.run.smithi176.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.1 MB/s | 142 kB 00:00 2024-04-25T15:23:19.816 INFO:teuthology.orchestra.run.smithi067.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.8 MB/s | 159 kB 00:00 2024-04-25T15:23:19.842 INFO:teuthology.orchestra.run.smithi067.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 6.5 MB/s | 279 kB 00:00 2024-04-25T15:23:19.858 INFO:teuthology.orchestra.run.smithi067.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-25T15:23:19.884 INFO:teuthology.orchestra.run.smithi067.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-04-25T15:23:19.926 INFO:teuthology.orchestra.run.smithi067.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-25T15:23:19.976 INFO:teuthology.orchestra.run.smithi067.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 928 kB/s | 44 kB 00:00 2024-04-25T15:23:20.043 INFO:teuthology.orchestra.run.smithi067.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.4 MB/s | 221 kB 00:00 2024-04-25T15:23:20.110 INFO:teuthology.orchestra.run.smithi067.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 560 kB/s | 36 kB 00:00 2024-04-25T15:23:20.302 INFO:teuthology.orchestra.run.smithi067.stdout:(95/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-25T15:23:20.327 INFO:teuthology.orchestra.run.smithi067.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 221 kB/s | 47 kB 00:00 2024-04-25T15:23:20.353 INFO:teuthology.orchestra.run.smithi067.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T15:23:20.378 INFO:teuthology.orchestra.run.smithi067.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T15:23:20.395 INFO:teuthology.orchestra.run.smithi067.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-25T15:23:20.420 INFO:teuthology.orchestra.run.smithi067.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-25T15:23:20.445 INFO:teuthology.orchestra.run.smithi067.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-04-25T15:23:20.470 INFO:teuthology.orchestra.run.smithi067.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-04-25T15:23:20.513 INFO:teuthology.orchestra.run.smithi067.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-04-25T15:23:20.755 INFO:teuthology.orchestra.run.smithi067.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-04-25T15:23:20.896 INFO:teuthology.orchestra.run.smithi067.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-25T15:23:20.922 INFO:teuthology.orchestra.run.smithi067.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-04-25T15:23:20.980 INFO:teuthology.orchestra.run.smithi067.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-04-25T15:23:21.039 INFO:teuthology.orchestra.run.smithi067.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-04-25T15:23:21.087 INFO:teuthology.orchestra.run.smithi176.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 2.8 MB/s | 3.5 MB 00:01 2024-04-25T15:23:21.130 INFO:teuthology.orchestra.run.smithi067.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-04-25T15:23:21.148 INFO:teuthology.orchestra.run.smithi067.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-04-25T15:23:21.181 INFO:teuthology.orchestra.run.smithi067.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-25T15:23:21.248 INFO:teuthology.orchestra.run.smithi067.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-25T15:23:21.282 INFO:teuthology.orchestra.run.smithi067.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-04-25T15:23:21.299 INFO:teuthology.orchestra.run.smithi067.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 863 kB/s | 43 kB 00:00 2024-04-25T15:23:21.332 INFO:teuthology.orchestra.run.smithi067.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-04-25T15:23:21.359 INFO:teuthology.orchestra.run.smithi067.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 247 kB/s | 15 kB 00:00 2024-04-25T15:23:21.393 INFO:teuthology.orchestra.run.smithi067.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.8 MB/s | 172 kB 00:00 2024-04-25T15:23:21.426 INFO:teuthology.orchestra.run.smithi067.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-25T15:23:21.452 INFO:teuthology.orchestra.run.smithi067.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-25T15:23:21.486 INFO:teuthology.orchestra.run.smithi067.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-04-25T15:23:21.511 INFO:teuthology.orchestra.run.smithi067.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-25T15:23:21.578 INFO:teuthology.orchestra.run.smithi067.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-25T15:23:21.645 INFO:teuthology.orchestra.run.smithi067.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-04-25T15:23:21.670 INFO:teuthology.orchestra.run.smithi067.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-25T15:23:21.695 INFO:teuthology.orchestra.run.smithi067.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-25T15:23:21.729 INFO:teuthology.orchestra.run.smithi067.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 395 kB/s | 23 kB 00:00 2024-04-25T15:23:21.754 INFO:teuthology.orchestra.run.smithi067.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 331 kB/s | 19 kB 00:00 2024-04-25T15:23:21.788 INFO:teuthology.orchestra.run.smithi067.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-04-25T15:23:21.821 INFO:teuthology.orchestra.run.smithi067.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-04-25T15:23:21.846 INFO:teuthology.orchestra.run.smithi067.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-04-25T15:23:21.880 INFO:teuthology.orchestra.run.smithi067.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-25T15:23:21.905 INFO:teuthology.orchestra.run.smithi067.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-04-25T15:23:21.939 INFO:teuthology.orchestra.run.smithi067.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-04-25T15:23:21.964 INFO:teuthology.orchestra.run.smithi067.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-04-25T15:23:21.979 INFO:teuthology.orchestra.run.smithi176.stdout:(31/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 2.2 MB/s | 18 MB 00:07 2024-04-25T15:23:21.998 INFO:teuthology.orchestra.run.smithi067.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-25T15:23:22.023 INFO:teuthology.orchestra.run.smithi067.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-04-25T15:23:22.057 INFO:teuthology.orchestra.run.smithi067.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 527 kB/s | 31 kB 00:00 2024-04-25T15:23:22.082 INFO:teuthology.orchestra.run.smithi067.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-25T15:23:22.115 INFO:teuthology.orchestra.run.smithi067.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-25T15:23:22.141 INFO:teuthology.orchestra.run.smithi067.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-04-25T15:23:22.174 INFO:teuthology.orchestra.run.smithi067.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 873 kB/s | 51 kB 00:00 2024-04-25T15:23:22.187 INFO:teuthology.orchestra.run.smithi176.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 1.2 MB/s | 249 kB 00:00 2024-04-25T15:23:22.199 INFO:teuthology.orchestra.run.smithi067.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-25T15:23:22.233 INFO:teuthology.orchestra.run.smithi067.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-25T15:23:22.258 INFO:teuthology.orchestra.run.smithi067.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-04-25T15:23:22.292 INFO:teuthology.orchestra.run.smithi067.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-04-25T15:23:22.313 INFO:teuthology.orchestra.run.smithi176.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 396 kB/s | 49 kB 00:00 2024-04-25T15:23:22.326 INFO:teuthology.orchestra.run.smithi067.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-25T15:23:22.376 INFO:teuthology.orchestra.run.smithi067.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-25T15:23:22.438 INFO:teuthology.orchestra.run.smithi176.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 131 kB/s | 16 kB 00:00 2024-04-25T15:23:22.646 INFO:teuthology.orchestra.run.smithi176.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.3 MB/s | 273 kB 00:00 2024-04-25T15:23:23.005 INFO:teuthology.orchestra.run.smithi176.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 3.9 MB/s | 7.4 MB 00:01 2024-04-25T15:23:23.081 INFO:teuthology.orchestra.run.smithi176.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 244 kB/s | 18 kB 00:00 2024-04-25T15:23:23.105 INFO:teuthology.orchestra.run.smithi176.stdout:(38/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 1.6 MB/s | 762 kB 00:00 2024-04-25T15:23:23.306 INFO:teuthology.orchestra.run.smithi176.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 357 kB/s | 80 kB 00:00 2024-04-25T15:23:23.326 INFO:teuthology.orchestra.run.smithi067.stdout:(148/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 3.3 MB/s | 3.1 MB 00:00 2024-04-25T15:23:23.398 INFO:teuthology.orchestra.run.smithi176.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-25T15:23:23.419 INFO:teuthology.orchestra.run.smithi067.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 15 MB/s | 1.3 MB 00:00 2024-04-25T15:23:23.423 INFO:teuthology.orchestra.run.smithi176.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-04-25T15:23:23.490 INFO:teuthology.orchestra.run.smithi176.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-04-25T15:23:23.615 INFO:teuthology.orchestra.run.smithi176.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 6.4 MB/s | 1.2 MB 00:00 2024-04-25T15:23:23.620 INFO:teuthology.orchestra.run.smithi067.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-25T15:23:23.632 INFO:teuthology.orchestra.run.smithi176.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 754 kB/s | 106 kB 00:00 2024-04-25T15:23:23.657 INFO:teuthology.orchestra.run.smithi176.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T15:23:23.682 INFO:teuthology.orchestra.run.smithi176.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T15:23:23.708 INFO:teuthology.orchestra.run.smithi176.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T15:23:23.758 INFO:teuthology.orchestra.run.smithi176.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 9.0 MB/s | 689 kB 00:00 2024-04-25T15:23:23.809 INFO:teuthology.orchestra.run.smithi176.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 9.2 MB/s | 943 kB 00:00 2024-04-25T15:23:23.834 INFO:teuthology.orchestra.run.smithi176.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-25T15:23:23.844 INFO:teuthology.orchestra.run.smithi067.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 2.2 MB/s | 3.3 MB 00:01 2024-04-25T15:23:23.859 INFO:teuthology.orchestra.run.smithi176.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-04-25T15:23:23.884 INFO:teuthology.orchestra.run.smithi176.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T15:23:23.910 INFO:teuthology.orchestra.run.smithi176.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 661 kB/s | 33 kB 00:00 2024-04-25T15:23:24.002 INFO:teuthology.orchestra.run.smithi176.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-04-25T15:23:24.027 INFO:teuthology.orchestra.run.smithi176.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 157 kB/s | 18 kB 00:00 2024-04-25T15:23:24.052 INFO:teuthology.orchestra.run.smithi176.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-25T15:23:24.078 INFO:teuthology.orchestra.run.smithi176.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T15:23:24.103 INFO:teuthology.orchestra.run.smithi176.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 911 kB/s | 45 kB 00:00 2024-04-25T15:23:24.136 INFO:teuthology.orchestra.run.smithi176.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T15:23:24.170 INFO:teuthology.orchestra.run.smithi176.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 582 kB 00:00 2024-04-25T15:23:24.220 INFO:teuthology.orchestra.run.smithi176.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 308 kB/s | 25 kB 00:00 2024-04-25T15:23:24.246 INFO:teuthology.orchestra.run.smithi176.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 631 kB/s | 47 kB 00:00 2024-04-25T15:23:24.279 INFO:teuthology.orchestra.run.smithi176.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-25T15:23:24.304 INFO:teuthology.orchestra.run.smithi176.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-04-25T15:23:24.688 INFO:teuthology.orchestra.run.smithi176.stdout:(65/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 4.8 MB/s | 43 MB 00:08 2024-04-25T15:23:24.713 INFO:teuthology.orchestra.run.smithi176.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 81 kB/s | 35 kB 00:00 2024-04-25T15:23:24.805 INFO:teuthology.orchestra.run.smithi176.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 9.5 MB/s | 4.7 MB 00:00 2024-04-25T15:23:24.930 INFO:teuthology.orchestra.run.smithi176.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-04-25T15:23:24.972 INFO:teuthology.orchestra.run.smithi176.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 6.2 MB/s | 1.6 MB 00:00 2024-04-25T15:23:25.006 INFO:teuthology.orchestra.run.smithi176.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 4.5 MB/s | 911 kB 00:00 2024-04-25T15:23:25.040 INFO:teuthology.orchestra.run.smithi176.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-25T15:23:25.215 INFO:teuthology.orchestra.run.smithi176.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.9 MB/s | 1.0 MB 00:00 2024-04-25T15:23:25.290 INFO:teuthology.orchestra.run.smithi176.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-04-25T15:23:25.315 INFO:teuthology.orchestra.run.smithi176.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 318 kB/s | 87 kB 00:00 2024-04-25T15:23:25.474 INFO:teuthology.orchestra.run.smithi176.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-04-25T15:23:25.499 INFO:teuthology.orchestra.run.smithi176.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-25T15:23:25.524 INFO:teuthology.orchestra.run.smithi176.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 248 kB 00:00 2024-04-25T15:23:25.541 INFO:teuthology.orchestra.run.smithi176.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 387 kB/s | 26 kB 00:00 2024-04-25T15:23:25.583 INFO:teuthology.orchestra.run.smithi176.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-04-25T15:23:25.617 INFO:teuthology.orchestra.run.smithi176.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-04-25T15:23:25.633 INFO:teuthology.orchestra.run.smithi176.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 376 kB/s | 35 kB 00:00 2024-04-25T15:23:25.859 INFO:teuthology.orchestra.run.smithi176.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-04-25T15:23:25.884 INFO:teuthology.orchestra.run.smithi176.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-04-25T15:23:25.909 INFO:teuthology.orchestra.run.smithi176.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 811 kB/s | 222 kB 00:00 2024-04-25T15:23:25.951 INFO:teuthology.orchestra.run.smithi176.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-04-25T15:23:25.968 INFO:teuthology.orchestra.run.smithi176.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 725 kB/s | 42 kB 00:00 2024-04-25T15:23:25.993 INFO:teuthology.orchestra.run.smithi176.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-04-25T15:23:26.018 INFO:teuthology.orchestra.run.smithi176.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-25T15:23:26.044 INFO:teuthology.orchestra.run.smithi176.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T15:23:26.060 INFO:teuthology.orchestra.run.smithi176.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 763 kB/s | 51 kB 00:00 2024-04-25T15:23:26.077 INFO:teuthology.orchestra.run.smithi176.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 922 kB/s | 54 kB 00:00 2024-04-25T15:23:26.419 INFO:teuthology.orchestra.run.smithi176.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 122 kB/s | 42 kB 00:00 2024-04-25T15:23:26.436 INFO:teuthology.orchestra.run.smithi176.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 123 kB/s | 44 kB 00:00 2024-04-25T15:23:26.636 INFO:teuthology.orchestra.run.smithi176.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-04-25T15:23:26.662 INFO:teuthology.orchestra.run.smithi176.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 912 kB/s | 221 kB 00:00 2024-04-25T15:23:26.686 INFO:teuthology.orchestra.run.smithi176.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 145 kB/s | 36 kB 00:00 2024-04-25T15:23:26.712 INFO:teuthology.orchestra.run.smithi176.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 636 kB/s | 47 kB 00:00 2024-04-25T15:23:26.737 INFO:teuthology.orchestra.run.smithi176.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-25T15:23:26.744 INFO:teuthology.orchestra.run.smithi067.stdout:(152/152): ceph-test-19.0.0-3291.g994e2f20.el9. 3.3 MB/s | 43 MB 00:13 2024-04-25T15:23:26.749 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:23:26.749 INFO:teuthology.orchestra.run.smithi067.stdout:Total 13 MB/s | 214 MB 00:16 2024-04-25T15:23:26.837 INFO:teuthology.orchestra.run.smithi176.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 385 kB/s | 58 kB 00:00 2024-04-25T15:23:26.863 INFO:teuthology.orchestra.run.smithi176.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 626 kB/s | 94 kB 00:00 2024-04-25T15:23:26.888 INFO:teuthology.orchestra.run.smithi176.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T15:23:26.913 INFO:teuthology.orchestra.run.smithi176.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 597 kB/s | 45 kB 00:00 2024-04-25T15:23:26.938 INFO:teuthology.orchestra.run.smithi176.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-04-25T15:23:27.005 INFO:teuthology.orchestra.run.smithi176.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.2 MB/s | 863 kB 00:00 2024-04-25T15:23:27.139 INFO:teuthology.orchestra.run.smithi176.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-04-25T15:23:27.256 INFO:teuthology.orchestra.run.smithi176.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 976 kB/s | 309 kB 00:00 2024-04-25T15:23:27.306 INFO:teuthology.orchestra.run.smithi176.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-04-25T15:23:27.340 INFO:teuthology.orchestra.run.smithi176.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 310 kB/s | 26 kB 00:00 2024-04-25T15:23:27.390 INFO:teuthology.orchestra.run.smithi176.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-04-25T15:23:27.415 INFO:teuthology.orchestra.run.smithi176.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 900 kB/s | 67 kB 00:00 2024-04-25T15:23:27.474 INFO:teuthology.orchestra.run.smithi176.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-25T15:23:27.582 INFO:teuthology.orchestra.run.smithi176.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-04-25T15:23:27.607 INFO:teuthology.orchestra.run.smithi176.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-25T15:23:27.700 INFO:teuthology.orchestra.run.smithi176.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 351 kB/s | 32 kB 00:00 2024-04-25T15:23:27.760 INFO:teuthology.orchestra.run.smithi176.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 248 kB/s | 15 kB 00:00 2024-04-25T15:23:27.844 INFO:teuthology.orchestra.run.smithi176.stdout:(116/152): libarrow-9.0.0-10.el9.x86_64.rpm 6.3 MB/s | 4.4 MB 00:00 2024-04-25T15:23:27.869 INFO:teuthology.orchestra.run.smithi176.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-25T15:23:27.890 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-25T15:23:27.902 INFO:teuthology.orchestra.run.smithi176.stdout:(118/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 136 kB/s | 43 kB 00:00 2024-04-25T15:23:27.928 INFO:teuthology.orchestra.run.smithi176.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T15:23:27.961 INFO:teuthology.orchestra.run.smithi176.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-25T15:23:27.978 INFO:teuthology.orchestra.run.smithi176.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-04-25T15:23:28.004 INFO:teuthology.orchestra.run.smithi176.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-25T15:23:28.050 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-25T15:23:28.050 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-25T15:23:28.054 INFO:teuthology.orchestra.run.smithi176.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-25T15:23:28.071 INFO:teuthology.orchestra.run.smithi176.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-25T15:23:28.104 INFO:teuthology.orchestra.run.smithi176.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-25T15:23:28.121 INFO:teuthology.orchestra.run.smithi176.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-25T15:23:28.171 INFO:teuthology.orchestra.run.smithi176.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-04-25T15:23:28.197 INFO:teuthology.orchestra.run.smithi176.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-25T15:23:28.222 INFO:teuthology.orchestra.run.smithi176.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-25T15:23:28.272 INFO:teuthology.orchestra.run.smithi176.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-25T15:23:28.297 INFO:teuthology.orchestra.run.smithi176.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 618 kB/s | 46 kB 00:00 2024-04-25T15:23:28.323 INFO:teuthology.orchestra.run.smithi176.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 786 kB/s | 79 kB 00:00 2024-04-25T15:23:28.348 INFO:teuthology.orchestra.run.smithi176.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-04-25T15:23:28.383 INFO:teuthology.orchestra.run.smithi176.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-04-25T15:23:28.407 INFO:teuthology.orchestra.run.smithi176.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-25T15:23:28.432 INFO:teuthology.orchestra.run.smithi176.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-25T15:23:28.457 INFO:teuthology.orchestra.run.smithi176.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-25T15:23:28.482 INFO:teuthology.orchestra.run.smithi176.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-25T15:23:28.508 INFO:teuthology.orchestra.run.smithi176.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T15:23:28.533 INFO:teuthology.orchestra.run.smithi176.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-04-25T15:23:28.558 INFO:teuthology.orchestra.run.smithi176.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-25T15:23:28.583 INFO:teuthology.orchestra.run.smithi176.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-25T15:23:28.609 INFO:teuthology.orchestra.run.smithi176.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-25T15:23:28.634 INFO:teuthology.orchestra.run.smithi176.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-25T15:23:28.676 INFO:teuthology.orchestra.run.smithi176.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-04-25T15:23:28.701 INFO:teuthology.orchestra.run.smithi176.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-25T15:23:28.718 INFO:teuthology.orchestra.run.smithi176.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-25T15:23:28.810 INFO:teuthology.orchestra.run.smithi176.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T15:23:28.894 INFO:teuthology.orchestra.run.smithi176.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 16 MB/s | 1.3 MB 00:00 2024-04-25T15:23:29.019 INFO:teuthology.orchestra.run.smithi176.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-04-25T15:23:29.194 INFO:teuthology.orchestra.run.smithi176.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-04-25T15:23:29.753 INFO:teuthology.orchestra.run.smithi176.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 3.2 MB/s | 3.3 MB 00:01 2024-04-25T15:23:29.758 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:23:29.758 INFO:teuthology.orchestra.run.smithi176.stdout:Total 11 MB/s | 214 MB 00:20 2024-04-25T15:23:29.879 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-25T15:23:29.879 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-25T15:23:30.888 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-04-25T15:23:31.033 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-04-25T15:23:31.033 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-04-25T15:23:32.750 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-04-25T15:23:32.750 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-04-25T15:23:35.919 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-25T15:23:36.023 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T15:23:36.272 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T15:23:36.327 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T15:23:36.367 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T15:23:36.391 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T15:23:36.436 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T15:23:36.795 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T15:23:36.917 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:23:37.037 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:23:37.068 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:23:37.160 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:23:37.188 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:23:37.347 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T15:23:37.394 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T15:23:37.421 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:23:37.528 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:23:37.556 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:23:37.618 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:23:37.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:23:37.664 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:23:37.849 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:23:37.882 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:23:37.992 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T15:23:38.089 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T15:23:38.122 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T15:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T15:23:38.234 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T15:23:38.273 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T15:23:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T15:23:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T15:23:38.377 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T15:23:38.450 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T15:23:38.500 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T15:23:38.530 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T15:23:38.574 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T15:23:38.660 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T15:23:38.685 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-04-25T15:23:38.692 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T15:23:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T15:23:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T15:23:38.798 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T15:23:38.857 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T15:23:38.905 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T15:23:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T15:23:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T15:23:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T15:23:39.129 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T15:23:39.141 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T15:23:39.166 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T15:23:39.191 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T15:23:39.210 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T15:23:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T15:23:39.658 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:23:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T15:23:39.783 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:23:39.810 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:23:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T15:23:39.873 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T15:23:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:23:39.916 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T15:23:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:23:39.954 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T15:23:39.995 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T15:23:40.081 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T15:23:40.136 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T15:23:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:23:40.171 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T15:23:40.231 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T15:23:40.267 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:23:40.288 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T15:23:40.289 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:23:40.335 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:23:40.357 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:23:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T15:23:40.382 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:23:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T15:23:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T15:23:40.483 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T15:23:40.518 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T15:23:40.532 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:23:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T15:23:40.565 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:23:40.604 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T15:23:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T15:23:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T15:23:40.768 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T15:23:40.792 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T15:23:40.805 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T15:23:40.822 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T15:23:40.841 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T15:23:40.864 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T15:23:40.914 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T15:23:40.947 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T15:23:40.962 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T15:23:40.992 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T15:23:41.004 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T15:23:41.024 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T15:23:41.045 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T15:23:41.052 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T15:23:41.078 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T15:23:41.103 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T15:23:41.122 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T15:23:41.175 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T15:23:41.182 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T15:23:41.205 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T15:23:41.249 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T15:23:41.312 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T15:23:41.327 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T15:23:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T15:23:41.360 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T15:23:41.371 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T15:23:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T15:23:41.412 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T15:23:41.440 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T15:23:41.453 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T15:23:41.486 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T15:23:41.510 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T15:23:41.555 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T15:23:41.598 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T15:23:41.628 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T15:23:41.633 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T15:23:41.666 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T15:23:41.708 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T15:23:41.742 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T15:23:41.762 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T15:23:41.815 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T15:23:41.841 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T15:23:41.889 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T15:23:41.928 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T15:23:41.976 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T15:23:42.086 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T15:23:42.139 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T15:23:42.352 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T15:23:42.436 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T15:23:42.464 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T15:23:42.498 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T15:23:42.537 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T15:23:42.578 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T15:23:42.726 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T15:23:42.789 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T15:23:42.844 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T15:23:42.908 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T15:23:42.917 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T15:23:42.948 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T15:23:42.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T15:23:42.984 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T15:23:42.985 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T15:23:43.031 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T15:23:43.067 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T15:23:43.091 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T15:23:43.111 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T15:23:43.153 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T15:23:43.289 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T15:23:43.303 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T15:23:43.333 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T15:23:43.363 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T15:23:43.392 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T15:23:43.404 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T15:23:43.429 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T15:23:43.499 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T15:23:43.529 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T15:23:43.569 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T15:23:43.603 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T15:23:43.613 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T15:23:43.628 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T15:23:43.643 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T15:23:43.703 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T15:23:43.722 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T15:23:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T15:23:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T15:23:43.896 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T15:23:43.930 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T15:23:43.969 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T15:23:44.001 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T15:23:44.031 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T15:23:44.065 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T15:23:44.099 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T15:23:44.132 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T15:23:44.183 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T15:23:44.270 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T15:23:44.313 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T15:23:44.345 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T15:23:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T15:23:44.414 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T15:23:44.515 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T15:23:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T15:23:44.571 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T15:23:45.300 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T15:23:45.343 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T15:23:45.376 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T15:23:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T15:23:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T15:23:45.615 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T15:23:45.653 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T15:23:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T15:23:45.858 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T15:23:45.859 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T15:23:45.936 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T15:23:45.938 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T15:23:45.966 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T15:23:46.000 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T15:23:46.336 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T15:23:46.371 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:23:46.479 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:23:46.500 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T15:23:46.537 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T15:23:46.583 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T15:23:46.699 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T15:23:47.051 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T15:23:47.082 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:23:47.145 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:23:47.175 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T15:23:47.936 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T15:23:48.012 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T15:23:48.040 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T15:23:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T15:23:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T15:23:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:23:48.534 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:23:48.566 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T15:23:48.602 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T15:23:49.091 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T15:23:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:23:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:23:49.215 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T15:23:49.503 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:23:49.538 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:23:50.127 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:23:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:23:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:23:50.268 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T15:23:50.268 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T15:23:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:23:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:23:51.393 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:23:51.428 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:23:51.991 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:23:52.040 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:23:52.132 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:23:52.132 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T15:23:52.132 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T15:23:52.132 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:23:52.185 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:24:02.405 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-04-25T15:24:02.406 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:02.596 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:24:02.613 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:24:02.613 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:02.614 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T15:24:02.614 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /root 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-04-25T15:24:03.671 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:24:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:24:03.867 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:03.867 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T15:24:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:24:04.276 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:24:04.276 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:04.276 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T15:24:04.276 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:24:04.276 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:24:04.277 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:04.507 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:24:04.531 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:24:04.531 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:04.532 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T15:24:04.532 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:24:04.532 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:24:04.532 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:05.017 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:24:05.041 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:24:05.041 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:05.041 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T15:24:05.041 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:24:05.041 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:24:05.041 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:05.148 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T15:24:05.206 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T15:24:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T15:24:05.437 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:24:05.463 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:24:05.463 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:05.463 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T15:24:05.464 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:24:05.464 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:24:05.464 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:05.464 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T15:24:05.518 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T15:24:05.599 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T15:24:05.688 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:24:05.696 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:24:05.710 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:24:05.710 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:05.710 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T15:24:05.711 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:24:05.711 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:24:05.711 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:05.839 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:24:05.876 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T15:24:05.900 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T15:24:05.927 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T15:24:06.172 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:24:06.195 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:24:06.195 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:06.195 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T15:24:06.195 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:24:06.195 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:24:06.195 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:06.293 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T15:24:06.342 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T15:24:06.397 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T15:24:06.398 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:24:06.425 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:24:06.458 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T15:24:06.521 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T15:24:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:24:06.583 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T15:24:06.630 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T15:24:06.709 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T15:24:06.797 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:24:06.946 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:24:06.998 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T15:24:07.029 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T15:24:07.057 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T15:24:07.443 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:24:07.472 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:24:07.504 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T15:24:07.567 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T15:24:07.603 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:24:07.654 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:24:07.700 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:24:08.643 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:24:08.680 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:24:08.722 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:24:08.750 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:24:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:24:09.042 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T15:24:09.069 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:24:09.094 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:24:09.094 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:09.094 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T15:24:09.094 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:24:09.094 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:24:09.095 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:09.145 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:24:09.156 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:24:09.259 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T15:24:09.311 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T15:24:09.643 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:24:09.672 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:24:09.774 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:24:09.940 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T15:24:09.982 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:24:10.005 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:24:10.005 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:10.005 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T15:24:10.006 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:24:10.006 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:24:10.006 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:10.041 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:24:10.052 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:24:10.143 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T15:24:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T15:24:10.300 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T15:24:10.331 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:24:10.355 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:24:10.355 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:10.355 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T15:24:10.355 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:24:10.355 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:24:10.356 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:10.398 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:24:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:24:10.421 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:10.421 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T15:24:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:24:10.498 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:24:10.498 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:10.498 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T15:24:10.498 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:24:10.498 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:24:10.498 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:10.824 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:24:10.848 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:24:10.848 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:10.848 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T15:24:10.848 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:24:10.848 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:24:10.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:11.131 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T15:24:11.160 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:24:11.183 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:24:11.183 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:11.183 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T15:24:11.183 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:24:11.183 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:24:11.184 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:11.227 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:24:11.250 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:24:11.250 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:11.250 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T15:24:11.250 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:11.294 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:24:11.317 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:24:11.317 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:11.318 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T15:24:11.318 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:24:11.318 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:24:11.318 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:11.629 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:24:11.651 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:24:11.652 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:24:11.652 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T15:24:11.652 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:24:11.652 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:24:11.652 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T15:24:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T15:24:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T15:24:15.576 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T15:24:15.603 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T15:24:15.797 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T15:24:15.841 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T15:24:15.871 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T15:24:15.907 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T15:24:15.948 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T15:24:15.998 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T15:24:16.029 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T15:24:16.057 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T15:24:16.242 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T15:24:16.286 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T15:24:16.316 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T15:24:16.463 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:24:16.509 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:24:16.531 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:24:16.889 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:24:16.921 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:24:16.941 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:24:17.443 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T15:24:17.531 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T15:24:17.572 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T15:24:17.656 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T15:24:17.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T15:24:17.693 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:24:17.777 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:24:17.788 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T15:24:17.810 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:24:17.810 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:24:17.875 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T15:24:17.906 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:24:17.916 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T15:24:17.979 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:24:17.998 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T15:24:18.029 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T15:24:18.029 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:24:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:24:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:24:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:24:18.245 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:24:18.312 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:24:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:24:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T15:24:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T15:24:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T15:24:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T15:24:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T15:24:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T15:24:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T15:24:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T15:24:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T15:24:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T15:24:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T15:24:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T15:24:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:24:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T15:24:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T15:24:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:24:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T15:24:24.306 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:24:24.306 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T15:24:24.306 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T15:24:24.306 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T15:24:24.306 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T15:24:24.306 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T15:24:24.306 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T15:24:24.307 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T15:24:24.308 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T15:24:24.308 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T15:24:24.308 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T15:24:24.308 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T15:24:24.308 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T15:24:24.308 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T15:24:24.308 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T15:24:24.319 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T15:24:24.319 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T15:24:24.319 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T15:24:24.319 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T15:24:24.320 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T15:24:24.321 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T15:24:24.322 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T15:24:24.323 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T15:24:24.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:24:24.325 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T15:24:24.326 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T15:24:24.327 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T15:24:24.328 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T15:24:24.328 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T15:24:24.328 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:24:24.328 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T15:24:26.171 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T15:24:26.172 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.173 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.174 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.175 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T15:24:26.176 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T15:24:26.177 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T15:24:26.178 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T15:24:26.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T15:24:26.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T15:24:26.181 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T15:24:26.182 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T15:24:26.183 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T15:24:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T15:24:26.184 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:26.184 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-25T15:24:26.399 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:24:26.399 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.400 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.401 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T15:24:26.402 INFO:teuthology.orchestra.run.smithi176.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.403 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T15:24:26.404 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.405 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T15:24:26.406 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T15:24:26.407 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T15:24:26.408 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T15:24:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T15:24:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T15:24:26.411 INFO:teuthology.orchestra.run.smithi176.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T15:24:26.411 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T15:24:26.411 INFO:teuthology.orchestra.run.smithi176.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T15:24:26.411 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:26.411 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-04-25T15:24:26.537 DEBUG:teuthology.parallel:result is None 2024-04-25T15:24:26.804 DEBUG:teuthology.parallel:result is None 2024-04-25T15:24:26.805 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:24:26.946 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T15:24:26.970 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T15:24:26.971 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T15:24:26.971 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T15:24:26.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:24:27.108 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T15:24:27.129 INFO:teuthology.orchestra.run.smithi176.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T15:24:27.129 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T15:24:27.129 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T15:24:27.131 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T15:24:27.131 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:27.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:24:27.160 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:27.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:24:27.196 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T15:24:27.197 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:27.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:24:27.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:24:27.292 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:27.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:24:27.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:24:27.380 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T15:24:27.381 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:27.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:24:27.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:24:27.473 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:27.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:24:27.497 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:24:27.560 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T15:24:27.561 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:27.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:24:27.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:24:27.650 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:27.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:24:27.673 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:24:27.736 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T15:24:27.828 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T15:24:27.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-04-25T15:24:27.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-04-25T15:24:27.854 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T15:24:27.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T15:24:27.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T15:24:27.919 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T15:24:27.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T15:24:27.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T15:24:27.991 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T15:24:27.991 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T15:24:27.991 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T15:24:27.991 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:27.991 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T15:24:28.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T15:24:28.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T15:24:28.172 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:28.173 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T15:24:28.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T15:24:28.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T15:24:28.340 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T15:24:28.340 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T15:24:28.341 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213', 'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7673213', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1880, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'sleep_before_teardown': 0, 'subset': '178/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240425.015853-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGF+6zczyNRzUwUq4mHVemqqq+3Eg6daGTgqLqwuCskvRzMbWi9rZ0yYZq5Vc/jIzXC0M2JXZdKhRjZjbI0AgDQ=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCIapig+HZqUrvPepgPs4Q3bSMMG1nHS8RdgRYWO3o9F/hA/qS8LyrkryWyh3jsIsCNM79MRqkO9ImLP90vm1Yw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_quota']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_10:00:23', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T15:24:28.341 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T15:24:28.344 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T15:24:28.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T15:24:28.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T15:24:28.454 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:28.454 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:24:28.476 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:24:28.476 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-04-25T15:24:28.531 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:24:28.531 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:28.531 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T15:24:28.532 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:28.532 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:28.532 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:24:18.362902175 +0000 2024-04-25T15:24:28.532 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:22:29.342755594 +0000 2024-04-25T15:24:28.532 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:22:29.342755594 +0000 2024-04-25T15:24:28.532 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-25 15:22:29.342755594 +0000 2024-04-25T15:24:28.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:24:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:24:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:24:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000245209 s, 2.1 MB/s 2024-04-25T15:24:28.595 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:24:28.652 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-04-25T15:24:28.706 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:24:18.363902153 +0000 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:22:29.698747071 +0000 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:22:29.698747071 +0000 2024-04-25T15:24:28.707 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-25 15:22:29.698747071 +0000 2024-04-25T15:24:28.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:24:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:24:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:24:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000174229 s, 2.9 MB/s 2024-04-25T15:24:28.770 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:24:28.827 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:24:18.363902153 +0000 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:22:30.055738525 +0000 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:22:30.055738525 +0000 2024-04-25T15:24:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-25 15:22:30.055738525 +0000 2024-04-25T15:24:28.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:24:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:24:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:24:28.945 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000189954 s, 2.7 MB/s 2024-04-25T15:24:28.946 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:24:29.002 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-04-25T15:24:29.057 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:24:29.057 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:29.058 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T15:24:29.058 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:29.058 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:29.058 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:24:18.364902131 +0000 2024-04-25T15:24:29.058 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:22:30.392730461 +0000 2024-04-25T15:24:29.058 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:22:30.392730461 +0000 2024-04-25T15:24:29.058 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-25 15:22:30.392730461 +0000 2024-04-25T15:24:29.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000233621 s, 2.2 MB/s 2024-04-25T15:24:29.122 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:24:29.177 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:29.177 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:29.177 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:24:29.193 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:24:29.194 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:24:18.688212225 +0000 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:22:24.978784531 +0000 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:22:24.978784531 +0000 2024-04-25T15:24:29.249 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-25 15:22:24.978784531 +0000 2024-04-25T15:24:29.249 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:24:29.311 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:24:29.311 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:24:29.311 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000212827 s, 2.4 MB/s 2024-04-25T15:24:29.312 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:24:29.368 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-04-25T15:24:29.423 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:24:29.423 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:29.423 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T15:24:29.424 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:29.424 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:29.424 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:24:18.688212225 +0000 2024-04-25T15:24:29.424 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:22:25.330781024 +0000 2024-04-25T15:24:29.424 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:22:25.330781024 +0000 2024-04-25T15:24:29.424 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-25 15:22:25.330781024 +0000 2024-04-25T15:24:29.424 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:24:29.485 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:24:29.485 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:24:29.485 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.0002076 s, 2.5 MB/s 2024-04-25T15:24:29.486 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:24:29.542 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-04-25T15:24:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:24:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:29.596 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T15:24:29.597 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:29.597 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:29.597 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:24:18.688212225 +0000 2024-04-25T15:24:29.597 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:22:25.659777747 +0000 2024-04-25T15:24:29.597 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:22:25.659777747 +0000 2024-04-25T15:24:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-25 15:22:25.659777747 +0000 2024-04-25T15:24:29.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:24:29.658 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:24:29.658 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:24:29.658 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000196838 s, 2.6 MB/s 2024-04-25T15:24:29.659 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:24:29.715 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-04-25T15:24:29.770 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:24:29.770 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:24:29.771 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T15:24:29.771 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:24:29.771 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:24:29.771 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:24:18.689212215 +0000 2024-04-25T15:24:29.771 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:22:25.982774530 +0000 2024-04-25T15:24:29.771 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:22:25.982774530 +0000 2024-04-25T15:24:29.771 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-25 15:22:25.982774530 +0000 2024-04-25T15:24:29.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:24:29.832 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:24:29.832 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:24:29.832 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000212229 s, 2.4 MB/s 2024-04-25T15:24:29.833 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:24:29.888 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:29.888 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T15:24:29.889 INFO:tasks.ceph:Generating config... 2024-04-25T15:24:29.890 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T15:24:29.890 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T15:24:29.890 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T15:24:29.890 INFO:tasks.ceph:[client] pid file = 2024-04-25T15:24:29.890 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T15:24:29.890 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T15:24:29.890 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-25T15:24:29.890 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T15:24:29.890 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T15:24:29.890 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T15:24:29.891 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T15:24:29.892 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T15:24:29.892 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T15:24:29.892 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T15:24:29.893 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T15:24:29.893 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T15:24:29.893 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T15:24:29.893 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T15:24:29.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T15:24:29.974 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T15:24:29.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T15:24:30.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:24:30.059 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.b', '172.21.15.176'), ('mon.c', '[v2:172.21.15.176:3301,v1:172.21.15.176:6790]')] 2024-04-25T15:24:30.060 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T15:24:30.061 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:30.061 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T15:24:30.115 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --add b 172.21.15.176 --addv c '[v2:172.21.15.176:3301,v1:172.21.15.176:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid b43d0acb-252a-4f7a-bf14-34d789b94b95 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:fsid b43d0acb-252a-4f7a-bf14-34d789b94b95 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-04-25T15:24:30.211018+0000 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-25T15:24:30.211018+0000 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-04-25T15:24:30.211 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] mon.c 2024-04-25T15:24:30.212 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T15:24:30.213 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T15:24:30.227 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b43d0acb-252a-4f7a-bf14-34d789b94b95... 2024-04-25T15:24:30.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T15:24:30.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-04-25T15:24:30.310 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-04-25T15:24:30.311 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-04-25T15:24:30.312 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]" 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = b43d0acb-252a-4f7a-bf14-34d789b94b95 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.313 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.314 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-04-25T15:24:30.315 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-04-25T15:24:30.316 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-04-25T15:24:30.317 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T15:24:30.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: mon down mkfs grace = 300 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-04-25T15:24:30.319 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-04-25T15:24:30.320 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:24:30.321 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-04-25T15:24:30.321 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-04-25T15:24:30.321 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-04-25T15:24:30.321 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-04-25T15:24:30.321 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-25T15:24:30.321 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-25T15:24:30.321 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.322 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-04-25T15:24:30.323 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-04-25T15:24:30.324 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "172.21.15.67,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]" 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on skipped message = False 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = b43d0acb-252a-4f7a-bf14-34d789b94b95 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-04-25T15:24:30.325 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-04-25T15:24:30.326 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.327 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = bitmap 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1/20 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1/20 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 4/10 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2024-04-25T15:24:30.328 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: debug client = 20 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:24:30.329 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T15:24:30.330 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: mon down mkfs grace = 300 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: mon op complaint time = 120 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T15:24:30.331 INFO:teuthology.orchestra.run.smithi176.stdout: client mount timeout = 600 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout: debug client = 20 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = "" 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 900 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 900 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2024-04-25T15:24:30.332 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.d] 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout:[mds] 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: debug mds = 20 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: debug mds balancer = 20 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug frag = True 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug scatterstat = True 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: mds op complaint time = 180 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: mds verify scatter = True 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 900 2024-04-25T15:24:30.333 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 900 2024-04-25T15:24:30.335 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T15:24:30.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T15:24:30.438 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T15:24:30.438 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:30.438 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T15:24:30.453 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:30.453 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T15:24:30.509 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:24:30.509 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:30.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T15:24:30.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:24:30.582 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:24:30.582 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:30.636 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:24:30.636 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:30.637 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T15:24:30.637 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:24:30.670 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:24:30.670 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:30.724 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T15:24:30.724 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T15:24:30.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T15:24:30.783 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T15:24:30.786 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T15:24:30.856 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T15:24:30.859 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T15:24:30.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T15:24:30.919 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T15:24:30.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T15:24:30.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T15:24:31.046 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T15:24:31.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T15:24:31.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T15:24:31.132 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T15:24:31.135 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T15:24:31.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T15:24:31.256 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T15:24:31.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T15:24:31.282 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T15:24:31.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T15:24:31.333 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T15:24:31.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-25T15:24:31.395 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-25T15:24:31.407 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T15:24:31.456 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T15:24:31.469 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-25T15:24:31.517 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-25T15:24:31.529 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T15:24:31.530 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T15:24:31.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T15:24:31.554 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:31.554 INFO:tasks.ceph:role: osd.0 2024-04-25T15:24:31.554 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:24:31.555 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T15:24:31.626 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:31.626 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:31.627 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:32.125 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:24:32.127 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:24:32.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T15:24:32.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T15:24:32.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T15:24:32.343 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:32.343 INFO:tasks.ceph:role: osd.1 2024-04-25T15:24:32.343 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:24:32.344 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T15:24:32.415 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:32.415 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:32.415 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:32.416 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:32.416 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:32.416 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:32.416 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:32.416 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:32.416 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:32.416 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:32.878 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:24:32.880 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:24:32.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T15:24:32.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T15:24:33.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T15:24:33.073 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:33.073 INFO:tasks.ceph:role: osd.2 2024-04-25T15:24:33.073 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:24:33.073 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T15:24:33.145 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:33.146 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:33.631 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:24:33.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:24:33.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T15:24:33.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T15:24:33.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T15:24:33.837 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:33.837 INFO:tasks.ceph:role: osd.3 2024-04-25T15:24:33.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:24:33.837 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T15:24:33.909 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:33.909 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:33.909 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:33.909 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:33.909 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:33.910 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:33.910 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:33.910 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:33.910 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:33.910 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:34.435 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:24:34.437 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:24:34.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T15:24:34.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T15:24:34.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:34.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:34.712+0000 7f14e51a7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T15:24:34.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:34.712+0000 7f14e51a7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T15:24:34.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:34.712+0000 7f14e51a7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:24:34.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:34.712+0000 7f14e51a7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:24:34.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:34.712+0000 7f14e51a7640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:24:34.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:34.713+0000 7f14e51a7640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T15:24:37.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T15:24:37.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:37.689+0000 7f1e0e7c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T15:24:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:37.689+0000 7f1e0e7c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T15:24:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:37.689+0000 7f1e0e7c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:24:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:37.689+0000 7f1e0e7c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:24:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:37.689+0000 7f1e0e7c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:24:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:37.690+0000 7f1e0e7c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T15:24:40.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T15:24:40.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:40.669+0000 7f873e60f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T15:24:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:40.669+0000 7f873e60f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T15:24:40.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:40.669+0000 7f873e60f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:24:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:40.669+0000 7f873e60f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:24:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:40.669+0000 7f873e60f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:24:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:40.670+0000 7f873e60f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T15:24:43.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T15:24:43.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:43.664+0000 7fdad761e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T15:24:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:43.665+0000 7fdad761e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T15:24:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:43.665+0000 7fdad761e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:24:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:43.665+0000 7fdad761e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:24:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:43.665+0000 7fdad761e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:24:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:24:43.665+0000 7fdad761e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T15:24:46.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T15:24:46.555 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T15:24:46.584 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:46.584 INFO:tasks.ceph:role: osd.4 2024-04-25T15:24:46.584 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:24:46.584 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T15:24:46.658 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:46.658 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:46.658 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:46.659 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:46.659 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:46.659 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:46.659 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:46.659 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:46.659 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:46.659 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:47.158 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:24:47.160 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:24:47.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T15:24:47.272 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T15:24:47.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T15:24:47.365 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:47.366 INFO:tasks.ceph:role: osd.5 2024-04-25T15:24:47.366 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:24:47.366 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T15:24:47.436 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:47.437 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:47.882 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:24:47.884 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:24:47.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T15:24:47.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T15:24:48.023 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T15:24:48.087 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:48.087 INFO:tasks.ceph:role: osd.6 2024-04-25T15:24:48.087 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:24:48.087 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T15:24:48.158 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:48.158 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:48.159 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:48.159 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:48.159 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:48.656 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:24:48.658 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:24:48.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T15:24:48.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T15:24:48.786 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T15:24:48.850 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:24:48.850 INFO:tasks.ceph:role: osd.7 2024-04-25T15:24:48.850 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:24:48.850 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:24:48.922 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T15:24:48.923 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:24:48.923 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:24:49.416 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:24:49.418 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:24:49.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T15:24:49.542 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T15:24:49.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:49.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:49.677+0000 7f422700f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T15:24:49.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:49.683+0000 7f422700f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T15:24:49.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:49.683+0000 7f422700f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:24:49.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:49.683+0000 7f422700f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:24:49.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:49.683+0000 7f422700f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:24:49.690 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:49.690+0000 7f422700f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T15:24:52.537 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T15:24:52.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:52.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:52.664+0000 7f3958017640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T15:24:52.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:52.664+0000 7f3958017640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T15:24:52.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:52.664+0000 7f3958017640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:24:52.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:52.664+0000 7f3958017640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:24:52.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:52.664+0000 7f3958017640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:24:52.665 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:52.664+0000 7f3958017640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T15:24:55.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T15:24:55.543 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:55.640 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:55.639+0000 7efea4a0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T15:24:55.640 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:55.639+0000 7efea4a0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T15:24:55.641 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:55.639+0000 7efea4a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:24:55.641 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:55.639+0000 7efea4a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:24:55.641 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:55.639+0000 7efea4a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:24:55.641 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:55.640+0000 7efea4a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T15:24:58.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:24:58.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:24:58.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T15:24:58.517 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:24:58.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:58.609+0000 7f0c1ea14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T15:24:58.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:58.610+0000 7f0c1ea14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T15:24:58.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:58.610+0000 7f0c1ea14640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:24:58.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:58.610+0000 7f0c1ea14640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:24:58.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:58.610+0000 7f0c1ea14640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:24:58.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:24:58.610+0000 7f0c1ea14640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T15:25:01.467 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T15:25:01.494 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T15:25:01.494 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:01.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T15:25:01.524 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:01.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T15:25:01.593 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:01.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T15:25:01.661 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:01.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T15:25:01.728 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:01.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T15:25:01.794 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:01.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T15:25:01.860 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:01.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T15:25:01.925 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:01.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T15:25:01.954 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:01.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T15:25:02.021 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:02.021 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T15:25:02.087 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:02.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T15:25:02.153 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:02.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T15:25:02.217 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:02.217 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T15:25:02.282 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:02.282 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T15:25:02.345 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:02.345 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T15:25:02.363 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:02.364 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-25T15:25:02.421 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:02.421 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T15:25:02.436 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:02.436 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-25T15:25:02.490 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T15:25:02.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T15:25:02.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T15:25:02.533 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag== 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg== 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q== 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg== 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw== 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA== 2024-04-25T15:25:02.534 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g== 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg== 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA== 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw== 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA== 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA== 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ== 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-04-25T15:25:02.535 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA== 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout:[client.2] 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg== 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ== 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout:[client.3] 2024-04-25T15:25:02.536 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA== 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.y] 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag== 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg== 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q== 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg== 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw== 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA== 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.3] 2024-04-25T15:25:02.553 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g== 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg== 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA== 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.d] 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw== 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.4] 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA== 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.5] 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA== 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.6] 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ== 2024-04-25T15:25:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.7] 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA== 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout:[client.2] 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg== 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ== 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout:[client.3] 2024-04-25T15:25:02.555 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA== 2024-04-25T15:25:02.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:25:02.580 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:25:02.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:02.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:02.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:02.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:02.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:02.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:02.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:02.849 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:02.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:02.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:02.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:02.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:25:03.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:25:03.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:03.092 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:03.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:03.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:25:03.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.220 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.281 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.342 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:25:03.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.654 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:25:03.717 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T15:25:03.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T15:25:03.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:25:03.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T15:25:03.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T15:25:03.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:25:04.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T15:25:04.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T15:25:04.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:25:04.193 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T15:25:04.218 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:25:04.222 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:25:04.273 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T15:25:04.273 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T15:25:04.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T15:25:04.278 INFO:tasks.ceph.mon.a:Started 2024-04-25T15:25:04.278 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T15:25:04.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T15:25:04.316 INFO:tasks.ceph.mon.b:Started 2024-04-25T15:25:04.316 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T15:25:04.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T15:25:04.320 INFO:tasks.ceph.mon.c:Started 2024-04-25T15:25:04.320 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T15:25:04.321 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T15:25:04.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T15:25:04.327 INFO:tasks.ceph.mgr.y:Started 2024-04-25T15:25:04.327 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T15:25:04.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T15:25:04.341 INFO:tasks.ceph.mgr.x:Started 2024-04-25T15:25:04.342 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T15:25:04.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T15:25:04.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.455+0000 7fdd34067640 1 Processor -- start 2024-04-25T15:25:04.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.455+0000 7fdd34067640 1 -- start start 2024-04-25T15:25:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd34067640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd2c0d2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd34067640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 0x7fdd2c0af740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd34067640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd2c162f20 con 0x7fdd2c161870 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd34067640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdd2c0d2aa0 con 0x7fdd2c0af330 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd34067640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd2c1623a0 con 0x7fdd2c0d2150 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd31e03640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdd2c161870 0x7fdd2c06c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46164/0 (socket says 172.21.15.67:46164) 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd32604640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 0x7fdd2c0af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.456+0000 7fdd31e03640 1 -- 172.21.15.67:0/625683689 learned_addr learned my addr 172.21.15.67:0/625683689 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:04.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd31602640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd2c0d2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd32604640 1 -- 172.21.15.67:0/625683689 >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 msgr2=0x7fdd2c0af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T15:25:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd32604640 1 -- 172.21.15.67:0/625683689 >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 msgr2=0x7fdd2c0af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:25:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd32604640 1 --2- 172.21.15.67:0/625683689 >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 0x7fdd2c0af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:25:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd32604640 1 --2- 172.21.15.67:0/625683689 >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 0x7fdd2c0af740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:25:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd31602640 1 -- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 msgr2=0x7fdd2c0d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T15:25:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd31602640 1 -- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 msgr2=0x7fdd2c0d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:25:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd31602640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd2c0d2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:25:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.457+0000 7fdd31602640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd2c0d2560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:25:04.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.598+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911231709 0 0) 0x7fdd2c162f20 con 0x7fdd2c161870 2024-04-25T15:25:04.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.598+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd18003520 con 0x7fdd2c161870 2024-04-25T15:25:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2424744621 0 0) 0x7fdd18003520 con 0x7fdd2c161870 2024-04-25T15:25:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 msgr2=0x7fdd2c0d2560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd2c0d2560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 msgr2=0x7fdd2c0af740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 --2- 172.21.15.67:0/625683689 >> v2:172.21.15.67:3300/0 conn(0x7fdd2c0af330 0x7fdd2c0af740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2c06cdf0 con 0x7fdd2c161870 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fdd1c001670 con 0x7fdd2c161870 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 >> v1:172.21.15.67:6789/0 conn(0x7fdd2c161870 legacy=0x7fdd2c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd18003fe0 0x7fdd18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd18005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c0d2aa0 con 0x7fdd18004d40 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c1623a0 con 0x7fdd18003fe0 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1800bb70 con 0x7fdd2c0d2150 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd32604640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd18005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd31e03640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.599+0000 7fdd31602640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd18003fe0 0x7fdd18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.600+0000 7fdd32604640 1 -- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 msgr2=0x7fdd180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.600+0000 7fdd32604640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d2150 0x7fdd180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.600+0000 7fdd32604640 1 -- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd18003fe0 msgr2=0x7fdd18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.600+0000 7fdd32604640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd18003fe0 0x7fdd18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.600+0000 7fdd32604640 1 -- 172.21.15.67:0/625683689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd18003520 con 0x7fdd18004d40 2024-04-25T15:25:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.600+0000 7fdd32604640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd18005150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009610 tx=0x7fdd18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5068f371f2eb0e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.602+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24033070 con 0x7fdd18004d40 2024-04-25T15:25:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.602+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd2402f650 con 0x7fdd18004d40 2024-04-25T15:25:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.602+0000 7fdd30e01640 1 -- 172.21.15.67:0/625683689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24009bd0 con 0x7fdd18004d40 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.603+0000 7fdd34067640 1 -- 172.21.15.67:0/625683689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 msgr2=0x7fdd18005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.603+0000 7fdd34067640 1 --2- 172.21.15.67:0/625683689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd18005150 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009610 tx=0x7fdd18005830 comp rx=0 tx=0).stop 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.603+0000 7fdd32604640 1 -- 172.21.15.67:0/625683689 reap_dead start 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.603+0000 7fdd34067640 1 -- 172.21.15.67:0/625683689 shutdown_connections 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.603+0000 7fdd34067640 1 -- 172.21.15.67:0/625683689 >> 172.21.15.67:0/625683689 conn(0x7fdd2c09b7d0 msgr2=0x7fdd2c0b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.603+0000 7fdd34067640 1 -- 172.21.15.67:0/625683689 shutdown_connections 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.603+0000 7fdd34067640 1 -- 172.21.15.67:0/625683689 wait complete. 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.605+0000 7fdd34067640 1 Processor -- start 2024-04-25T15:25:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.605+0000 7fdd34067640 1 -- start start 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.606+0000 7fdd34067640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd2c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.606+0000 7fdd34067640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c09fec0 0x7fdd2c09cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.606+0000 7fdd34067640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c09d250 0x7fdd2c0b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.606+0000 7fdd34067640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c162c40 con 0x7fdd18004d40 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.606+0000 7fdd34067640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c162960 con 0x7fdd2c09fec0 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.606+0000 7fdd34067640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd2c162000 con 0x7fdd2c09d250 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.606+0000 7fdd31e03640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd2c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd2c156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51632/0 (socket says 172.21.15.67:51632) 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 -- 172.21.15.67:0/4048844280 learned_addr learned my addr 172.21.15.67:0/4048844280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31602640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c09fec0 0x7fdd2c09cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd32604640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c09d250 0x7fdd2c0b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 -- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c09d250 msgr2=0x7fdd2c0b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c09d250 0x7fdd2c0b0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 -- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c09fec0 msgr2=0x7fdd2c09cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c09fec0 0x7fdd2c09cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 -- 172.21.15.67:0/4048844280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2c09da00 con 0x7fdd18004d40 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.607+0000 7fdd31e03640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd2c156950 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c001ae0 tx=0x7fdd1c007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.608+0000 7fdd227fc640 1 -- 172.21.15.67:0/4048844280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00f910 con 0x7fdd18004d40 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.608+0000 7fdd227fc640 1 -- 172.21.15.67:0/4048844280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1c018070 con 0x7fdd18004d40 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.608+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2c09f100 con 0x7fdd18004d40 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.608+0000 7fdd227fc640 1 -- 172.21.15.67:0/4048844280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00fd30 con 0x7fdd18004d40 2024-04-25T15:25:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.608+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd2c09f6c0 con 0x7fdd18004d40 2024-04-25T15:25:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.608+0000 7fdd227fc640 1 -- 172.21.15.67:0/4048844280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdd1c01d460 con 0x7fdd18004d40 2024-04-25T15:25:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.608+0000 7fdd227fc640 1 -- 172.21.15.67:0/4048844280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdd1c013760 con 0x7fdd18004d40 2024-04-25T15:25:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.609+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2c09ee60 con 0x7fdd18004d40 2024-04-25T15:25:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.611+0000 7fdd227fc640 1 -- 172.21.15.67:0/4048844280 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdd2c09da00 con 0x7fdd18004d40 2024-04-25T15:25:04.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.679+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fdd2c050740 con 0x7fdd18004d40 2024-04-25T15:25:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.681+0000 7fdd227fc640 1 -- 172.21.15.67:0/4048844280 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fdd1c0139b0 con 0x7fdd18004d40 2024-04-25T15:25:04.689 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-04-25T15:25:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 msgr2=0x7fdd2c156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd2c156950 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c001ae0 tx=0x7fdd1c007720 comp rx=0 tx=0).stop 2024-04-25T15:25:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 shutdown_connections 2024-04-25T15:25:04.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c09d250 0x7fdd2c0b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c09fec0 0x7fdd2c09cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 --2- 172.21.15.67:0/4048844280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd18004d40 0x7fdd2c156950 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 >> 172.21.15.67:0/4048844280 conn(0x7fdd2c09b7d0 msgr2=0x7fdd2c070880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 shutdown_connections 2024-04-25T15:25:04.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:04.682+0000 7fdd34067640 1 -- 172.21.15.67:0/4048844280 wait complete. 2024-04-25T15:25:04.703 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T15:25:04.703 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T15:25:04.704 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:04.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T15:25:04.704 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:04.702+0000 7f52077b8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:25:04.728 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:04.715+0000 7f97bf9e3040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:25:04.743 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:04.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T15:25:04.814 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:04.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T15:25:04.855 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:04.854+0000 7f52077b8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:25:04.886 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:04.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T15:25:04.889 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:04.888+0000 7f97bf9e3040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:25:04.907 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:04.906+0000 7f52077b8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:25:04.947 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:04.945+0000 7f97bf9e3040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:25:04.959 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:04.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T15:25:04.984 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:04.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T15:25:05.054 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:05.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T15:25:05.122 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:05.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T15:25:05.155 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:05.154+0000 7f52077b8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:25:05.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7 0 2024-04-25T15:25:05.204 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:05.202+0000 7f52077b8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:25:05.212 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:05.211+0000 7f97bf9e3040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:25:05.261 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:05.259+0000 7f97bf9e3040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:25:05.315 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.313+0000 7f33aca34640 1 Processor -- start 2024-04-25T15:25:05.315 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.313+0000 7f33aca34640 1 -- start start 2024-04-25T15:25:05.316 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33aca34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33a80b2d70 0x7f33a80b3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.316 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33aca34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80cc800 0x7f33a80ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33aca34640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a8163f60 con 0x7f33a8164a20 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33aca34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f33a8164e30 con 0x7f33a80b2d70 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33aca34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f33a80cf660 con 0x7f33a80cc800 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f33a8164a20 0x7f33a80cebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:57444/0 (socket says 172.21.15.176:57444) 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a6d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80cc800 0x7f33a80ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a659c640 1 -- 172.21.15.176:0/3967425073 learned_addr learned my addr 172.21.15.176:0/3967425073 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a5d9b640 1 --2- 172.21.15.176:0/3967425073 >> v2:172.21.15.67:3300/0 conn(0x7f33a80b2d70 0x7f33a80b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a559a640 1 -- 172.21.15.176:0/3967425073 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224482566 0 0) 0x7f33a8163f60 con 0x7f33a8164a20 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a6d9d640 1 -- 172.21.15.176:0/3967425073 >> v2:172.21.15.67:3300/0 conn(0x7f33a80b2d70 msgr2=0x7f33a80b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a6d9d640 1 --2- 172.21.15.176:0/3967425073 >> v2:172.21.15.67:3300/0 conn(0x7f33a80b2d70 0x7f33a80b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a6d9d640 1 -- 172.21.15.176:0/3967425073 >> v1:172.21.15.176:6789/0 conn(0x7f33a8164a20 legacy=0x7f33a80cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a6d9d640 1 -- 172.21.15.176:0/3967425073 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a80cf940 con 0x7f33a80cc800 2024-04-25T15:25:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.315+0000 7f33a5d9b640 1 --2- 172.21.15.176:0/3967425073 >> v2:172.21.15.67:3300/0 conn(0x7f33a80b2d70 0x7f33a80b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:05.318 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.316+0000 7f33a6d9d640 1 --2- 172.21.15.176:0/3967425073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80cc800 0x7f33a80ccc10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009d90 tx=0x7f33a002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d846f8af55c91f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.316+0000 7f33a559a640 1 -- 172.21.15.176:0/3967425073 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0031600 con 0x7f33a80cc800 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.316+0000 7f33a559a640 1 -- 172.21.15.176:0/3967425073 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a00317a0 con 0x7f33a80cc800 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.316+0000 7f33a559a640 1 -- 172.21.15.176:0/3967425073 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0031b70 con 0x7f33a80cc800 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 -- 172.21.15.176:0/3967425073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80cc800 msgr2=0x7f33a80ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 --2- 172.21.15.176:0/3967425073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80cc800 0x7f33a80ccc10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009d90 tx=0x7f33a002a5d0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 -- 172.21.15.176:0/3967425073 shutdown_connections 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 --2- 172.21.15.176:0/3967425073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80cc800 0x7f33a80ccc10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 --2- 172.21.15.176:0/3967425073 >> v2:172.21.15.67:3300/0 conn(0x7f33a80b2d70 0x7f33a80b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 -- 172.21.15.176:0/3967425073 >> 172.21.15.176:0/3967425073 conn(0x7f33a809b7d0 msgr2=0x7f33a80b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 -- 172.21.15.176:0/3967425073 shutdown_connections 2024-04-25T15:25:05.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.317+0000 7f33aca34640 1 -- 172.21.15.176:0/3967425073 wait complete. 2024-04-25T15:25:05.320 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.319+0000 7f33aca34640 1 Processor -- start 2024-04-25T15:25:05.320 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.319+0000 7f33aca34640 1 -- start start 2024-04-25T15:25:05.323 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33aca34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80b2d70 0x7f33a815d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.323 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33aca34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a80cc800 0x7f33a809c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.323 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33aca34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33a8164a20 0x7f33a80d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.323 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33aca34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f33a80b36c0 con 0x7f33a80cc800 2024-04-25T15:25:05.323 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33aca34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8164500 con 0x7f33a8164a20 2024-04-25T15:25:05.323 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33aca34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f33a80cf2d0 con 0x7f33a80b2d70 2024-04-25T15:25:05.323 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33a659c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80b2d70 0x7f33a815d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33a6d9d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33a8164a20 0x7f33a80d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.320+0000 7f33a659c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80b2d70 0x7f33a815d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34500/0 (socket says 172.21.15.176:34500) 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a659c640 1 -- 172.21.15.176:0/504096856 learned_addr learned my addr 172.21.15.176:0/504096856 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a5d9b640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a80cc800 0x7f33a809c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a6d9d640 1 -- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80b2d70 msgr2=0x7f33a815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a6d9d640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80b2d70 0x7f33a815d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a6d9d640 1 -- 172.21.15.176:0/504096856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a80cc800 msgr2=0x7f33a809c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a6d9d640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a80cc800 0x7f33a809c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a6d9d640 1 -- 172.21.15.176:0/504096856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a809f1f0 con 0x7f33a8164a20 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a659c640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80b2d70 0x7f33a815d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33a6d9d640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33a8164a20 0x7f33a80d1f50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f33a8163ec0 tx=0x7f33a0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f3396ffd640 1 -- 172.21.15.176:0/504096856 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a003f070 con 0x7f33a8164a20 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f3396ffd640 1 -- 172.21.15.176:0/504096856 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a0030b60 con 0x7f33a8164a20 2024-04-25T15:25:05.324 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f3396ffd640 1 -- 172.21.15.176:0/504096856 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0009450 con 0x7f33a8164a20 2024-04-25T15:25:05.325 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a809f4d0 con 0x7f33a8164a20 2024-04-25T15:25:05.325 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a809fa90 con 0x7f33a8164a20 2024-04-25T15:25:05.325 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.321+0000 7f3396ffd640 1 -- 172.21.15.176:0/504096856 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f33a0044460 con 0x7f33a8164a20 2024-04-25T15:25:05.325 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.322+0000 7f3396ffd640 1 -- 172.21.15.176:0/504096856 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f33a003b410 con 0x7f33a8164a20 2024-04-25T15:25:05.325 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.322+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a8050740 con 0x7f33a8164a20 2024-04-25T15:25:05.325 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.325+0000 7f3396ffd640 1 -- 172.21.15.176:0/504096856 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f33a0052020 con 0x7f33a8164a20 2024-04-25T15:25:05.398 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.397+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7", "id": 0} v 0) -- 0x7f33a80b32b0 con 0x7f33a8164a20 2024-04-25T15:25:05.402 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.401+0000 7f3396ffd640 1 -- 172.21.15.176:0/504096856 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f33a0044890 con 0x7f33a8164a20 2024-04-25T15:25:05.402 INFO:teuthology.orchestra.run.smithi176.stdout:0 2024-04-25T15:25:05.403 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33a8164a20 msgr2=0x7f33a80d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33a8164a20 0x7f33a80d1f50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f33a8163ec0 tx=0x7f33a0004a40 comp rx=0 tx=0).stop 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 shutdown_connections 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33a8164a20 0x7f33a80d1f50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a80cc800 0x7f33a809c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 --2- 172.21.15.176:0/504096856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33a80b2d70 0x7f33a815d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 >> 172.21.15.176:0/504096856 conn(0x7f33a809b7d0 msgr2=0x7f33a806fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 shutdown_connections 2024-04-25T15:25:05.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.402+0000 7f33aca34640 1 -- 172.21.15.176:0/504096856 wait complete. 2024-04-25T15:25:05.414 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new bc9efa57-f337-42d2-b403-f6c8d5e2a4c4 1 2024-04-25T15:25:05.521 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.520+0000 7f9946934640 1 Processor -- start 2024-04-25T15:25:05.521 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.520+0000 7f9946934640 1 -- start start 2024-04-25T15:25:05.523 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9946934640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f994006e0e0 0x7f99400ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.523 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9946934640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f994006eaa0 0x7f99400d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.523 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9946934640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f99400d2e30 0x7f99400b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9946934640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f994006e6c0 con 0x7f994006e0e0 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9946934640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f99400d3240 con 0x7f99400d2e30 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9946934640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f99400b3e90 con 0x7f994006eaa0 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9944ed1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f99400d2e30 0x7f99400b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f993f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f994006eaa0 0x7f99400d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9944ed1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f99400d2e30 0x7f99400b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37160/0 (socket says 172.21.15.176:37160) 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f9944ed1640 1 -- 172.21.15.176:0/830389641 learned_addr learned my addr 172.21.15.176:0/830389641 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.521+0000 7f993ffff640 1 --2- 172.21.15.176:0/830389641 >> v2:172.21.15.67:3300/0 conn(0x7f994006e0e0 0x7f99400ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993f7fe640 1 -- 172.21.15.176:0/830389641 >> v2:172.21.15.176:3300/0 conn(0x7f99400d2e30 msgr2=0x7f99400b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993f7fe640 1 --2- 172.21.15.176:0/830389641 >> v2:172.21.15.176:3300/0 conn(0x7f99400d2e30 0x7f99400b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993f7fe640 1 -- 172.21.15.176:0/830389641 >> v2:172.21.15.67:3300/0 conn(0x7f994006e0e0 msgr2=0x7f99400ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993f7fe640 1 --2- 172.21.15.176:0/830389641 >> v2:172.21.15.67:3300/0 conn(0x7f994006e0e0 0x7f99400ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993f7fe640 1 -- 172.21.15.176:0/830389641 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99400b2410 con 0x7f994006eaa0 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f9944ed1640 1 --2- 172.21.15.176:0/830389641 >> v2:172.21.15.176:3300/0 conn(0x7f99400d2e30 0x7f99400b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:05.524 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993f7fe640 1 --2- 172.21.15.176:0/830389641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f994006eaa0 0x7f99400d28f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99400b2250 tx=0x7f993402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d185948c6ba43d85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.525 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993effd640 1 -- 172.21.15.176:0/830389641 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934037070 con 0x7f994006eaa0 2024-04-25T15:25:05.525 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.522+0000 7f993effd640 1 -- 172.21.15.176:0/830389641 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f993402ae50 con 0x7f994006eaa0 2024-04-25T15:25:05.525 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.523+0000 7f993effd640 1 -- 172.21.15.176:0/830389641 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934032660 con 0x7f994006eaa0 2024-04-25T15:25:05.525 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 -- 172.21.15.176:0/830389641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f994006eaa0 msgr2=0x7f99400d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 --2- 172.21.15.176:0/830389641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f994006eaa0 0x7f99400d28f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f99400b2250 tx=0x7f993402a100 comp rx=0 tx=0).stop 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 -- 172.21.15.176:0/830389641 shutdown_connections 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 --2- 172.21.15.176:0/830389641 >> v2:172.21.15.176:3300/0 conn(0x7f99400d2e30 0x7f99400b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 --2- 172.21.15.176:0/830389641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f994006eaa0 0x7f99400d28f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 --2- 172.21.15.176:0/830389641 >> v2:172.21.15.67:3300/0 conn(0x7f994006e0e0 0x7f99400ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 -- 172.21.15.176:0/830389641 >> 172.21.15.176:0/830389641 conn(0x7f994009b7d0 msgr2=0x7f994009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 -- 172.21.15.176:0/830389641 shutdown_connections 2024-04-25T15:25:05.526 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.524+0000 7f9946934640 1 -- 172.21.15.176:0/830389641 wait complete. 2024-04-25T15:25:05.527 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.527+0000 7f9946934640 1 Processor -- start 2024-04-25T15:25:05.528 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.527+0000 7f9946934640 1 -- start start 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f9946934640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f994006e0e0 0x7f99401e5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f9946934640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994006eaa0 0x7f99401e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f9946934640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99400d2e30 0x7f994006c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f9946934640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f99400b4130 con 0x7f994006eaa0 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f9946934640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f99400b29d0 con 0x7f994006e0e0 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f9946934640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f994006eeb0 con 0x7f99400d2e30 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f993ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f994006e0e0 0x7f99401e5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f993ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f994006e0e0 0x7f99401e5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37164/0 (socket says 172.21.15.176:37164) 2024-04-25T15:25:05.532 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f993ffff640 1 -- 172.21.15.176:0/4227121866 learned_addr learned my addr 172.21.15.176:0/4227121866 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f9944ed1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99400d2e30 0x7f994006c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.529+0000 7f993f7fe640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994006eaa0 0x7f99401e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993ffff640 1 -- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99400d2e30 msgr2=0x7f994006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993ffff640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99400d2e30 0x7f994006c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993ffff640 1 -- 172.21.15.176:0/4227121866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994006eaa0 msgr2=0x7f99401e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993ffff640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994006eaa0 0x7f99401e5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993ffff640 1 -- 172.21.15.176:0/4227121866 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99401f4580 con 0x7f994006e0e0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f9944ed1640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99400d2e30 0x7f994006c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993ffff640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f994006e0e0 0x7f99401e5910 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f993000e840 tx=0x7f993000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993cff9640 1 -- 172.21.15.176:0/4227121866 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930005990 con 0x7f994006e0e0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993cff9640 1 -- 172.21.15.176:0/4227121866 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9930005b50 con 0x7f994006e0e0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99401f4860 con 0x7f994006e0e0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993cff9640 1 -- 172.21.15.176:0/4227121866 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993000c940 con 0x7f994006e0e0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99401f4e20 con 0x7f994006e0e0 2024-04-25T15:25:05.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993cff9640 1 -- 172.21.15.176:0/4227121866 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9930006030 con 0x7f994006e0e0 2024-04-25T15:25:05.534 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.530+0000 7f993cff9640 1 -- 172.21.15.176:0/4227121866 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f9930013c60 con 0x7f994006e0e0 2024-04-25T15:25:05.534 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.531+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9940050740 con 0x7f994006e0e0 2024-04-25T15:25:05.534 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.533+0000 7f993cff9640 1 -- 172.21.15.176:0/4227121866 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9930017020 con 0x7f994006e0e0 2024-04-25T15:25:05.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.607+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bc9efa57-f337-42d2-b403-f6c8d5e2a4c4", "id": 1} v 0) -- 0x7f994015d0a0 con 0x7f994006e0e0 2024-04-25T15:25:05.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.613+0000 7f993cff9640 1 -- 172.21.15.176:0/4227121866 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bc9efa57-f337-42d2-b403-f6c8d5e2a4c4", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f9930006500 con 0x7f994006e0e0 2024-04-25T15:25:05.614 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:25:05.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f994006e0e0 msgr2=0x7f99401e5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f994006e0e0 0x7f99401e5910 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f993000e840 tx=0x7f993000ed00 comp rx=0 tx=0).stop 2024-04-25T15:25:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 shutdown_connections 2024-04-25T15:25:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99400d2e30 0x7f994006c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f994006eaa0 0x7f99401e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 --2- 172.21.15.176:0/4227121866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f994006e0e0 0x7f99401e5910 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 >> 172.21.15.176:0/4227121866 conn(0x7f994009b7d0 msgr2=0x7f99400b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 shutdown_connections 2024-04-25T15:25:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.614+0000 7f9946934640 1 -- 172.21.15.176:0/4227121866 wait complete. 2024-04-25T15:25:05.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c 2 2024-04-25T15:25:05.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.732+0000 7f819c051640 1 Processor -- start 2024-04-25T15:25:05.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.733+0000 7f819c051640 1 -- start start 2024-04-25T15:25:05.736 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819c051640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f81940cec70 0x7f81940b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.736 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819c051640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81940cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.736 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819c051640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81941642e0 con 0x7f8194164d50 2024-04-25T15:25:05.736 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819c051640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8194165160 con 0x7f81940cec70 2024-04-25T15:25:05.736 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819c051640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f81941648c0 con 0x7f81940b3e80 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f81995ec640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f81940cec70 0x7f81940b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819a5ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81940cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819a5ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81940cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34530/0 (socket says 172.21.15.176:34530) 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819a5ee640 1 -- 172.21.15.176:0/3955675052 learned_addr learned my addr 172.21.15.176:0/3955675052 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f81995ec640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f81940cec70 0x7f81940b7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37172/0 (socket says 172.21.15.176:37172) 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819a5ee640 1 -- 172.21.15.176:0/3955675052 >> v2:172.21.15.176:3300/0 conn(0x7f81940cec70 msgr2=0x7f81940b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819a5ee640 1 --2- 172.21.15.176:0/3955675052 >> v2:172.21.15.176:3300/0 conn(0x7f81940cec70 0x7f81940b7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819a5ee640 1 -- 172.21.15.176:0/3955675052 >> v1:172.21.15.67:6789/0 conn(0x7f8194164d50 legacy=0x7f81940cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f819a5ee640 1 -- 172.21.15.176:0/3955675052 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81940cf080 con 0x7f81940b3e80 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.734+0000 7f81995ec640 1 --2- 172.21.15.176:0/3955675052 >> v2:172.21.15.176:3300/0 conn(0x7f81940cec70 0x7f81940b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.735+0000 7f819a5ee640 1 --2- 172.21.15.176:0/3955675052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81940cd760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f818c009e50 tx=0x7f818c02a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d39bd1f96780c82b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.735+0000 7f8198deb640 1 -- 172.21.15.176:0/3955675052 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c036070 con 0x7f81940b3e80 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.735+0000 7f8198deb640 1 -- 172.21.15.176:0/3955675052 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f818c031600 con 0x7f81940b3e80 2024-04-25T15:25:05.737 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.735+0000 7f8198deb640 1 -- 172.21.15.176:0/3955675052 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c0319d0 con 0x7f81940b3e80 2024-04-25T15:25:05.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 -- 172.21.15.176:0/3955675052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 msgr2=0x7f81940cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 --2- 172.21.15.176:0/3955675052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81940cd760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f818c009e50 tx=0x7f818c02a7a0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 -- 172.21.15.176:0/3955675052 shutdown_connections 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 --2- 172.21.15.176:0/3955675052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81940cd760 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 --2- 172.21.15.176:0/3955675052 >> v2:172.21.15.176:3300/0 conn(0x7f81940cec70 0x7f81940b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 -- 172.21.15.176:0/3955675052 >> 172.21.15.176:0/3955675052 conn(0x7f819409b7d0 msgr2=0x7f819409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 -- 172.21.15.176:0/3955675052 shutdown_connections 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.736+0000 7f819c051640 1 -- 172.21.15.176:0/3955675052 wait complete. 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.738+0000 7f819c051640 1 Processor -- start 2024-04-25T15:25:05.739 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.738+0000 7f819c051640 1 -- start start 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f819c051640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81941550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f819c051640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81940cec70 0x7f81940c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f819c051640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8194164d50 0x7f81940cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f819c051640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81940cf630 con 0x7f81940cec70 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f819c051640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8194164550 con 0x7f8194164d50 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f819c051640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f81940cdca0 con 0x7f81940b3e80 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f8199ded640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81941550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.739+0000 7f819a5ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8194164d50 0x7f81940cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f819a5ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8194164d50 0x7f81940cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37178/0 (socket says 172.21.15.176:37178) 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f819a5ee640 1 -- 172.21.15.176:0/2774851526 learned_addr learned my addr 172.21.15.176:0/2774851526 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f81995ec640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81940cec70 0x7f81940c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.742 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f8199ded640 1 -- 172.21.15.176:0/2774851526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8194164d50 msgr2=0x7f81940cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f8199ded640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8194164d50 0x7f81940cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f8199ded640 1 -- 172.21.15.176:0/2774851526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81940cec70 msgr2=0x7f81940c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f8199ded640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81940cec70 0x7f81940c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f8199ded640 1 -- 172.21.15.176:0/2774851526 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81940caa80 con 0x7f81940b3e80 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f8199ded640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81941550e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f819000c3f0 tx=0x7f819000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f818a7fc640 1 -- 172.21.15.176:0/2774851526 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190017070 con 0x7f81940b3e80 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f818a7fc640 1 -- 172.21.15.176:0/2774851526 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81900052d0 con 0x7f81940b3e80 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f818a7fc640 1 -- 172.21.15.176:0/2774851526 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190003830 con 0x7f81940b3e80 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81940cad60 con 0x7f81940b3e80 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81940cb320 con 0x7f81940b3e80 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f818a7fc640 1 -- 172.21.15.176:0/2774851526 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f819001a070 con 0x7f81940b3e80 2024-04-25T15:25:05.743 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.740+0000 7f818a7fc640 1 -- 172.21.15.176:0/2774851526 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f8190012420 con 0x7f81940b3e80 2024-04-25T15:25:05.744 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.741+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8194050740 con 0x7f81940b3e80 2024-04-25T15:25:05.745 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.744+0000 7f818a7fc640 1 -- 172.21.15.176:0/2774851526 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8190021070 con 0x7f81940b3e80 2024-04-25T15:25:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.817+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c", "id": 2} v 0) -- 0x7f81940b8530 con 0x7f81940b3e80 2024-04-25T15:25:05.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.823+0000 7f818a7fc640 1 -- 172.21.15.176:0/2774851526 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f8190011590 con 0x7f81940b3e80 2024-04-25T15:25:05.824 INFO:teuthology.orchestra.run.smithi176.stdout:2 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 msgr2=0x7f81941550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81941550e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f819000c3f0 tx=0x7f819000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 shutdown_connections 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8194164d50 0x7f81940cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81940cec70 0x7f81940c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 --2- 172.21.15.176:0/2774851526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81940b3e80 0x7f81941550e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 >> 172.21.15.176:0/2774851526 conn(0x7f819409b7d0 msgr2=0x7f819406e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:05.825 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 shutdown_connections 2024-04-25T15:25:05.826 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.824+0000 7f819c051640 1 -- 172.21.15.176:0/2774851526 wait complete. 2024-04-25T15:25:05.836 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 42f8d3ce-92d8-4b80-90ab-f847ee917452 3 2024-04-25T15:25:05.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.942+0000 7f20dfb46640 1 Processor -- start 2024-04-25T15:25:05.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.943+0000 7f20dfb46640 1 -- start start 2024-04-25T15:25:05.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20dfb46640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20d80d20a0 0x7f20d80b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20dfb46640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80b8140 con 0x7f20d806e0e0 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20dfb46640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d806e660 con 0x7f20d80b7d30 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20dfb46640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f20d80d32d0 con 0x7f20d80d20a0 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20dd0e1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20d80d20a0 0x7f20d80b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20dd0e1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20d80d20a0 0x7f20d80b77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37184/0 (socket says 172.21.15.176:37184) 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20de0e3640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f20d80b7d30 0x7f20d80d0610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:57450/0 (socket says 172.21.15.176:57450) 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.944+0000 7f20dd0e1640 1 -- 172.21.15.176:0/1665519128 learned_addr learned my addr 172.21.15.176:0/1665519128 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.945+0000 7f20dd0e1640 1 -- 172.21.15.176:0/1665519128 >> v1:172.21.15.176:6789/0 conn(0x7f20d80b7d30 legacy=0x7f20d80d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.945+0000 7f20dd0e1640 1 -- 172.21.15.176:0/1665519128 >> v1:172.21.15.67:6789/0 conn(0x7f20d806e0e0 legacy=0x7f20d806ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.945+0000 7f20dc8e0640 1 -- 172.21.15.176:0/1665519128 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995250510 0 0) 0x7f20d806e660 con 0x7f20d80b7d30 2024-04-25T15:25:05.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.945+0000 7f20dd0e1640 1 -- 172.21.15.176:0/1665519128 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d80d29a0 con 0x7f20d80d20a0 2024-04-25T15:25:05.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.946+0000 7f20dd0e1640 1 --2- 172.21.15.176:0/1665519128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80d20a0 0x7f20d80b77f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009870 tx=0x7f20d402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff87c37756487f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.946+0000 7f20dc8e0640 1 -- 172.21.15.176:0/1665519128 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4037070 con 0x7f20d80d20a0 2024-04-25T15:25:05.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.946+0000 7f20dc8e0640 1 -- 172.21.15.176:0/1665519128 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d4032440 con 0x7f20d80d20a0 2024-04-25T15:25:05.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.946+0000 7f20dc8e0640 1 -- 172.21.15.176:0/1665519128 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4032810 con 0x7f20d80d20a0 2024-04-25T15:25:05.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.947+0000 7f20dfb46640 1 -- 172.21.15.176:0/1665519128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80d20a0 msgr2=0x7f20d80b77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.947+0000 7f20dfb46640 1 --2- 172.21.15.176:0/1665519128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80d20a0 0x7f20d80b77f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009870 tx=0x7f20d402a270 comp rx=0 tx=0).stop 2024-04-25T15:25:05.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.947+0000 7f20dfb46640 1 -- 172.21.15.176:0/1665519128 shutdown_connections 2024-04-25T15:25:05.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.947+0000 7f20dfb46640 1 --2- 172.21.15.176:0/1665519128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80d20a0 0x7f20d80b77f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.947+0000 7f20dfb46640 1 -- 172.21.15.176:0/1665519128 >> 172.21.15.176:0/1665519128 conn(0x7f20d809b7d0 msgr2=0x7f20d809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:05.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.947+0000 7f20dfb46640 1 -- 172.21.15.176:0/1665519128 shutdown_connections 2024-04-25T15:25:05.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.947+0000 7f20dfb46640 1 -- 172.21.15.176:0/1665519128 wait complete. 2024-04-25T15:25:05.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.948+0000 7f20dfb46640 1 Processor -- start 2024-04-25T15:25:05.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.948+0000 7f20dfb46640 1 -- start start 2024-04-25T15:25:05.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dfb46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20d806e0e0 0x7f20d81f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dfb46640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80b7d30 0x7f20d815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dfb46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80d20a0 0x7f20d81519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:05.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dfb46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20d80d2f60 con 0x7f20d80d20a0 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dfb46640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f20d80d24b0 con 0x7f20d80b7d30 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dfb46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f20d80d3760 con 0x7f20d806e0e0 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80b7d30 0x7f20d815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd8e2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20d806e0e0 0x7f20d81f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80b7d30 0x7f20d815f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37198/0 (socket says 172.21.15.176:37198) 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 -- 172.21.15.176:0/48903024 learned_addr learned my addr 172.21.15.176:0/48903024 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20de0e3640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80d20a0 0x7f20d81519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 -- 172.21.15.176:0/48903024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20d806e0e0 msgr2=0x7f20d81f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20d806e0e0 0x7f20d81f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 -- 172.21.15.176:0/48903024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80d20a0 msgr2=0x7f20d81519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80d20a0 0x7f20d81519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 -- 172.21.15.176:0/48903024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d815d820 con 0x7f20d80b7d30 2024-04-25T15:25:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dd0e1640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80b7d30 0x7f20d815f8c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009870 tx=0x7f20d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20ce7fc640 1 -- 172.21.15.176:0/48903024 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4041020 con 0x7f20d80b7d30 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20ce7fc640 1 -- 172.21.15.176:0/48903024 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d4030970 con 0x7f20d80b7d30 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.950+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d815db00 con 0x7f20d80b7d30 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.951+0000 7f20ce7fc640 1 -- 172.21.15.176:0/48903024 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4037040 con 0x7f20d80b7d30 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.951+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d815e0c0 con 0x7f20d80b7d30 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.951+0000 7f20ce7fc640 1 -- 172.21.15.176:0/48903024 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f20d4044020 con 0x7f20d80b7d30 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.951+0000 7f20ce7fc640 1 -- 172.21.15.176:0/48903024 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f20d4030e00 con 0x7f20d80b7d30 2024-04-25T15:25:05.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.951+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d8050740 con 0x7f20d80b7d30 2024-04-25T15:25:05.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:05.955+0000 7f20ce7fc640 1 -- 172.21.15.176:0/48903024 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f20d404b030 con 0x7f20d80b7d30 2024-04-25T15:25:05.972 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:05.972+0000 7f52077b8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:25:05.986 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:05.984+0000 7f97bf9e3040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:25:06.029 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.028+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "42f8d3ce-92d8-4b80-90ab-f847ee917452", "id": 3} v 0) -- 0x7f20d80cf670 con 0x7f20d80b7d30 2024-04-25T15:25:06.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.034+0000 7f20ce7fc640 1 -- 172.21.15.176:0/48903024 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42f8d3ce-92d8-4b80-90ab-f847ee917452", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f20d4046400 con 0x7f20d80b7d30 2024-04-25T15:25:06.035 INFO:teuthology.orchestra.run.smithi176.stdout:3 2024-04-25T15:25:06.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80b7d30 msgr2=0x7f20d815f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80b7d30 0x7f20d815f8c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009870 tx=0x7f20d40046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 shutdown_connections 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80d20a0 0x7f20d81519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20d80b7d30 0x7f20d815f8c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 --2- 172.21.15.176:0/48903024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20d806e0e0 0x7f20d81f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 >> 172.21.15.176:0/48903024 conn(0x7f20d809b7d0 msgr2=0x7f20d815ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 shutdown_connections 2024-04-25T15:25:06.037 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.035+0000 7f20dfb46640 1 -- 172.21.15.176:0/48903024 wait complete. 2024-04-25T15:25:06.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new a422a859-f7f2-4ade-ae44-67a905058938 4 2024-04-25T15:25:06.066 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:06.065+0000 7f52077b8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:25:06.079 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:06.077+0000 7f97bf9e3040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:25:06.111 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:06.110+0000 7f52077b8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:25:06.124 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:06.123+0000 7f97bf9e3040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:25:06.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.154+0000 7f0a9d3c5640 1 Processor -- start 2024-04-25T15:25:06.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.154+0000 7f0a9d3c5640 1 -- start start 2024-04-25T15:25:06.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.155+0000 7f0a9d3c5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0a980d0de0 0x7f0a980d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.155+0000 7f0a9d3c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a9806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.155+0000 7f0a9d3c5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a9806cf10 con 0x7f0a980b77f0 2024-04-25T15:25:06.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.155+0000 7f0a9d3c5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0a980d5a40 con 0x7f0a980d0de0 2024-04-25T15:25:06.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.155+0000 7f0a9d3c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0a980d1830 con 0x7f0a9806c5c0 2024-04-25T15:25:06.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a967fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a9806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a96ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0a980d0de0 0x7f0a980d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a967fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a9806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34560/0 (socket says 172.21.15.176:34560) 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a96ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0a980d0de0 0x7f0a980d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37202/0 (socket says 172.21.15.176:37202) 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a967fc640 1 -- 172.21.15.176:0/1456510296 learned_addr learned my addr 172.21.15.176:0/1456510296 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a967fc640 1 -- 172.21.15.176:0/1456510296 >> v2:172.21.15.176:3300/0 conn(0x7f0a980d0de0 msgr2=0x7f0a980d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a967fc640 1 --2- 172.21.15.176:0/1456510296 >> v2:172.21.15.176:3300/0 conn(0x7f0a980d0de0 0x7f0a980d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a967fc640 1 -- 172.21.15.176:0/1456510296 >> v1:172.21.15.176:6789/0 conn(0x7f0a980b77f0 legacy=0x7f0a980b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a967fc640 1 -- 172.21.15.176:0/1456510296 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a98160dc0 con 0x7f0a9806c5c0 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a95ffb640 1 -- 172.21.15.176:0/1456510296 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2543366160 0 0) 0x7f0a90001490 con 0x7f0a980b77f0 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.156+0000 7f0a96ffd640 1 --2- 172.21.15.176:0/1456510296 >> v2:172.21.15.176:3300/0 conn(0x7f0a980d0de0 0x7f0a980d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.157+0000 7f0a967fc640 1 --2- 172.21.15.176:0/1456510296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a9806c9d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009870 tx=0x7f0a8802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da54c8187efb82d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.157+0000 7f0a95ffb640 1 -- 172.21.15.176:0/1456510296 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88037070 con 0x7f0a9806c5c0 2024-04-25T15:25:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.157+0000 7f0a95ffb640 1 -- 172.21.15.176:0/1456510296 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88032440 con 0x7f0a9806c5c0 2024-04-25T15:25:06.160 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.157+0000 7f0a95ffb640 1 -- 172.21.15.176:0/1456510296 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a88032810 con 0x7f0a9806c5c0 2024-04-25T15:25:06.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/1456510296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 msgr2=0x7f0a9806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 --2- 172.21.15.176:0/1456510296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a9806c9d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009870 tx=0x7f0a8802a250 comp rx=0 tx=0).stop 2024-04-25T15:25:06.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/1456510296 shutdown_connections 2024-04-25T15:25:06.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 --2- 172.21.15.176:0/1456510296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a9806c9d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 --2- 172.21.15.176:0/1456510296 >> v2:172.21.15.176:3300/0 conn(0x7f0a980d0de0 0x7f0a980d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/1456510296 >> 172.21.15.176:0/1456510296 conn(0x7f0a980d5d70 msgr2=0x7f0a980d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.161 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/1456510296 shutdown_connections 2024-04-25T15:25:06.162 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.158+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/1456510296 wait complete. 2024-04-25T15:25:06.162 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.160+0000 7f0a9d3c5640 1 Processor -- start 2024-04-25T15:25:06.162 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.160+0000 7f0a9d3c5640 1 -- start start 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a9d3c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a981eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a9d3c5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a980b77f0 0x7f0a9815a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a9d3c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a980d0de0 0x7f0a9815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a9d3c5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0a98160ab0 con 0x7f0a980d0de0 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a9d3c5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0a98161380 con 0x7f0a980b77f0 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a9d3c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0a980b5300 con 0x7f0a9806c5c0 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a981eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a967fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a980b77f0 0x7f0a9815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a967fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a980b77f0 0x7f0a9815a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37210/0 (socket says 172.21.15.176:37210) 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a981eb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34574/0 (socket says 172.21.15.176:34574) 2024-04-25T15:25:06.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 -- 172.21.15.176:0/3675282124 learned_addr learned my addr 172.21.15.176:0/3675282124 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a977fe640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a980d0de0 0x7f0a9815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 -- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a980b77f0 msgr2=0x7f0a9815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a980b77f0 0x7f0a9815a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 -- 172.21.15.176:0/3675282124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a980d0de0 msgr2=0x7f0a9815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a980d0de0 0x7f0a9815a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a96ffd640 1 -- 172.21.15.176:0/3675282124 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a980cb180 con 0x7f0a9806c5c0 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a977fe640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a980d0de0 0x7f0a9815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.161+0000 7f0a967fc640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a980b77f0 0x7f0a9815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a96ffd640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a981eb9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00e990 tx=0x7f0a8c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a7f7fe640 1 -- 172.21.15.176:0/3675282124 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c017070 con 0x7f0a9806c5c0 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a7f7fe640 1 -- 172.21.15.176:0/3675282124 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c00c9e0 con 0x7f0a9806c5c0 2024-04-25T15:25:06.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a980ca600 con 0x7f0a9806c5c0 2024-04-25T15:25:06.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a7f7fe640 1 -- 172.21.15.176:0/3675282124 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c00cdb0 con 0x7f0a9806c5c0 2024-04-25T15:25:06.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a980cabc0 con 0x7f0a9806c5c0 2024-04-25T15:25:06.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a7f7fe640 1 -- 172.21.15.176:0/3675282124 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0a8c01a070 con 0x7f0a9806c5c0 2024-04-25T15:25:06.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.162+0000 7f0a7f7fe640 1 -- 172.21.15.176:0/3675282124 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f0a8c013470 con 0x7f0a9806c5c0 2024-04-25T15:25:06.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.163+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a98050740 con 0x7f0a9806c5c0 2024-04-25T15:25:06.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.167+0000 7f0a7f7fe640 1 -- 172.21.15.176:0/3675282124 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0a8c006020 con 0x7f0a9806c5c0 2024-04-25T15:25:06.240 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.239+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a422a859-f7f2-4ade-ae44-67a905058938", "id": 4} v 0) -- 0x7f0a980d4300 con 0x7f0a9806c5c0 2024-04-25T15:25:06.246 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.246+0000 7f0a7f7fe640 1 -- 172.21.15.176:0/3675282124 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a422a859-f7f2-4ade-ae44-67a905058938", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f0a8c0064d0 con 0x7f0a9806c5c0 2024-04-25T15:25:06.247 INFO:teuthology.orchestra.run.smithi176.stdout:4 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 msgr2=0x7f0a981eb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a981eb9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00e990 tx=0x7f0a8c00ee50 comp rx=0 tx=0).stop 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 shutdown_connections 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a980d0de0 0x7f0a9815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a980b77f0 0x7f0a9815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 --2- 172.21.15.176:0/3675282124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9806c5c0 0x7f0a981eb9c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 >> 172.21.15.176:0/3675282124 conn(0x7f0a980d5d70 msgr2=0x7f0a98162070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 shutdown_connections 2024-04-25T15:25:06.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.247+0000 7f0a9d3c5640 1 -- 172.21.15.176:0/3675282124 wait complete. 2024-04-25T15:25:06.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new da442e17-9ae8-4b0a-939f-64578aa41f7a 5 2024-04-25T15:25:06.296 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:06.295+0000 7f52077b8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:25:06.300 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:06.299+0000 7f97bf9e3040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:25:06.350 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:06.349+0000 7f52077b8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:25:06.357 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:06.355+0000 7f97bf9e3040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:25:06.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.403+0000 7ff21e882640 1 Processor -- start 2024-04-25T15:25:06.404 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.403+0000 7ff21e882640 1 -- start start 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff21e882640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21806d140 0x7ff21806d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff21e882640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff218162780 con 0x7ff21809b7d0 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff21e882640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff218166820 con 0x7ff2180b1f10 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff21e882640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff21806c660 con 0x7ff21806d140 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff2177fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21806d140 0x7ff21806d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff2177fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21806d140 0x7ff21806d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34584/0 (socket says 172.21.15.176:34584) 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff2177fe640 1 -- 172.21.15.176:0/3191475066 learned_addr learned my addr 172.21.15.176:0/3191475066 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:06.406 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.404+0000 7ff21ce1f640 1 --1- 172.21.15.176:0/3191475066 >> v1:172.21.15.176:6789/0 conn(0x7ff2180b1f10 0x7ff2180b2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:57474/0 (socket says 172.21.15.176:57474) 2024-04-25T15:25:06.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.405+0000 7ff216ffd640 1 -- 172.21.15.176:0/3191475066 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711244431 0 0) 0x7ff218166820 con 0x7ff2180b1f10 2024-04-25T15:25:06.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.405+0000 7ff216ffd640 1 -- 172.21.15.176:0/3191475066 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff200003520 con 0x7ff2180b1f10 2024-04-25T15:25:06.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.405+0000 7ff2177fe640 1 -- 172.21.15.176:0/3191475066 >> v1:172.21.15.176:6789/0 conn(0x7ff2180b1f10 legacy=0x7ff2180b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.405+0000 7ff2177fe640 1 -- 172.21.15.176:0/3191475066 >> v1:172.21.15.67:6789/0 conn(0x7ff21809b7d0 legacy=0x7ff218166110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.407 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.405+0000 7ff2177fe640 1 -- 172.21.15.176:0/3191475066 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21806c940 con 0x7ff21806d140 2024-04-25T15:25:06.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.406+0000 7ff2177fe640 1 --2- 172.21.15.176:0/3191475066 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21806d140 0x7ff21806d550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0098a0 tx=0x7ff20c02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac9a188cbf1d7ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.406+0000 7ff216ffd640 1 -- 172.21.15.176:0/3191475066 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c037070 con 0x7ff21806d140 2024-04-25T15:25:06.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.406+0000 7ff216ffd640 1 -- 172.21.15.176:0/3191475066 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20c032440 con 0x7ff21806d140 2024-04-25T15:25:06.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.406+0000 7ff216ffd640 1 -- 172.21.15.176:0/3191475066 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0327f0 con 0x7ff21806d140 2024-04-25T15:25:06.408 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.407+0000 7ff21e882640 1 -- 172.21.15.176:0/3191475066 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21806d140 msgr2=0x7ff21806d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.407+0000 7ff21e882640 1 --2- 172.21.15.176:0/3191475066 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21806d140 0x7ff21806d550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0098a0 tx=0x7ff20c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.407+0000 7ff21e882640 1 -- 172.21.15.176:0/3191475066 shutdown_connections 2024-04-25T15:25:06.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.407+0000 7ff21e882640 1 --2- 172.21.15.176:0/3191475066 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21806d140 0x7ff21806d550 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.407+0000 7ff21e882640 1 -- 172.21.15.176:0/3191475066 >> 172.21.15.176:0/3191475066 conn(0x7ff218073b60 msgr2=0x7ff2180b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.407+0000 7ff21e882640 1 -- 172.21.15.176:0/3191475066 shutdown_connections 2024-04-25T15:25:06.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.407+0000 7ff21e882640 1 -- 172.21.15.176:0/3191475066 wait complete. 2024-04-25T15:25:06.409 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:06.407+0000 7f52077b8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:25:06.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.409+0000 7ff21e882640 1 Processor -- start 2024-04-25T15:25:06.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.409+0000 7ff21e882640 1 -- start start 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21e882640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff21806d140 0x7ff21809f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21e882640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21809b7d0 0x7ff2181fb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21e882640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2180b1f10 0x7ff2181fb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21e882640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff218165000 con 0x7ff21806d140 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21e882640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff218164ce0 con 0x7ff2180b1f10 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21e882640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff2180b2a30 con 0x7ff21809b7d0 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21ce1f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2180b1f10 0x7ff2181fb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff2177fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21809b7d0 0x7ff2181fb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff21ce1f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2180b1f10 0x7ff2181fb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37214/0 (socket says 172.21.15.176:37214) 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff2177fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21809b7d0 0x7ff2181fb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34588/0 (socket says 172.21.15.176:34588) 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff2177fe640 1 -- 172.21.15.176:0/3221061503 learned_addr learned my addr 172.21.15.176:0/3221061503 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:06.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff217fff640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff21806d140 0x7ff21809f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff2177fe640 1 -- 172.21.15.176:0/3221061503 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2180b1f10 msgr2=0x7ff2181fb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff2177fe640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2180b1f10 0x7ff2181fb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff2177fe640 1 -- 172.21.15.176:0/3221061503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff21806d140 msgr2=0x7ff21809f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.410+0000 7ff2177fe640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff21806d140 0x7ff21809f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff2177fe640 1 -- 172.21.15.176:0/3221061503 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2181fbf70 con 0x7ff21809b7d0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff2177fe640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21809b7d0 0x7ff2181fb150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0098a0 tx=0x7ff20c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff214ff9640 1 -- 172.21.15.176:0/3221061503 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c037040 con 0x7ff21809b7d0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff214ff9640 1 -- 172.21.15.176:0/3221061503 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20c004230 con 0x7ff21809b7d0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2181fc210 con 0x7ff21809b7d0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff214ff9640 1 -- 172.21.15.176:0/3221061503 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c003610 con 0x7ff21809b7d0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2181fc7d0 con 0x7ff21809b7d0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff214ff9640 1 -- 172.21.15.176:0/3221061503 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff20c02ad10 con 0x7ff21809b7d0 2024-04-25T15:25:06.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.411+0000 7ff214ff9640 1 -- 172.21.15.176:0/3221061503 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7ff20c046400 con 0x7ff21809b7d0 2024-04-25T15:25:06.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.412+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff218050740 con 0x7ff21809b7d0 2024-04-25T15:25:06.415 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:06.411+0000 7f97bf9e3040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:25:06.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.416+0000 7ff214ff9640 1 -- 172.21.15.176:0/3221061503 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff2181fbf70 con 0x7ff21809b7d0 2024-04-25T15:25:06.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.488+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "da442e17-9ae8-4b0a-939f-64578aa41f7a", "id": 5} v 0) -- 0x7ff218074540 con 0x7ff21809b7d0 2024-04-25T15:25:06.495 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.495+0000 7ff214ff9640 1 -- 172.21.15.176:0/3221061503 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da442e17-9ae8-4b0a-939f-64578aa41f7a", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7ff20c03d4e0 con 0x7ff21809b7d0 2024-04-25T15:25:06.496 INFO:teuthology.orchestra.run.smithi176.stdout:5 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.495+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21809b7d0 msgr2=0x7ff2181fb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.495+0000 7ff21e882640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21809b7d0 0x7ff2181fb150 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0098a0 tx=0x7ff20c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.496+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 shutdown_connections 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.496+0000 7ff21e882640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2180b1f10 0x7ff2181fb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.496+0000 7ff21e882640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff21809b7d0 0x7ff2181fb150 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.496+0000 7ff21e882640 1 --2- 172.21.15.176:0/3221061503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff21806d140 0x7ff21809f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.496+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 >> 172.21.15.176:0/3221061503 conn(0x7ff218073b60 msgr2=0x7ff21815e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.496+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 shutdown_connections 2024-04-25T15:25:06.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.496+0000 7ff21e882640 1 -- 172.21.15.176:0/3221061503 wait complete. 2024-04-25T15:25:06.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new c6374d82-7a32-4b64-b4ba-e33737cb15f6 6 2024-04-25T15:25:06.530 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:06.529+0000 7f52077b8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:25:06.532 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:06.531+0000 7f97bf9e3040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:25:06.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.615+0000 7f95d92f2640 1 Processor -- start 2024-04-25T15:25:06.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.616+0000 7f95d92f2640 1 -- start start 2024-04-25T15:25:06.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95d92f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d406ccd0 0x7f95d415e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95d92f2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f95d4163fa0 0x7f95d41643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95d92f2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95d41609b0 con 0x7f95d4161710 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95d92f2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f95d4164cd0 con 0x7f95d4163fa0 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95d92f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f95d4160c30 con 0x7f95d406ccd0 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95cbfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d406ccd0 0x7f95d415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95cbfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d406ccd0 0x7f95d415e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34598/0 (socket says 172.21.15.176:34598) 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95cbfff640 1 -- 172.21.15.176:0/182110424 learned_addr learned my addr 172.21.15.176:0/182110424 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95d37fe640 1 --2- 172.21.15.176:0/182110424 >> v2:172.21.15.67:3300/0 conn(0x7f95d4163fa0 0x7f95d41643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95cbfff640 1 -- 172.21.15.176:0/182110424 >> v2:172.21.15.67:3300/0 conn(0x7f95d4163fa0 msgr2=0x7f95d41643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95cbfff640 1 --2- 172.21.15.176:0/182110424 >> v2:172.21.15.67:3300/0 conn(0x7f95d4163fa0 0x7f95d41643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95cbfff640 1 -- 172.21.15.176:0/182110424 >> v1:172.21.15.67:6789/0 conn(0x7f95d4161710 legacy=0x7f95d406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.617+0000 7f95cbfff640 1 -- 172.21.15.176:0/182110424 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d4160f10 con 0x7f95d406ccd0 2024-04-25T15:25:06.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.618+0000 7f95cbfff640 1 --2- 172.21.15.176:0/182110424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d406ccd0 0x7f95d415e080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f02c4b9c9782b93b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.618+0000 7f95d27fc640 1 -- 172.21.15.176:0/182110424 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc037070 con 0x7f95d406ccd0 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.618+0000 7f95d27fc640 1 -- 172.21.15.176:0/182110424 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95bc032440 con 0x7f95d406ccd0 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.618+0000 7f95d27fc640 1 -- 172.21.15.176:0/182110424 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc032810 con 0x7f95d406ccd0 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 -- 172.21.15.176:0/182110424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d406ccd0 msgr2=0x7f95d415e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 --2- 172.21.15.176:0/182110424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d406ccd0 0x7f95d415e080 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 -- 172.21.15.176:0/182110424 shutdown_connections 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 --2- 172.21.15.176:0/182110424 >> v2:172.21.15.67:3300/0 conn(0x7f95d4163fa0 0x7f95d41643b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 --2- 172.21.15.176:0/182110424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d406ccd0 0x7f95d415e080 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 -- 172.21.15.176:0/182110424 >> 172.21.15.176:0/182110424 conn(0x7f95d40d5d90 msgr2=0x7f95d40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 -- 172.21.15.176:0/182110424 shutdown_connections 2024-04-25T15:25:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.619+0000 7f95d92f2640 1 -- 172.21.15.176:0/182110424 wait complete. 2024-04-25T15:25:06.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.621+0000 7f95d92f2640 1 Processor -- start 2024-04-25T15:25:06.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.621+0000 7f95d92f2640 1 -- start start 2024-04-25T15:25:06.624 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d92f2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d4161710 0x7f95d41569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.624 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d92f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95d4163fa0 0x7f95d415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d92f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d415bbe0 0x7f95d4155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d92f2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f95d4164a70 con 0x7f95d4161710 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d92f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f95d41648f0 con 0x7f95d4163fa0 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d92f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f95d40d49f0 con 0x7f95d415bbe0 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d415bbe0 0x7f95d4155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95cbfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95d4163fa0 0x7f95d415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d415bbe0 0x7f95d4155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34602/0 (socket says 172.21.15.176:34602) 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d4161710 0x7f95d41569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95cbfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95d4163fa0 0x7f95d415b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37226/0 (socket says 172.21.15.176:37226) 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.622+0000 7f95d37fe640 1 -- 172.21.15.176:0/194097031 learned_addr learned my addr 172.21.15.176:0/194097031 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d37fe640 1 -- 172.21.15.176:0/194097031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95d4163fa0 msgr2=0x7f95d415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d37fe640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95d4163fa0 0x7f95d415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d37fe640 1 -- 172.21.15.176:0/194097031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d4161710 msgr2=0x7f95d41569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d37fe640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d4161710 0x7f95d41569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d37fe640 1 -- 172.21.15.176:0/194097031 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d40ce080 con 0x7f95d415bbe0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d2ffd640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d4161710 0x7f95d41569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d37fe640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d415bbe0 0x7f95d4155e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009cb0 tx=0x7f95cc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95cb7fe640 1 -- 172.21.15.176:0/194097031 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc018020 con 0x7f95d415bbe0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95cb7fe640 1 -- 172.21.15.176:0/194097031 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95cc014480 con 0x7f95d415bbe0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d40c9df0 con 0x7f95d415bbe0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95cb7fe640 1 -- 172.21.15.176:0/194097031 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc014870 con 0x7f95d415bbe0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d40cc1a0 con 0x7f95d415bbe0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95cb7fe640 1 -- 172.21.15.176:0/194097031 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f95cc005d10 con 0x7f95d415bbe0 2024-04-25T15:25:06.626 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.623+0000 7f95cb7fe640 1 -- 172.21.15.176:0/194097031 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f95cc01ddb0 con 0x7f95d415bbe0 2024-04-25T15:25:06.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.624+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95d40c9b60 con 0x7f95d415bbe0 2024-04-25T15:25:06.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.626+0000 7f95cb7fe640 1 -- 172.21.15.176:0/194097031 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f95cc022030 con 0x7f95d415bbe0 2024-04-25T15:25:06.698 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.697+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c6374d82-7a32-4b64-b4ba-e33737cb15f6", "id": 6} v 0) -- 0x7f95d4050740 con 0x7f95d415bbe0 2024-04-25T15:25:06.706 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.705+0000 7f95cb7fe640 1 -- 172.21.15.176:0/194097031 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c6374d82-7a32-4b64-b4ba-e33737cb15f6", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f95d40ce080 con 0x7f95d415bbe0 2024-04-25T15:25:06.706 INFO:teuthology.orchestra.run.smithi176.stdout:6 2024-04-25T15:25:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d415bbe0 msgr2=0x7f95d4155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d415bbe0 0x7f95d4155e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009cb0 tx=0x7f95cc00c490 comp rx=0 tx=0).stop 2024-04-25T15:25:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 shutdown_connections 2024-04-25T15:25:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95d415bbe0 0x7f95d4155e70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95d4163fa0 0x7f95d415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 --2- 172.21.15.176:0/194097031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95d4161710 0x7f95d41569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 >> 172.21.15.176:0/194097031 conn(0x7f95d40d5d90 msgr2=0x7f95d4162f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 shutdown_connections 2024-04-25T15:25:06.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.706+0000 7f95d92f2640 1 -- 172.21.15.176:0/194097031 wait complete. 2024-04-25T15:25:06.718 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 6546bb86-09ac-4dc9-b996-84004d2444d9 7 2024-04-25T15:25:06.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.866+0000 7ffaa6851640 1 Processor -- start 2024-04-25T15:25:06.867 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.866+0000 7ffaa6851640 1 -- start start 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffaa6851640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa006d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffaa6851640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffaa00b7760 0x7ffaa00b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffaa6851640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaa00b80b0 con 0x7ffaa015e170 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffaa6851640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ffaa015e650 con 0x7ffaa00b7760 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffaa6851640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa00cf320 con 0x7ffaa015d240 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa006d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffa9ffff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ffaa015e170 0x7ffaa00b3d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:57484/0 (socket says 172.21.15.176:57484) 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffa9ffff640 1 -- 172.21.15.176:0/2480908980 learned_addr learned my addr 172.21.15.176:0/2480908980 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-04-25T15:25:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.867+0000 7ffaa4dee640 1 --2- 172.21.15.176:0/2480908980 >> v2:172.21.15.67:3300/0 conn(0x7ffaa00b7760 0x7ffaa00b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.868+0000 7ffa9effd640 1 -- 172.21.15.176:0/2480908980 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3442234009 0 0) 0x7ffa90001110 con 0x7ffaa015e170 2024-04-25T15:25:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.868+0000 7ffa9effd640 1 -- 172.21.15.176:0/2480908980 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa88003520 con 0x7ffaa015e170 2024-04-25T15:25:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.868+0000 7ffa9f7fe640 1 -- 172.21.15.176:0/2480908980 >> v2:172.21.15.67:3300/0 conn(0x7ffaa00b7760 msgr2=0x7ffaa00b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.868+0000 7ffa9f7fe640 1 --2- 172.21.15.176:0/2480908980 >> v2:172.21.15.67:3300/0 conn(0x7ffaa00b7760 0x7ffaa00b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.868+0000 7ffa9f7fe640 1 -- 172.21.15.176:0/2480908980 >> v1:172.21.15.176:6789/0 conn(0x7ffaa015e170 legacy=0x7ffaa00b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.868+0000 7ffa9f7fe640 1 -- 172.21.15.176:0/2480908980 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa00cf600 con 0x7ffaa015d240 2024-04-25T15:25:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.868+0000 7ffa9f7fe640 1 --2- 172.21.15.176:0/2480908980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa006d6d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffa940097a0 tx=0x7ffa9402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa8144b26f87ef2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffa9effd640 1 -- 172.21.15.176:0/2480908980 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94037070 con 0x7ffaa015d240 2024-04-25T15:25:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffa9effd640 1 -- 172.21.15.176:0/2480908980 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa94032440 con 0x7ffaa015d240 2024-04-25T15:25:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffa9effd640 1 -- 172.21.15.176:0/2480908980 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa940327f0 con 0x7ffaa015d240 2024-04-25T15:25:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffaa6851640 1 -- 172.21.15.176:0/2480908980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 msgr2=0x7ffaa006d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffaa6851640 1 --2- 172.21.15.176:0/2480908980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa006d6d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffa940097a0 tx=0x7ffa9402a180 comp rx=0 tx=0).stop 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffaa6851640 1 -- 172.21.15.176:0/2480908980 shutdown_connections 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffaa6851640 1 --2- 172.21.15.176:0/2480908980 >> v2:172.21.15.67:3300/0 conn(0x7ffaa00b7760 0x7ffaa00b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffaa6851640 1 --2- 172.21.15.176:0/2480908980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa006d6d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.869+0000 7ffaa6851640 1 -- 172.21.15.176:0/2480908980 >> 172.21.15.176:0/2480908980 conn(0x7ffaa009b7d0 msgr2=0x7ffaa009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.870+0000 7ffaa6851640 1 -- 172.21.15.176:0/2480908980 shutdown_connections 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.870+0000 7ffaa6851640 1 -- 172.21.15.176:0/2480908980 wait complete. 2024-04-25T15:25:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.871+0000 7ffaa6851640 1 Processor -- start 2024-04-25T15:25:06.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.871+0000 7ffaa6851640 1 -- start start 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffaa6851640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaa00b7760 0x7ffaa00d3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffaa6851640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffaa6851640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa015e170 0x7ffaa00b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffaa6851640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa006dc10 con 0x7ffaa015e170 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffaa6851640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa0160ce0 con 0x7ffaa00b7760 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffaa6851640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa006dd90 con 0x7ffaa015d240 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaa00b7760 0x7ffaa00d3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaa00b7760 0x7ffaa00d3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:37228/0 (socket says 172.21.15.176:37228) 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa014ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:34632/0 (socket says 172.21.15.176:34632) 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9ffff640 1 -- 172.21.15.176:0/2509436892 learned_addr learned my addr 172.21.15.176:0/2509436892 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffaa4dee640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa015e170 0x7ffaa00b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 -- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaa00b7760 msgr2=0x7ffaa00d3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaa00b7760 0x7ffaa00d3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 -- 172.21.15.176:0/2509436892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa015e170 msgr2=0x7ffaa00b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa015e170 0x7ffaa00b1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 -- 172.21.15.176:0/2509436892 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa00d13b0 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9ffff640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaa00b7760 0x7ffaa00d3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.873+0000 7ffa9f7fe640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa014ee60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffa94031000 tx=0x7ffa940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.874+0000 7ffa9cff9640 1 -- 172.21.15.176:0/2509436892 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94041020 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.874+0000 7ffa9cff9640 1 -- 172.21.15.176:0/2509436892 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9403d620 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.874+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaa00d1620 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.874+0000 7ffa9cff9640 1 -- 172.21.15.176:0/2509436892 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94037040 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.874+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaa00d1be0 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.874+0000 7ffa9cff9640 1 -- 172.21.15.176:0/2509436892 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffa94044020 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.874+0000 7ffa9cff9640 1 -- 172.21.15.176:0/2509436892 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7ffa94046c20 con 0x7ffaa015d240 2024-04-25T15:25:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.875+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa0050740 con 0x7ffaa015d240 2024-04-25T15:25:06.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.877+0000 7ffa9cff9640 1 -- 172.21.15.176:0/2509436892 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ffa9404b030 con 0x7ffaa015d240 2024-04-25T15:25:06.931 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:06.930+0000 7f52077b8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:25:06.939 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:06.938+0000 7f97bf9e3040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:25:06.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.949+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6546bb86-09ac-4dc9-b996-84004d2444d9", "id": 7} v 0) -- 0x7ffaa00b8530 con 0x7ffaa015d240 2024-04-25T15:25:06.959 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.958+0000 7ffa9cff9640 1 -- 172.21.15.176:0/2509436892 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6546bb86-09ac-4dc9-b996-84004d2444d9", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7ffa94046400 con 0x7ffaa015d240 2024-04-25T15:25:06.959 INFO:teuthology.orchestra.run.smithi176.stdout:7 2024-04-25T15:25:06.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 msgr2=0x7ffaa014ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:06.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa014ee60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffa94031000 tx=0x7ffa940046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 shutdown_connections 2024-04-25T15:25:06.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaa015e170 0x7ffaa00b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaa015d240 0x7ffaa014ee60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 --2- 172.21.15.176:0/2509436892 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaa00b7760 0x7ffaa00d3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:06.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 >> 172.21.15.176:0/2509436892 conn(0x7ffaa009b7d0 msgr2=0x7ffaa015f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:06.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 shutdown_connections 2024-04-25T15:25:06.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:06.959+0000 7ffaa6851640 1 -- 172.21.15.176:0/2509436892 wait complete. 2024-04-25T15:25:06.970 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T15:25:06.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T15:25:06.973 INFO:tasks.ceph.osd.0:Started 2024-04-25T15:25:06.973 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T15:25:06.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T15:25:06.975 INFO:tasks.ceph.osd.1:Started 2024-04-25T15:25:06.975 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T15:25:06.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T15:25:06.977 INFO:tasks.ceph.osd.2:Started 2024-04-25T15:25:06.977 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T15:25:06.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T15:25:06.980 INFO:tasks.ceph.osd.3:Started 2024-04-25T15:25:06.980 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T15:25:06.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T15:25:06.982 INFO:tasks.ceph.osd.4:Started 2024-04-25T15:25:06.982 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T15:25:06.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T15:25:06.984 INFO:tasks.ceph.osd.5:Started 2024-04-25T15:25:06.984 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T15:25:06.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T15:25:06.986 INFO:tasks.ceph.osd.6:Started 2024-04-25T15:25:06.986 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T15:25:06.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T15:25:06.990 INFO:tasks.ceph.osd.7:Started 2024-04-25T15:25:06.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:07.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.126+0000 7f4ff5492640 1 Processor -- start 2024-04-25T15:25:07.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.126+0000 7f4ff5492640 1 -- start start 2024-04-25T15:25:07.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4ff5492640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4ff00cff90 0x7f4ff00b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:07.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4ff5492640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ff00ced20 0x7f4ff015e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4ff5492640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff015eff0 con 0x7f4ff00d2930 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4ff5492640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff00b25b0 con 0x7f4ff00ced20 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4ff5492640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff015f2d0 con 0x7f4ff00cff90 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4fee7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ff00ced20 0x7f4ff015e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4fee7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ff00ced20 0x7f4ff015e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51646/0 (socket says 172.21.15.67:51646) 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4fee7fc640 1 -- 172.21.15.67:0/935106786 learned_addr learned my addr 172.21.15.67:0/935106786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4fef7fe640 1 --1- 172.21.15.67:0/935106786 >> v1:172.21.15.67:6789/0 conn(0x7f4ff00d2930 0x7f4ff00d2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46202/0 (socket says 172.21.15.67:46202) 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.127+0000 7f4feeffd640 1 --2- 172.21.15.67:0/935106786 >> v2:172.21.15.176:3300/0 conn(0x7f4ff00cff90 0x7f4ff00b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.128+0000 7f4fee7fc640 1 -- 172.21.15.67:0/935106786 >> v2:172.21.15.176:3300/0 conn(0x7f4ff00cff90 msgr2=0x7f4ff00b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.128+0000 7f4fee7fc640 1 --2- 172.21.15.67:0/935106786 >> v2:172.21.15.176:3300/0 conn(0x7f4ff00cff90 0x7f4ff00b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.128+0000 7f4fee7fc640 1 -- 172.21.15.67:0/935106786 >> v1:172.21.15.67:6789/0 conn(0x7f4ff00d2930 legacy=0x7f4ff00d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:07.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.128+0000 7f4fee7fc640 1 -- 172.21.15.67:0/935106786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff015f5b0 con 0x7f4ff00ced20 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.128+0000 7f4fee7fc640 1 --2- 172.21.15.67:0/935106786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00ced20 0x7f4ff015e7c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00d3690 tx=0x7f4fe402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9968c542753c75a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4fedffb640 1 -- 172.21.15.67:0/935106786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4037070 con 0x7f4ff00ced20 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4fedffb640 1 -- 172.21.15.67:0/935106786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe402ae10 con 0x7f4ff00ced20 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4fedffb640 1 -- 172.21.15.67:0/935106786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4032660 con 0x7f4ff00ced20 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4ff5492640 1 -- 172.21.15.67:0/935106786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00ced20 msgr2=0x7f4ff015e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4ff5492640 1 --2- 172.21.15.67:0/935106786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00ced20 0x7f4ff015e7c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00d3690 tx=0x7f4fe402a180 comp rx=0 tx=0).stop 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4ff5492640 1 -- 172.21.15.67:0/935106786 shutdown_connections 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4ff5492640 1 --2- 172.21.15.67:0/935106786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00ced20 0x7f4ff015e7c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4ff5492640 1 --2- 172.21.15.67:0/935106786 >> v2:172.21.15.176:3300/0 conn(0x7f4ff00cff90 0x7f4ff00b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.129+0000 7f4ff5492640 1 -- 172.21.15.67:0/935106786 >> 172.21.15.67:0/935106786 conn(0x7f4ff009b7d0 msgr2=0x7f4ff009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.130+0000 7f4ff5492640 1 -- 172.21.15.67:0/935106786 shutdown_connections 2024-04-25T15:25:07.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.130+0000 7f4ff5492640 1 -- 172.21.15.67:0/935106786 wait complete. 2024-04-25T15:25:07.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.132+0000 7f4ff5492640 1 Processor -- start 2024-04-25T15:25:07.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.132+0000 7f4ff5492640 1 -- start start 2024-04-25T15:25:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4ff5492640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ff00ced20 0x7f4ff00c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4ff5492640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ff00cff90 0x7f4ff00af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4ff5492640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00d2930 0x7f4ff006d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4ff5492640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff00b2a40 con 0x7f4ff00d2930 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4ff5492640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff00cf130 con 0x7f4ff00cff90 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4ff5492640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff00b2bc0 con 0x7f4ff00ced20 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4feeffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ff00ced20 0x7f4ff00c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4fee7fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ff00cff90 0x7f4ff00af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4feeffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ff00ced20 0x7f4ff00c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:33242/0 (socket says 172.21.15.67:33242) 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.133+0000 7f4fee7fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ff00cff90 0x7f4ff00af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:48778/0 (socket says 172.21.15.67:48778) 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4feeffd640 1 -- 172.21.15.67:0/3316284644 learned_addr learned my addr 172.21.15.67:0/3316284644 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fef7fe640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00d2930 0x7f4ff006d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fee7fc640 1 -- 172.21.15.67:0/3316284644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ff00ced20 msgr2=0x7f4ff00c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fee7fc640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ff00ced20 0x7f4ff00c9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fee7fc640 1 -- 172.21.15.67:0/3316284644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00d2930 msgr2=0x7f4ff006d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fee7fc640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00d2930 0x7f4ff006d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fee7fc640 1 -- 172.21.15.67:0/3316284644 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff00b0030 con 0x7f4ff00cff90 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fee7fc640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ff00cff90 0x7f4ff00af750 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4031000 tx=0x7f4fe40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fd77fe640 1 -- 172.21.15.67:0/3316284644 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4041070 con 0x7f4ff00cff90 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fd77fe640 1 -- 172.21.15.67:0/3316284644 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe403c460 con 0x7f4ff00cff90 2024-04-25T15:25:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4ff5492640 1 -- 172.21.15.67:0/3316284644 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff00b0310 con 0x7f4ff00cff90 2024-04-25T15:25:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4fd77fe640 1 -- 172.21.15.67:0/3316284644 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4037040 con 0x7f4ff00cff90 2024-04-25T15:25:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.134+0000 7f4ff5492640 1 -- 172.21.15.67:0/3316284644 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff00c66b0 con 0x7f4ff00cff90 2024-04-25T15:25:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.135+0000 7f4fd77fe640 1 -- 172.21.15.67:0/3316284644 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4fe4044070 con 0x7f4ff00cff90 2024-04-25T15:25:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.135+0000 7f4fd77fe640 1 -- 172.21.15.67:0/3316284644 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f4fe4046450 con 0x7f4ff00cff90 2024-04-25T15:25:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.136+0000 7f4fd57fa640 1 -- 172.21.15.67:0/3316284644 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc8005290 con 0x7f4ff00cff90 2024-04-25T15:25:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.138+0000 7f4fd77fe640 1 -- 172.21.15.67:0/3316284644 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4fe4046900 con 0x7f4ff00cff90 2024-04-25T15:25:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.200+0000 7f4fd57fa640 1 -- 172.21.15.67:0/3316284644 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fc8005850 con 0x7f4ff00cff90 2024-04-25T15:25:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.201+0000 7f4fd77fe640 1 -- 172.21.15.67:0/3316284644 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4fe4046ae0 con 0x7f4ff00cff90 2024-04-25T15:25:07.203 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:07.203 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:06.951778+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 -- 172.21.15.67:0/3316284644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ff00cff90 msgr2=0x7f4ff00af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ff00cff90 0x7f4ff00af750 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4031000 tx=0x7f4fe40046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 -- 172.21.15.67:0/3316284644 shutdown_connections 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ff00d2930 0x7f4ff006d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ff00cff90 0x7f4ff00af750 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 --2- 172.21.15.67:0/3316284644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ff00ced20 0x7f4ff00c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 -- 172.21.15.67:0/3316284644 >> 172.21.15.67:0/3316284644 conn(0x7f4ff009b7d0 msgr2=0x7f4ff009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 -- 172.21.15.67:0/3316284644 shutdown_connections 2024-04-25T15:25:07.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:07.202+0000 7f4fd57fa640 1 -- 172.21.15.67:0/3316284644 wait complete. 2024-04-25T15:25:07.215 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T15:25:07.215 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T15:25:07.236 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:07.235+0000 7f52077b8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:25:07.264 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:07.263+0000 7f97bf9e3040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:25:07.287 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:07.287+0000 7f52077b8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:25:07.320 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:07.318+0000 7f97bf9e3040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:25:07.406 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:07.405+0000 7f52077b8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:25:07.443 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:07.443+0000 7f97bf9e3040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:25:07.591 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:25:07.591+0000 7f880e21c640 -1 Falling back to public interface 2024-04-25T15:25:07.593 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:25:07.593+0000 7f77b5686640 -1 Falling back to public interface 2024-04-25T15:25:07.603 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:25:07.602+0000 7fec95420640 -1 Falling back to public interface 2024-04-25T15:25:07.603 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:25:07.603+0000 7fe5686db640 -1 Falling back to public interface 2024-04-25T15:25:07.608 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:25:07.607+0000 7f9e23c1b640 -1 Falling back to public interface 2024-04-25T15:25:07.613 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:25:07.612+0000 7fcb52bd4640 -1 Falling back to public interface 2024-04-25T15:25:07.627 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:25:07.627+0000 7fa08cc10640 -1 Falling back to public interface 2024-04-25T15:25:07.628 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:25:07.627+0000 7f2b70ee1640 -1 Falling back to public interface 2024-04-25T15:25:07.628 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:25:07.627+0000 7fa08b51d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2024-04-25T15:25:07.685 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:07.684+0000 7f52077b8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:25:07.731 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:07.730+0000 7f52077b8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:25:07.745 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:07.745+0000 7f97bf9e3040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:25:07.796 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:07.795+0000 7f97bf9e3040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:25:07.886 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:07.885+0000 7f52077b8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:25:07.963 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:07.962+0000 7f97bf9e3040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:25:08.021 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:08.023 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:08.022+0000 7f52077b8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:25:08.072 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:08.071+0000 7f52077b8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:25:08.111 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:08.110+0000 7f97bf9e3040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:25:08.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.124+0000 7f1909346640 1 Processor -- start 2024-04-25T15:25:08.125 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.124+0000 7f1909346640 1 -- start start 2024-04-25T15:25:08.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.125+0000 7f1909346640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f190406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:08.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.125+0000 7f1909346640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f190406e9b0 con 0x7f19040d08f0 2024-04-25T15:25:08.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.125+0000 7f1909346640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f190406ec90 con 0x7f19040b77f0 2024-04-25T15:25:08.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.125+0000 7f1909346640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f190406ef70 con 0x7f19040b5210 2024-04-25T15:25:08.127 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f1902ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f19040d08f0 0x7f19041612b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58718/0 (socket says 172.21.15.67:58718) 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f1902ffd640 1 -- 172.21.15.67:0/1425087239 learned_addr learned my addr 172.21.15.67:0/1425087239 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f19027fc640 1 --2- 172.21.15.67:0/1425087239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f190406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f1901ffb640 1 -- 172.21.15.67:0/1425087239 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729074846 0 0) 0x7f190406e9b0 con 0x7f19040d08f0 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f1901ffb640 1 -- 172.21.15.67:0/1425087239 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18f0003520 con 0x7f19040d08f0 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f1901ffb640 1 -- 172.21.15.67:0/1425087239 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480125092 0 0) 0x7f190406ec90 con 0x7f19040b77f0 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f1901ffb640 1 -- 172.21.15.67:0/1425087239 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f190406e9b0 con 0x7f19040b77f0 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f19027fc640 1 -- 172.21.15.67:0/1425087239 >> v1:172.21.15.176:6789/0 conn(0x7f19040b77f0 legacy=0x7f19040b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f19027fc640 1 -- 172.21.15.67:0/1425087239 >> v1:172.21.15.67:6789/0 conn(0x7f19040d08f0 legacy=0x7f19041612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:08.128 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.126+0000 7f19027fc640 1 -- 172.21.15.67:0/1425087239 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1904160ab0 con 0x7f19040b5210 2024-04-25T15:25:08.129 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.127+0000 7f19027fc640 1 --2- 172.21.15.67:0/1425087239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f190406e0b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18f40097d0 tx=0x7f18f402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8d50b6af2ad22b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.127+0000 7f1901ffb640 1 -- 172.21.15.67:0/1425087239 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4037070 con 0x7f19040b5210 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.127+0000 7f1901ffb640 1 -- 172.21.15.67:0/1425087239 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f4032440 con 0x7f19040b5210 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.127+0000 7f1901ffb640 1 -- 172.21.15.67:0/1425087239 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4032810 con 0x7f19040b5210 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.128+0000 7f1909346640 1 -- 172.21.15.67:0/1425087239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 msgr2=0x7f190406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.128+0000 7f1909346640 1 --2- 172.21.15.67:0/1425087239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f190406e0b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18f40097d0 tx=0x7f18f402a210 comp rx=0 tx=0).stop 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.128+0000 7f1909346640 1 -- 172.21.15.67:0/1425087239 shutdown_connections 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.128+0000 7f1909346640 1 --2- 172.21.15.67:0/1425087239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f190406e0b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.128+0000 7f1909346640 1 -- 172.21.15.67:0/1425087239 >> 172.21.15.67:0/1425087239 conn(0x7f19040d29d0 msgr2=0x7f19040d2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.128+0000 7f1909346640 1 -- 172.21.15.67:0/1425087239 shutdown_connections 2024-04-25T15:25:08.130 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.128+0000 7f1909346640 1 -- 172.21.15.67:0/1425087239 wait complete. 2024-04-25T15:25:08.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.130+0000 7f1909346640 1 Processor -- start 2024-04-25T15:25:08.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.130+0000 7f1909346640 1 -- start start 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f1909346640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f19041fa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f1909346640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19040b77f0 0x7f19040ba660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f1909346640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19040d08f0 0x7f19040baba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f1909346640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19040b5620 con 0x7f19040d08f0 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f1909346640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1904160e20 con 0x7f19040b77f0 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f1909346640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1904163e70 con 0x7f19040b5210 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f19037fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19040d08f0 0x7f19040baba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f19037fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19040d08f0 0x7f19040baba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46170/0 (socket says 172.21.15.67:46170) 2024-04-25T15:25:08.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f19037fe640 1 -- 172.21.15.67:0/3918238749 learned_addr learned my addr 172.21.15.67:0/3918238749 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f1902ffd640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f19041fa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.131+0000 7f19027fc640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19040b77f0 0x7f19040ba660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1902ffd640 1 -- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19040b77f0 msgr2=0x7f19040ba660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1902ffd640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19040b77f0 0x7f19040ba660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1902ffd640 1 -- 172.21.15.67:0/3918238749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19040d08f0 msgr2=0x7f19040baba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1902ffd640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19040d08f0 0x7f19040baba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1902ffd640 1 -- 172.21.15.67:0/3918238749 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19041ea200 con 0x7f19040b5210 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f19037fe640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19040d08f0 0x7f19040baba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1902ffd640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f19041fa700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00c3c0 tx=0x7f18ec001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f18fb7fe640 1 -- 172.21.15.67:0/3918238749 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec017070 con 0x7f19040b5210 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f18fb7fe640 1 -- 172.21.15.67:0/3918238749 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18ec005ce0 con 0x7f19040b5210 2024-04-25T15:25:08.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19041ea4e0 con 0x7f19040b5210 2024-04-25T15:25:08.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f18fb7fe640 1 -- 172.21.15.67:0/3918238749 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0072d0 con 0x7f19040b5210 2024-04-25T15:25:08.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19041eaaa0 con 0x7f19040b5210 2024-04-25T15:25:08.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f18fb7fe640 1 -- 172.21.15.67:0/3918238749 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f18ec01a070 con 0x7f19040b5210 2024-04-25T15:25:08.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.132+0000 7f18fb7fe640 1 -- 172.21.15.67:0/3918238749 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f18ec01cbc0 con 0x7f19040b5210 2024-04-25T15:25:08.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.133+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f190406a160 con 0x7f19040b5210 2024-04-25T15:25:08.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.135+0000 7f18fb7fe640 1 -- 172.21.15.67:0/3918238749 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f18ec007470 con 0x7f19040b5210 2024-04-25T15:25:08.143 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:08.142+0000 7f52077b8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:25:08.166 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:08.165+0000 7f97bf9e3040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:25:08.189 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:08.189+0000 7f52077b8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:25:08.200 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.199+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19040d4100 con 0x7f19040b5210 2024-04-25T15:25:08.201 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.200+0000 7f18fb7fe640 1 -- 172.21.15.67:0/3918238749 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f18ec0076b0 con 0x7f19040b5210 2024-04-25T15:25:08.201 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-25T15:25:08.201 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:06.951778+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:08.202 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 msgr2=0x7f19041fa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:08.202 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f19041fa700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f18ec00c3c0 tx=0x7f18ec001e70 comp rx=0 tx=0).stop 2024-04-25T15:25:08.203 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 shutdown_connections 2024-04-25T15:25:08.203 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19040d08f0 0x7f19040baba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:08.203 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19040b77f0 0x7f19040ba660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:08.203 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 --2- 172.21.15.67:0/3918238749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19040b5210 0x7f19041fa700 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:08.203 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 >> 172.21.15.67:0/3918238749 conn(0x7f19040d29d0 msgr2=0x7f1904075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:08.203 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 shutdown_connections 2024-04-25T15:25:08.203 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:08.201+0000 7f1909346640 1 -- 172.21.15.67:0/3918238749 wait complete. 2024-04-25T15:25:08.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T15:25:08.241 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:08.241+0000 7f97bf9e3040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:25:08.290 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:08.289+0000 7f97bf9e3040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:25:08.665 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:08.664+0000 7f52077b8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:25:08.781 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:08.780+0000 7f52077b8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:25:08.801 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:08.801+0000 7f97bf9e3040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:25:08.925 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:25:08.924+0000 7f97bf9e3040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:25:09.801 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:09.800+0000 7f51a8253640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T15:25:09.801 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:25:09.800+0000 7f51a8253640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T15:25:11.062 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:25:11.061+0000 7f880e21c640 -1 osd.1 0 log_to_monitors true 2024-04-25T15:25:11.085 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:25:11.084+0000 7f77b5686640 -1 osd.6 0 log_to_monitors true 2024-04-25T15:25:11.094 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:25:11.093+0000 7fe5686db640 -1 osd.0 0 log_to_monitors true 2024-04-25T15:25:11.117 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:25:11.116+0000 7fec95420640 -1 osd.7 0 log_to_monitors true 2024-04-25T15:25:11.128 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:25:11.127+0000 7f2b70ee1640 -1 osd.3 0 log_to_monitors true 2024-04-25T15:25:11.134 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:25:11.133+0000 7fa08cc10640 -1 osd.2 0 log_to_monitors true 2024-04-25T15:25:11.140 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:25:11.140+0000 7fcb52bd4640 -1 osd.5 0 log_to_monitors true 2024-04-25T15:25:11.149 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:25:11.149+0000 7f9e23c1b640 -1 osd.4 0 log_to_monitors true 2024-04-25T15:25:14.869 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:25:14.868+0000 7f8809b0c640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T15:25:14.887 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:25:14.887+0000 7f8801907640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:14.919 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:25:14.918+0000 7fec90e12640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T15:25:14.925 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:25:14.924+0000 7f77b161c640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T15:25:14.927 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:25:14.927+0000 7fcb4eb6a640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T15:25:14.932 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:25:14.931+0000 7fec88c0d640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:14.943 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:25:14.942+0000 7fcb46965640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:14.943 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:25:14.942+0000 7f77a9417640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:14.951 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:25:14.950+0000 7f9e1f607640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T15:25:14.962 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:25:14.961+0000 7f9e17402640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:14.987 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:25:14.987+0000 7fe564671640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T15:25:14.998 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:25:14.998+0000 7fe55c46c640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:15.019 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:15.035 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:25:15.035+0000 7fa088504640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T15:25:15.048 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:25:15.048+0000 7fa0802ff640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:15.049 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:25:15.048+0000 7f2b6ce77640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T15:25:15.061 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:25:15.061+0000 7f2b64c72640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:25:15.129 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.128+0000 7f52ebaa8640 1 Processor -- start 2024-04-25T15:25:15.129 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.128+0000 7f52ebaa8640 1 -- start start 2024-04-25T15:25:15.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ebaa8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52e4074110 0x7f52e4167380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:15.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ebaa8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e40b6ed0 con 0x7f52e40b67d0 2024-04-25T15:25:15.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ebaa8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e40b4af0 con 0x7f52e40b58b0 2024-04-25T15:25:15.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ebaa8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f52e40b4dd0 con 0x7f52e4074110 2024-04-25T15:25:15.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52e9043640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f52e40b67d0 0x7f52e4073a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58734/0 (socket says 172.21.15.67:58734) 2024-04-25T15:25:15.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52e9043640 1 -- 172.21.15.67:0/1940945245 learned_addr learned my addr 172.21.15.67:0/1940945245 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:15.132 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ea045640 1 --2- 172.21.15.67:0/1940945245 >> v2:172.21.15.67:3300/0 conn(0x7f52e4074110 0x7f52e4167380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:15.132 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ea045640 1 -- 172.21.15.67:0/1940945245 >> v1:172.21.15.176:6789/0 conn(0x7f52e40b58b0 legacy=0x7f52e406f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:15.132 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ea045640 1 -- 172.21.15.67:0/1940945245 >> v1:172.21.15.67:6789/0 conn(0x7f52e40b67d0 legacy=0x7f52e4073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:15.132 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52ea045640 1 -- 172.21.15.67:0/1940945245 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e40b50b0 con 0x7f52e4074110 2024-04-25T15:25:15.132 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.130+0000 7f52e8842640 1 -- 172.21.15.67:0/1940945245 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4180344264 0 0) 0x7f52e40b6ed0 con 0x7f52e40b67d0 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.131+0000 7f52ea045640 1 --2- 172.21.15.67:0/1940945245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e4074110 0x7f52e4167380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009d90 tx=0x7f52d402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c224d50af7ad8dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.131+0000 7f52e8842640 1 -- 172.21.15.67:0/1940945245 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4007740 con 0x7f52e4074110 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.131+0000 7f52e8842640 1 -- 172.21.15.67:0/1940945245 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d40078e0 con 0x7f52e4074110 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.131+0000 7f52e8842640 1 -- 172.21.15.67:0/1940945245 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4007cb0 con 0x7f52e4074110 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.132+0000 7f52ebaa8640 1 -- 172.21.15.67:0/1940945245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e4074110 msgr2=0x7f52e4167380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.132+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/1940945245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e4074110 0x7f52e4167380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009d90 tx=0x7f52d402a660 comp rx=0 tx=0).stop 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.132+0000 7f52ebaa8640 1 -- 172.21.15.67:0/1940945245 shutdown_connections 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.132+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/1940945245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e4074110 0x7f52e4167380 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:15.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.132+0000 7f52ebaa8640 1 -- 172.21.15.67:0/1940945245 >> 172.21.15.67:0/1940945245 conn(0x7f52e406cfe0 msgr2=0x7f52e40b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:15.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.132+0000 7f52ebaa8640 1 -- 172.21.15.67:0/1940945245 shutdown_connections 2024-04-25T15:25:15.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.132+0000 7f52ebaa8640 1 -- 172.21.15.67:0/1940945245 wait complete. 2024-04-25T15:25:15.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.134+0000 7f52ebaa8640 1 Processor -- start 2024-04-25T15:25:15.135 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.134+0000 7f52ebaa8640 1 -- start start 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ebaa8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52e40b58b0 0x7f52e415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ebaa8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52e40b67d0 0x7f52e414c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ebaa8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e415f880 0x7f52e414b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ebaa8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52e415c9d0 con 0x7f52e415f880 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ebaa8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f52e40b6be0 con 0x7f52e40b58b0 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ebaa8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f52e415cb50 con 0x7f52e40b67d0 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ea045640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e415f880 0x7f52e414b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:15.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ea045640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e415f880 0x7f52e414b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46230/0 (socket says 172.21.15.67:46230) 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.135+0000 7f52ea045640 1 -- 172.21.15.67:0/4079111176 learned_addr learned my addr 172.21.15.67:0/4079111176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52e9043640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52e40b67d0 0x7f52e414c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52e9844640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52e40b58b0 0x7f52e415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ea045640 1 -- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52e40b67d0 msgr2=0x7f52e414c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ea045640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52e40b67d0 0x7f52e414c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ea045640 1 -- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52e40b58b0 msgr2=0x7f52e415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ea045640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52e40b58b0 0x7f52e415da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ea045640 1 -- 172.21.15.67:0/4079111176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e4149c00 con 0x7f52e415f880 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ea045640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e415f880 0x7f52e414b1a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009eb0 tx=0x7f52d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52e27fc640 1 -- 172.21.15.67:0/4079111176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d403f070 con 0x7f52e415f880 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52e27fc640 1 -- 172.21.15.67:0/4079111176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d4031c90 con 0x7f52e415f880 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e414cb90 con 0x7f52e415f880 2024-04-25T15:25:15.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52e27fc640 1 -- 172.21.15.67:0/4079111176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4043630 con 0x7f52e415f880 2024-04-25T15:25:15.139 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.136+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e414a3e0 con 0x7f52e415f880 2024-04-25T15:25:15.139 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.137+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e40cdc10 con 0x7f52e415f880 2024-04-25T15:25:15.139 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.138+0000 7f52e27fc640 1 -- 172.21.15.67:0/4079111176 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 96726+0+0 (secure 0 0 0) 0x7f52d403b410 con 0x7f52e415f880 2024-04-25T15:25:15.139 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.138+0000 7f52e27fc640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52d0074420 0x7f52d0076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:15.140 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.139+0000 7f52e27fc640 1 -- 172.21.15.67:0/4079111176 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f52d40b5970 con 0x7f52e415f880 2024-04-25T15:25:15.140 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.139+0000 7f52e9844640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52d0074420 0x7f52d0076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:15.140 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.139+0000 7f52e9844640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52d0074420 0x7f52d0076930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52dc000f60 tx=0x7f52dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:15.144 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.144+0000 7f52e27fc640 1 -- 172.21.15.67:0/4079111176 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52e40cdc10 con 0x7f52e415f880 2024-04-25T15:25:15.333 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.332+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52e40aecf0 con 0x7f52e415f880 2024-04-25T15:25:15.333 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.333+0000 7f52e27fc640 1 -- 172.21.15.67:0/4079111176 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f52e40aecf0 con 0x7f52e415f880 2024-04-25T15:25:15.334 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-25T15:25:15.334 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":12,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:12.800151+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.334+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52d0074420 msgr2=0x7f52d0076930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.334+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52d0074420 0x7f52d0076930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52dc000f60 tx=0x7f52dc009040 comp rx=0 tx=0).stop 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.334+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e415f880 msgr2=0x7f52e414b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.334+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e415f880 0x7f52e414b1a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009eb0 tx=0x7f52d4004a40 comp rx=0 tx=0).stop 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 shutdown_connections 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52d0074420 0x7f52d0076930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52e415f880 0x7f52e414b1a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52e40b67d0 0x7f52e414c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 --2- 172.21.15.67:0/4079111176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52e40b58b0 0x7f52e415da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 >> 172.21.15.67:0/4079111176 conn(0x7f52e406cfe0 msgr2=0x7f52e415a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 shutdown_connections 2024-04-25T15:25:15.336 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:15.335+0000 7f52ebaa8640 1 -- 172.21.15.67:0/4079111176 wait complete. 2024-04-25T15:25:15.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T15:25:22.152 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:22.269 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.268+0000 7fbe627c0640 1 Processor -- start 2024-04-25T15:25:22.269 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.268+0000 7fbe627c0640 1 -- start start 2024-04-25T15:25:22.271 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe627c0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbe5c0d08f0 0x7fbe5c0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.271 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe627c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe5c1612b0 0x7fbe5c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.271 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe627c0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.271 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe627c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5c06ef70 con 0x7fbe5c1612b0 2024-04-25T15:25:22.271 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe627c0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5c160e20 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe627c0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5c0b8140 con 0x7fbe5c0b77f0 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe5c1612b0 0x7fbe5c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe5c1612b0 0x7fbe5c06e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59890/0 (socket says 172.21.15.67:59890) 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/3167678289 learned_addr learned my addr 172.21.15.67:0/3167678289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe60d5d640 1 --2- 172.21.15.67:0/3167678289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5bfff640 1 --2- 172.21.15.67:0/3167678289 >> v2:172.21.15.176:3300/0 conn(0x7fbe5c0d08f0 0x7fbe5c0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/3167678289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 msgr2=0x7fbe5c0b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 --2- 172.21.15.67:0/3167678289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0b7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/3167678289 >> v2:172.21.15.176:3300/0 conn(0x7fbe5c0d08f0 msgr2=0x7fbe5c0b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 --2- 172.21.15.67:0/3167678289 >> v2:172.21.15.176:3300/0 conn(0x7fbe5c0d08f0 0x7fbe5c0b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.272 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.270+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/3167678289 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5c06e9b0 con 0x7fbe5c1612b0 2024-04-25T15:25:22.273 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.271+0000 7fbe5b7fe640 1 --2- 172.21.15.67:0/3167678289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c1612b0 0x7fbe5c06e0b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c06f280 tx=0x7fbe4c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8a31d5c1765d98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.273 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.271+0000 7fbe5affd640 1 -- 172.21.15.67:0/3167678289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c037070 con 0x7fbe5c1612b0 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.271+0000 7fbe5affd640 1 -- 172.21.15.67:0/3167678289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4c032440 con 0x7fbe5c1612b0 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.271+0000 7fbe5affd640 1 -- 172.21.15.67:0/3167678289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c032810 con 0x7fbe5c1612b0 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 -- 172.21.15.67:0/3167678289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c1612b0 msgr2=0x7fbe5c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 --2- 172.21.15.67:0/3167678289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c1612b0 0x7fbe5c06e0b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c06f280 tx=0x7fbe4c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 -- 172.21.15.67:0/3167678289 shutdown_connections 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 --2- 172.21.15.67:0/3167678289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 --2- 172.21.15.67:0/3167678289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c1612b0 0x7fbe5c06e0b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 --2- 172.21.15.67:0/3167678289 >> v2:172.21.15.176:3300/0 conn(0x7fbe5c0d08f0 0x7fbe5c0b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 -- 172.21.15.67:0/3167678289 >> 172.21.15.67:0/3167678289 conn(0x7fbe5c0d29d0 msgr2=0x7fbe5c0d2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 -- 172.21.15.67:0/3167678289 shutdown_connections 2024-04-25T15:25:22.274 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.272+0000 7fbe627c0640 1 -- 172.21.15.67:0/3167678289 wait complete. 2024-04-25T15:25:22.280 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.274+0000 7fbe627c0640 1 Processor -- start 2024-04-25T15:25:22.280 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.274+0000 7fbe627c0640 1 -- start start 2024-04-25T15:25:22.280 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.275+0000 7fbe627c0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.275+0000 7fbe627c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c0d08f0 0x7fbe5c14f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.275+0000 7fbe627c0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe5c1612b0 0x7fbe5c0ba2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.275+0000 7fbe627c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c06e5f0 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.275+0000 7fbe627c0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c160ab0 con 0x7fbe5c1612b0 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.275+0000 7fbe627c0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5c06e770 con 0x7fbe5c0b77f0 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c0d08f0 0x7fbe5c14f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c0d08f0 0x7fbe5c14f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59902/0 (socket says 172.21.15.67:59902) 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/172596320 learned_addr learned my addr 172.21.15.67:0/172596320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5bfff640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe60d5d640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe5c1612b0 0x7fbe5c0ba2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 msgr2=0x7fbe5c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe5c1612b0 msgr2=0x7fbe5c0ba2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.281 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe5c1612b0 0x7fbe5c0ba2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 -- 172.21.15.67:0/172596320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5c14f650 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe5b7fe640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c0d08f0 0x7fbe5c14f110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c000c00 tx=0x7fbe4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe58ff9640 1 -- 172.21.15.67:0/172596320 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c041020 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe58ff9640 1 -- 172.21.15.67:0/172596320 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4c03d620 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5c1f7700 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.276+0000 7fbe58ff9640 1 -- 172.21.15.67:0/172596320 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c037040 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.277+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5c1f7cc0 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.277+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe5c06a160 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.278+0000 7fbe58ff9640 1 -- 172.21.15.67:0/172596320 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbe4c044020 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.279+0000 7fbe58ff9640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe3c0744b0 0x7fbe3c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.279+0000 7fbe58ff9640 1 -- 172.21.15.67:0/172596320 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbe4c0b54b0 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.279+0000 7fbe5bfff640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe3c0744b0 0x7fbe3c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.282 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.279+0000 7fbe5bfff640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe3c0744b0 0x7fbe3c0769c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0bb9b0 tx=0x7fbe44009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.284 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.284+0000 7fbe58ff9640 1 -- 172.21.15.67:0/172596320 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe4c04b030 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.473 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.473+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe5c0d4100 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.474 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.474+0000 7fbe58ff9640 1 -- 172.21.15.67:0/172596320 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7fbe4c07f940 con 0x7fbe5c0d08f0 2024-04-25T15:25:22.474 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-25T15:25:22.475 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:18.877686+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe3c0744b0 msgr2=0x7fbe3c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe3c0744b0 0x7fbe3c0769c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0bb9b0 tx=0x7fbe44009190 comp rx=0 tx=0).stop 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c0d08f0 msgr2=0x7fbe5c14f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c0d08f0 0x7fbe5c14f110 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c000c00 tx=0x7fbe4c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 shutdown_connections 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe3c0744b0 0x7fbe3c0769c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe5c1612b0 0x7fbe5c0ba2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe5c0d08f0 0x7fbe5c14f110 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 --2- 172.21.15.67:0/172596320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe5c0b77f0 0x7fbe5c0cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.477 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 >> 172.21.15.67:0/172596320 conn(0x7fbe5c0d29d0 msgr2=0x7fbe5c0d53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:22.478 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 shutdown_connections 2024-04-25T15:25:22.478 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:25:22.476+0000 7fbe627c0640 1 -- 172.21.15.67:0/172596320 wait complete. 2024-04-25T15:25:22.487 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T15:25:22.487 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T15:25:22.487 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:25:22.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:25:22.490 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:25:22.490 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:25:22.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:25:22.492 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:25:22.492 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:25:22.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:25:22.494 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:25:22.495 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:25:22.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:25:22.497 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:25:22.497 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T15:25:22.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:22.577 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:25:22.577 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:25:22.593 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:25:22.593 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:25:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.614+0000 7fa7f53b4640 1 Processor -- start 2024-04-25T15:25:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.614+0000 7fa7f53b4640 1 -- start start 2024-04-25T15:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7f53b4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa7f00b3d60 0x7fa7f00b7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7f53b4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7f00b8190 con 0x7fa7f00d2570 2024-04-25T15:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7f53b4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7f00b2140 con 0x7fa7f00b7d80 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7f53b4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa7f00cee00 con 0x7fa7f00b3d60 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7eeffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa7f00d2570 0x7fa7f00cfe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43194/0 (socket says 172.21.15.67:43194) 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7eeffd640 1 -- 172.21.15.67:0/1018588221 learned_addr learned my addr 172.21.15.67:0/1018588221 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7ee7fc640 1 --2- 172.21.15.67:0/1018588221 >> v2:172.21.15.176:3300/0 conn(0x7fa7f00b3d60 0x7fa7f00b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391094150 0 0) 0x7fa7e4001110 con 0x7fa7f00d2570 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d8003520 con 0x7fa7f00d2570 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026258580 0 0) 0x7fa7f00b2140 con 0x7fa7f00b7d80 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e4001110 con 0x7fa7f00b7d80 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7ee7fc640 1 -- 172.21.15.67:0/1018588221 >> v1:172.21.15.176:6789/0 conn(0x7fa7f00b7d80 legacy=0x7fa7f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7ee7fc640 1 -- 172.21.15.67:0/1018588221 >> v1:172.21.15.67:6789/0 conn(0x7fa7f00d2570 legacy=0x7fa7f00cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 404336392 0 0) 0x7fa7d8003520 con 0x7fa7f00d2570 2024-04-25T15:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.616+0000 7fa7ee7fc640 1 -- 172.21.15.67:0/1018588221 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f00cf0e0 con 0x7fa7f00b3d60 2024-04-25T15:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.617+0000 7fa7ee7fc640 1 --2- 172.21.15.67:0/1018588221 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00b3d60 0x7fa7f00b7840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00097a0 tx=0x7fa7e002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94e3b4778f2db54f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.617+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0037070 con 0x7fa7f00b3d60 2024-04-25T15:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.617+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7e002ac50 con 0x7fa7f00b3d60 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.617+0000 7fa7edffb640 1 -- 172.21.15.67:0/1018588221 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4001840 con 0x7fa7f00b3d60 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.618+0000 7fa7f53b4640 1 -- 172.21.15.67:0/1018588221 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00b3d60 msgr2=0x7fa7f00b7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.618+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/1018588221 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00b3d60 0x7fa7f00b7840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00097a0 tx=0x7fa7e002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.618+0000 7fa7f53b4640 1 -- 172.21.15.67:0/1018588221 shutdown_connections 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.618+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/1018588221 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00b3d60 0x7fa7f00b7840 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.618+0000 7fa7f53b4640 1 -- 172.21.15.67:0/1018588221 >> 172.21.15.67:0/1018588221 conn(0x7fa7f009b7d0 msgr2=0x7fa7f009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.618+0000 7fa7f53b4640 1 -- 172.21.15.67:0/1018588221 shutdown_connections 2024-04-25T15:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.618+0000 7fa7f53b4640 1 -- 172.21.15.67:0/1018588221 wait complete. 2024-04-25T15:25:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.620+0000 7fa7f53b4640 1 Processor -- start 2024-04-25T15:25:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.620+0000 7fa7f53b4640 1 -- start start 2024-04-25T15:25:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7f53b4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7f00b3d60 0x7fa7f0156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7f53b4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7f00b7d80 0x7fa7f00c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7f53b4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00d2570 0x7fa7f00c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7f53b4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f00cd130 con 0x7fa7f00b7d80 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7f53b4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f00b4340 con 0x7fa7f00d2570 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7f53b4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa7f00cd2b0 con 0x7fa7f00b3d60 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7ee7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7f00b7d80 0x7fa7f00c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.621+0000 7fa7ee7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7f00b7d80 0x7fa7f00c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59948/0 (socket says 172.21.15.67:59948) 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ee7fc640 1 -- 172.21.15.67:0/3534660905 learned_addr learned my addr 172.21.15.67:0/3534660905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7eeffd640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7f00b3d60 0x7fa7f0156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ef7fe640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00d2570 0x7fa7f00c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ee7fc640 1 -- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7f00b3d60 msgr2=0x7fa7f0156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ee7fc640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7f00b3d60 0x7fa7f0156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ee7fc640 1 -- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00d2570 msgr2=0x7fa7f00c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ee7fc640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00d2570 0x7fa7f00c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ee7fc640 1 -- 172.21.15.67:0/3534660905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f00b1d40 con 0x7fa7f00b7d80 2024-04-25T15:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7ee7fc640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7f00b7d80 0x7fa7f00c78e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00098c0 tx=0x7fa7e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7d77fe640 1 -- 172.21.15.67:0/3534660905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0041070 con 0x7fa7f00b7d80 2024-04-25T15:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7d77fe640 1 -- 172.21.15.67:0/3534660905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7e003c460 con 0x7fa7f00b7d80 2024-04-25T15:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f00c8480 con 0x7fa7f00b7d80 2024-04-25T15:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7d77fe640 1 -- 172.21.15.67:0/3534660905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0037040 con 0x7fa7f00b7d80 2024-04-25T15:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.622+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f00c8a40 con 0x7fa7f00b7d80 2024-04-25T15:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.623+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7f0156560 con 0x7fa7f00b7d80 2024-04-25T15:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.624+0000 7fa7d77fe640 1 -- 172.21.15.67:0/3534660905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa7e0044070 con 0x7fa7f00b7d80 2024-04-25T15:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.624+0000 7fa7d77fe640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7dc0744b0 0x7fa7dc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.624+0000 7fa7d77fe640 1 -- 172.21.15.67:0/3534660905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa7e00b50e0 con 0x7fa7f00b7d80 2024-04-25T15:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.624+0000 7fa7eeffd640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7dc0744b0 0x7fa7dc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.625+0000 7fa7eeffd640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7dc0744b0 0x7fa7dc0769c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4000fd0 tx=0x7fa7e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.629+0000 7fa7d77fe640 1 -- 172.21.15.67:0/3534660905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa7f0156560 con 0x7fa7f00b7d80 2024-04-25T15:25:22.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.818+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7f0050740 con 0x7fa7f00b7d80 2024-04-25T15:25:22.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.818+0000 7fa7d77fe640 1 -- 172.21.15.67:0/3534660905 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7fa7e007fce0 con 0x7fa7f00b7d80 2024-04-25T15:25:22.819 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:22.820 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:18.877686+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7dc0744b0 msgr2=0x7fa7dc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7dc0744b0 0x7fa7dc0769c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4000fd0 tx=0x7fa7e4009040 comp rx=0 tx=0).stop 2024-04-25T15:25:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7f00b7d80 msgr2=0x7fa7f00c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7f00b7d80 0x7fa7f00c78e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00098c0 tx=0x7fa7e00046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 shutdown_connections 2024-04-25T15:25:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7dc0744b0 0x7fa7dc0769c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7f00d2570 0x7fa7f00c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7f00b7d80 0x7fa7f00c78e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 --2- 172.21.15.67:0/3534660905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7f00b3d60 0x7fa7f0156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 >> 172.21.15.67:0/3534660905 conn(0x7fa7f009b7d0 msgr2=0x7fa7f015fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 shutdown_connections 2024-04-25T15:25:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.821+0000 7fa7f53b4640 1 -- 172.21.15.67:0/3534660905 wait complete. 2024-04-25T15:25:22.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:22.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.947+0000 7f7f41a1a640 1 Processor -- start 2024-04-25T15:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.947+0000 7f7f41a1a640 1 -- start start 2024-04-25T15:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f41a1a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f3c0b8290 0x7f7f3c160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f41a1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c15df80 0x7f7f3c160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f41a1a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3c1615f0 con 0x7f7f3c164b00 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f41a1a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7f3c164670 con 0x7f7f3c0b8290 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f41a1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7f3c164020 con 0x7f7f3c15df80 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f33fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f3c0b8290 0x7f7f3c160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f33fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f3c0b8290 0x7f7f3c160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59950/0 (socket says 172.21.15.67:59950) 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.948+0000 7f7f33fff640 1 -- 172.21.15.67:0/3632828298 learned_addr learned my addr 172.21.15.67:0/3632828298 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.949+0000 7f7f3b7fe640 1 --2- 172.21.15.67:0/3632828298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c15df80 0x7f7f3c160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.949+0000 7f7f33fff640 1 -- 172.21.15.67:0/3632828298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c15df80 msgr2=0x7f7f3c160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.949+0000 7f7f33fff640 1 --2- 172.21.15.67:0/3632828298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c15df80 0x7f7f3c160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.949+0000 7f7f33fff640 1 -- 172.21.15.67:0/3632828298 >> v1:172.21.15.176:6789/0 conn(0x7f7f3c164b00 legacy=0x7f7f3c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.949+0000 7f7f33fff640 1 -- 172.21.15.67:0/3632828298 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c164300 con 0x7f7f3c0b8290 2024-04-25T15:25:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.949+0000 7f7f33fff640 1 --2- 172.21.15.67:0/3632828298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c0b8290 0x7f7f3c160890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009870 tx=0x7f7f2402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4939a107a810b460 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f3a7fc640 1 -- 172.21.15.67:0/3632828298 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24037070 con 0x7f7f3c0b8290 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f3a7fc640 1 -- 172.21.15.67:0/3632828298 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f24032440 con 0x7f7f3c0b8290 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f3a7fc640 1 -- 172.21.15.67:0/3632828298 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24032810 con 0x7f7f3c0b8290 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 -- 172.21.15.67:0/3632828298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c0b8290 msgr2=0x7f7f3c160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/3632828298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c0b8290 0x7f7f3c160890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009870 tx=0x7f7f2402a270 comp rx=0 tx=0).stop 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 -- 172.21.15.67:0/3632828298 shutdown_connections 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/3632828298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c15df80 0x7f7f3c160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/3632828298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c0b8290 0x7f7f3c160890 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 -- 172.21.15.67:0/3632828298 >> 172.21.15.67:0/3632828298 conn(0x7f7f3c0d2bc0 msgr2=0x7f7f3c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 -- 172.21.15.67:0/3632828298 shutdown_connections 2024-04-25T15:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.950+0000 7f7f41a1a640 1 -- 172.21.15.67:0/3632828298 wait complete. 2024-04-25T15:25:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.952+0000 7f7f41a1a640 1 Processor -- start 2024-04-25T15:25:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.952+0000 7f7f41a1a640 1 -- start start 2024-04-25T15:25:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.953+0000 7f7f41a1a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f3c15df80 0x7f7f3c1faa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.953+0000 7f7f41a1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c164b00 0x7f7f3c154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.953+0000 7f7f41a1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c152bc0 0x7f7f3c152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.953+0000 7f7f41a1a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c0b4a00 con 0x7f7f3c164b00 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.953+0000 7f7f41a1a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7f3c0b4880 con 0x7f7f3c15df80 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.953+0000 7f7f41a1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7f3c0b4b80 con 0x7f7f3c152bc0 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.953+0000 7f7f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c164b00 0x7f7f3c154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c164b00 0x7f7f3c154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59960/0 (socket says 172.21.15.67:59960) 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 -- 172.21.15.67:0/2840494280 learned_addr learned my addr 172.21.15.67:0/2840494280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f3b7fe640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c152bc0 0x7f7f3c152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f3affd640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f3c15df80 0x7f7f3c1faa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 -- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c152bc0 msgr2=0x7f7f3c152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c152bc0 0x7f7f3c152fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 -- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f3c15df80 msgr2=0x7f7f3c1faa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f3c15df80 0x7f7f3c1faa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 -- 172.21.15.67:0/2840494280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f3c153510 con 0x7f7f3c164b00 2024-04-25T15:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f33fff640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c164b00 0x7f7f3c154190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7f24000c00 tx=0x7f7f240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f38ff9640 1 -- 172.21.15.67:0/2840494280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24037040 con 0x7f7f3c164b00 2024-04-25T15:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f38ff9640 1 -- 172.21.15.67:0/2840494280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f24042070 con 0x7f7f3c164b00 2024-04-25T15:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f38ff9640 1 -- 172.21.15.67:0/2840494280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24030880 con 0x7f7f3c164b00 2024-04-25T15:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3c1f74b0 con 0x7f7f3c164b00 2024-04-25T15:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.954+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3c1fa280 con 0x7f7f3c164b00 2024-04-25T15:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.955+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f3c050740 con 0x7f7f3c164b00 2024-04-25T15:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.956+0000 7f7f38ff9640 1 -- 172.21.15.67:0/2840494280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7f24030a20 con 0x7f7f3c164b00 2024-04-25T15:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.956+0000 7f7f38ff9640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7f1c0744b0 0x7f7f1c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.957+0000 7f7f38ff9640 1 -- 172.21.15.67:0/2840494280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f7f240b5660 con 0x7f7f3c164b00 2024-04-25T15:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.957+0000 7f7f3affd640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7f1c0744b0 0x7f7f1c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.957+0000 7f7f3affd640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7f1c0744b0 0x7f7f1c0769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0016e0 tx=0x7f7f2c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:22.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:22.961+0000 7f7f38ff9640 1 -- 172.21.15.67:0/2840494280 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f2407faf0 con 0x7f7f3c164b00 2024-04-25T15:25:23.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.144+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f3c0d4170 con 0x7f7f3c164b00 2024-04-25T15:25:23.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.145+0000 7f7f38ff9640 1 -- 172.21.15.67:0/2840494280 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f7f24083df0 con 0x7f7f3c164b00 2024-04-25T15:25:23.146 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7f1c0744b0 msgr2=0x7f7f1c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7f1c0744b0 0x7f7f1c0769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0016e0 tx=0x7f7f2c009040 comp rx=0 tx=0).stop 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c164b00 msgr2=0x7f7f3c154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c164b00 0x7f7f3c154190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7f24000c00 tx=0x7f7f240046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 shutdown_connections 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7f1c0744b0 0x7f7f1c0769c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f3c152bc0 0x7f7f3c152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f3c164b00 0x7f7f3c154190 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.146+0000 7f7f41a1a640 1 --2- 172.21.15.67:0/2840494280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f3c15df80 0x7f7f3c1faa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.147+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 >> 172.21.15.67:0/2840494280 conn(0x7f7f3c0d2bc0 msgr2=0x7f7f3c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.147+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 shutdown_connections 2024-04-25T15:25:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.147+0000 7f7f41a1a640 1 -- 172.21.15.67:0/2840494280 wait complete. 2024-04-25T15:25:23.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:25:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.271+0000 7f1bf1a20640 1 Processor -- start 2024-04-25T15:25:23.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.271+0000 7f1bf1a20640 1 -- start start 2024-04-25T15:25:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1bf1a20640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1bec0d0e00 0x7f1bec0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1bf1a20640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1bec06c5c0 0x7f1bec06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1bf1a20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1bf1a20640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1bec06cf10 con 0x7f1bec0d0e00 2024-04-25T15:25:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1bf1a20640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1bec0d1850 con 0x7f1bec06c5c0 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1bf1a20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec160de0 con 0x7f1bec0b7810 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1bec0d0e00 0x7f1bec0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1bec0d0e00 0x7f1bec0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59972/0 (socket says 172.21.15.67:59972) 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 -- 172.21.15.67:0/4074884459 learned_addr learned my addr 172.21.15.67:0/4074884459 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1bea7fc640 1 --2- 172.21.15.67:0/4074884459 >> v2:172.21.15.176:3300/0 conn(0x7f1bec06c5c0 0x7f1bec06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beb7fe640 1 --2- 172.21.15.67:0/4074884459 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 -- 172.21.15.67:0/4074884459 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 msgr2=0x7f1bec0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 --2- 172.21.15.67:0/4074884459 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 -- 172.21.15.67:0/4074884459 >> v2:172.21.15.176:3300/0 conn(0x7f1bec06c5c0 msgr2=0x7f1bec06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 --2- 172.21.15.67:0/4074884459 >> v2:172.21.15.176:3300/0 conn(0x7f1bec06c5c0 0x7f1bec06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.273+0000 7f1beaffd640 1 -- 172.21.15.67:0/4074884459 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec1610c0 con 0x7f1bec0d0e00 2024-04-25T15:25:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.274+0000 7f1beaffd640 1 --2- 172.21.15.67:0/4074884459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0d1210 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1bec08f170 tx=0x7f1be402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93253f46985c10b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.274+0000 7f1be9ffb640 1 -- 172.21.15.67:0/4074884459 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4037070 con 0x7f1bec0d0e00 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.274+0000 7f1be9ffb640 1 -- 172.21.15.67:0/4074884459 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be4032440 con 0x7f1bec0d0e00 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.274+0000 7f1be9ffb640 1 -- 172.21.15.67:0/4074884459 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4032810 con 0x7f1bec0d0e00 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 -- 172.21.15.67:0/4074884459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 msgr2=0x7f1bec0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/4074884459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0d1210 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1bec08f170 tx=0x7f1be402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 -- 172.21.15.67:0/4074884459 shutdown_connections 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/4074884459 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/4074884459 >> v2:172.21.15.176:3300/0 conn(0x7f1bec06c5c0 0x7f1bec06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/4074884459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0d1210 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 -- 172.21.15.67:0/4074884459 >> 172.21.15.67:0/4074884459 conn(0x7f1bec0d5d90 msgr2=0x7f1bec0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 -- 172.21.15.67:0/4074884459 shutdown_connections 2024-04-25T15:25:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.275+0000 7f1bf1a20640 1 -- 172.21.15.67:0/4074884459 wait complete. 2024-04-25T15:25:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.277+0000 7f1bf1a20640 1 Processor -- start 2024-04-25T15:25:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.277+0000 7f1bf1a20640 1 -- start start 2024-04-25T15:25:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1bf1a20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bec06c5c0 0x7f1bec15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1bf1a20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec1e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1bf1a20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1bf1a20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec0b5320 con 0x7f1bec0d0e00 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1bf1a20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec160ad0 con 0x7f1bec06c5c0 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1bf1a20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec164e00 con 0x7f1bec0b7810 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1beb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1beb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0ce3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59978/0 (socket says 172.21.15.67:59978) 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.278+0000 7f1beb7fe640 1 -- 172.21.15.67:0/852357789 learned_addr learned my addr 172.21.15.67:0/852357789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1bea7fc640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec1e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beaffd640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bec06c5c0 0x7f1bec15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beb7fe640 1 -- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 msgr2=0x7f1bec1e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beb7fe640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec1e9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beb7fe640 1 -- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bec06c5c0 msgr2=0x7f1bec15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beb7fe640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bec06c5c0 0x7f1bec15a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beb7fe640 1 -- 172.21.15.67:0/852357789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec15abe0 con 0x7f1bec0d0e00 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beaffd640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bec06c5c0 0x7f1bec15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:23.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1beb7fe640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0ce3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc007e90 tx=0x7f1bdc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1be3fff640 1 -- 172.21.15.67:0/852357789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc014420 con 0x7f1bec0d0e00 2024-04-25T15:25:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1be3fff640 1 -- 172.21.15.67:0/852357789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bdc019070 con 0x7f1bec0d0e00 2024-04-25T15:25:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1be3fff640 1 -- 172.21.15.67:0/852357789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc014800 con 0x7f1bec0d0e00 2024-04-25T15:25:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bec1fb3b0 con 0x7f1bec0d0e00 2024-04-25T15:25:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.279+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bec1fab10 con 0x7f1bec0d0e00 2024-04-25T15:25:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.280+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bec050740 con 0x7f1bec0d0e00 2024-04-25T15:25:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.281+0000 7f1be3fff640 1 -- 172.21.15.67:0/852357789 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1bdc0149a0 con 0x7f1bec0d0e00 2024-04-25T15:25:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.281+0000 7f1be3fff640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bcc0744b0 0x7f1bcc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.281+0000 7f1be3fff640 1 -- 172.21.15.67:0/852357789 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f1bdc095660 con 0x7f1bec0d0e00 2024-04-25T15:25:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.281+0000 7f1beaffd640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bcc0744b0 0x7f1bcc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.282+0000 7f1beaffd640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bcc0744b0 0x7f1bcc0769c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1be4034000 tx=0x7f1be402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.286+0000 7f1be3fff640 1 -- 172.21.15.67:0/852357789 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1bec15abe0 con 0x7f1bec0d0e00 2024-04-25T15:25:23.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.510+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bec0d6340 con 0x7f1bec0d0e00 2024-04-25T15:25:23.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.510+0000 7f1be3fff640 1 -- 172.21.15.67:0/852357789 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f1bdc05fb80 con 0x7f1bec0d0e00 2024-04-25T15:25:23.512 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:23.512 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:25:23.512 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bcc0744b0 msgr2=0x7f1bcc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bcc0744b0 0x7f1bcc0769c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1be4034000 tx=0x7f1be402e000 comp rx=0 tx=0).stop 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 msgr2=0x7f1bec0ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0ce3c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc007e90 tx=0x7f1bdc00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 shutdown_connections 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bcc0744b0 0x7f1bcc0769c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bec0d0e00 0x7f1bec0ce3c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bec0b7810 0x7f1bec1e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.512+0000 7f1bf1a20640 1 --2- 172.21.15.67:0/852357789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bec06c5c0 0x7f1bec15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.513+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 >> 172.21.15.67:0/852357789 conn(0x7f1bec0d5d90 msgr2=0x7f1bec1641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.513+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 shutdown_connections 2024-04-25T15:25:23.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:23.513+0000 7f1bf1a20640 1 -- 172.21.15.67:0/852357789 wait complete. 2024-04-25T15:25:25.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:25:25.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.651+0000 7f9e3d53f640 1 Processor -- start 2024-04-25T15:25:25.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.651+0000 7f9e3d53f640 1 -- start start 2024-04-25T15:25:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.652+0000 7f9e3d53f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e380d0de0 0x7f9e380d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.652+0000 7f9e3d53f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9e3806c5c0 0x7f9e3806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.652+0000 7f9e3d53f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3806cf10 con 0x7f9e380b77f0 2024-04-25T15:25:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.652+0000 7f9e3d53f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9e380d5a40 con 0x7f9e3806c5c0 2024-04-25T15:25:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.652+0000 7f9e3d53f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9e380d1830 con 0x7f9e380d0de0 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e377fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9e380b77f0 0x7f9e380b7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43202/0 (socket says 172.21.15.67:43202) 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e377fe640 1 -- 172.21.15.67:0/1405685014 learned_addr learned my addr 172.21.15.67:0/1405685014 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e380d0de0 0x7f9e380d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e367fc640 1 --2- 172.21.15.67:0/1405685014 >> v2:172.21.15.176:3300/0 conn(0x7f9e3806c5c0 0x7f9e3806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 504095517 0 0) 0x7f9e30001490 con 0x7f9e380b77f0 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e20003520 con 0x7f9e380b77f0 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2982616774 0 0) 0x7f9e20003520 con 0x7f9e380b77f0 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e380d0de0 msgr2=0x7f9e380d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e380d0de0 0x7f9e380d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 >> v2:172.21.15.176:3300/0 conn(0x7f9e3806c5c0 msgr2=0x7f9e3806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 --2- 172.21.15.67:0/1405685014 >> v2:172.21.15.176:3300/0 conn(0x7f9e3806c5c0 0x7f9e3806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e38160dc0 con 0x7f9e380b77f0 2024-04-25T15:25:25.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f9e30001bc0 con 0x7f9e380b77f0 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 >> v1:172.21.15.67:6789/0 conn(0x7f9e380b77f0 legacy=0x7f9e380b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e200039b0 0x7f9e20003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e20004800 0x7f9e20004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e20005150 0x7f9e2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e380d5a40 con 0x7f9e200039b0 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9e380d1830 con 0x7f9e20005150 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2000bf80 con 0x7f9e20004800 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.653+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e380d0de0 0x7f9e380d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e200039b0 0x7f9e20003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e377fe640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e20005150 0x7f9e2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e367fc640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e20004800 0x7f9e20004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e36ffd640 1 -- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e20004800 msgr2=0x7f9e20004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:25.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e20004800 0x7f9e20004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e36ffd640 1 -- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e20005150 msgr2=0x7f9e2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e20005150 0x7f9e2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.654+0000 7f9e36ffd640 1 -- 172.21.15.67:0/1405685014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e20003520 con 0x7f9e200039b0 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e200039b0 0x7f9e20003e20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9e380d6b70 tx=0x7f9e20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6476c1a04f6a878b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28036070 con 0x7f9e200039b0 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e28004030 con 0x7f9e200039b0 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e35ffb640 1 -- 172.21.15.67:0/1405685014 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e280097d0 con 0x7f9e200039b0 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e3d53f640 1 -- 172.21.15.67:0/1405685014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e200039b0 msgr2=0x7f9e20003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e3d53f640 1 --2- 172.21.15.67:0/1405685014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e200039b0 0x7f9e20003e20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9e380d6b70 tx=0x7f9e20005830 comp rx=0 tx=0).stop 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e377fe640 1 -- 172.21.15.67:0/1405685014 reap_dead start 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e3d53f640 1 -- 172.21.15.67:0/1405685014 shutdown_connections 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.655+0000 7f9e3d53f640 1 -- 172.21.15.67:0/1405685014 >> 172.21.15.67:0/1405685014 conn(0x7f9e380d5d70 msgr2=0x7f9e380d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.656+0000 7f9e3d53f640 1 -- 172.21.15.67:0/1405685014 shutdown_connections 2024-04-25T15:25:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.656+0000 7f9e3d53f640 1 -- 172.21.15.67:0/1405685014 wait complete. 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.657+0000 7f9e3d53f640 1 Processor -- start 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.658+0000 7f9e3d53f640 1 -- start start 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e3d53f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e200039b0 0x7f9e380cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e3d53f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e38152eb0 0x7f9e38162910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e3d53f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e38156100 0x7f9e38156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e3d53f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e38160ab0 con 0x7f9e38152eb0 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e3d53f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9e38161380 con 0x7f9e38156100 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e3d53f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9e380b5300 con 0x7f9e200039b0 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e38152eb0 0x7f9e38162910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e38152eb0 0x7f9e38162910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59996/0 (socket says 172.21.15.67:59996) 2024-04-25T15:25:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 -- 172.21.15.67:0/708079657 learned_addr learned my addr 172.21.15.67:0/708079657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e200039b0 0x7f9e380cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e377fe640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e38156100 0x7f9e38156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 -- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e200039b0 msgr2=0x7f9e380cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e200039b0 0x7f9e380cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 -- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e38156100 msgr2=0x7f9e38156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e38156100 0x7f9e38156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 -- 172.21.15.67:0/708079657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e380caaf0 con 0x7f9e38152eb0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.659+0000 7f9e367fc640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e38152eb0 0x7f9e38162910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c011990 tx=0x7f9e2c011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.660+0000 7f9e1f7fe640 1 -- 172.21.15.67:0/708079657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c00fa30 con 0x7f9e38152eb0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.660+0000 7f9e1f7fe640 1 -- 172.21.15.67:0/708079657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e2c00fbd0 con 0x7f9e38152eb0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.660+0000 7f9e1f7fe640 1 -- 172.21.15.67:0/708079657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c014440 con 0x7f9e38152eb0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.660+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e380cb020 con 0x7f9e38152eb0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.660+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e380cb5e0 con 0x7f9e38152eb0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.661+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e38050740 con 0x7f9e38152eb0 2024-04-25T15:25:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.661+0000 7f9e1f7fe640 1 -- 172.21.15.67:0/708079657 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9e2c012040 con 0x7f9e38152eb0 2024-04-25T15:25:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.662+0000 7f9e1f7fe640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e240744b0 0x7f9e240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.662+0000 7f9e1f7fe640 1 -- 172.21.15.67:0/708079657 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9e2c095310 con 0x7f9e38152eb0 2024-04-25T15:25:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.662+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e240744b0 0x7f9e240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.663+0000 7f9e36ffd640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e240744b0 0x7f9e240769c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e28033000 tx=0x7f9e28008070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.667+0000 7f9e1f7fe640 1 -- 172.21.15.67:0/708079657 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e2c019030 con 0x7f9e38152eb0 2024-04-25T15:25:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.891+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e380cadb0 con 0x7f9e38152eb0 2024-04-25T15:25:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.892+0000 7f9e1f7fe640 1 -- 172.21.15.67:0/708079657 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3870 (secure 0 0 0) 0x7f9e2c05ff10 con 0x7f9e38152eb0 2024-04-25T15:25:25.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:25.893 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T15:25:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e240744b0 msgr2=0x7f9e240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e240744b0 0x7f9e240769c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9e28033000 tx=0x7f9e28008070 comp rx=0 tx=0).stop 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e38152eb0 msgr2=0x7f9e38162910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e38152eb0 0x7f9e38162910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c011990 tx=0x7f9e2c011d90 comp rx=0 tx=0).stop 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 shutdown_connections 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e240744b0 0x7f9e240769c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e38156100 0x7f9e38156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e38152eb0 0x7f9e38162910 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 --2- 172.21.15.67:0/708079657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e200039b0 0x7f9e380cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 >> 172.21.15.67:0/708079657 conn(0x7f9e380d5d70 msgr2=0x7f9e380bb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 shutdown_connections 2024-04-25T15:25:25.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:25.894+0000 7f9e3d53f640 1 -- 172.21.15.67:0/708079657 wait complete. 2024-04-25T15:25:25.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.016+0000 7f294c4b1640 1 Processor -- start 2024-04-25T15:25:26.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.016+0000 7f294c4b1640 1 -- start start 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f294c4b1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2944164b00 0x7f29440b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f294c4b1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f294415df80 0x7f29440b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f294c4b1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29441615f0 con 0x7f2944160890 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f294c4b1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f29441613b0 con 0x7f294415df80 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f294c4b1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2944164670 con 0x7f2944164b00 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f2949a4c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f294415df80 0x7f29440b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f2949a4c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f294415df80 0x7f29440b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60008/0 (socket says 172.21.15.67:60008) 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f2949a4c640 1 -- 172.21.15.67:0/3860772784 learned_addr learned my addr 172.21.15.67:0/3860772784 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f294a24d640 1 --2- 172.21.15.67:0/3860772784 >> v2:172.21.15.176:3300/0 conn(0x7f2944164b00 0x7f29440b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f2949a4c640 1 -- 172.21.15.67:0/3860772784 >> v2:172.21.15.176:3300/0 conn(0x7f2944164b00 msgr2=0x7f29440b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f2949a4c640 1 --2- 172.21.15.67:0/3860772784 >> v2:172.21.15.176:3300/0 conn(0x7f2944164b00 0x7f29440b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f2949a4c640 1 -- 172.21.15.67:0/3860772784 >> v1:172.21.15.176:6789/0 conn(0x7f2944160890 legacy=0x7f2944160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f2949a4c640 1 -- 172.21.15.67:0/3860772784 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2944164020 con 0x7f294415df80 2024-04-25T15:25:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.018+0000 7f294a24d640 1 --2- 172.21.15.67:0/3860772784 >> v2:172.21.15.176:3300/0 conn(0x7f2944164b00 0x7f29440b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.019+0000 7f2949a4c640 1 --2- 172.21.15.67:0/3860772784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294415df80 0x7f29440b80c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f294408e410 tx=0x7f293802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52940f9122c50e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.019+0000 7f294924b640 1 -- 172.21.15.67:0/3860772784 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938037070 con 0x7f294415df80 2024-04-25T15:25:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.019+0000 7f294924b640 1 -- 172.21.15.67:0/3860772784 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2938032440 con 0x7f294415df80 2024-04-25T15:25:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.019+0000 7f294924b640 1 -- 172.21.15.67:0/3860772784 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938032810 con 0x7f294415df80 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 -- 172.21.15.67:0/3860772784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294415df80 msgr2=0x7f29440b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 --2- 172.21.15.67:0/3860772784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294415df80 0x7f29440b80c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f294408e410 tx=0x7f293802a270 comp rx=0 tx=0).stop 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 -- 172.21.15.67:0/3860772784 shutdown_connections 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 --2- 172.21.15.67:0/3860772784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f294415df80 0x7f29440b80c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 --2- 172.21.15.67:0/3860772784 >> v2:172.21.15.176:3300/0 conn(0x7f2944164b00 0x7f29440b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 -- 172.21.15.67:0/3860772784 >> 172.21.15.67:0/3860772784 conn(0x7f29440d2bc0 msgr2=0x7f29440d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 -- 172.21.15.67:0/3860772784 shutdown_connections 2024-04-25T15:25:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.020+0000 7f294c4b1640 1 -- 172.21.15.67:0/3860772784 wait complete. 2024-04-25T15:25:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.022+0000 7f294c4b1640 1 Processor -- start 2024-04-25T15:25:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.022+0000 7f294c4b1640 1 -- start start 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294c4b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f294415df80 0x7f2944155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294c4b1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2944160890 0x7f29440c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294c4b1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2944164b00 0x7f2944162e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294c4b1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f29440b4a00 con 0x7f2944164b00 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294c4b1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f29440b4880 con 0x7f2944160890 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294c4b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f29440b4b80 con 0x7f294415df80 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2944164b00 0x7f2944162e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2944164b00 0x7f2944162e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60016/0 (socket says 172.21.15.67:60016) 2024-04-25T15:25:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 -- 172.21.15.67:0/2858929619 learned_addr learned my addr 172.21.15.67:0/2858929619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294a24d640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f294415df80 0x7f2944155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f2949a4c640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2944160890 0x7f29440c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 -- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f294415df80 msgr2=0x7f2944155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f294415df80 0x7f2944155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 -- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2944160890 msgr2=0x7f29440c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2944160890 0x7f29440c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.023+0000 7f294aa4e640 1 -- 172.21.15.67:0/2858929619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29441544f0 con 0x7f2944164b00 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.024+0000 7f294aa4e640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2944164b00 0x7f2944162e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f293400c6b0 tx=0x7f293400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.024+0000 7f2942ffd640 1 -- 172.21.15.67:0/2858929619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934003b70 con 0x7f2944164b00 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.024+0000 7f2942ffd640 1 -- 172.21.15.67:0/2858929619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2934003d10 con 0x7f2944164b00 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.024+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2944155ca0 con 0x7f2944164b00 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.024+0000 7f2942ffd640 1 -- 172.21.15.67:0/2858929619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934005310 con 0x7f2944164b00 2024-04-25T15:25:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.024+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29441ea150 con 0x7f2944164b00 2024-04-25T15:25:26.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.025+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2944050740 con 0x7f2944164b00 2024-04-25T15:25:26.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.026+0000 7f2942ffd640 1 -- 172.21.15.67:0/2858929619 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f29340054b0 con 0x7f2944164b00 2024-04-25T15:25:26.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.026+0000 7f2942ffd640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f29300744e0 0x7f29300769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.026+0000 7f2942ffd640 1 -- 172.21.15.67:0/2858929619 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f2934094b70 con 0x7f2944164b00 2024-04-25T15:25:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.026+0000 7f294a24d640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f29300744e0 0x7f29300769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.027+0000 7f294a24d640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f29300744e0 0x7f29300769f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f293c006fd0 tx=0x7f293c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.031+0000 7f2942ffd640 1 -- 172.21.15.67:0/2858929619 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f293405f090 con 0x7f2944164b00 2024-04-25T15:25:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.218+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2944155a30 con 0x7f2944164b00 2024-04-25T15:25:26.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.219+0000 7f2942ffd640 1 -- 172.21.15.67:0/2858929619 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7f2934063390 con 0x7f2944164b00 2024-04-25T15:25:26.220 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:26.220 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:18.877686+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f29300744e0 msgr2=0x7f29300769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f29300744e0 0x7f29300769f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f293c006fd0 tx=0x7f293c008040 comp rx=0 tx=0).stop 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2944164b00 msgr2=0x7f2944162e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2944164b00 0x7f2944162e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f293400c6b0 tx=0x7f293400cb70 comp rx=0 tx=0).stop 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 shutdown_connections 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f29300744e0 0x7f29300769f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2944164b00 0x7f2944162e40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2944160890 0x7f29440c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 --2- 172.21.15.67:0/2858929619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f294415df80 0x7f2944155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 >> 172.21.15.67:0/2858929619 conn(0x7f29440d2bc0 msgr2=0x7f29440d5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:26.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 shutdown_connections 2024-04-25T15:25:26.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.221+0000 7f294c4b1640 1 -- 172.21.15.67:0/2858929619 wait complete. 2024-04-25T15:25:26.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:26.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:26.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.348+0000 7ff3fe50d640 1 Processor -- start 2024-04-25T15:25:26.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.348+0000 7ff3fe50d640 1 -- start start 2024-04-25T15:25:26.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.349+0000 7ff3fe50d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff3f80ce040 0x7ff3f80b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3fe50d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 0x7ff3f80b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3fe50d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3f806d770 con 0x7ff3f806e350 2024-04-25T15:25:26.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3fe50d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff3f806da50 con 0x7ff3f80ce040 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3fe50d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f80d2200 con 0x7ff3f81649f0 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3fcaaa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff3f806e350 0x7ff3f806e760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43206/0 (socket says 172.21.15.67:43206) 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3fcaaa640 1 -- 172.21.15.67:0/3382208130 learned_addr learned my addr 172.21.15.67:0/3382208130 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f77fe640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 0x7ff3f80b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f7fff640 1 --2- 172.21.15.67:0/3382208130 >> v2:172.21.15.176:3300/0 conn(0x7ff3f80ce040 0x7ff3f80b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2670576502 0 0) 0x7ff3f806d770 con 0x7ff3f806e350 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3e0003520 con 0x7ff3f806e350 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1678506739 0 0) 0x7ff3e0003520 con 0x7ff3f806e350 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 msgr2=0x7ff3f80b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 0x7ff3f80b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 >> v2:172.21.15.176:3300/0 conn(0x7ff3f80ce040 msgr2=0x7ff3f80b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 --2- 172.21.15.67:0/3382208130 >> v2:172.21.15.176:3300/0 conn(0x7ff3f80ce040 0x7ff3f80b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f80d24e0 con 0x7ff3f806e350 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7ff3f00019f0 con 0x7ff3f806e350 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f7fff640 1 --2- 172.21.15.67:0/3382208130 >> v2:172.21.15.176:3300/0 conn(0x7ff3f80ce040 0x7ff3f80b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 >> v1:172.21.15.67:6789/0 conn(0x7ff3f806e350 legacy=0x7ff3f806e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00039b0 0x7ff3e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 0x7ff3e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e0004d40 0x7ff3e0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f806da50 con 0x7ff3e0004d40 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f80d2200 con 0x7ff3e00039b0 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.350+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e000ba40 con 0x7ff3f81649f0 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3fcaaa640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e0004d40 0x7ff3e0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3f7fff640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00039b0 0x7ff3e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3f77fe640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 0x7ff3e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3fcaaa640 1 -- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 msgr2=0x7ff3e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3fcaaa640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81649f0 0x7ff3e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3fcaaa640 1 -- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00039b0 msgr2=0x7ff3e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3fcaaa640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00039b0 0x7ff3e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3fcaaa640 1 -- 172.21.15.67:0/3382208130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e0003520 con 0x7ff3e0004d40 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.351+0000 7ff3fcaaa640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e0004d40 0x7ff3e0005150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000a8a0 tx=0x7ff3e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ececde86b25aeb8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f000a4c0 con 0x7ff3e0004d40 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f002f070 con 0x7ff3e0004d40 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3f6ffd640 1 -- 172.21.15.67:0/3382208130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f002bd30 con 0x7ff3e0004d40 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3fe50d640 1 -- 172.21.15.67:0/3382208130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e0004d40 msgr2=0x7ff3e0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3fe50d640 1 --2- 172.21.15.67:0/3382208130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e0004d40 0x7ff3e0005150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000a8a0 tx=0x7ff3e0005830 comp rx=0 tx=0).stop 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3fcaaa640 1 -- 172.21.15.67:0/3382208130 reap_dead start 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3fe50d640 1 -- 172.21.15.67:0/3382208130 shutdown_connections 2024-04-25T15:25:26.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3fe50d640 1 -- 172.21.15.67:0/3382208130 >> 172.21.15.67:0/3382208130 conn(0x7ff3f80d3a80 msgr2=0x7ff3f80d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3fe50d640 1 -- 172.21.15.67:0/3382208130 shutdown_connections 2024-04-25T15:25:26.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.352+0000 7ff3fe50d640 1 -- 172.21.15.67:0/3382208130 wait complete. 2024-04-25T15:25:26.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.354+0000 7ff3fe50d640 1 Processor -- start 2024-04-25T15:25:26.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.354+0000 7ff3fe50d640 1 -- start start 2024-04-25T15:25:26.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3fe50d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3f81faaa0 0x7ff3f80c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3fe50d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3f81f62f0 0x7ff3f8163020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3fe50d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81f6700 0x7ff3f80c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3fe50d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f80b90a0 con 0x7ff3f81f62f0 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3fe50d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f8164e00 con 0x7ff3f81faaa0 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3fe50d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f80b9220 con 0x7ff3f81f6700 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3f81f62f0 0x7ff3f8163020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3f81f62f0 0x7ff3f8163020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60030/0 (socket says 172.21.15.67:60030) 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 -- 172.21.15.67:0/570541092 learned_addr learned my addr 172.21.15.67:0/570541092 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3fcaaa640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81f6700 0x7ff3f80c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f7fff640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3f81faaa0 0x7ff3f80c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 -- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81f6700 msgr2=0x7ff3f80c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81f6700 0x7ff3f80c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 -- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3f81faaa0 msgr2=0x7ff3f80c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3f81faaa0 0x7ff3f80c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 -- 172.21.15.67:0/570541092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f81faeb0 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f77fe640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3f81f62f0 0x7ff3f8163020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80119a0 tx=0x7ff3e8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f4ff9640 1 -- 172.21.15.67:0/570541092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e800faf0 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.356+0000 7ff3f4ff9640 1 -- 172.21.15.67:0/570541092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3e800fc90 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.357+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f8162260 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.357+0000 7ff3f4ff9640 1 -- 172.21.15.67:0/570541092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8014440 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.357+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f8162820 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.357+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f8069990 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.358+0000 7ff3f4ff9640 1 -- 172.21.15.67:0/570541092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff3e8012040 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.359+0000 7ff3f4ff9640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3e40744b0 0x7ff3e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.359+0000 7ff3f4ff9640 1 -- 172.21.15.67:0/570541092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff3e8095420 con 0x7ff3f81f62f0 2024-04-25T15:25:26.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.359+0000 7ff3f7fff640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3e40744b0 0x7ff3e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.359+0000 7ff3f7fff640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3e40744b0 0x7ff3e40769c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80cee40 tx=0x7ff3ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.363+0000 7ff3f4ff9640 1 -- 172.21.15.67:0/570541092 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff3f8069990 con 0x7ff3f81f62f0 2024-04-25T15:25:26.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.547+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff3f80d4780 con 0x7ff3f81f62f0 2024-04-25T15:25:26.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.547+0000 7ff3f4ff9640 1 -- 172.21.15.67:0/570541092 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7ff3f81faeb0 con 0x7ff3f81f62f0 2024-04-25T15:25:26.548 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:26.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3e40744b0 msgr2=0x7ff3e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3e40744b0 0x7ff3e40769c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80cee40 tx=0x7ff3ec008040 comp rx=0 tx=0).stop 2024-04-25T15:25:26.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3f81f62f0 msgr2=0x7ff3f8163020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3f81f62f0 0x7ff3f8163020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80119a0 tx=0x7ff3e8011e60 comp rx=0 tx=0).stop 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 shutdown_connections 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3e40744b0 0x7ff3e40769c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3f81f6700 0x7ff3f80c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3f81f62f0 0x7ff3f8163020 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 --2- 172.21.15.67:0/570541092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3f81faaa0 0x7ff3f80c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 >> 172.21.15.67:0/570541092 conn(0x7ff3f80d3a80 msgr2=0x7ff3f80686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 shutdown_connections 2024-04-25T15:25:26.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.549+0000 7ff3fe50d640 1 -- 172.21.15.67:0/570541092 wait complete. 2024-04-25T15:25:26.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:25:26.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.672+0000 7f3c3fe9a640 1 Processor -- start 2024-04-25T15:25:26.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.672+0000 7f3c3fe9a640 1 -- start start 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.673+0000 7f3c3fe9a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c3806ccd0 0x7f3c380d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.673+0000 7f3c3fe9a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c380d1830 con 0x7f3c380d0de0 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.673+0000 7f3c3fe9a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c38160dc0 con 0x7f3c380d4a30 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.673+0000 7f3c3fe9a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c381610a0 con 0x7f3c3806ccd0 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.673+0000 7f3c3dc36640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3c380d0de0 0x7f3c3806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43220/0 (socket says 172.21.15.67:43220) 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.673+0000 7f3c3dc36640 1 -- 172.21.15.67:0/2449901781 learned_addr learned my addr 172.21.15.67:0/2449901781 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.673+0000 7f3c3d435640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c3806ccd0 0x7f3c380d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940124801 0 0) 0x7f3c380d1830 con 0x7f3c380d0de0 2024-04-25T15:25:26.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c24003520 con 0x7f3c380d0de0 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210230921 0 0) 0x7f3c38160dc0 con 0x7f3c380d4a30 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c380d1830 con 0x7f3c380d4a30 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1850485954 0 0) 0x7f3c24003520 con 0x7f3c380d0de0 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c3806ccd0 msgr2=0x7f3c380d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c3806ccd0 0x7f3c380d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 >> v1:172.21.15.176:6789/0 conn(0x7f3c380d4a30 legacy=0x7f3c380d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c38161380 con 0x7f3c380d0de0 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f3c340019f0 con 0x7f3c380d0de0 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3d435640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c3806ccd0 0x7f3c380d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 >> v1:172.21.15.67:6789/0 conn(0x7f3c380d0de0 legacy=0x7f3c3806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24003b70 0x7f3c24003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c24004800 0x7f3c24004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24005150 0x7f3c2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3c381610a0 con 0x7f3c24005150 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2400bf00 con 0x7f3c24003b70 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c2400c100 con 0x7f3c24004800 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3e437640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24005150 0x7f3c2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3d435640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c24004800 0x7f3c24004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3dc36640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24003b70 0x7f3c24003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3e437640 1 -- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c24004800 msgr2=0x7f3c24004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3e437640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c24004800 0x7f3c24004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3e437640 1 -- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24003b70 msgr2=0x7f3c24003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3e437640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24003b70 0x7f3c24003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.674+0000 7f3c3e437640 1 -- 172.21.15.67:0/2449901781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c24003520 con 0x7f3c24005150 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.675+0000 7f3c3e437640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24005150 0x7f3c2400b9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c300018a0 tx=0x7f3c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a78826951ee5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.675+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c300014f0 con 0x7f3c24005150 2024-04-25T15:25:26.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.675+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c300052d0 con 0x7f3c24005150 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.675+0000 7f3c3cc34640 1 -- 172.21.15.67:0/2449901781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3002ac60 con 0x7f3c24005150 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.676+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2449901781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24005150 msgr2=0x7f3c2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.676+0000 7f3c3fe9a640 1 --2- 172.21.15.67:0/2449901781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24005150 0x7f3c2400b9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c300018a0 tx=0x7f3c24005830 comp rx=0 tx=0).stop 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.676+0000 7f3c3e437640 1 -- 172.21.15.67:0/2449901781 reap_dead start 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.676+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2449901781 shutdown_connections 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.676+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2449901781 >> 172.21.15.67:0/2449901781 conn(0x7f3c380d5d70 msgr2=0x7f3c380d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.676+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2449901781 shutdown_connections 2024-04-25T15:25:26.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.676+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2449901781 wait complete. 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.678+0000 7f3c3fe9a640 1 Processor -- start 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.678+0000 7f3c3fe9a640 1 -- start start 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3fe9a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24005150 0x7f3c38157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3fe9a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c380cce20 0x7f3c380cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3fe9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c380ced70 0x7f3c380cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3fe9a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3c38164de0 con 0x7f3c380ced70 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3fe9a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c380b5300 con 0x7f3c24005150 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3fe9a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c38164f60 con 0x7f3c380cce20 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3e437640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c380ced70 0x7f3c380cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3e437640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c380ced70 0x7f3c380cf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60042/0 (socket says 172.21.15.67:60042) 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.679+0000 7f3c3e437640 1 -- 172.21.15.67:0/2769080267 learned_addr learned my addr 172.21.15.67:0/2769080267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3d435640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c380cce20 0x7f3c380cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3dc36640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24005150 0x7f3c38157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3e437640 1 -- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c380cce20 msgr2=0x7f3c380cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3e437640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c380cce20 0x7f3c380cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3e437640 1 -- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24005150 msgr2=0x7f3c38157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3e437640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24005150 0x7f3c38157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3e437640 1 -- 172.21.15.67:0/2769080267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c380cf6c0 con 0x7f3c380ced70 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3e437640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c380ced70 0x7f3c380cf180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c30005040 tx=0x7f3c300012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c2e7fc640 1 -- 172.21.15.67:0/2769080267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30037040 con 0x7f3c380ced70 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c2e7fc640 1 -- 172.21.15.67:0/2769080267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c30040070 con 0x7f3c380ced70 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c380cdb90 con 0x7f3c380ced70 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c2e7fc640 1 -- 172.21.15.67:0/2769080267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c300014f0 con 0x7f3c380ced70 2024-04-25T15:25:26.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.680+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c380ce150 con 0x7f3c380ced70 2024-04-25T15:25:26.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.681+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c380cd900 con 0x7f3c380ced70 2024-04-25T15:25:26.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.682+0000 7f3c2e7fc640 1 -- 172.21.15.67:0/2769080267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3c300052d0 con 0x7f3c380ced70 2024-04-25T15:25:26.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.682+0000 7f3c2e7fc640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c1c0744e0 0x7f3c1c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:26.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.682+0000 7f3c2e7fc640 1 -- 172.21.15.67:0/2769080267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3c300b49f0 con 0x7f3c380ced70 2024-04-25T15:25:26.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.682+0000 7f3c3dc36640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c1c0744e0 0x7f3c1c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:26.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.683+0000 7f3c3dc36640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c1c0744e0 0x7f3c1c0769f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c34002230 tx=0x7f3c34006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:26.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.687+0000 7f3c2e7fc640 1 -- 172.21.15.67:0/2769080267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c380cf6c0 con 0x7f3c380ced70 2024-04-25T15:25:26.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.873+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3c38050740 con 0x7f3c380ced70 2024-04-25T15:25:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.874+0000 7f3c2e7fc640 1 -- 172.21.15.67:0/2769080267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f3c3007f4d0 con 0x7f3c380ced70 2024-04-25T15:25:26.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:26.875 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:25:26.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:25:26.875 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:25:26.875 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:25:26.875 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:25:26.876 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c1c0744e0 msgr2=0x7f3c1c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c1c0744e0 0x7f3c1c0769f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3c34002230 tx=0x7f3c34006040 comp rx=0 tx=0).stop 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c380ced70 msgr2=0x7f3c380cf180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c380ced70 0x7f3c380cf180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c30005040 tx=0x7f3c300012d0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 shutdown_connections 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c1c0744e0 0x7f3c1c0769f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c380ced70 0x7f3c380cf180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c380cce20 0x7f3c380cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 --2- 172.21.15.67:0/2769080267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c24005150 0x7f3c38157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:26.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 >> 172.21.15.67:0/2769080267 conn(0x7f3c380d5d70 msgr2=0x7f3c38164390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:26.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 shutdown_connections 2024-04-25T15:25:26.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:26.876+0000 7f3c3fe9a640 1 -- 172.21.15.67:0/2769080267 wait complete. 2024-04-25T15:25:26.887 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:25:26.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:25:27.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.003+0000 7f4e91abd640 1 Processor -- start 2024-04-25T15:25:27.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.003+0000 7f4e91abd640 1 -- start start 2024-04-25T15:25:27.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e91abd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c06ccd0 0x7f4e8c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e91abd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e8c0d4a50 0x7f4e8c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e91abd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e8c0d1850 con 0x7f4e8c0d0e00 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e91abd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4e8c160de0 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e91abd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c1610c0 con 0x7f4e8c06ccd0 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e8c0d4a50 0x7f4e8c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e8c0d4a50 0x7f4e8c0d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60056/0 (socket says 172.21.15.67:60056) 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8bfff640 1 -- 172.21.15.67:0/174871287 learned_addr learned my addr 172.21.15.67:0/174871287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8affd640 1 --2- 172.21.15.67:0/174871287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c06ccd0 0x7f4e8c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8bfff640 1 -- 172.21.15.67:0/174871287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c06ccd0 msgr2=0x7f4e8c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8bfff640 1 --2- 172.21.15.67:0/174871287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c06ccd0 0x7f4e8c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8bfff640 1 -- 172.21.15.67:0/174871287 >> v1:172.21.15.176:6789/0 conn(0x7f4e8c0d0e00 legacy=0x7f4e8c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.004+0000 7f4e8bfff640 1 -- 172.21.15.67:0/174871287 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c1613a0 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.005+0000 7f4e8bfff640 1 --2- 172.21.15.67:0/174871287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0d4e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009e50 tx=0x7f4e8402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5e03a75b0e80dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.005+0000 7f4e8a7fc640 1 -- 172.21.15.67:0/174871287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84036070 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.005+0000 7f4e8a7fc640 1 -- 172.21.15.67:0/174871287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e84031600 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.005+0000 7f4e8a7fc640 1 -- 172.21.15.67:0/174871287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e840319d0 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 -- 172.21.15.67:0/174871287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 msgr2=0x7f4e8c0d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 --2- 172.21.15.67:0/174871287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0d4e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009e50 tx=0x7f4e8402a710 comp rx=0 tx=0).stop 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 -- 172.21.15.67:0/174871287 shutdown_connections 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 --2- 172.21.15.67:0/174871287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0d4e60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 --2- 172.21.15.67:0/174871287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c06ccd0 0x7f4e8c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 -- 172.21.15.67:0/174871287 >> 172.21.15.67:0/174871287 conn(0x7f4e8c0d5d90 msgr2=0x7f4e8c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 -- 172.21.15.67:0/174871287 shutdown_connections 2024-04-25T15:25:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.006+0000 7f4e91abd640 1 -- 172.21.15.67:0/174871287 wait complete. 2024-04-25T15:25:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.008+0000 7f4e91abd640 1 Processor -- start 2024-04-25T15:25:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.008+0000 7f4e91abd640 1 -- start start 2024-04-25T15:25:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e91abd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e8c0d0e00 0x7f4e8c150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e91abd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e91abd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c0cf200 0x7f4e8c1f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e91abd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c164e00 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e91abd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c0b5320 con 0x7f4e8c0d0e00 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e91abd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8c164f80 con 0x7f4e8c0cf200 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60072/0 (socket says 172.21.15.67:60072) 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.009+0000 7f4e8affd640 1 -- 172.21.15.67:0/716026752 learned_addr learned my addr 172.21.15.67:0/716026752 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8bfff640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c0cf200 0x7f4e8c1f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8b7fe640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e8c0d0e00 0x7f4e8c150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8affd640 1 -- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c0cf200 msgr2=0x7f4e8c1f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8affd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c0cf200 0x7f4e8c1f9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8affd640 1 -- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e8c0d0e00 msgr2=0x7f4e8c150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8affd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e8c0d0e00 0x7f4e8c150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8affd640 1 -- 172.21.15.67:0/716026752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c151450 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8bfff640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c0cf200 0x7f4e8c1f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e8affd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0cecc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000e870 tx=0x7f4e8000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e73fff640 1 -- 172.21.15.67:0/716026752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80017070 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e73fff640 1 -- 172.21.15.67:0/716026752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8000c940 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8c0cde80 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e73fff640 1 -- 172.21.15.67:0/716026752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000cd30 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.010+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8c0ce160 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.011+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8c050740 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.012+0000 7f4e73fff640 1 -- 172.21.15.67:0/716026752 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4e8001a070 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.012+0000 7f4e73fff640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e780744b0 0x7f4e780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.013+0000 7f4e73fff640 1 -- 172.21.15.67:0/716026752 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f4e80094dc0 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.013+0000 7f4e8b7fe640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e780744b0 0x7f4e780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.013+0000 7f4e8b7fe640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e780744b0 0x7f4e780769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c007fd0 tx=0x7f4e7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:27.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.017+0000 7f4e73fff640 1 -- 172.21.15.67:0/716026752 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e8005f890 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.203+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4e8c0d4320 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.274+0000 7f4e73fff640 1 -- 172.21.15.67:0/716026752 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f4e8005f190 con 0x7f4e8c0d4a50 2024-04-25T15:25:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.276+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e780744b0 msgr2=0x7f4e780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.276+0000 7f4e91abd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e780744b0 0x7f4e780769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c007fd0 tx=0x7f4e7c009040 comp rx=0 tx=0).stop 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.276+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 msgr2=0x7f4e8c0cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.276+0000 7f4e91abd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0cecc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000e870 tx=0x7f4e8000ed30 comp rx=0 tx=0).stop 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 shutdown_connections 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e780744b0 0x7f4e780769c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8c0cf200 0x7f4e8c1f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8c0d4a50 0x7f4e8c0cecc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 --2- 172.21.15.67:0/716026752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e8c0d0e00 0x7f4e8c150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 >> 172.21.15.67:0/716026752 conn(0x7f4e8c0d5d90 msgr2=0x7f4e8c1638b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:27.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 shutdown_connections 2024-04-25T15:25:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.277+0000 7f4e91abd640 1 -- 172.21.15.67:0/716026752 wait complete. 2024-04-25T15:25:27.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:25:27.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.409+0000 7f3b91c42640 1 Processor -- start 2024-04-25T15:25:27.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.409+0000 7f3b91c42640 1 -- start start 2024-04-25T15:25:27.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.410+0000 7f3b91c42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c164b00 0x7f3b8c0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.410+0000 7f3b91c42640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3b8c160fa0 0x7f3b8c0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.410+0000 7f3b91c42640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b8c1615f0 con 0x7f3b8c15df80 2024-04-25T15:25:27.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.410+0000 7f3b91c42640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3b8c1613b0 con 0x7f3b8c160fa0 2024-04-25T15:25:27.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.410+0000 7f3b91c42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8c164670 con 0x7f3b8c164b00 2024-04-25T15:25:27.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8bfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3b8c160fa0 0x7f3b8c0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c164b00 0x7f3b8c0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8bfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3b8c160fa0 0x7f3b8c0b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:58014/0 (socket says 172.21.15.67:58014) 2024-04-25T15:25:27.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8bfff640 1 -- 172.21.15.67:0/306311457 learned_addr learned my addr 172.21.15.67:0/306311457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8a7fc640 1 -- 172.21.15.67:0/306311457 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2597643784 0 0) 0x7f3b8c1615f0 con 0x7f3b8c15df80 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8a7fc640 1 -- 172.21.15.67:0/306311457 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b74003520 con 0x7f3b8c15df80 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8bfff640 1 -- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c164b00 msgr2=0x7f3b8c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8bfff640 1 --2- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c164b00 0x7f3b8c0b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8bfff640 1 -- 172.21.15.67:0/306311457 >> v1:172.21.15.176:6789/0 conn(0x7f3b8c15df80 legacy=0x7f3b8c160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8bfff640 1 -- 172.21.15.67:0/306311457 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c164020 con 0x7f3b8c160fa0 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.411+0000 7f3b8b7fe640 1 --2- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c164b00 0x7f3b8c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.412+0000 7f3b8bfff640 1 --2- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c160fa0 0x7f3b8c0b80c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009e50 tx=0x7f3b8402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb6b8d01b6429c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.412+0000 7f3b8a7fc640 1 -- 172.21.15.67:0/306311457 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b84036070 con 0x7f3b8c160fa0 2024-04-25T15:25:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.412+0000 7f3b8a7fc640 1 -- 172.21.15.67:0/306311457 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b84031600 con 0x7f3b8c160fa0 2024-04-25T15:25:27.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.412+0000 7f3b8a7fc640 1 -- 172.21.15.67:0/306311457 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b840319b0 con 0x7f3b8c160fa0 2024-04-25T15:25:27.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 -- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c160fa0 msgr2=0x7f3b8c0b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 --2- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c160fa0 0x7f3b8c0b80c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3b84009e50 tx=0x7f3b8402a710 comp rx=0 tx=0).stop 2024-04-25T15:25:27.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 -- 172.21.15.67:0/306311457 shutdown_connections 2024-04-25T15:25:27.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 --2- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c160fa0 0x7f3b8c0b80c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 --2- 172.21.15.67:0/306311457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c164b00 0x7f3b8c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 -- 172.21.15.67:0/306311457 >> 172.21.15.67:0/306311457 conn(0x7f3b8c0d2bc0 msgr2=0x7f3b8c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 -- 172.21.15.67:0/306311457 shutdown_connections 2024-04-25T15:25:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.413+0000 7f3b91c42640 1 -- 172.21.15.67:0/306311457 wait complete. 2024-04-25T15:25:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.415+0000 7f3b91c42640 1 Processor -- start 2024-04-25T15:25:27.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.415+0000 7f3b91c42640 1 -- start start 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b91c42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8c15df80 0x7f3b8c0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b91c42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c160fa0 0x7f3b8c156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b91c42640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c164b00 0x7f3b8c156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b91c42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c0b4a00 con 0x7f3b8c15df80 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b91c42640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c0b4880 con 0x7f3b8c164b00 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b91c42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8c0b4b80 con 0x7f3b8c160fa0 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8c15df80 0x7f3b8c0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8c15df80 0x7f3b8c0c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60086/0 (socket says 172.21.15.67:60086) 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b8b7fe640 1 -- 172.21.15.67:0/865242096 learned_addr learned my addr 172.21.15.67:0/865242096 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:27.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b8affd640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c160fa0 0x7f3b8c156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.416+0000 7f3b8bfff640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c164b00 0x7f3b8c156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b8b7fe640 1 -- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c160fa0 msgr2=0x7f3b8c156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b8b7fe640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c160fa0 0x7f3b8c156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b8b7fe640 1 -- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c164b00 msgr2=0x7f3b8c156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b8b7fe640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c164b00 0x7f3b8c156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b8b7fe640 1 -- 172.21.15.67:0/865242096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c158240 con 0x7f3b8c15df80 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b8b7fe640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8c15df80 0x7f3b8c0c8cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00e960 tx=0x7f3b7c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b73fff640 1 -- 172.21.15.67:0/865242096 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c017070 con 0x7f3b8c15df80 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b73fff640 1 -- 172.21.15.67:0/865242096 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7c00ca00 con 0x7f3b8c15df80 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8c158520 con 0x7f3b8c15df80 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b73fff640 1 -- 172.21.15.67:0/865242096 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c00cdd0 con 0x7f3b8c15df80 2024-04-25T15:25:27.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.417+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8c162af0 con 0x7f3b8c15df80 2024-04-25T15:25:27.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.418+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8c050740 con 0x7f3b8c15df80 2024-04-25T15:25:27.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.419+0000 7f3b73fff640 1 -- 172.21.15.67:0/865242096 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3b7c01a070 con 0x7f3b8c15df80 2024-04-25T15:25:27.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.419+0000 7f3b73fff640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b780744e0 0x7f3b780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:27.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.419+0000 7f3b73fff640 1 -- 172.21.15.67:0/865242096 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f3b7c095290 con 0x7f3b8c15df80 2024-04-25T15:25:27.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.420+0000 7f3b8affd640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b780744e0 0x7f3b780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:27.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.420+0000 7f3b8affd640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b780744e0 0x7f3b780769f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b80007fd0 tx=0x7f3b80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:27.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.424+0000 7f3b73fff640 1 -- 172.21.15.67:0/865242096 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b7c05f5b0 con 0x7f3b8c15df80 2024-04-25T15:25:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:27.611+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3b8c0d4170 con 0x7f3b8c15df80 2024-04-25T15:25:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.277+0000 7f3b73fff640 1 -- 172.21.15.67:0/865242096 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f3b7c0638b0 con 0x7f3b8c15df80 2024-04-25T15:25:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b780744e0 msgr2=0x7f3b780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b780744e0 0x7f3b780769f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b80007fd0 tx=0x7f3b80009040 comp rx=0 tx=0).stop 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8c15df80 msgr2=0x7f3b8c0c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8c15df80 0x7f3b8c0c8cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00e960 tx=0x7f3b7c00ee20 comp rx=0 tx=0).stop 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 shutdown_connections 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b780744e0 0x7f3b780769f0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b80007fd0 tx=0x7f3b80009040 comp rx=0 tx=0).stop 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b8c164b00 0x7f3b8c156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b8c160fa0 0x7f3b8c156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 --2- 172.21.15.67:0/865242096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8c15df80 0x7f3b8c0c8cc0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00e960 tx=0x7f3b7c00ee20 comp rx=0 tx=0).stop 2024-04-25T15:25:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.283+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 >> 172.21.15.67:0/865242096 conn(0x7f3b8c0d2bc0 msgr2=0x7f3b8c15bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.286+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 shutdown_connections 2024-04-25T15:25:28.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.288+0000 7f3b91c42640 1 -- 172.21.15.67:0/865242096 wait complete. 2024-04-25T15:25:28.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:25:28.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:25:28.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:25:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.431+0000 7fa0fe3b2640 1 Processor -- start 2024-04-25T15:25:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.431+0000 7fa0fe3b2640 1 -- start start 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.432+0000 7fa0fe3b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa0f80b40b0 0x7fa0f80b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.432+0000 7fa0fe3b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f80b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.432+0000 7fa0fe3b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0f80d2710 0x7fa0f80d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.432+0000 7fa0fe3b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f80b1b10 con 0x7fa0f80d2710 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.432+0000 7fa0fe3b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f80cea10 con 0x7fa0f80b40b0 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.432+0000 7fa0fe3b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f80cecf0 con 0x7fa0f8163b10 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0f80d2710 0x7fa0f80d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0f80d2710 0x7fa0f80d2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36892/0 (socket says 172.21.15.67:36892) 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 -- 172.21.15.67:0/2513546852 learned_addr learned my addr 172.21.15.67:0/2513546852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0f77fe640 1 --2- 172.21.15.67:0/2513546852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f80b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0f7fff640 1 --2- 172.21.15.67:0/2513546852 >> v2:172.21.15.176:3300/0 conn(0x7fa0f80b40b0 0x7fa0f80b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 -- 172.21.15.67:0/2513546852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 msgr2=0x7fa0f80b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 --2- 172.21.15.67:0/2513546852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f80b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 -- 172.21.15.67:0/2513546852 >> v2:172.21.15.176:3300/0 conn(0x7fa0f80b40b0 msgr2=0x7fa0f80b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 --2- 172.21.15.67:0/2513546852 >> v2:172.21.15.176:3300/0 conn(0x7fa0f80b40b0 0x7fa0f80b77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.433+0000 7fa0fc94f640 1 -- 172.21.15.67:0/2513546852 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f80d1f10 con 0x7fa0f80d2710 2024-04-25T15:25:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.434+0000 7fa0fc94f640 1 --2- 172.21.15.67:0/2513546852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f80d2b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80d3ab0 tx=0x7fa0f002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29981dc4dd361e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.434+0000 7fa0f6ffd640 1 -- 172.21.15.67:0/2513546852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0036070 con 0x7fa0f80d2710 2024-04-25T15:25:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.434+0000 7fa0f6ffd640 1 -- 172.21.15.67:0/2513546852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f0031600 con 0x7fa0f80d2710 2024-04-25T15:25:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.434+0000 7fa0f6ffd640 1 -- 172.21.15.67:0/2513546852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00319d0 con 0x7fa0f80d2710 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/2513546852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 msgr2=0x7fa0f80d2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/2513546852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f80d2b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80d3ab0 tx=0x7fa0f002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/2513546852 shutdown_connections 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/2513546852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f80d2b20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/2513546852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f80b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/2513546852 >> v2:172.21.15.176:3300/0 conn(0x7fa0f80b40b0 0x7fa0f80b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/2513546852 >> 172.21.15.67:0/2513546852 conn(0x7fa0f809b7d0 msgr2=0x7fa0f809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/2513546852 shutdown_connections 2024-04-25T15:25:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.435+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/2513546852 wait complete. 2024-04-25T15:25:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.437+0000 7fa0fe3b2640 1 Processor -- start 2024-04-25T15:25:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.437+0000 7fa0fe3b2640 1 -- start start 2024-04-25T15:25:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.438+0000 7fa0fe3b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0f80b40b0 0x7fa0f81f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.438+0000 7fa0fe3b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f81f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.438+0000 7fa0fe3b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f81f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.438+0000 7fa0fe3b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f80d1cb0 con 0x7fa0f80d2710 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.438+0000 7fa0fe3b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f80d1b30 con 0x7fa0f80b40b0 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.438+0000 7fa0fe3b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f80d3060 con 0x7fa0f8163b10 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f81f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f81f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36898/0 (socket says 172.21.15.67:36898) 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0f77fe640 1 -- 172.21.15.67:0/3631299367 learned_addr learned my addr 172.21.15.67:0/3631299367 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0fc94f640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f81f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0f7fff640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0f80b40b0 0x7fa0f81f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0fc94f640 1 -- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0f80b40b0 msgr2=0x7fa0f81f5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0fc94f640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0f80b40b0 0x7fa0f81f5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0fc94f640 1 -- 172.21.15.67:0/3631299367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 msgr2=0x7fa0f81f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0fc94f640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f81f66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0fc94f640 1 -- 172.21.15.67:0/3631299367 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f81f9c50 con 0x7fa0f8163b10 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0f77fe640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f81f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0fc94f640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f81f6c20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8072c30 tx=0x7fa0f0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.439+0000 7fa0f4ff9640 1 -- 172.21.15.67:0/3631299367 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f003f020 con 0x7fa0f8163b10 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.440+0000 7fa0f4ff9640 1 -- 172.21.15.67:0/3631299367 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f0037040 con 0x7fa0f8163b10 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.440+0000 7fa0f4ff9640 1 -- 172.21.15.67:0/3631299367 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0036070 con 0x7fa0f8163b10 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.440+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f81f9f30 con 0x7fa0f8163b10 2024-04-25T15:25:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.440+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f809d6d0 con 0x7fa0f8163b10 2024-04-25T15:25:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.441+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f8050740 con 0x7fa0f8163b10 2024-04-25T15:25:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.441+0000 7fa0f4ff9640 1 -- 172.21.15.67:0/3631299367 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa0f0042020 con 0x7fa0f8163b10 2024-04-25T15:25:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.442+0000 7fa0f4ff9640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0e40744b0 0x7fa0e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.442+0000 7fa0f4ff9640 1 -- 172.21.15.67:0/3631299367 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fa0f00b5470 con 0x7fa0f8163b10 2024-04-25T15:25:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.442+0000 7fa0f7fff640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0e40744b0 0x7fa0e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.442+0000 7fa0f7fff640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0e40744b0 0x7fa0e40769c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80b8ac0 tx=0x7fa0e8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:28.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.447+0000 7fa0f4ff9640 1 -- 172.21.15.67:0/3631299367 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0f0049030 con 0x7fa0f8163b10 2024-04-25T15:25:28.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:28.674+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa0f81f7230 con 0x7fa0f8163b10 2024-04-25T15:25:29.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.308+0000 7fa0f4ff9640 1 -- 172.21.15.67:0/3631299367 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fa0f00b3e20 con 0x7fa0f8163b10 2024-04-25T15:25:29.309 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0e40744b0 msgr2=0x7fa0e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0e40744b0 0x7fa0e40769c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80b8ac0 tx=0x7fa0e8009190 comp rx=0 tx=0).stop 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 msgr2=0x7fa0f81f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f81f6c20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8072c30 tx=0x7fa0f0004a40 comp rx=0 tx=0).stop 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 shutdown_connections 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0e40744b0 0x7fa0e40769c0 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80b8ac0 tx=0x7fa0e8009190 comp rx=0 tx=0).stop 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0f8163b10 0x7fa0f81f6c20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f80d2710 0x7fa0f81f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 --2- 172.21.15.67:0/3631299367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0f80b40b0 0x7fa0f81f5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.316+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 >> 172.21.15.67:0/3631299367 conn(0x7fa0f809b7d0 msgr2=0x7fa0f8163190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.320+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 shutdown_connections 2024-04-25T15:25:29.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.320+0000 7fa0fe3b2640 1 -- 172.21.15.67:0/3631299367 wait complete. 2024-04-25T15:25:29.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:29.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.454+0000 7fb4de270640 1 Processor -- start 2024-04-25T15:25:29.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.454+0000 7fb4de270640 1 -- start start 2024-04-25T15:25:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4de270640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4d806ccd0 0x7fb4d8164040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4de270640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d8164db0 con 0x7fb4d815abd0 2024-04-25T15:25:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4de270640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d8160d30 con 0x7fb4d8164580 2024-04-25T15:25:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4de270640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb4d8161380 con 0x7fb4d806ccd0 2024-04-25T15:25:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4d77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4d806ccd0 0x7fb4d8164040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4d7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4d815abd0 0x7fb4d806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59360/0 (socket says 172.21.15.67:59360) 2024-04-25T15:25:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4d7fff640 1 -- 172.21.15.67:0/1557914280 learned_addr learned my addr 172.21.15.67:0/1557914280 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4d77fe640 1 -- 172.21.15.67:0/1557914280 >> v1:172.21.15.176:6789/0 conn(0x7fb4d8164580 legacy=0x7fb4d80d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4d77fe640 1 -- 172.21.15.67:0/1557914280 >> v1:172.21.15.67:6789/0 conn(0x7fb4d815abd0 legacy=0x7fb4d806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.456+0000 7fb4d77fe640 1 -- 172.21.15.67:0/1557914280 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d8161010 con 0x7fb4d806ccd0 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.457+0000 7fb4d77fe640 1 --2- 172.21.15.67:0/1557914280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d806ccd0 0x7fb4d8164040 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc0097d0 tx=0x7fb4cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e364c0504a09a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.457+0000 7fb4d6ffd640 1 -- 172.21.15.67:0/1557914280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc037070 con 0x7fb4d806ccd0 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.457+0000 7fb4d6ffd640 1 -- 172.21.15.67:0/1557914280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4cc032440 con 0x7fb4d806ccd0 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.457+0000 7fb4d6ffd640 1 -- 172.21.15.67:0/1557914280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc032810 con 0x7fb4d806ccd0 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.458+0000 7fb4de270640 1 -- 172.21.15.67:0/1557914280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d806ccd0 msgr2=0x7fb4d8164040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.458+0000 7fb4de270640 1 --2- 172.21.15.67:0/1557914280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d806ccd0 0x7fb4d8164040 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc0097d0 tx=0x7fb4cc02a270 comp rx=0 tx=0).stop 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.458+0000 7fb4de270640 1 -- 172.21.15.67:0/1557914280 shutdown_connections 2024-04-25T15:25:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.458+0000 7fb4de270640 1 --2- 172.21.15.67:0/1557914280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d806ccd0 0x7fb4d8164040 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.458+0000 7fb4de270640 1 -- 172.21.15.67:0/1557914280 >> 172.21.15.67:0/1557914280 conn(0x7fb4d80d5df0 msgr2=0x7fb4d80d6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.458+0000 7fb4de270640 1 -- 172.21.15.67:0/1557914280 shutdown_connections 2024-04-25T15:25:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.458+0000 7fb4de270640 1 -- 172.21.15.67:0/1557914280 wait complete. 2024-04-25T15:25:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.460+0000 7fb4de270640 1 Processor -- start 2024-04-25T15:25:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.460+0000 7fb4de270640 1 -- start start 2024-04-25T15:25:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4de270640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4d815abd0 0x7fb4d81f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4de270640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d8164580 0x7fb4d80ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4de270640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d80cf9e0 0x7fb4d80cee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4de270640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d8161810 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4de270640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d8164990 con 0x7fb4d8164580 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4de270640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb4d814f720 con 0x7fb4d815abd0 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4dc80d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d80cf9e0 0x7fb4d80cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4dc80d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d80cf9e0 0x7fb4d80cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36912/0 (socket says 172.21.15.67:36912) 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4dc80d640 1 -- 172.21.15.67:0/2338738798 learned_addr learned my addr 172.21.15.67:0/2338738798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4d77fe640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d8164580 0x7fb4d80ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.461+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4d815abd0 0x7fb4d81f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4dc80d640 1 -- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4d815abd0 msgr2=0x7fb4d81f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4dc80d640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4d815abd0 0x7fb4d81f6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4dc80d640 1 -- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d8164580 msgr2=0x7fb4d80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4dc80d640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d8164580 0x7fb4d80ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4dc80d640 1 -- 172.21.15.67:0/2338738798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d8156540 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4d815abd0 0x7fb4d81f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4dc80d640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d80cf9e0 0x7fb4d80cee70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c6b0 tx=0x7fb4d000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4d4ff9640 1 -- 172.21.15.67:0/2338738798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0003b70 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4d4ff9640 1 -- 172.21.15.67:0/2338738798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4d0003d30 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4d4ff9640 1 -- 172.21.15.67:0/2338738798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0005310 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d814dbe0 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.462+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d814f460 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.463+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d8050740 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.464+0000 7fb4d4ff9640 1 -- 172.21.15.67:0/2338738798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb4d00054b0 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.464+0000 7fb4d4ff9640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb4c40744b0 0x7fb4c40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.464+0000 7fb4d4ff9640 1 -- 172.21.15.67:0/2338738798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fb4d0094f70 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.465+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb4c40744b0 0x7fb4c40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.465+0000 7fb4d7fff640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb4c40744b0 0x7fb4c40769c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8005880 tx=0x7fb4c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:29.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.469+0000 7fb4d4ff9640 1 -- 172.21.15.67:0/2338738798 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb4d005f790 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.658+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4d80ca330 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.659+0000 7fb4d4ff9640 1 -- 172.21.15.67:0/2338738798 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16582 (secure 0 0 0) 0x7fb4d005f090 con 0x7fb4d80cf9e0 2024-04-25T15:25:29.660 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:29.660 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:28.675902+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb4c40744b0 msgr2=0x7fb4c40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb4c40744b0 0x7fb4c40769c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8005880 tx=0x7fb4c8009040 comp rx=0 tx=0).stop 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d80cf9e0 msgr2=0x7fb4d80cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d80cf9e0 0x7fb4d80cee70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4d000c6b0 tx=0x7fb4d000cb70 comp rx=0 tx=0).stop 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 shutdown_connections 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb4c40744b0 0x7fb4c40769c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d80cf9e0 0x7fb4d80cee70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4d8164580 0x7fb4d80ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 --2- 172.21.15.67:0/2338738798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4d815abd0 0x7fb4d81f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 >> 172.21.15.67:0/2338738798 conn(0x7fb4d80d5df0 msgr2=0x7fb4d8163520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 shutdown_connections 2024-04-25T15:25:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.661+0000 7fb4de270640 1 -- 172.21.15.67:0/2338738798 wait complete. 2024-04-25T15:25:29.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:29.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.784+0000 7efcfc700640 1 Processor -- start 2024-04-25T15:25:29.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.784+0000 7efcfc700640 1 -- start start 2024-04-25T15:25:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.785+0000 7efcfc700640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcf40b8290 0x7efcf4160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.785+0000 7efcfc700640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcf41615f0 con 0x7efcf415df80 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.785+0000 7efcfc700640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcf4164020 con 0x7efcf4164b00 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.785+0000 7efcfc700640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efcf4164670 con 0x7efcf40b8290 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.785+0000 7efcf9c9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efcf40b8290 0x7efcf4160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.785+0000 7efcfac9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efcf415df80 0x7efcf4160dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59370/0 (socket says 172.21.15.67:59370) 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.785+0000 7efcfac9d640 1 -- 172.21.15.67:0/2618560033 learned_addr learned my addr 172.21.15.67:0/2618560033 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.786+0000 7efcf9c9b640 1 -- 172.21.15.67:0/2618560033 >> v1:172.21.15.176:6789/0 conn(0x7efcf4164b00 legacy=0x7efcf40b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.786+0000 7efcf9c9b640 1 -- 172.21.15.67:0/2618560033 >> v1:172.21.15.67:6789/0 conn(0x7efcf415df80 legacy=0x7efcf4160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.786+0000 7efcf949a640 1 -- 172.21.15.67:0/2618560033 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679893380 0 0) 0x7efcf41615f0 con 0x7efcf415df80 2024-04-25T15:25:29.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.786+0000 7efcf9c9b640 1 -- 172.21.15.67:0/2618560033 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf4164300 con 0x7efcf40b8290 2024-04-25T15:25:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.787+0000 7efcf9c9b640 1 --2- 172.21.15.67:0/2618560033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf40b8290 0x7efcf4160890 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efcf41647f0 tx=0x7efce402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73880071fdc0446 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.787+0000 7efcf949a640 1 -- 172.21.15.67:0/2618560033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4037070 con 0x7efcf40b8290 2024-04-25T15:25:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.787+0000 7efcf949a640 1 -- 172.21.15.67:0/2618560033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce4032440 con 0x7efcf40b8290 2024-04-25T15:25:29.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.787+0000 7efcf949a640 1 -- 172.21.15.67:0/2618560033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4032810 con 0x7efcf40b8290 2024-04-25T15:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.787+0000 7efcfc700640 1 -- 172.21.15.67:0/2618560033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf40b8290 msgr2=0x7efcf4160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.787+0000 7efcfc700640 1 --2- 172.21.15.67:0/2618560033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf40b8290 0x7efcf4160890 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efcf41647f0 tx=0x7efce402a210 comp rx=0 tx=0).stop 2024-04-25T15:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.788+0000 7efcfc700640 1 -- 172.21.15.67:0/2618560033 shutdown_connections 2024-04-25T15:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.788+0000 7efcfc700640 1 --2- 172.21.15.67:0/2618560033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf40b8290 0x7efcf4160890 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.788+0000 7efcfc700640 1 -- 172.21.15.67:0/2618560033 >> 172.21.15.67:0/2618560033 conn(0x7efcf40d2bc0 msgr2=0x7efcf40d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.788+0000 7efcfc700640 1 -- 172.21.15.67:0/2618560033 shutdown_connections 2024-04-25T15:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.788+0000 7efcfc700640 1 -- 172.21.15.67:0/2618560033 wait complete. 2024-04-25T15:25:29.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.789+0000 7efcfc700640 1 Processor -- start 2024-04-25T15:25:29.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.789+0000 7efcfc700640 1 -- start start 2024-04-25T15:25:29.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efcf415df80 0x7efcf40c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf4164b00 0x7efcf4157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efcf40d4cf0 0x7efcf40d5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efcf40b4c90 con 0x7efcf4164b00 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efcf40b4b10 con 0x7efcf40d4cf0 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efcf40b4e10 con 0x7efcf415df80 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf4164b00 0x7efcf4157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf4164b00 0x7efcf4157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36928/0 (socket says 172.21.15.67:36928) 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 -- 172.21.15.67:0/1955909211 learned_addr learned my addr 172.21.15.67:0/1955909211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfac9d640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efcf40d4cf0 0x7efcf40d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfa49c640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efcf415df80 0x7efcf40c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 -- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efcf415df80 msgr2=0x7efcf40c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efcf415df80 0x7efcf40c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 -- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efcf40d4cf0 msgr2=0x7efcf40d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efcf40d4cf0 0x7efcf40d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 -- 172.21.15.67:0/1955909211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf40d5640 con 0x7efcf4164b00 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfac9d640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efcf40d4cf0 0x7efcf40d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcf9c9b640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf4164b00 0x7efcf4157e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efce4035040 tx=0x7efce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efceaffd640 1 -- 172.21.15.67:0/1955909211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4037050 con 0x7efcf4164b00 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efceaffd640 1 -- 172.21.15.67:0/1955909211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce4038040 con 0x7efcf4164b00 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf41f9ea0 con 0x7efcf4164b00 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efceaffd640 1 -- 172.21.15.67:0/1955909211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce4004450 con 0x7efcf4164b00 2024-04-25T15:25:29.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.791+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf41fa460 con 0x7efcf4164b00 2024-04-25T15:25:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.792+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf4050740 con 0x7efcf4164b00 2024-04-25T15:25:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.793+0000 7efceaffd640 1 -- 172.21.15.67:0/1955909211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efce4030900 con 0x7efcf4164b00 2024-04-25T15:25:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.794+0000 7efceaffd640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efcd80744e0 0x7efcd80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.794+0000 7efceaffd640 1 -- 172.21.15.67:0/1955909211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7efce40b5dc0 con 0x7efcf4164b00 2024-04-25T15:25:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.794+0000 7efcfa49c640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efcd80744e0 0x7efcd80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:29.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.794+0000 7efcfa49c640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efcd80744e0 0x7efcd80769f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcf0000f60 tx=0x7efcf0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:29.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.798+0000 7efceaffd640 1 -- 172.21.15.67:0/1955909211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcf40d5640 con 0x7efcf4164b00 2024-04-25T15:25:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.989+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efcf40d4170 con 0x7efcf4164b00 2024-04-25T15:25:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.990+0000 7efceaffd640 1 -- 172.21.15.67:0/1955909211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16582 (secure 0 0 0) 0x7efce4080550 con 0x7efcf4164b00 2024-04-25T15:25:29.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:29.991 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:28.675902+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efcd80744e0 msgr2=0x7efcd80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efcd80744e0 0x7efcd80769f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcf0000f60 tx=0x7efcf0009040 comp rx=0 tx=0).stop 2024-04-25T15:25:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf4164b00 msgr2=0x7efcf4157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf4164b00 0x7efcf4157e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efce4035040 tx=0x7efce40046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 shutdown_connections 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efcd80744e0 0x7efcd80769f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efcf40d4cf0 0x7efcf40d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf4164b00 0x7efcf4157e40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 --2- 172.21.15.67:0/1955909211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efcf415df80 0x7efcf40c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 >> 172.21.15.67:0/1955909211 conn(0x7efcf40d2bc0 msgr2=0x7efcf4160720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 shutdown_connections 2024-04-25T15:25:29.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:29.992+0000 7efcfc700640 1 -- 172.21.15.67:0/1955909211 wait complete. 2024-04-25T15:25:30.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:25:30.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.118+0000 7f62d5f00640 1 Processor -- start 2024-04-25T15:25:30.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.118+0000 7f62d5f00640 1 -- start start 2024-04-25T15:25:30.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.119+0000 7f62d5f00640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 0x7f62d00d2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.119+0000 7f62d5f00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d00ce5d0 con 0x7f62d00ce1c0 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.119+0000 7f62d5f00640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d00cf3c0 con 0x7f62d00b3f20 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.119+0000 7f62d5f00640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f62d00cf6a0 con 0x7f62d00cec90 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.119+0000 7f62cf7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f62d00ce1c0 0x7f62d00d2580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59380/0 (socket says 172.21.15.67:59380) 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.119+0000 7f62cf7fe640 1 -- 172.21.15.67:0/2481128811 learned_addr learned my addr 172.21.15.67:0/2481128811 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ceffd640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 0x7f62d00d2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048880613 0 0) 0x7f62d00ce5d0 con 0x7f62d00ce1c0 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62b8003520 con 0x7f62d00ce1c0 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3903694706 0 0) 0x7f62b8003520 con 0x7f62d00ce1c0 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 msgr2=0x7f62d00d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 0x7f62d00d2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 >> v1:172.21.15.176:6789/0 conn(0x7f62d00b3f20 legacy=0x7f62d00b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:30.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d00cf980 con 0x7f62d00ce1c0 2024-04-25T15:25:30.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f62c0001b70 con 0x7f62d00ce1c0 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 >> v1:172.21.15.67:6789/0 conn(0x7f62d00ce1c0 legacy=0x7f62d00d2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 0x7f62b80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62b8003f40 0x7f62b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b8004d40 0x7f62b8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00cf6a0 con 0x7f62b8004d40 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f62b800bb80 con 0x7f62b8003f40 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f62b800bd80 con 0x7f62d00cec90 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62cffff640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b8004d40 0x7f62b8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62cf7fe640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 0x7f62b80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62ceffd640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62b8003f40 0x7f62b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62cffff640 1 -- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 msgr2=0x7f62b80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62cffff640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62d00cec90 0x7f62b80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62cffff640 1 -- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62b8003f40 msgr2=0x7f62b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:30.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62cffff640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62b8003f40 0x7f62b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.120+0000 7f62cffff640 1 -- 172.21.15.67:0/2481128811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b8003520 con 0x7f62b8004d40 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.121+0000 7f62cffff640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b8004d40 0x7f62b8005150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62d00c9a20 tx=0x7f62b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb90a5f53cfa209 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.121+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8037070 con 0x7f62b8004d40 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.121+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62c802abe0 con 0x7f62b8004d40 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.121+0000 7f62ce7fc640 1 -- 172.21.15.67:0/2481128811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c80014e0 con 0x7f62b8004d40 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.122+0000 7f62d5f00640 1 -- 172.21.15.67:0/2481128811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b8004d40 msgr2=0x7f62b8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.122+0000 7f62d5f00640 1 --2- 172.21.15.67:0/2481128811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62b8004d40 0x7f62b8005150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62d00c9a20 tx=0x7f62b8005830 comp rx=0 tx=0).stop 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.122+0000 7f62cffff640 1 -- 172.21.15.67:0/2481128811 reap_dead start 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.122+0000 7f62d5f00640 1 -- 172.21.15.67:0/2481128811 shutdown_connections 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.122+0000 7f62d5f00640 1 -- 172.21.15.67:0/2481128811 >> 172.21.15.67:0/2481128811 conn(0x7f62d009b7d0 msgr2=0x7f62d009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.122+0000 7f62d5f00640 1 -- 172.21.15.67:0/2481128811 shutdown_connections 2024-04-25T15:25:30.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.122+0000 7f62d5f00640 1 -- 172.21.15.67:0/2481128811 wait complete. 2024-04-25T15:25:30.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.124+0000 7f62d5f00640 1 Processor -- start 2024-04-25T15:25:30.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.124+0000 7f62d5f00640 1 -- start start 2024-04-25T15:25:30.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.125+0000 7f62d5f00640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62b8004d40 0x7f62d0154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.125+0000 7f62d5f00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d00c6380 0x7f62d0161460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.125+0000 7f62d5f00640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62d00c7b50 0x7f62d0161f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.125+0000 7f62d5f00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00d2c90 con 0x7f62d00c6380 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.125+0000 7f62d5f00640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00cf0a0 con 0x7f62d00c7b50 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.125+0000 7f62d5f00640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f62d00d2e10 con 0x7f62b8004d40 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d00c6380 0x7f62d0161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d00c6380 0x7f62d0161460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36946/0 (socket says 172.21.15.67:36946) 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 -- 172.21.15.67:0/2430573664 learned_addr learned my addr 172.21.15.67:0/2430573664 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62cf7fe640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62b8004d40 0x7f62d0154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62cffff640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62d00c7b50 0x7f62d0161f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 -- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62b8004d40 msgr2=0x7f62d0154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62b8004d40 0x7f62d0154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 -- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62d00c7b50 msgr2=0x7f62d0161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:30.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62d00c7b50 0x7f62d0161f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 -- 172.21.15.67:0/2430573664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d00c6ab0 con 0x7f62d00c6380 2024-04-25T15:25:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62ceffd640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d00c6380 0x7f62d0161460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62c4011940 tx=0x7f62c4011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62b7fff640 1 -- 172.21.15.67:0/2430573664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4019070 con 0x7f62d00c6380 2024-04-25T15:25:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62b7fff640 1 -- 172.21.15.67:0/2430573664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62c4004300 con 0x7f62d00c6380 2024-04-25T15:25:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d00c6d90 con 0x7f62d00c6380 2024-04-25T15:25:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62b7fff640 1 -- 172.21.15.67:0/2430573664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c400fa00 con 0x7f62d00c6380 2024-04-25T15:25:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.126+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d00c7350 con 0x7f62d00c6380 2024-04-25T15:25:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.127+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d0050740 con 0x7f62d00c6380 2024-04-25T15:25:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.128+0000 7f62b7fff640 1 -- 172.21.15.67:0/2430573664 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f62c401c070 con 0x7f62d00c6380 2024-04-25T15:25:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.128+0000 7f62b7fff640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f62bc0744e0 0x7f62bc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.128+0000 7f62b7fff640 1 -- 172.21.15.67:0/2430573664 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f62c4095550 con 0x7f62d00c6380 2024-04-25T15:25:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.129+0000 7f62cf7fe640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f62bc0744e0 0x7f62bc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:30.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.129+0000 7f62cf7fe640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f62bc0744e0 0x7f62bc0769f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62c0001270 tx=0x7f62c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:30.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.133+0000 7f62b7fff640 1 -- 172.21.15.67:0/2430573664 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62c405f720 con 0x7f62d00c6380 2024-04-25T15:25:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:30.363+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f62d00c9520 con 0x7f62d00c6380 2024-04-25T15:25:31.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.308+0000 7f62b7fff640 1 -- 172.21.15.67:0/2430573664 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) ==== 118+0+0 (secure 0 0 0) 0x7f62c4063a20 con 0x7f62d00c6380 2024-04-25T15:25:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f62bc0744e0 msgr2=0x7f62bc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f62bc0744e0 0x7f62bc0769f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f62c0001270 tx=0x7f62c0009040 comp rx=0 tx=0).stop 2024-04-25T15:25:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d00c6380 msgr2=0x7f62d0161460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d00c6380 0x7f62d0161460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f62c4011940 tx=0x7f62c4011e00 comp rx=0 tx=0).stop 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 shutdown_connections 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f62bc0744e0 0x7f62bc0769f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f62d00c7b50 0x7f62d0161f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62d00c6380 0x7f62d0161460 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 --2- 172.21.15.67:0/2430573664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62b8004d40 0x7f62d0154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 >> 172.21.15.67:0/2430573664 conn(0x7f62d009b7d0 msgr2=0x7f62d0071390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 shutdown_connections 2024-04-25T15:25:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.311+0000 7f62d5f00640 1 -- 172.21.15.67:0/2430573664 wait complete. 2024-04-25T15:25:31.325 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:25:31.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:25:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.447+0000 7f79c0413640 1 Processor -- start 2024-04-25T15:25:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.447+0000 7f79c0413640 1 -- start start 2024-04-25T15:25:31.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79c0413640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79b80d0de0 0x7f79b80d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:31.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79c0413640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b80d4a30 0x7f79b80d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:31.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79c0413640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b80d1830 con 0x7f79b806c5c0 2024-04-25T15:25:31.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79c0413640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f79b8160dc0 con 0x7f79b80d0de0 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79c0413640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79b81610a0 con 0x7f79b80d4a30 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79be1af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79b80d0de0 0x7f79b80d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79bd9ae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79b806c5c0 0x7f79b806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59384/0 (socket says 172.21.15.67:59384) 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79bd9ae640 1 -- 172.21.15.67:0/1144198482 learned_addr learned my addr 172.21.15.67:0/1144198482 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.448+0000 7f79be9b0640 1 --2- 172.21.15.67:0/1144198482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b80d4a30 0x7f79b80d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79be1af640 1 -- 172.21.15.67:0/1144198482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b80d4a30 msgr2=0x7f79b80d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79be1af640 1 --2- 172.21.15.67:0/1144198482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b80d4a30 0x7f79b80d4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79be1af640 1 -- 172.21.15.67:0/1144198482 >> v1:172.21.15.67:6789/0 conn(0x7f79b806c5c0 legacy=0x7f79b806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79be1af640 1 -- 172.21.15.67:0/1144198482 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b8161380 con 0x7f79b80d0de0 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79bd1ad640 1 -- 172.21.15.67:0/1144198482 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 166305445 0 0) 0x7f79b80d1830 con 0x7f79b806c5c0 2024-04-25T15:25:31.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79be9b0640 1 --2- 172.21.15.67:0/1144198482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b80d4a30 0x7f79b80d4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:31.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79be1af640 1 --2- 172.21.15.67:0/1144198482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80d11f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79b00097a0 tx=0x7f79b002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f386c4645e13d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79bd1ad640 1 -- 172.21.15.67:0/1144198482 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0037070 con 0x7f79b80d0de0 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79bd1ad640 1 -- 172.21.15.67:0/1144198482 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b0032440 con 0x7f79b80d0de0 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.449+0000 7f79bd1ad640 1 -- 172.21.15.67:0/1144198482 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0032810 con 0x7f79b80d0de0 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 -- 172.21.15.67:0/1144198482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 msgr2=0x7f79b80d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 --2- 172.21.15.67:0/1144198482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80d11f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79b00097a0 tx=0x7f79b002a180 comp rx=0 tx=0).stop 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 -- 172.21.15.67:0/1144198482 shutdown_connections 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 --2- 172.21.15.67:0/1144198482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b80d4a30 0x7f79b80d4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 --2- 172.21.15.67:0/1144198482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80d11f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 -- 172.21.15.67:0/1144198482 >> 172.21.15.67:0/1144198482 conn(0x7f79b80d5d70 msgr2=0x7f79b80d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 -- 172.21.15.67:0/1144198482 shutdown_connections 2024-04-25T15:25:31.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.450+0000 7f79c0413640 1 -- 172.21.15.67:0/1144198482 wait complete. 2024-04-25T15:25:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.452+0000 7f79c0413640 1 Processor -- start 2024-04-25T15:25:31.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.452+0000 7f79c0413640 1 -- start start 2024-04-25T15:25:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79c0413640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b806c5c0 0x7f79b81e9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79c0413640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79c0413640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79b80d4a30 0x7f79b81f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79c0413640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79b8164de0 con 0x7f79b80d0de0 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79c0413640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f79b80b5300 con 0x7f79b80d4a30 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79c0413640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79b8164f60 con 0x7f79b806c5c0 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79bd9ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79bd9ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80b7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36954/0 (socket says 172.21.15.67:36954) 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.453+0000 7f79bd9ae640 1 -- 172.21.15.67:0/1865414501 learned_addr learned my addr 172.21.15.67:0/1865414501 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79be1af640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b806c5c0 0x7f79b81e9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79be9b0640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79b80d4a30 0x7f79b81f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79bd9ae640 1 -- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b806c5c0 msgr2=0x7f79b81e9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79bd9ae640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b806c5c0 0x7f79b81e9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79bd9ae640 1 -- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79b80d4a30 msgr2=0x7f79b81f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79bd9ae640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79b80d4a30 0x7f79b81f9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:31.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79bd9ae640 1 -- 172.21.15.67:0/1865414501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b80ce9b0 con 0x7f79b80d0de0 2024-04-25T15:25:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79be9b0640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79b80d4a30 0x7f79b81f9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79bd9ae640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80b7ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00c430 tx=0x7f79ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79b6ffd640 1 -- 172.21.15.67:0/1865414501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac0074e0 con 0x7f79b80d0de0 2024-04-25T15:25:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79b6ffd640 1 -- 172.21.15.67:0/1865414501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79ac0047e0 con 0x7f79b80d0de0 2024-04-25T15:25:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b80cec90 con 0x7f79b80d0de0 2024-04-25T15:25:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79b6ffd640 1 -- 172.21.15.67:0/1865414501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac004bd0 con 0x7f79b80d0de0 2024-04-25T15:25:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.454+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b80cf250 con 0x7f79b80d0de0 2024-04-25T15:25:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.455+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b8050740 con 0x7f79b80d0de0 2024-04-25T15:25:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.456+0000 7f79b6ffd640 1 -- 172.21.15.67:0/1865414501 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f79ac011070 con 0x7f79b80d0de0 2024-04-25T15:25:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.456+0000 7f79b6ffd640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79a40744e0 0x7f79a40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.457+0000 7f79b6ffd640 1 -- 172.21.15.67:0/1865414501 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f79ac094ea0 con 0x7f79b80d0de0 2024-04-25T15:25:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.457+0000 7f79be1af640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79a40744e0 0x7f79a40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:31.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.457+0000 7f79be1af640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79a40744e0 0x7f79a40769f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f79b0034000 tx=0x7f79b002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:31.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.461+0000 7f79b6ffd640 1 -- 172.21.15.67:0/1865414501 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79ac05f630 con 0x7f79b80d0de0 2024-04-25T15:25:31.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:31.685+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f79b80d4300 con 0x7f79b80d0de0 2024-04-25T15:25:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.310+0000 7f79b6ffd640 1 -- 172.21.15.67:0/1865414501 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7f79ac05ef30 con 0x7f79b80d0de0 2024-04-25T15:25:32.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79a40744e0 msgr2=0x7f79a40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79a40744e0 0x7f79a40769f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f79b0034000 tx=0x7f79b002e000 comp rx=0 tx=0).stop 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 msgr2=0x7f79b80b7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80b7ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00c430 tx=0x7f79ac00c8f0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 shutdown_connections 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79a40744e0 0x7f79a40769f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79b80d4a30 0x7f79b81f9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80d0de0 0x7f79b80b7ec0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 --2- 172.21.15.67:0/1865414501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79b806c5c0 0x7f79b81e9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.312+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 >> 172.21.15.67:0/1865414501 conn(0x7f79b80d5d70 msgr2=0x7f79b80ba1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.313+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 shutdown_connections 2024-04-25T15:25:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.313+0000 7f79c0413640 1 -- 172.21.15.67:0/1865414501 wait complete. 2024-04-25T15:25:32.327 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:25:32.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:25:32.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.445+0000 7fd620c53640 1 Processor -- start 2024-04-25T15:25:32.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.445+0000 7fd620c53640 1 -- start start 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd620c53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd620c53640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd61c06c5c0 0x7fd61c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd620c53640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd61c0b7810 0x7fd61c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd620c53640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd61c06cf10 con 0x7fd61c0b7810 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd620c53640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd61c0d1850 con 0x7fd61c06c5c0 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd620c53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd61c160de0 con 0x7fd61c0d0e00 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd61ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd61c0b7810 0x7fd61c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd61ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd61c0b7810 0x7fd61c0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36966/0 (socket says 172.21.15.67:36966) 2024-04-25T15:25:32.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.446+0000 7fd61ad9d640 1 -- 172.21.15.67:0/1798287228 learned_addr learned my addr 172.21.15.67:0/1798287228 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd613fff640 1 --2- 172.21.15.67:0/1798287228 >> v2:172.21.15.176:3300/0 conn(0x7fd61c06c5c0 0x7fd61c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd61a59c640 1 --2- 172.21.15.67:0/1798287228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd61ad9d640 1 -- 172.21.15.67:0/1798287228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 msgr2=0x7fd61c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd61ad9d640 1 --2- 172.21.15.67:0/1798287228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd61ad9d640 1 -- 172.21.15.67:0/1798287228 >> v2:172.21.15.176:3300/0 conn(0x7fd61c06c5c0 msgr2=0x7fd61c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd61ad9d640 1 --2- 172.21.15.67:0/1798287228 >> v2:172.21.15.176:3300/0 conn(0x7fd61c06c5c0 0x7fd61c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd61ad9d640 1 -- 172.21.15.67:0/1798287228 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61c1610c0 con 0x7fd61c0b7810 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd613fff640 1 --2- 172.21.15.67:0/1798287228 >> v2:172.21.15.176:3300/0 conn(0x7fd61c06c5c0 0x7fd61c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:32.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.447+0000 7fd61ad9d640 1 --2- 172.21.15.67:0/1798287228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0b7c20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd61c06f0a0 tx=0x7fd61402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6436be72ff2291 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd619d9b640 1 -- 172.21.15.67:0/1798287228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd614036070 con 0x7fd61c0b7810 2024-04-25T15:25:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd619d9b640 1 -- 172.21.15.67:0/1798287228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd614031600 con 0x7fd61c0b7810 2024-04-25T15:25:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd619d9b640 1 -- 172.21.15.67:0/1798287228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140319d0 con 0x7fd61c0b7810 2024-04-25T15:25:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 -- 172.21.15.67:0/1798287228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 msgr2=0x7fd61c0b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 --2- 172.21.15.67:0/1798287228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0b7c20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd61c06f0a0 tx=0x7fd61402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 -- 172.21.15.67:0/1798287228 shutdown_connections 2024-04-25T15:25:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 --2- 172.21.15.67:0/1798287228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0b7c20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 --2- 172.21.15.67:0/1798287228 >> v2:172.21.15.176:3300/0 conn(0x7fd61c06c5c0 0x7fd61c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 --2- 172.21.15.67:0/1798287228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 -- 172.21.15.67:0/1798287228 >> 172.21.15.67:0/1798287228 conn(0x7fd61c0d5d90 msgr2=0x7fd61c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 -- 172.21.15.67:0/1798287228 shutdown_connections 2024-04-25T15:25:32.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.448+0000 7fd620c53640 1 -- 172.21.15.67:0/1798287228 wait complete. 2024-04-25T15:25:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.450+0000 7fd620c53640 1 Processor -- start 2024-04-25T15:25:32.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.450+0000 7fd620c53640 1 -- start start 2024-04-25T15:25:32.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd620c53640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd61c06c5c0 0x7fd61c0cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd620c53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd620c53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c1fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd620c53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c0b5320 con 0x7fd61c0b7810 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd620c53640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c160ad0 con 0x7fd61c06c5c0 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd620c53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd61c164e00 con 0x7fd61c0d0e00 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36982/0 (socket says 172.21.15.67:36982) 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 -- 172.21.15.67:0/3723046590 learned_addr learned my addr 172.21.15.67:0/3723046590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd61a59c640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd61c06c5c0 0x7fd61c0cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd61ad9d640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c1fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 -- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 msgr2=0x7fd61c1fa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c1fa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 -- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd61c06c5c0 msgr2=0x7fd61c0cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd61c06c5c0 0x7fd61c0cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 -- 172.21.15.67:0/3723046590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61c155840 con 0x7fd61c0b7810 2024-04-25T15:25:32.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd613fff640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0ca670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd60400d800 tx=0x7fd60400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:32.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd612ffd640 1 -- 172.21.15.67:0/3723046590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604018070 con 0x7fd61c0b7810 2024-04-25T15:25:32.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd612ffd640 1 -- 172.21.15.67:0/3723046590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd604004030 con 0x7fd61c0b7810 2024-04-25T15:25:32.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.452+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61c0cec00 con 0x7fd61c0b7810 2024-04-25T15:25:32.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.453+0000 7fd612ffd640 1 -- 172.21.15.67:0/3723046590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604004400 con 0x7fd61c0b7810 2024-04-25T15:25:32.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.453+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd61c0ceea0 con 0x7fd61c0b7810 2024-04-25T15:25:32.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.453+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd61c050740 con 0x7fd61c0b7810 2024-04-25T15:25:32.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.454+0000 7fd612ffd640 1 -- 172.21.15.67:0/3723046590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd6040096a0 con 0x7fd61c0b7810 2024-04-25T15:25:32.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.454+0000 7fd612ffd640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5fc0744b0 0x7fd5fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.455+0000 7fd612ffd640 1 -- 172.21.15.67:0/3723046590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd6040953b0 con 0x7fd61c0b7810 2024-04-25T15:25:32.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.455+0000 7fd61a59c640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5fc0744b0 0x7fd5fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.455+0000 7fd61a59c640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5fc0744b0 0x7fd5fc0769c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0079a0 tx=0x7fd60c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:32.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.459+0000 7fd612ffd640 1 -- 172.21.15.67:0/3723046590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd61c155840 con 0x7fd61c0b7810 2024-04-25T15:25:32.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.682+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd61c1ea610 con 0x7fd61c0b7810 2024-04-25T15:25:32.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.683+0000 7fd612ffd640 1 -- 172.21.15.67:0/3723046590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7fd60405f580 con 0x7fd61c0b7810 2024-04-25T15:25:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:32.684 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:25:32.684 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:25:32.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5fc0744b0 msgr2=0x7fd5fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5fc0744b0 0x7fd5fc0769c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0079a0 tx=0x7fd60c008040 comp rx=0 tx=0).stop 2024-04-25T15:25:32.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 msgr2=0x7fd61c0ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0ca670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd60400d800 tx=0x7fd60400dcc0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 shutdown_connections 2024-04-25T15:25:32.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5fc0744b0 0x7fd5fc0769c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd61c0d0e00 0x7fd61c1fa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61c0b7810 0x7fd61c0ca670 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 --2- 172.21.15.67:0/3723046590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd61c06c5c0 0x7fd61c0cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 >> 172.21.15.67:0/3723046590 conn(0x7fd61c0d5d90 msgr2=0x7fd61c0bb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:32.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 shutdown_connections 2024-04-25T15:25:32.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.685+0000 7fd620c53640 1 -- 172.21.15.67:0/3723046590 wait complete. 2024-04-25T15:25:32.697 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.808+0000 7f9bfad9d640 1 Processor -- start 2024-04-25T15:25:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.808+0000 7f9bfad9d640 1 -- start start 2024-04-25T15:25:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bfad9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9bf4161760 0x7f9bf40b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bfad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bf40b82d0 0x7f9bf40d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bfad9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40d2030 0x7f9bf40d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bfad9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9bf40d00c0 con 0x7f9bf40b82d0 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bfad9d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9bf40cf4b0 con 0x7f9bf4161760 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bfad9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf40cf790 con 0x7f9bf40d2030 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bf3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bf40b82d0 0x7f9bf40d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bf3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bf40b82d0 0x7f9bf40d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36994/0 (socket says 172.21.15.67:36994) 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bf3fff640 1 -- 172.21.15.67:0/1196587288 learned_addr learned my addr 172.21.15.67:0/1196587288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.809+0000 7f9bf8b39640 1 --2- 172.21.15.67:0/1196587288 >> v2:172.21.15.176:3300/0 conn(0x7f9bf4161760 0x7f9bf40b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.810+0000 7f9bf933a640 1 --2- 172.21.15.67:0/1196587288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40d2030 0x7f9bf40d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.810+0000 7f9bf3fff640 1 -- 172.21.15.67:0/1196587288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40d2030 msgr2=0x7f9bf40d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.810+0000 7f9bf3fff640 1 --2- 172.21.15.67:0/1196587288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40d2030 0x7f9bf40d2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.810+0000 7f9bf3fff640 1 -- 172.21.15.67:0/1196587288 >> v2:172.21.15.176:3300/0 conn(0x7f9bf4161760 msgr2=0x7f9bf40b7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.810+0000 7f9bf3fff640 1 --2- 172.21.15.67:0/1196587288 >> v2:172.21.15.176:3300/0 conn(0x7f9bf4161760 0x7f9bf40b7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.810+0000 7f9bf3fff640 1 -- 172.21.15.67:0/1196587288 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf40cfa70 con 0x7f9bf40b82d0 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.810+0000 7f9bf3fff640 1 --2- 172.21.15.67:0/1196587288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40b82d0 0x7f9bf40d1af0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40d3a70 tx=0x7f9be002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784dcacb954484f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bf37fe640 1 -- 172.21.15.67:0/1196587288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0037070 con 0x7f9bf40b82d0 2024-04-25T15:25:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bf37fe640 1 -- 172.21.15.67:0/1196587288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9be0032440 con 0x7f9bf40b82d0 2024-04-25T15:25:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bf37fe640 1 -- 172.21.15.67:0/1196587288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0032810 con 0x7f9bf40b82d0 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 -- 172.21.15.67:0/1196587288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40b82d0 msgr2=0x7f9bf40d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/1196587288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40b82d0 0x7f9bf40d1af0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40d3a70 tx=0x7f9be002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 -- 172.21.15.67:0/1196587288 shutdown_connections 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/1196587288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40d2030 0x7f9bf40d2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/1196587288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40b82d0 0x7f9bf40d1af0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/1196587288 >> v2:172.21.15.176:3300/0 conn(0x7f9bf4161760 0x7f9bf40b7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 -- 172.21.15.67:0/1196587288 >> 172.21.15.67:0/1196587288 conn(0x7f9bf409b7d0 msgr2=0x7f9bf409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 -- 172.21.15.67:0/1196587288 shutdown_connections 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.811+0000 7f9bfad9d640 1 -- 172.21.15.67:0/1196587288 wait complete. 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.813+0000 7f9bfad9d640 1 Processor -- start 2024-04-25T15:25:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.813+0000 7f9bfad9d640 1 -- start start 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.814+0000 7f9bfad9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40b82d0 0x7f9bf4158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.814+0000 7f9bfad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40d2030 0x7f9bf415bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.814+0000 7f9bfad9d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bf4161760 0x7f9bf415c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.814+0000 7f9bfad9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf40b7940 con 0x7f9bf40d2030 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.814+0000 7f9bfad9d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf4161b70 con 0x7f9bf4161760 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.814+0000 7f9bfad9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf40d2440 con 0x7f9bf40b82d0 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40d2030 0x7f9bf415bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40d2030 0x7f9bf415bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37002/0 (socket says 172.21.15.67:37002) 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 -- 172.21.15.67:0/3212355718 learned_addr learned my addr 172.21.15.67:0/3212355718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf933a640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bf4161760 0x7f9bf415c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf8b39640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40b82d0 0x7f9bf4158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 -- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40b82d0 msgr2=0x7f9bf4158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40b82d0 0x7f9bf4158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 -- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bf4161760 msgr2=0x7f9bf415c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bf4161760 0x7f9bf415c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 -- 172.21.15.67:0/3212355718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf415ea50 con 0x7f9bf40d2030 2024-04-25T15:25:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf8b39640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40b82d0 0x7f9bf4158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf933a640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bf4161760 0x7f9bf415c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf3fff640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40d2030 0x7f9bf415bf20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf17fa640 1 -- 172.21.15.67:0/3212355718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0037040 con 0x7f9bf40d2030 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf17fa640 1 -- 172.21.15.67:0/3212355718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9be0042070 con 0x7f9bf40d2030 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf415ed30 con 0x7f9bf40d2030 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bf17fa640 1 -- 172.21.15.67:0/3212355718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0030880 con 0x7f9bf40d2030 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.815+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf415f2f0 con 0x7f9bf40d2030 2024-04-25T15:25:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.816+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf4160200 con 0x7f9bf40d2030 2024-04-25T15:25:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.817+0000 7f9bf17fa640 1 -- 172.21.15.67:0/3212355718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9be0030a20 con 0x7f9bf40d2030 2024-04-25T15:25:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.818+0000 7f9bf17fa640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bd80744e0 0x7f9bd80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.818+0000 7f9bf17fa640 1 -- 172.21.15.67:0/3212355718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9be00b5a90 con 0x7f9bf40d2030 2024-04-25T15:25:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.818+0000 7f9bf8b39640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bd80744e0 0x7f9bd80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:32.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.818+0000 7f9bf8b39640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bd80744e0 0x7f9bd80769f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9be80099a0 tx=0x7f9be8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:32.822+0000 7f9bf17fa640 1 -- 172.21.15.67:0/3212355718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9be0080220 con 0x7f9bf40d2030 2024-04-25T15:25:33.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.008+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9bf4050740 con 0x7f9bf40d2030 2024-04-25T15:25:33.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.009+0000 7f9bf17fa640 1 -- 172.21.15.67:0/3212355718 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f9be00882a0 con 0x7f9bf40d2030 2024-04-25T15:25:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:33.010 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:33.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bd80744e0 msgr2=0x7f9bd80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bd80744e0 0x7f9bd80769f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9be80099a0 tx=0x7f9be8008040 comp rx=0 tx=0).stop 2024-04-25T15:25:33.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40d2030 msgr2=0x7f9bf415bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40d2030 0x7f9bf415bf20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9be0000c00 tx=0x7f9be00046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 shutdown_connections 2024-04-25T15:25:33.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bd80744e0 0x7f9bd80769f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bf4161760 0x7f9bf415c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bf40d2030 0x7f9bf415bf20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 --2- 172.21.15.67:0/3212355718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bf40b82d0 0x7f9bf4158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 >> 172.21.15.67:0/3212355718 conn(0x7f9bf409b7d0 msgr2=0x7f9bf40d2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 shutdown_connections 2024-04-25T15:25:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.011+0000 7f9bfad9d640 1 -- 172.21.15.67:0/3212355718 wait complete. 2024-04-25T15:25:33.023 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:25:33.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:25:33.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.135+0000 7ff044651640 1 Processor -- start 2024-04-25T15:25:33.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.135+0000 7ff044651640 1 -- start start 2024-04-25T15:25:33.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff044651640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff03c0cf770 0x7ff03c162590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff044651640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff03c162ad0 0x7ff03c0b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff044651640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03c0cfc30 con 0x7ff03c0b91d0 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff044651640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff03c0d1ff0 con 0x7ff03c0cf770 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff044651640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff03c0d2890 con 0x7ff03c162ad0 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff041bec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff03c0cf770 0x7ff03c162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff041bec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff03c0cf770 0x7ff03c162590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37018/0 (socket says 172.21.15.67:37018) 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.136+0000 7ff041bec640 1 -- 172.21.15.67:0/513578595 learned_addr learned my addr 172.21.15.67:0/513578595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.137+0000 7ff042bee640 1 --2- 172.21.15.67:0/513578595 >> v2:172.21.15.176:3300/0 conn(0x7ff03c162ad0 0x7ff03c0b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.137+0000 7ff041bec640 1 -- 172.21.15.67:0/513578595 >> v2:172.21.15.176:3300/0 conn(0x7ff03c162ad0 msgr2=0x7ff03c0b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.137+0000 7ff041bec640 1 --2- 172.21.15.67:0/513578595 >> v2:172.21.15.176:3300/0 conn(0x7ff03c162ad0 0x7ff03c0b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.137+0000 7ff041bec640 1 -- 172.21.15.67:0/513578595 >> v1:172.21.15.176:6789/0 conn(0x7ff03c0b91d0 legacy=0x7ff03c161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.137+0000 7ff041bec640 1 -- 172.21.15.67:0/513578595 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c0d2240 con 0x7ff03c0cf770 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.138+0000 7ff041bec640 1 --2- 172.21.15.67:0/513578595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0cf770 0x7ff03c162590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0caa70 tx=0x7ff02c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7926aa86ce72248a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.138+0000 7ff0413eb640 1 -- 172.21.15.67:0/513578595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c037070 con 0x7ff03c0cf770 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.138+0000 7ff0413eb640 1 -- 172.21.15.67:0/513578595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02c032440 con 0x7ff03c0cf770 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.138+0000 7ff0413eb640 1 -- 172.21.15.67:0/513578595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c032810 con 0x7ff03c0cf770 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.138+0000 7ff044651640 1 -- 172.21.15.67:0/513578595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0cf770 msgr2=0x7ff03c162590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.138+0000 7ff044651640 1 --2- 172.21.15.67:0/513578595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0cf770 0x7ff03c162590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0caa70 tx=0x7ff02c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.139+0000 7ff044651640 1 -- 172.21.15.67:0/513578595 shutdown_connections 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.139+0000 7ff044651640 1 --2- 172.21.15.67:0/513578595 >> v2:172.21.15.176:3300/0 conn(0x7ff03c162ad0 0x7ff03c0b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.139+0000 7ff044651640 1 --2- 172.21.15.67:0/513578595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0cf770 0x7ff03c162590 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.139+0000 7ff044651640 1 -- 172.21.15.67:0/513578595 >> 172.21.15.67:0/513578595 conn(0x7ff03c0d3aa0 msgr2=0x7ff03c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.139+0000 7ff044651640 1 -- 172.21.15.67:0/513578595 shutdown_connections 2024-04-25T15:25:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.139+0000 7ff044651640 1 -- 172.21.15.67:0/513578595 wait complete. 2024-04-25T15:25:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.140+0000 7ff044651640 1 Processor -- start 2024-04-25T15:25:33.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.140+0000 7ff044651640 1 -- start start 2024-04-25T15:25:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff044651640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0b91d0 0x7ff03c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff044651640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c162ad0 0x7ff03c160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff044651640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c15ce20 0x7ff03c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff044651640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c0b24a0 con 0x7ff03c0b91d0 2024-04-25T15:25:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff044651640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c0b2320 con 0x7ff03c15ce20 2024-04-25T15:25:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff044651640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff03c0b0bd0 con 0x7ff03c162ad0 2024-04-25T15:25:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0b91d0 0x7ff03c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0b91d0 0x7ff03c152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37034/0 (socket says 172.21.15.67:37034) 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 -- 172.21.15.67:0/3851365852 learned_addr learned my addr 172.21.15.67:0/3851365852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff042bee640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c15ce20 0x7ff03c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff041bec640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c162ad0 0x7ff03c160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 -- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c162ad0 msgr2=0x7ff03c160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c162ad0 0x7ff03c160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 -- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c15ce20 msgr2=0x7ff03c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c15ce20 0x7ff03c1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 -- 172.21.15.67:0/3851365852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c0b79c0 con 0x7ff03c0b91d0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff0423ed640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0b91d0 0x7ff03c152fa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c3c0 tx=0x7ff03800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff032ffd640 1 -- 172.21.15.67:0/3851365852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0380052d0 con 0x7ff03c0b91d0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff032ffd640 1 -- 172.21.15.67:0/3851365852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff038005490 con 0x7ff03c0b91d0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff032ffd640 1 -- 172.21.15.67:0/3851365852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff038007260 con 0x7ff03c0b91d0 2024-04-25T15:25:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.142+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03c0c9370 con 0x7ff03c0b91d0 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.143+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03c0c9930 con 0x7ff03c0b91d0 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.144+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03c050740 con 0x7ff03c0b91d0 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.144+0000 7ff032ffd640 1 -- 172.21.15.67:0/3851365852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff038007400 con 0x7ff03c0b91d0 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.145+0000 7ff032ffd640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0200744e0 0x7ff0200769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.145+0000 7ff032ffd640 1 -- 172.21.15.67:0/3851365852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff038094640 con 0x7ff03c0b91d0 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.145+0000 7ff041bec640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0200744e0 0x7ff0200769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.146+0000 7ff041bec640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0200744e0 0x7ff0200769f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff02c034000 tx=0x7ff02c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.150+0000 7ff032ffd640 1 -- 172.21.15.67:0/3851365852 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff03805e780 con 0x7ff03c0b91d0 2024-04-25T15:25:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.333+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff03c0730c0 con 0x7ff0200744e0 2024-04-25T15:25:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.340+0000 7ff032ffd640 1 -- 172.21.15.67:0/3851365852 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251936 (secure 0 0 0) 0x7ff03c0730c0 con 0x7ff0200744e0 2024-04-25T15:25:33.344 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:25:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.343+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0200744e0 msgr2=0x7ff0200769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.343+0000 7ff044651640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0200744e0 0x7ff0200769f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff02c034000 tx=0x7ff02c02e000 comp rx=0 tx=0).stop 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.343+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0b91d0 msgr2=0x7ff03c152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.343+0000 7ff044651640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0b91d0 0x7ff03c152fa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff03800c3c0 tx=0x7ff03800c880 comp rx=0 tx=0).stop 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.343+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 shutdown_connections 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.344+0000 7ff044651640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0200744e0 0x7ff0200769f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.344+0000 7ff044651640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c15ce20 0x7ff03c1608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.344+0000 7ff044651640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c162ad0 0x7ff03c160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.344+0000 7ff044651640 1 --2- 172.21.15.67:0/3851365852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c0b91d0 0x7ff03c152fa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.344+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 >> 172.21.15.67:0/3851365852 conn(0x7ff03c0d3aa0 msgr2=0x7ff03c0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.344+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 shutdown_connections 2024-04-25T15:25:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.344+0000 7ff044651640 1 -- 172.21.15.67:0/3851365852 wait complete. 2024-04-25T15:25:33.356 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-25T15:25:32.795770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.518961"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659051+0000","last_change":"2024-04-25T15:25:29.325672+0000","last_active":"2024-04-25T15:25:30.659051+0000","last_peered":"2024-04-25T15:25:30.659051+0000","last_clean":"2024-04-25T15:25:30.659051+0000","last_became_active":"2024-04-25T15:25:29.324999+0000","last_became_peered":"2024-04-25T15:25:29.324999+0000","last_unstale":"2024-04-25T15:25:30.659051+0000","last_undegraded":"2024-04-25T15:25:30.659051+0000","last_fullsized":"2024-04-25T15:25:30.659051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293966+0000","last_change":"2024-04-25T15:25:29.303032+0000","last_active":"2024-04-25T15:25:30.293966+0000","last_peered":"2024-04-25T15:25:30.293966+0000","last_clean":"2024-04-25T15:25:30.293966+0000","last_became_active":"2024-04-25T15:25:29.302027+0000","last_became_peered":"2024-04-25T15:25:29.302027+0000","last_unstale":"2024-04-25T15:25:30.293966+0000","last_undegraded":"2024-04-25T15:25:30.293966+0000","last_fullsized":"2024-04-25T15:25:30.293966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297058+0000","last_change":"2024-04-25T15:25:29.307526+0000","last_active":"2024-04-25T15:25:30.297058+0000","last_peered":"2024-04-25T15:25:30.297058+0000","last_clean":"2024-04-25T15:25:30.297058+0000","last_became_active":"2024-04-25T15:25:29.307109+0000","last_became_peered":"2024-04-25T15:25:29.307109+0000","last_unstale":"2024-04-25T15:25:30.297058+0000","last_undegraded":"2024-04-25T15:25:30.297058+0000","last_fullsized":"2024-04-25T15:25:30.297058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323540+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322901+0000","last_became_peered":"2024-04-25T15:25:29.322901+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658248+0000","last_change":"2024-04-25T15:25:29.313182+0000","last_active":"2024-04-25T15:25:30.658248+0000","last_peered":"2024-04-25T15:25:30.658248+0000","last_clean":"2024-04-25T15:25:30.658248+0000","last_became_active":"2024-04-25T15:25:29.312850+0000","last_became_peered":"2024-04-25T15:25:29.312850+0000","last_unstale":"2024-04-25T15:25:30.658248+0000","last_undegraded":"2024-04-25T15:25:30.658248+0000","last_fullsized":"2024-04-25T15:25:30.658248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658795+0000","last_change":"2024-04-25T15:25:29.319442+0000","last_active":"2024-04-25T15:25:30.658795+0000","last_peered":"2024-04-25T15:25:30.658795+0000","last_clean":"2024-04-25T15:25:30.658795+0000","last_became_active":"2024-04-25T15:25:29.319141+0000","last_became_peered":"2024-04-25T15:25:29.319141+0000","last_unstale":"2024-04-25T15:25:30.658795+0000","last_undegraded":"2024-04-25T15:25:30.658795+0000","last_fullsized":"2024-04-25T15:25:30.658795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657979+0000","last_change":"2024-04-25T15:25:29.322616+0000","last_active":"2024-04-25T15:25:30.657979+0000","last_peered":"2024-04-25T15:25:30.657979+0000","last_clean":"2024-04-25T15:25:30.657979+0000","last_became_active":"2024-04-25T15:25:29.322311+0000","last_became_peered":"2024-04-25T15:25:29.322311+0000","last_unstale":"2024-04-25T15:25:30.657979+0000","last_undegraded":"2024-04-25T15:25:30.657979+0000","last_fullsized":"2024-04-25T15:25:30.657979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323286+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322911+0000","last_became_peered":"2024-04-25T15:25:29.322911+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659143+0000","last_change":"2024-04-25T15:25:29.325686+0000","last_active":"2024-04-25T15:25:30.659143+0000","last_peered":"2024-04-25T15:25:30.659143+0000","last_clean":"2024-04-25T15:25:30.659143+0000","last_became_active":"2024-04-25T15:25:29.325332+0000","last_became_peered":"2024-04-25T15:25:29.325332+0000","last_unstale":"2024-04-25T15:25:30.659143+0000","last_undegraded":"2024-04-25T15:25:30.659143+0000","last_fullsized":"2024-04-25T15:25:30.659143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:20.087185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313378+0000","last_change":"2024-04-25T15:25:29.323852+0000","last_active":"2024-04-25T15:25:30.313378+0000","last_peered":"2024-04-25T15:25:30.313378+0000","last_clean":"2024-04-25T15:25:30.313378+0000","last_became_active":"2024-04-25T15:25:29.323524+0000","last_became_peered":"2024-04-25T15:25:29.323524+0000","last_unstale":"2024-04-25T15:25:30.313378+0000","last_undegraded":"2024-04-25T15:25:30.313378+0000","last_fullsized":"2024-04-25T15:25:30.313378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313308+0000","last_change":"2024-04-25T15:25:29.325815+0000","last_active":"2024-04-25T15:25:30.313308+0000","last_peered":"2024-04-25T15:25:30.313308+0000","last_clean":"2024-04-25T15:25:30.313308+0000","last_became_active":"2024-04-25T15:25:29.325436+0000","last_became_peered":"2024-04-25T15:25:29.325436+0000","last_unstale":"2024-04-25T15:25:30.313308+0000","last_undegraded":"2024-04-25T15:25:30.313308+0000","last_fullsized":"2024-04-25T15:25:30.313308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300011+0000","last_change":"2024-04-25T15:25:29.311301+0000","last_active":"2024-04-25T15:25:30.300011+0000","last_peered":"2024-04-25T15:25:30.300011+0000","last_clean":"2024-04-25T15:25:30.300011+0000","last_became_active":"2024-04-25T15:25:29.310966+0000","last_became_peered":"2024-04-25T15:25:29.310966+0000","last_unstale":"2024-04-25T15:25:30.300011+0000","last_undegraded":"2024-04-25T15:25:30.300011+0000","last_fullsized":"2024-04-25T15:25:30.300011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:44.511850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657661+0000","last_change":"2024-04-25T15:25:29.315008+0000","last_active":"2024-04-25T15:25:30.657661+0000","last_peered":"2024-04-25T15:25:30.657661+0000","last_clean":"2024-04-25T15:25:30.657661+0000","last_became_active":"2024-04-25T15:25:29.314675+0000","last_became_peered":"2024-04-25T15:25:29.314675+0000","last_unstale":"2024-04-25T15:25:30.657661+0000","last_undegraded":"2024-04-25T15:25:30.657661+0000","last_fullsized":"2024-04-25T15:25:30.657661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296694+0000","last_change":"2024-04-25T15:25:29.301255+0000","last_active":"2024-04-25T15:25:30.296694+0000","last_peered":"2024-04-25T15:25:30.296694+0000","last_clean":"2024-04-25T15:25:30.296694+0000","last_became_active":"2024-04-25T15:25:29.300877+0000","last_became_peered":"2024-04-25T15:25:29.300877+0000","last_unstale":"2024-04-25T15:25:30.296694+0000","last_undegraded":"2024-04-25T15:25:30.296694+0000","last_fullsized":"2024-04-25T15:25:30.296694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:40.562364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293711+0000","last_change":"2024-04-25T15:25:29.300064+0000","last_active":"2024-04-25T15:25:30.293711+0000","last_peered":"2024-04-25T15:25:30.293711+0000","last_clean":"2024-04-25T15:25:30.293711+0000","last_became_active":"2024-04-25T15:25:29.299759+0000","last_became_peered":"2024-04-25T15:25:29.299759+0000","last_unstale":"2024-04-25T15:25:30.293711+0000","last_undegraded":"2024-04-25T15:25:30.293711+0000","last_fullsized":"2024-04-25T15:25:30.293711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657884+0000","last_change":"2024-04-25T15:25:29.314325+0000","last_active":"2024-04-25T15:25:30.657884+0000","last_peered":"2024-04-25T15:25:30.657884+0000","last_clean":"2024-04-25T15:25:30.657884+0000","last_became_active":"2024-04-25T15:25:29.311827+0000","last_became_peered":"2024-04-25T15:25:29.311827+0000","last_unstale":"2024-04-25T15:25:30.657884+0000","last_undegraded":"2024-04-25T15:25:30.657884+0000","last_fullsized":"2024-04-25T15:25:30.657884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658676+0000","last_change":"2024-04-25T15:25:29.327856+0000","last_active":"2024-04-25T15:25:30.658676+0000","last_peered":"2024-04-25T15:25:30.658676+0000","last_clean":"2024-04-25T15:25:30.658676+0000","last_became_active":"2024-04-25T15:25:29.327408+0000","last_became_peered":"2024-04-25T15:25:29.327408+0000","last_unstale":"2024-04-25T15:25:30.658676+0000","last_undegraded":"2024-04-25T15:25:30.658676+0000","last_fullsized":"2024-04-25T15:25:30.658676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657624+0000","last_change":"2024-04-25T15:25:29.313009+0000","last_active":"2024-04-25T15:25:30.657624+0000","last_peered":"2024-04-25T15:25:30.657624+0000","last_clean":"2024-04-25T15:25:30.657624+0000","last_became_active":"2024-04-25T15:25:29.312684+0000","last_became_peered":"2024-04-25T15:25:29.312684+0000","last_unstale":"2024-04-25T15:25:30.657624+0000","last_undegraded":"2024-04-25T15:25:30.657624+0000","last_fullsized":"2024-04-25T15:25:30.657624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:26.944400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658764+0000","last_change":"2024-04-25T15:25:29.319894+0000","last_active":"2024-04-25T15:25:30.658764+0000","last_peered":"2024-04-25T15:25:30.658764+0000","last_clean":"2024-04-25T15:25:30.658764+0000","last_became_active":"2024-04-25T15:25:29.319566+0000","last_became_peered":"2024-04-25T15:25:29.319566+0000","last_unstale":"2024-04-25T15:25:30.658764+0000","last_undegraded":"2024-04-25T15:25:30.658764+0000","last_fullsized":"2024-04-25T15:25:30.658764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313203+0000","last_change":"2024-04-25T15:25:29.312046+0000","last_active":"2024-04-25T15:25:30.313203+0000","last_peered":"2024-04-25T15:25:30.313203+0000","last_clean":"2024-04-25T15:25:30.313203+0000","last_became_active":"2024-04-25T15:25:29.310121+0000","last_became_peered":"2024-04-25T15:25:29.310121+0000","last_unstale":"2024-04-25T15:25:30.313203+0000","last_undegraded":"2024-04-25T15:25:30.313203+0000","last_fullsized":"2024-04-25T15:25:30.313203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296808+0000","last_change":"2024-04-25T15:25:29.317661+0000","last_active":"2024-04-25T15:25:30.296808+0000","last_peered":"2024-04-25T15:25:30.296808+0000","last_clean":"2024-04-25T15:25:30.296808+0000","last_became_active":"2024-04-25T15:25:29.317302+0000","last_became_peered":"2024-04-25T15:25:29.317302+0000","last_unstale":"2024-04-25T15:25:30.296808+0000","last_undegraded":"2024-04-25T15:25:30.296808+0000","last_fullsized":"2024-04-25T15:25:30.296808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658934+0000","last_change":"2024-04-25T15:25:29.327012+0000","last_active":"2024-04-25T15:25:30.658934+0000","last_peered":"2024-04-25T15:25:30.658934+0000","last_clean":"2024-04-25T15:25:30.658934+0000","last_became_active":"2024-04-25T15:25:29.326589+0000","last_became_peered":"2024-04-25T15:25:29.326589+0000","last_unstale":"2024-04-25T15:25:30.658934+0000","last_undegraded":"2024-04-25T15:25:30.658934+0000","last_fullsized":"2024-04-25T15:25:30.658934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657788+0000","last_change":"2024-04-25T15:25:29.321180+0000","last_active":"2024-04-25T15:25:30.657788+0000","last_peered":"2024-04-25T15:25:30.657788+0000","last_clean":"2024-04-25T15:25:30.657788+0000","last_became_active":"2024-04-25T15:25:29.320679+0000","last_became_peered":"2024-04-25T15:25:29.320679+0000","last_unstale":"2024-04-25T15:25:30.657788+0000","last_undegraded":"2024-04-25T15:25:30.657788+0000","last_fullsized":"2024-04-25T15:25:30.657788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658226+0000","last_change":"2024-04-25T15:25:29.311398+0000","last_active":"2024-04-25T15:25:30.658226+0000","last_peered":"2024-04-25T15:25:30.658226+0000","last_clean":"2024-04-25T15:25:30.658226+0000","last_became_active":"2024-04-25T15:25:29.311069+0000","last_became_peered":"2024-04-25T15:25:29.311069+0000","last_unstale":"2024-04-25T15:25:30.658226+0000","last_undegraded":"2024-04-25T15:25:30.658226+0000","last_fullsized":"2024-04-25T15:25:30.658226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658383+0000","last_change":"2024-04-25T15:25:29.320926+0000","last_active":"2024-04-25T15:25:30.658383+0000","last_peered":"2024-04-25T15:25:30.658383+0000","last_clean":"2024-04-25T15:25:30.658383+0000","last_became_active":"2024-04-25T15:25:29.320574+0000","last_became_peered":"2024-04-25T15:25:29.320574+0000","last_unstale":"2024-04-25T15:25:30.658383+0000","last_undegraded":"2024-04-25T15:25:30.658383+0000","last_fullsized":"2024-04-25T15:25:30.658383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313448+0000","last_change":"2024-04-25T15:25:29.330048+0000","last_active":"2024-04-25T15:25:30.313448+0000","last_peered":"2024-04-25T15:25:30.313448+0000","last_clean":"2024-04-25T15:25:30.313448+0000","last_became_active":"2024-04-25T15:25:29.329720+0000","last_became_peered":"2024-04-25T15:25:29.329720+0000","last_unstale":"2024-04-25T15:25:30.313448+0000","last_undegraded":"2024-04-25T15:25:30.313448+0000","last_fullsized":"2024-04-25T15:25:30.313448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293796+0000","last_change":"2024-04-25T15:25:29.309985+0000","last_active":"2024-04-25T15:25:30.293796+0000","last_peered":"2024-04-25T15:25:30.293796+0000","last_clean":"2024-04-25T15:25:30.293796+0000","last_became_active":"2024-04-25T15:25:29.309619+0000","last_became_peered":"2024-04-25T15:25:29.309619+0000","last_unstale":"2024-04-25T15:25:30.293796+0000","last_undegraded":"2024-04-25T15:25:30.293796+0000","last_fullsized":"2024-04-25T15:25:30.293796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658375+0000","last_change":"2024-04-25T15:25:29.321037+0000","last_active":"2024-04-25T15:25:30.658375+0000","last_peered":"2024-04-25T15:25:30.658375+0000","last_clean":"2024-04-25T15:25:30.658375+0000","last_became_active":"2024-04-25T15:25:29.320680+0000","last_became_peered":"2024-04-25T15:25:29.320680+0000","last_unstale":"2024-04-25T15:25:30.658375+0000","last_undegraded":"2024-04-25T15:25:30.658375+0000","last_fullsized":"2024-04-25T15:25:30.658375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296855+0000","last_change":"2024-04-25T15:25:29.309805+0000","last_active":"2024-04-25T15:25:30.296855+0000","last_peered":"2024-04-25T15:25:30.296855+0000","last_clean":"2024-04-25T15:25:30.296855+0000","last_became_active":"2024-04-25T15:25:29.309408+0000","last_became_peered":"2024-04-25T15:25:29.309408+0000","last_unstale":"2024-04-25T15:25:30.296855+0000","last_undegraded":"2024-04-25T15:25:30.296855+0000","last_fullsized":"2024-04-25T15:25:30.296855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659006+0000","last_change":"2024-04-25T15:25:29.318254+0000","last_active":"2024-04-25T15:25:30.659006+0000","last_peered":"2024-04-25T15:25:30.659006+0000","last_clean":"2024-04-25T15:25:30.659006+0000","last_became_active":"2024-04-25T15:25:29.317944+0000","last_became_peered":"2024-04-25T15:25:29.317944+0000","last_unstale":"2024-04-25T15:25:30.659006+0000","last_undegraded":"2024-04-25T15:25:30.659006+0000","last_fullsized":"2024-04-25T15:25:30.659006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657620+0000","last_change":"2024-04-25T15:25:29.313901+0000","last_active":"2024-04-25T15:25:30.657620+0000","last_peered":"2024-04-25T15:25:30.657620+0000","last_clean":"2024-04-25T15:25:30.657620+0000","last_became_active":"2024-04-25T15:25:29.313526+0000","last_became_peered":"2024-04-25T15:25:29.313526+0000","last_unstale":"2024-04-25T15:25:30.657620+0000","last_undegraded":"2024-04-25T15:25:30.657620+0000","last_fullsized":"2024-04-25T15:25:30.657620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313041+0000","last_change":"2024-04-25T15:25:29.312724+0000","last_active":"2024-04-25T15:25:30.313041+0000","last_peered":"2024-04-25T15:25:30.313041+0000","last_clean":"2024-04-25T15:25:30.313041+0000","last_became_active":"2024-04-25T15:25:29.312425+0000","last_became_peered":"2024-04-25T15:25:29.312425+0000","last_unstale":"2024-04-25T15:25:30.313041+0000","last_undegraded":"2024-04-25T15:25:30.313041+0000","last_fullsized":"2024-04-25T15:25:30.313041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299747+0000","last_change":"2024-04-25T15:25:29.298478+0000","last_active":"2024-04-25T15:25:30.299747+0000","last_peered":"2024-04-25T15:25:30.299747+0000","last_clean":"2024-04-25T15:25:30.299747+0000","last_became_active":"2024-04-25T15:25:29.298178+0000","last_became_peered":"2024-04-25T15:25:29.298178+0000","last_unstale":"2024-04-25T15:25:30.299747+0000","last_undegraded":"2024-04-25T15:25:30.299747+0000","last_fullsized":"2024-04-25T15:25:30.299747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:53.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658039+0000","last_change":"2024-04-25T15:25:28.301149+0000","last_active":"2024-04-25T15:25:30.658039+0000","last_peered":"2024-04-25T15:25:30.658039+0000","last_clean":"2024-04-25T15:25:30.658039+0000","last_became_active":"2024-04-25T15:25:28.300804+0000","last_became_peered":"2024-04-25T15:25:28.300804+0000","last_unstale":"2024-04-25T15:25:30.658039+0000","last_undegraded":"2024-04-25T15:25:30.658039+0000","last_fullsized":"2024-04-25T15:25:30.658039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296993+0000","last_change":"2024-04-25T15:25:29.306337+0000","last_active":"2024-04-25T15:25:30.296993+0000","last_peered":"2024-04-25T15:25:30.296993+0000","last_clean":"2024-04-25T15:25:30.296993+0000","last_became_active":"2024-04-25T15:25:29.306039+0000","last_became_peered":"2024-04-25T15:25:29.306039+0000","last_unstale":"2024-04-25T15:25:30.296993+0000","last_undegraded":"2024-04-25T15:25:30.296993+0000","last_fullsized":"2024-04-25T15:25:30.296993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313038+0000","last_change":"2024-04-25T15:25:28.301076+0000","last_active":"2024-04-25T15:25:30.313038+0000","last_peered":"2024-04-25T15:25:30.313038+0000","last_clean":"2024-04-25T15:25:30.313038+0000","last_became_active":"2024-04-25T15:25:28.300670+0000","last_became_peered":"2024-04-25T15:25:28.300670+0000","last_unstale":"2024-04-25T15:25:30.313038+0000","last_undegraded":"2024-04-25T15:25:30.313038+0000","last_fullsized":"2024-04-25T15:25:30.313038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657997+0000","last_change":"2024-04-25T15:25:28.305288+0000","last_active":"2024-04-25T15:25:30.657997+0000","last_peered":"2024-04-25T15:25:30.657997+0000","last_clean":"2024-04-25T15:25:30.657997+0000","last_became_active":"2024-04-25T15:25:28.304900+0000","last_became_peered":"2024-04-25T15:25:28.304900+0000","last_unstale":"2024-04-25T15:25:30.657997+0000","last_undegraded":"2024-04-25T15:25:30.657997+0000","last_fullsized":"2024-04-25T15:25:30.657997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:04.261360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658085+0000","last_change":"2024-04-25T15:25:29.310703+0000","last_active":"2024-04-25T15:25:30.658085+0000","last_peered":"2024-04-25T15:25:30.658085+0000","last_clean":"2024-04-25T15:25:30.658085+0000","last_became_active":"2024-04-25T15:25:29.309942+0000","last_became_peered":"2024-04-25T15:25:29.309942+0000","last_unstale":"2024-04-25T15:25:30.658085+0000","last_undegraded":"2024-04-25T15:25:30.658085+0000","last_fullsized":"2024-04-25T15:25:30.658085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658034+0000","last_change":"2024-04-25T15:25:28.292273+0000","last_active":"2024-04-25T15:25:30.658034+0000","last_peered":"2024-04-25T15:25:30.658034+0000","last_clean":"2024-04-25T15:25:30.658034+0000","last_became_active":"2024-04-25T15:25:28.291890+0000","last_became_peered":"2024-04-25T15:25:28.291890+0000","last_unstale":"2024-04-25T15:25:30.658034+0000","last_undegraded":"2024-04-25T15:25:30.658034+0000","last_fullsized":"2024-04-25T15:25:30.658034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293637+0000","last_change":"2024-04-25T15:25:29.310761+0000","last_active":"2024-04-25T15:25:30.293637+0000","last_peered":"2024-04-25T15:25:30.293637+0000","last_clean":"2024-04-25T15:25:30.293637+0000","last_became_active":"2024-04-25T15:25:29.310403+0000","last_became_peered":"2024-04-25T15:25:29.310403+0000","last_unstale":"2024-04-25T15:25:30.293637+0000","last_undegraded":"2024-04-25T15:25:30.293637+0000","last_fullsized":"2024-04-25T15:25:30.293637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313366+0000","last_change":"2024-04-25T15:25:28.306210+0000","last_active":"2024-04-25T15:25:30.313366+0000","last_peered":"2024-04-25T15:25:30.313366+0000","last_clean":"2024-04-25T15:25:30.313366+0000","last_became_active":"2024-04-25T15:25:28.305931+0000","last_became_peered":"2024-04-25T15:25:28.305931+0000","last_unstale":"2024-04-25T15:25:30.313366+0000","last_undegraded":"2024-04-25T15:25:30.313366+0000","last_fullsized":"2024-04-25T15:25:30.313366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658132+0000","last_change":"2024-04-25T15:25:29.328500+0000","last_active":"2024-04-25T15:25:30.658132+0000","last_peered":"2024-04-25T15:25:30.658132+0000","last_clean":"2024-04-25T15:25:30.658132+0000","last_became_active":"2024-04-25T15:25:29.328150+0000","last_became_peered":"2024-04-25T15:25:29.328150+0000","last_unstale":"2024-04-25T15:25:30.658132+0000","last_undegraded":"2024-04-25T15:25:30.658132+0000","last_fullsized":"2024-04-25T15:25:30.658132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658259+0000","last_change":"2024-04-25T15:25:28.305011+0000","last_active":"2024-04-25T15:25:30.658259+0000","last_peered":"2024-04-25T15:25:30.658259+0000","last_clean":"2024-04-25T15:25:30.658259+0000","last_became_active":"2024-04-25T15:25:28.304682+0000","last_became_peered":"2024-04-25T15:25:28.304682+0000","last_unstale":"2024-04-25T15:25:30.658259+0000","last_undegraded":"2024-04-25T15:25:30.658259+0000","last_fullsized":"2024-04-25T15:25:30.658259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297172+0000","last_change":"2024-04-25T15:25:29.318333+0000","last_active":"2024-04-25T15:25:30.297172+0000","last_peered":"2024-04-25T15:25:30.297172+0000","last_clean":"2024-04-25T15:25:30.297172+0000","last_became_active":"2024-04-25T15:25:29.317557+0000","last_became_peered":"2024-04-25T15:25:29.317557+0000","last_unstale":"2024-04-25T15:25:30.297172+0000","last_undegraded":"2024-04-25T15:25:30.297172+0000","last_fullsized":"2024-04-25T15:25:30.297172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293936+0000","last_change":"2024-04-25T15:25:28.297270+0000","last_active":"2024-04-25T15:25:30.293936+0000","last_peered":"2024-04-25T15:25:30.293936+0000","last_clean":"2024-04-25T15:25:30.293936+0000","last_became_active":"2024-04-25T15:25:28.296865+0000","last_became_peered":"2024-04-25T15:25:28.296865+0000","last_unstale":"2024-04-25T15:25:30.293936+0000","last_undegraded":"2024-04-25T15:25:30.293936+0000","last_fullsized":"2024-04-25T15:25:30.293936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300012+0000","last_change":"2024-04-25T15:25:29.324749+0000","last_active":"2024-04-25T15:25:30.300012+0000","last_peered":"2024-04-25T15:25:30.300012+0000","last_clean":"2024-04-25T15:25:30.300012+0000","last_became_active":"2024-04-25T15:25:29.324454+0000","last_became_peered":"2024-04-25T15:25:29.324454+0000","last_unstale":"2024-04-25T15:25:30.300012+0000","last_undegraded":"2024-04-25T15:25:30.300012+0000","last_fullsized":"2024-04-25T15:25:30.300012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313205+0000","last_change":"2024-04-25T15:25:28.302276+0000","last_active":"2024-04-25T15:25:30.313205+0000","last_peered":"2024-04-25T15:25:30.313205+0000","last_clean":"2024-04-25T15:25:30.313205+0000","last_became_active":"2024-04-25T15:25:28.302039+0000","last_became_peered":"2024-04-25T15:25:28.302039+0000","last_unstale":"2024-04-25T15:25:30.313205+0000","last_undegraded":"2024-04-25T15:25:30.313205+0000","last_fullsized":"2024-04-25T15:25:30.313205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658280+0000","last_change":"2024-04-25T15:25:29.324177+0000","last_active":"2024-04-25T15:25:30.658280+0000","last_peered":"2024-04-25T15:25:30.658280+0000","last_clean":"2024-04-25T15:25:30.658280+0000","last_became_active":"2024-04-25T15:25:29.323727+0000","last_became_peered":"2024-04-25T15:25:29.323727+0000","last_unstale":"2024-04-25T15:25:30.658280+0000","last_undegraded":"2024-04-25T15:25:30.658280+0000","last_fullsized":"2024-04-25T15:25:30.658280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296971+0000","last_change":"2024-04-25T15:25:28.288837+0000","last_active":"2024-04-25T15:25:30.296971+0000","last_peered":"2024-04-25T15:25:30.296971+0000","last_clean":"2024-04-25T15:25:30.296971+0000","last_became_active":"2024-04-25T15:25:28.288485+0000","last_became_peered":"2024-04-25T15:25:28.288485+0000","last_unstale":"2024-04-25T15:25:30.296971+0000","last_undegraded":"2024-04-25T15:25:30.296971+0000","last_fullsized":"2024-04-25T15:25:30.296971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:58.179326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296812+0000","last_change":"2024-04-25T15:25:29.323297+0000","last_active":"2024-04-25T15:25:30.296812+0000","last_peered":"2024-04-25T15:25:30.296812+0000","last_clean":"2024-04-25T15:25:30.296812+0000","last_became_active":"2024-04-25T15:25:29.322955+0000","last_became_peered":"2024-04-25T15:25:29.322955+0000","last_unstale":"2024-04-25T15:25:30.296812+0000","last_undegraded":"2024-04-25T15:25:30.296812+0000","last_fullsized":"2024-04-25T15:25:30.296812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659064+0000","last_change":"2024-04-25T15:25:28.300177+0000","last_active":"2024-04-25T15:25:30.659064+0000","last_peered":"2024-04-25T15:25:30.659064+0000","last_clean":"2024-04-25T15:25:30.659064+0000","last_became_active":"2024-04-25T15:25:28.298626+0000","last_became_peered":"2024-04-25T15:25:28.298626+0000","last_unstale":"2024-04-25T15:25:30.659064+0000","last_undegraded":"2024-04-25T15:25:30.659064+0000","last_fullsized":"2024-04-25T15:25:30.659064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657985+0000","last_change":"2024-04-25T15:25:29.316614+0000","last_active":"2024-04-25T15:25:30.657985+0000","last_peered":"2024-04-25T15:25:30.657985+0000","last_clean":"2024-04-25T15:25:30.657985+0000","last_became_active":"2024-04-25T15:25:29.316293+0000","last_became_peered":"2024-04-25T15:25:29.316293+0000","last_unstale":"2024-04-25T15:25:30.657985+0000","last_undegraded":"2024-04-25T15:25:30.657985+0000","last_fullsized":"2024-04-25T15:25:30.657985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312968+0000","last_change":"2024-04-25T15:25:28.304313+0000","last_active":"2024-04-25T15:25:30.312968+0000","last_peered":"2024-04-25T15:25:30.312968+0000","last_clean":"2024-04-25T15:25:30.312968+0000","last_became_active":"2024-04-25T15:25:28.303943+0000","last_became_peered":"2024-04-25T15:25:28.303943+0000","last_unstale":"2024-04-25T15:25:30.312968+0000","last_undegraded":"2024-04-25T15:25:30.312968+0000","last_fullsized":"2024-04-25T15:25:30.312968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313012+0000","last_change":"2024-04-25T15:25:29.310260+0000","last_active":"2024-04-25T15:25:30.313012+0000","last_peered":"2024-04-25T15:25:30.313012+0000","last_clean":"2024-04-25T15:25:30.313012+0000","last_became_active":"2024-04-25T15:25:29.309916+0000","last_became_peered":"2024-04-25T15:25:29.309916+0000","last_unstale":"2024-04-25T15:25:30.313012+0000","last_undegraded":"2024-04-25T15:25:30.313012+0000","last_fullsized":"2024-04-25T15:25:30.313012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657409+0000","last_change":"2024-04-25T15:25:28.293959+0000","last_active":"2024-04-25T15:25:30.657409+0000","last_peered":"2024-04-25T15:25:30.657409+0000","last_clean":"2024-04-25T15:25:30.657409+0000","last_became_active":"2024-04-25T15:25:28.293594+0000","last_became_peered":"2024-04-25T15:25:28.293594+0000","last_unstale":"2024-04-25T15:25:30.657409+0000","last_undegraded":"2024-04-25T15:25:30.657409+0000","last_fullsized":"2024-04-25T15:25:30.657409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658071+0000","last_change":"2024-04-25T15:25:29.318796+0000","last_active":"2024-04-25T15:25:30.658071+0000","last_peered":"2024-04-25T15:25:30.658071+0000","last_clean":"2024-04-25T15:25:30.658071+0000","last_became_active":"2024-04-25T15:25:29.318218+0000","last_became_peered":"2024-04-25T15:25:29.318218+0000","last_unstale":"2024-04-25T15:25:30.658071+0000","last_undegraded":"2024-04-25T15:25:30.658071+0000","last_fullsized":"2024-04-25T15:25:30.658071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300098+0000","last_change":"2024-04-25T15:25:28.293813+0000","last_active":"2024-04-25T15:25:30.300098+0000","last_peered":"2024-04-25T15:25:30.300098+0000","last_clean":"2024-04-25T15:25:30.300098+0000","last_became_active":"2024-04-25T15:25:28.293498+0000","last_became_peered":"2024-04-25T15:25:28.293498+0000","last_unstale":"2024-04-25T15:25:30.300098+0000","last_undegraded":"2024-04-25T15:25:30.300098+0000","last_fullsized":"2024-04-25T15:25:30.300098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:31:00.192290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658957+0000","last_change":"2024-04-25T15:25:29.322896+0000","last_active":"2024-04-25T15:25:30.658957+0000","last_peered":"2024-04-25T15:25:30.658957+0000","last_clean":"2024-04-25T15:25:30.658957+0000","last_became_active":"2024-04-25T15:25:29.321170+0000","last_became_peered":"2024-04-25T15:25:29.321170+0000","last_unstale":"2024-04-25T15:25:30.658957+0000","last_undegraded":"2024-04-25T15:25:30.658957+0000","last_fullsized":"2024-04-25T15:25:30.658957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657752+0000","last_change":"2024-04-25T15:25:28.294831+0000","last_active":"2024-04-25T15:25:30.657752+0000","last_peered":"2024-04-25T15:25:30.657752+0000","last_clean":"2024-04-25T15:25:30.657752+0000","last_became_active":"2024-04-25T15:25:28.294257+0000","last_became_peered":"2024-04-25T15:25:28.294257+0000","last_unstale":"2024-04-25T15:25:30.657752+0000","last_undegraded":"2024-04-25T15:25:30.657752+0000","last_fullsized":"2024-04-25T15:25:30.657752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659232+0000","last_change":"2024-04-25T15:25:29.325425+0000","last_active":"2024-04-25T15:25:30.659232+0000","last_peered":"2024-04-25T15:25:30.659232+0000","last_clean":"2024-04-25T15:25:30.659232+0000","last_became_active":"2024-04-25T15:25:29.324916+0000","last_became_peered":"2024-04-25T15:25:29.324916+0000","last_unstale":"2024-04-25T15:25:30.659232+0000","last_undegraded":"2024-04-25T15:25:30.659232+0000","last_fullsized":"2024-04-25T15:25:30.659232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:46.643371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.294024+0000","last_change":"2024-04-25T15:25:28.297384+0000","last_active":"2024-04-25T15:25:30.294024+0000","last_peered":"2024-04-25T15:25:30.294024+0000","last_clean":"2024-04-25T15:25:30.294024+0000","last_became_active":"2024-04-25T15:25:28.297035+0000","last_became_peered":"2024-04-25T15:25:28.297035+0000","last_unstale":"2024-04-25T15:25:30.294024+0000","last_undegraded":"2024-04-25T15:25:30.294024+0000","last_fullsized":"2024-04-25T15:25:30.294024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658488+0000","last_change":"2024-04-25T15:25:29.323269+0000","last_active":"2024-04-25T15:25:30.658488+0000","last_peered":"2024-04-25T15:25:30.658488+0000","last_clean":"2024-04-25T15:25:30.658488+0000","last_became_active":"2024-04-25T15:25:29.322481+0000","last_became_peered":"2024-04-25T15:25:29.322481+0000","last_unstale":"2024-04-25T15:25:30.658488+0000","last_undegraded":"2024-04-25T15:25:30.658488+0000","last_fullsized":"2024-04-25T15:25:30.658488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657594+0000","last_change":"2024-04-25T15:25:28.293938+0000","last_active":"2024-04-25T15:25:30.657594+0000","last_peered":"2024-04-25T15:25:30.657594+0000","last_clean":"2024-04-25T15:25:30.657594+0000","last_became_active":"2024-04-25T15:25:28.293520+0000","last_became_peered":"2024-04-25T15:25:28.293520+0000","last_unstale":"2024-04-25T15:25:30.657594+0000","last_undegraded":"2024-04-25T15:25:30.657594+0000","last_fullsized":"2024-04-25T15:25:30.657594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658465+0000","last_change":"2024-04-25T15:25:29.325130+0000","last_active":"2024-04-25T15:25:30.658465+0000","last_peered":"2024-04-25T15:25:30.658465+0000","last_clean":"2024-04-25T15:25:30.658465+0000","last_became_active":"2024-04-25T15:25:29.324783+0000","last_became_peered":"2024-04-25T15:25:29.324783+0000","last_unstale":"2024-04-25T15:25:30.658465+0000","last_undegraded":"2024-04-25T15:25:30.658465+0000","last_fullsized":"2024-04-25T15:25:30.658465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657696+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:25:30.657696+0000","last_peered":"2024-04-25T15:25:30.657696+0000","last_clean":"2024-04-25T15:25:30.657696+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:25:30.657696+0000","last_undegraded":"2024-04-25T15:25:30.657696+0000","last_fullsized":"2024-04-25T15:25:30.657696+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657863+0000","last_change":"2024-04-25T15:25:28.293187+0000","last_active":"2024-04-25T15:25:30.657863+0000","last_peered":"2024-04-25T15:25:30.657863+0000","last_clean":"2024-04-25T15:25:30.657863+0000","last_became_active":"2024-04-25T15:25:28.292748+0000","last_became_peered":"2024-04-25T15:25:28.292748+0000","last_unstale":"2024-04-25T15:25:30.657863+0000","last_undegraded":"2024-04-25T15:25:30.657863+0000","last_fullsized":"2024-04-25T15:25:30.657863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657964+0000","last_change":"2024-04-25T15:25:29.323272+0000","last_active":"2024-04-25T15:25:30.657964+0000","last_peered":"2024-04-25T15:25:30.657964+0000","last_clean":"2024-04-25T15:25:30.657964+0000","last_became_active":"2024-04-25T15:25:29.322937+0000","last_became_peered":"2024-04-25T15:25:29.322937+0000","last_unstale":"2024-04-25T15:25:30.657964+0000","last_undegraded":"2024-04-25T15:25:30.657964+0000","last_fullsized":"2024-04-25T15:25:30.657964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657646+0000","last_change":"2024-04-25T15:25:28.301819+0000","last_active":"2024-04-25T15:25:30.657646+0000","last_peered":"2024-04-25T15:25:30.657646+0000","last_clean":"2024-04-25T15:25:30.657646+0000","last_became_active":"2024-04-25T15:25:28.301519+0000","last_became_peered":"2024-04-25T15:25:28.301519+0000","last_unstale":"2024-04-25T15:25:30.657646+0000","last_undegraded":"2024-04-25T15:25:30.657646+0000","last_fullsized":"2024-04-25T15:25:30.657646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296763+0000","last_change":"2024-04-25T15:25:29.306397+0000","last_active":"2024-04-25T15:25:30.296763+0000","last_peered":"2024-04-25T15:25:30.296763+0000","last_clean":"2024-04-25T15:25:30.296763+0000","last_became_active":"2024-04-25T15:25:29.306088+0000","last_became_peered":"2024-04-25T15:25:29.306088+0000","last_unstale":"2024-04-25T15:25:30.296763+0000","last_undegraded":"2024-04-25T15:25:30.296763+0000","last_fullsized":"2024-04-25T15:25:30.296763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:05.665096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657506+0000","last_change":"2024-04-25T15:25:28.295278+0000","last_active":"2024-04-25T15:25:30.657506+0000","last_peered":"2024-04-25T15:25:30.657506+0000","last_clean":"2024-04-25T15:25:30.657506+0000","last_became_active":"2024-04-25T15:25:28.294876+0000","last_became_peered":"2024-04-25T15:25:28.294876+0000","last_unstale":"2024-04-25T15:25:30.657506+0000","last_undegraded":"2024-04-25T15:25:30.657506+0000","last_fullsized":"2024-04-25T15:25:30.657506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293669+0000","last_change":"2024-04-25T15:25:29.307860+0000","last_active":"2024-04-25T15:25:30.293669+0000","last_peered":"2024-04-25T15:25:30.293669+0000","last_clean":"2024-04-25T15:25:30.293669+0000","last_became_active":"2024-04-25T15:25:29.307525+0000","last_became_peered":"2024-04-25T15:25:29.307525+0000","last_unstale":"2024-04-25T15:25:30.293669+0000","last_undegraded":"2024-04-25T15:25:30.293669+0000","last_fullsized":"2024-04-25T15:25:30.293669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659149+0000","last_change":"2024-04-25T15:25:28.298243+0000","last_active":"2024-04-25T15:25:30.659149+0000","last_peered":"2024-04-25T15:25:30.659149+0000","last_clean":"2024-04-25T15:25:30.659149+0000","last_became_active":"2024-04-25T15:25:28.297837+0000","last_became_peered":"2024-04-25T15:25:28.297837+0000","last_unstale":"2024-04-25T15:25:30.659149+0000","last_undegraded":"2024-04-25T15:25:30.659149+0000","last_fullsized":"2024-04-25T15:25:30.659149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299949+0000","last_change":"2024-04-25T15:25:29.308455+0000","last_active":"2024-04-25T15:25:30.299949+0000","last_peered":"2024-04-25T15:25:30.299949+0000","last_clean":"2024-04-25T15:25:30.299949+0000","last_became_active":"2024-04-25T15:25:29.307968+0000","last_became_peered":"2024-04-25T15:25:29.307968+0000","last_unstale":"2024-04-25T15:25:30.299949+0000","last_undegraded":"2024-04-25T15:25:30.299949+0000","last_fullsized":"2024-04-25T15:25:30.299949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:53.177957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293919+0000","last_change":"2024-04-25T15:25:28.298284+0000","last_active":"2024-04-25T15:25:30.293919+0000","last_peered":"2024-04-25T15:25:30.293919+0000","last_clean":"2024-04-25T15:25:30.293919+0000","last_became_active":"2024-04-25T15:25:28.297792+0000","last_became_peered":"2024-04-25T15:25:28.297792+0000","last_unstale":"2024-04-25T15:25:30.293919+0000","last_undegraded":"2024-04-25T15:25:30.293919+0000","last_fullsized":"2024-04-25T15:25:30.293919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658750+0000","last_change":"2024-04-25T15:25:29.313891+0000","last_active":"2024-04-25T15:25:30.658750+0000","last_peered":"2024-04-25T15:25:30.658750+0000","last_clean":"2024-04-25T15:25:30.658750+0000","last_became_active":"2024-04-25T15:25:29.313488+0000","last_became_peered":"2024-04-25T15:25:29.313488+0000","last_unstale":"2024-04-25T15:25:30.658750+0000","last_undegraded":"2024-04-25T15:25:30.658750+0000","last_fullsized":"2024-04-25T15:25:30.658750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658847+0000","last_change":"2024-04-25T15:25:28.301563+0000","last_active":"2024-04-25T15:25:30.658847+0000","last_peered":"2024-04-25T15:25:30.658847+0000","last_clean":"2024-04-25T15:25:30.658847+0000","last_became_active":"2024-04-25T15:25:28.301186+0000","last_became_peered":"2024-04-25T15:25:28.301186+0000","last_unstale":"2024-04-25T15:25:30.658847+0000","last_undegraded":"2024-04-25T15:25:30.658847+0000","last_fullsized":"2024-04-25T15:25:30.658847+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:25.937421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293726+0000","last_change":"2024-04-25T15:25:29.308391+0000","last_active":"2024-04-25T15:25:30.293726+0000","last_peered":"2024-04-25T15:25:30.293726+0000","last_clean":"2024-04-25T15:25:30.293726+0000","last_became_active":"2024-04-25T15:25:29.308010+0000","last_became_peered":"2024-04-25T15:25:29.308010+0000","last_unstale":"2024-04-25T15:25:30.293726+0000","last_undegraded":"2024-04-25T15:25:30.293726+0000","last_fullsized":"2024-04-25T15:25:30.293726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312496+0000","last_change":"2024-04-25T15:25:28.304872+0000","last_active":"2024-04-25T15:25:30.312496+0000","last_peered":"2024-04-25T15:25:30.312496+0000","last_clean":"2024-04-25T15:25:30.312496+0000","last_became_active":"2024-04-25T15:25:28.304443+0000","last_became_peered":"2024-04-25T15:25:28.304443+0000","last_unstale":"2024-04-25T15:25:30.312496+0000","last_undegraded":"2024-04-25T15:25:30.312496+0000","last_fullsized":"2024-04-25T15:25:30.312496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658447+0000","last_change":"2024-04-25T15:25:29.326344+0000","last_active":"2024-04-25T15:25:30.658447+0000","last_peered":"2024-04-25T15:25:30.658447+0000","last_clean":"2024-04-25T15:25:30.658447+0000","last_became_active":"2024-04-25T15:25:29.326016+0000","last_became_peered":"2024-04-25T15:25:29.326016+0000","last_unstale":"2024-04-25T15:25:30.658447+0000","last_undegraded":"2024-04-25T15:25:30.658447+0000","last_fullsized":"2024-04-25T15:25:30.658447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313461+0000","last_change":"2024-04-25T15:25:28.307284+0000","last_active":"2024-04-25T15:25:30.313461+0000","last_peered":"2024-04-25T15:25:30.313461+0000","last_clean":"2024-04-25T15:25:30.313461+0000","last_became_active":"2024-04-25T15:25:28.307013+0000","last_became_peered":"2024-04-25T15:25:28.307013+0000","last_unstale":"2024-04-25T15:25:30.313461+0000","last_undegraded":"2024-04-25T15:25:30.313461+0000","last_fullsized":"2024-04-25T15:25:30.313461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297033+0000","last_change":"2024-04-25T15:25:29.301384+0000","last_active":"2024-04-25T15:25:30.297033+0000","last_peered":"2024-04-25T15:25:30.297033+0000","last_clean":"2024-04-25T15:25:30.297033+0000","last_became_active":"2024-04-25T15:25:29.301021+0000","last_became_peered":"2024-04-25T15:25:29.301021+0000","last_unstale":"2024-04-25T15:25:30.297033+0000","last_undegraded":"2024-04-25T15:25:30.297033+0000","last_fullsized":"2024-04-25T15:25:30.297033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657611+0000","last_change":"2024-04-25T15:25:28.304086+0000","last_active":"2024-04-25T15:25:30.657611+0000","last_peered":"2024-04-25T15:25:30.657611+0000","last_clean":"2024-04-25T15:25:30.657611+0000","last_became_active":"2024-04-25T15:25:28.303771+0000","last_became_peered":"2024-04-25T15:25:28.303771+0000","last_unstale":"2024-04-25T15:25:30.657611+0000","last_undegraded":"2024-04-25T15:25:30.657611+0000","last_fullsized":"2024-04-25T15:25:30.657611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299860+0000","last_change":"2024-04-25T15:25:29.310784+0000","last_active":"2024-04-25T15:25:30.299860+0000","last_peered":"2024-04-25T15:25:30.299860+0000","last_clean":"2024-04-25T15:25:30.299860+0000","last_became_active":"2024-04-25T15:25:29.309711+0000","last_became_peered":"2024-04-25T15:25:29.309711+0000","last_unstale":"2024-04-25T15:25:30.299860+0000","last_undegraded":"2024-04-25T15:25:30.299860+0000","last_fullsized":"2024-04-25T15:25:30.299860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:16.888909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293756+0000","last_change":"2024-04-25T15:25:28.300848+0000","last_active":"2024-04-25T15:25:30.293756+0000","last_peered":"2024-04-25T15:25:30.293756+0000","last_clean":"2024-04-25T15:25:30.293756+0000","last_became_active":"2024-04-25T15:25:28.300218+0000","last_became_peered":"2024-04-25T15:25:28.300218+0000","last_unstale":"2024-04-25T15:25:30.293756+0000","last_undegraded":"2024-04-25T15:25:30.293756+0000","last_fullsized":"2024-04-25T15:25:30.293756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658212+0000","last_change":"2024-04-25T15:25:29.313291+0000","last_active":"2024-04-25T15:25:30.658212+0000","last_peered":"2024-04-25T15:25:30.658212+0000","last_clean":"2024-04-25T15:25:30.658212+0000","last_became_active":"2024-04-25T15:25:29.312960+0000","last_became_peered":"2024-04-25T15:25:29.312960+0000","last_unstale":"2024-04-25T15:25:30.658212+0000","last_undegraded":"2024-04-25T15:25:30.658212+0000","last_fullsized":"2024-04-25T15:25:30.658212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293893+0000","last_change":"2024-04-25T15:25:28.301806+0000","last_active":"2024-04-25T15:25:30.293893+0000","last_peered":"2024-04-25T15:25:30.293893+0000","last_clean":"2024-04-25T15:25:30.293893+0000","last_became_active":"2024-04-25T15:25:28.301447+0000","last_became_peered":"2024-04-25T15:25:28.301447+0000","last_unstale":"2024-04-25T15:25:30.293893+0000","last_undegraded":"2024-04-25T15:25:30.293893+0000","last_fullsized":"2024-04-25T15:25:30.293893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657861+0000","last_change":"2024-04-25T15:25:29.310336+0000","last_active":"2024-04-25T15:25:30.657861+0000","last_peered":"2024-04-25T15:25:30.657861+0000","last_clean":"2024-04-25T15:25:30.657861+0000","last_became_active":"2024-04-25T15:25:29.309914+0000","last_became_peered":"2024-04-25T15:25:29.309914+0000","last_unstale":"2024-04-25T15:25:30.657861+0000","last_undegraded":"2024-04-25T15:25:30.657861+0000","last_fullsized":"2024-04-25T15:25:30.657861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658401+0000","last_change":"2024-04-25T15:25:28.300815+0000","last_active":"2024-04-25T15:25:30.658401+0000","last_peered":"2024-04-25T15:25:30.658401+0000","last_clean":"2024-04-25T15:25:30.658401+0000","last_became_active":"2024-04-25T15:25:28.300487+0000","last_became_peered":"2024-04-25T15:25:28.300487+0000","last_unstale":"2024-04-25T15:25:30.658401+0000","last_undegraded":"2024-04-25T15:25:30.658401+0000","last_fullsized":"2024-04-25T15:25:30.658401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313011+0000","last_change":"2024-04-25T15:25:29.310500+0000","last_active":"2024-04-25T15:25:30.313011+0000","last_peered":"2024-04-25T15:25:30.313011+0000","last_clean":"2024-04-25T15:25:30.313011+0000","last_became_active":"2024-04-25T15:25:29.310111+0000","last_became_peered":"2024-04-25T15:25:29.310111+0000","last_unstale":"2024-04-25T15:25:30.313011+0000","last_undegraded":"2024-04-25T15:25:30.313011+0000","last_fullsized":"2024-04-25T15:25:30.313011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658618+0000","last_change":"2024-04-25T15:25:29.314651+0000","last_active":"2024-04-25T15:25:30.658618+0000","last_peered":"2024-04-25T15:25:30.658618+0000","last_clean":"2024-04-25T15:25:30.658618+0000","last_became_active":"2024-04-25T15:25:29.314239+0000","last_became_peered":"2024-04-25T15:25:29.314239+0000","last_unstale":"2024-04-25T15:25:30.658618+0000","last_undegraded":"2024-04-25T15:25:30.658618+0000","last_fullsized":"2024-04-25T15:25:30.658618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658320+0000","last_change":"2024-04-25T15:25:28.304792+0000","last_active":"2024-04-25T15:25:30.658320+0000","last_peered":"2024-04-25T15:25:30.658320+0000","last_clean":"2024-04-25T15:25:30.658320+0000","last_became_active":"2024-04-25T15:25:28.304489+0000","last_became_peered":"2024-04-25T15:25:28.304489+0000","last_unstale":"2024-04-25T15:25:30.658320+0000","last_undegraded":"2024-04-25T15:25:30.658320+0000","last_fullsized":"2024-04-25T15:25:30.658320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658967+0000","last_change":"2024-04-25T15:25:29.318608+0000","last_active":"2024-04-25T15:25:30.658967+0000","last_peered":"2024-04-25T15:25:30.658967+0000","last_clean":"2024-04-25T15:25:30.658967+0000","last_became_active":"2024-04-25T15:25:29.317330+0000","last_became_peered":"2024-04-25T15:25:29.317330+0000","last_unstale":"2024-04-25T15:25:30.658967+0000","last_undegraded":"2024-04-25T15:25:30.658967+0000","last_fullsized":"2024-04-25T15:25:30.658967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658591+0000","last_change":"2024-04-25T15:25:28.305376+0000","last_active":"2024-04-25T15:25:30.658591+0000","last_peered":"2024-04-25T15:25:30.658591+0000","last_clean":"2024-04-25T15:25:30.658591+0000","last_became_active":"2024-04-25T15:25:28.305083+0000","last_became_peered":"2024-04-25T15:25:28.305083+0000","last_unstale":"2024-04-25T15:25:30.658591+0000","last_undegraded":"2024-04-25T15:25:30.658591+0000","last_fullsized":"2024-04-25T15:25:30.658591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297061+0000","last_change":"2024-04-25T15:25:29.319184+0000","last_active":"2024-04-25T15:25:30.297061+0000","last_peered":"2024-04-25T15:25:30.297061+0000","last_clean":"2024-04-25T15:25:30.297061+0000","last_became_active":"2024-04-25T15:25:29.317757+0000","last_became_peered":"2024-04-25T15:25:29.317757+0000","last_unstale":"2024-04-25T15:25:30.297061+0000","last_undegraded":"2024-04-25T15:25:30.297061+0000","last_fullsized":"2024-04-25T15:25:30.297061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658361+0000","last_change":"2024-04-25T15:25:28.298019+0000","last_active":"2024-04-25T15:25:30.658361+0000","last_peered":"2024-04-25T15:25:30.658361+0000","last_clean":"2024-04-25T15:25:30.658361+0000","last_became_active":"2024-04-25T15:25:28.297570+0000","last_became_peered":"2024-04-25T15:25:28.297570+0000","last_unstale":"2024-04-25T15:25:30.658361+0000","last_undegraded":"2024-04-25T15:25:30.658361+0000","last_fullsized":"2024-04-25T15:25:30.658361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658351+0000","last_change":"2024-04-25T15:25:29.316068+0000","last_active":"2024-04-25T15:25:30.658351+0000","last_peered":"2024-04-25T15:25:30.658351+0000","last_clean":"2024-04-25T15:25:30.658351+0000","last_became_active":"2024-04-25T15:25:29.315705+0000","last_became_peered":"2024-04-25T15:25:29.315705+0000","last_unstale":"2024-04-25T15:25:30.658351+0000","last_undegraded":"2024-04-25T15:25:30.658351+0000","last_fullsized":"2024-04-25T15:25:30.658351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297170+0000","last_change":"2024-04-25T15:25:28.300551+0000","last_active":"2024-04-25T15:25:30.297170+0000","last_peered":"2024-04-25T15:25:30.297170+0000","last_clean":"2024-04-25T15:25:30.297170+0000","last_became_active":"2024-04-25T15:25:28.299984+0000","last_became_peered":"2024-04-25T15:25:28.299984+0000","last_unstale":"2024-04-25T15:25:30.297170+0000","last_undegraded":"2024-04-25T15:25:30.297170+0000","last_fullsized":"2024-04-25T15:25:30.297170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:08:02.904549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:25:33.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:25:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.506+0000 7fcb6a2b6640 1 Processor -- start 2024-04-25T15:25:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.506+0000 7fcb6a2b6640 1 -- start start 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb6a2b6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcb641646d0 0x7fcb6415ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb6a2b6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb64164c80 con 0x7fcb641616a0 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb6a2b6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb64160bc0 con 0x7fcb6406e740 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb6a2b6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcb64160980 con 0x7fcb641646d0 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb63fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb641616a0 0x7fcb6406e030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59390/0 (socket says 172.21.15.67:59390) 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb63fff640 1 -- 172.21.15.67:0/3533100092 learned_addr learned my addr 172.21.15.67:0/3533100092 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb68853640 1 --2- 172.21.15.67:0/3533100092 >> v2:172.21.15.176:3300/0 conn(0x7fcb641646d0 0x7fcb6415ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.507+0000 7fcb62ffd640 1 -- 172.21.15.67:0/3533100092 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1715666552 0 0) 0x7fcb64164c80 con 0x7fcb641616a0 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.508+0000 7fcb62ffd640 1 -- 172.21.15.67:0/3533100092 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb4c003520 con 0x7fcb641616a0 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.508+0000 7fcb68853640 1 -- 172.21.15.67:0/3533100092 >> v1:172.21.15.176:6789/0 conn(0x7fcb6406e740 legacy=0x7fcb64163fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.508+0000 7fcb68853640 1 -- 172.21.15.67:0/3533100092 >> v1:172.21.15.67:6789/0 conn(0x7fcb641616a0 legacy=0x7fcb6406e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.508+0000 7fcb62ffd640 1 -- 172.21.15.67:0/3533100092 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2243249829 0 0) 0x7fcb4c003520 con 0x7fcb641616a0 2024-04-25T15:25:33.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.508+0000 7fcb68853640 1 -- 172.21.15.67:0/3533100092 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb64160ea0 con 0x7fcb641646d0 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb68853640 1 --2- 172.21.15.67:0/3533100092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415ded0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009dc0 tx=0x7fcb5c02a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cead2bf3db8ff505 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb62ffd640 1 -- 172.21.15.67:0/3533100092 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c007740 con 0x7fcb641646d0 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb62ffd640 1 -- 172.21.15.67:0/3533100092 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0078e0 con 0x7fcb641646d0 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb62ffd640 1 -- 172.21.15.67:0/3533100092 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c007c90 con 0x7fcb641646d0 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/3533100092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 msgr2=0x7fcb6415ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/3533100092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415ded0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009dc0 tx=0x7fcb5c02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/3533100092 shutdown_connections 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/3533100092 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415ded0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/3533100092 >> 172.21.15.67:0/3533100092 conn(0x7fcb640d27c0 msgr2=0x7fcb640d2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.509+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/3533100092 shutdown_connections 2024-04-25T15:25:33.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.510+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/3533100092 wait complete. 2024-04-25T15:25:33.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.511+0000 7fcb6a2b6640 1 Processor -- start 2024-04-25T15:25:33.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.511+0000 7fcb6a2b6640 1 -- start start 2024-04-25T15:25:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb6a2b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb641616a0 0x7fcb6413d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb6a2b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb6a2b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb640cfad0 0x7fcb64155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb6a2b6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6416b280 con 0x7fcb641616a0 2024-04-25T15:25:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb6a2b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6416b100 con 0x7fcb641646d0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb6a2b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6416b400 con 0x7fcb640cfad0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb641616a0 0x7fcb6413d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb641616a0 0x7fcb6413d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37046/0 (socket says 172.21.15.67:37046) 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 -- 172.21.15.67:0/4276166340 learned_addr learned my addr 172.21.15.67:0/4276166340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb637fe640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb68853640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb640cfad0 0x7fcb64155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 -- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb640cfad0 msgr2=0x7fcb64155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb640cfad0 0x7fcb64155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 -- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 msgr2=0x7fcb6415d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 -- 172.21.15.67:0/4276166340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6413dcb0 con 0x7fcb641616a0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb637fe640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb63fff640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb641616a0 0x7fcb6413d770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c410 tx=0x7fcb54001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb60ff9640 1 -- 172.21.15.67:0/4276166340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54017070 con 0x7fcb641616a0 2024-04-25T15:25:33.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb60ff9640 1 -- 172.21.15.67:0/4276166340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb54005ce0 con 0x7fcb641616a0 2024-04-25T15:25:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.513+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb64158980 con 0x7fcb641616a0 2024-04-25T15:25:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.514+0000 7fcb60ff9640 1 -- 172.21.15.67:0/4276166340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54007310 con 0x7fcb641616a0 2024-04-25T15:25:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.514+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb64155160 con 0x7fcb641616a0 2024-04-25T15:25:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.515+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb64050740 con 0x7fcb641616a0 2024-04-25T15:25:33.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.515+0000 7fcb60ff9640 1 -- 172.21.15.67:0/4276166340 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb5401a070 con 0x7fcb641616a0 2024-04-25T15:25:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.516+0000 7fcb60ff9640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb500744b0 0x7fcb500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.516+0000 7fcb60ff9640 1 -- 172.21.15.67:0/4276166340 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcb54094570 con 0x7fcb641616a0 2024-04-25T15:25:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.516+0000 7fcb637fe640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb500744b0 0x7fcb500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.516+0000 7fcb637fe640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb500744b0 0x7fcb500769c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb64159150 tx=0x7fcb58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.520+0000 7fcb60ff9640 1 -- 172.21.15.67:0/4276166340 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb5405ed90 con 0x7fcb641616a0 2024-04-25T15:25:33.592 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-04-25T15:25:33.591+0000 7efc9845a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:25:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.701+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcb64160380 con 0x7fcb500744b0 2024-04-25T15:25:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.708+0000 7fcb60ff9640 1 -- 172.21.15.67:0/4276166340 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251936 (secure 0 0 0) 0x7fcb64160380 con 0x7fcb500744b0 2024-04-25T15:25:33.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:33.712 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:25:33.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb500744b0 msgr2=0x7fcb500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb500744b0 0x7fcb500769c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcb64159150 tx=0x7fcb58009040 comp rx=0 tx=0).stop 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb641616a0 msgr2=0x7fcb6413d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb641616a0 0x7fcb6413d770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c410 tx=0x7fcb54001e70 comp rx=0 tx=0).stop 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 shutdown_connections 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb500744b0 0x7fcb500769c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb640cfad0 0x7fcb64155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb641646d0 0x7fcb6415d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 --2- 172.21.15.67:0/4276166340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb641616a0 0x7fcb6413d770 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 >> 172.21.15.67:0/4276166340 conn(0x7fcb640d27c0 msgr2=0x7fcb640d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 shutdown_connections 2024-04-25T15:25:33.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.711+0000 7fcb6a2b6640 1 -- 172.21.15.67:0/4276166340 wait complete. 2024-04-25T15:25:33.723 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-25T15:25:32.795770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.518961"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659051+0000","last_change":"2024-04-25T15:25:29.325672+0000","last_active":"2024-04-25T15:25:30.659051+0000","last_peered":"2024-04-25T15:25:30.659051+0000","last_clean":"2024-04-25T15:25:30.659051+0000","last_became_active":"2024-04-25T15:25:29.324999+0000","last_became_peered":"2024-04-25T15:25:29.324999+0000","last_unstale":"2024-04-25T15:25:30.659051+0000","last_undegraded":"2024-04-25T15:25:30.659051+0000","last_fullsized":"2024-04-25T15:25:30.659051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293966+0000","last_change":"2024-04-25T15:25:29.303032+0000","last_active":"2024-04-25T15:25:30.293966+0000","last_peered":"2024-04-25T15:25:30.293966+0000","last_clean":"2024-04-25T15:25:30.293966+0000","last_became_active":"2024-04-25T15:25:29.302027+0000","last_became_peered":"2024-04-25T15:25:29.302027+0000","last_unstale":"2024-04-25T15:25:30.293966+0000","last_undegraded":"2024-04-25T15:25:30.293966+0000","last_fullsized":"2024-04-25T15:25:30.293966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297058+0000","last_change":"2024-04-25T15:25:29.307526+0000","last_active":"2024-04-25T15:25:30.297058+0000","last_peered":"2024-04-25T15:25:30.297058+0000","last_clean":"2024-04-25T15:25:30.297058+0000","last_became_active":"2024-04-25T15:25:29.307109+0000","last_became_peered":"2024-04-25T15:25:29.307109+0000","last_unstale":"2024-04-25T15:25:30.297058+0000","last_undegraded":"2024-04-25T15:25:30.297058+0000","last_fullsized":"2024-04-25T15:25:30.297058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323540+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322901+0000","last_became_peered":"2024-04-25T15:25:29.322901+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658248+0000","last_change":"2024-04-25T15:25:29.313182+0000","last_active":"2024-04-25T15:25:30.658248+0000","last_peered":"2024-04-25T15:25:30.658248+0000","last_clean":"2024-04-25T15:25:30.658248+0000","last_became_active":"2024-04-25T15:25:29.312850+0000","last_became_peered":"2024-04-25T15:25:29.312850+0000","last_unstale":"2024-04-25T15:25:30.658248+0000","last_undegraded":"2024-04-25T15:25:30.658248+0000","last_fullsized":"2024-04-25T15:25:30.658248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658795+0000","last_change":"2024-04-25T15:25:29.319442+0000","last_active":"2024-04-25T15:25:30.658795+0000","last_peered":"2024-04-25T15:25:30.658795+0000","last_clean":"2024-04-25T15:25:30.658795+0000","last_became_active":"2024-04-25T15:25:29.319141+0000","last_became_peered":"2024-04-25T15:25:29.319141+0000","last_unstale":"2024-04-25T15:25:30.658795+0000","last_undegraded":"2024-04-25T15:25:30.658795+0000","last_fullsized":"2024-04-25T15:25:30.658795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657979+0000","last_change":"2024-04-25T15:25:29.322616+0000","last_active":"2024-04-25T15:25:30.657979+0000","last_peered":"2024-04-25T15:25:30.657979+0000","last_clean":"2024-04-25T15:25:30.657979+0000","last_became_active":"2024-04-25T15:25:29.322311+0000","last_became_peered":"2024-04-25T15:25:29.322311+0000","last_unstale":"2024-04-25T15:25:30.657979+0000","last_undegraded":"2024-04-25T15:25:30.657979+0000","last_fullsized":"2024-04-25T15:25:30.657979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323286+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322911+0000","last_became_peered":"2024-04-25T15:25:29.322911+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659143+0000","last_change":"2024-04-25T15:25:29.325686+0000","last_active":"2024-04-25T15:25:30.659143+0000","last_peered":"2024-04-25T15:25:30.659143+0000","last_clean":"2024-04-25T15:25:30.659143+0000","last_became_active":"2024-04-25T15:25:29.325332+0000","last_became_peered":"2024-04-25T15:25:29.325332+0000","last_unstale":"2024-04-25T15:25:30.659143+0000","last_undegraded":"2024-04-25T15:25:30.659143+0000","last_fullsized":"2024-04-25T15:25:30.659143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:20.087185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313378+0000","last_change":"2024-04-25T15:25:29.323852+0000","last_active":"2024-04-25T15:25:30.313378+0000","last_peered":"2024-04-25T15:25:30.313378+0000","last_clean":"2024-04-25T15:25:30.313378+0000","last_became_active":"2024-04-25T15:25:29.323524+0000","last_became_peered":"2024-04-25T15:25:29.323524+0000","last_unstale":"2024-04-25T15:25:30.313378+0000","last_undegraded":"2024-04-25T15:25:30.313378+0000","last_fullsized":"2024-04-25T15:25:30.313378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313308+0000","last_change":"2024-04-25T15:25:29.325815+0000","last_active":"2024-04-25T15:25:30.313308+0000","last_peered":"2024-04-25T15:25:30.313308+0000","last_clean":"2024-04-25T15:25:30.313308+0000","last_became_active":"2024-04-25T15:25:29.325436+0000","last_became_peered":"2024-04-25T15:25:29.325436+0000","last_unstale":"2024-04-25T15:25:30.313308+0000","last_undegraded":"2024-04-25T15:25:30.313308+0000","last_fullsized":"2024-04-25T15:25:30.313308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300011+0000","last_change":"2024-04-25T15:25:29.311301+0000","last_active":"2024-04-25T15:25:30.300011+0000","last_peered":"2024-04-25T15:25:30.300011+0000","last_clean":"2024-04-25T15:25:30.300011+0000","last_became_active":"2024-04-25T15:25:29.310966+0000","last_became_peered":"2024-04-25T15:25:29.310966+0000","last_unstale":"2024-04-25T15:25:30.300011+0000","last_undegraded":"2024-04-25T15:25:30.300011+0000","last_fullsized":"2024-04-25T15:25:30.300011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:44.511850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657661+0000","last_change":"2024-04-25T15:25:29.315008+0000","last_active":"2024-04-25T15:25:30.657661+0000","last_peered":"2024-04-25T15:25:30.657661+0000","last_clean":"2024-04-25T15:25:30.657661+0000","last_became_active":"2024-04-25T15:25:29.314675+0000","last_became_peered":"2024-04-25T15:25:29.314675+0000","last_unstale":"2024-04-25T15:25:30.657661+0000","last_undegraded":"2024-04-25T15:25:30.657661+0000","last_fullsized":"2024-04-25T15:25:30.657661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296694+0000","last_change":"2024-04-25T15:25:29.301255+0000","last_active":"2024-04-25T15:25:30.296694+0000","last_peered":"2024-04-25T15:25:30.296694+0000","last_clean":"2024-04-25T15:25:30.296694+0000","last_became_active":"2024-04-25T15:25:29.300877+0000","last_became_peered":"2024-04-25T15:25:29.300877+0000","last_unstale":"2024-04-25T15:25:30.296694+0000","last_undegraded":"2024-04-25T15:25:30.296694+0000","last_fullsized":"2024-04-25T15:25:30.296694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:40.562364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293711+0000","last_change":"2024-04-25T15:25:29.300064+0000","last_active":"2024-04-25T15:25:30.293711+0000","last_peered":"2024-04-25T15:25:30.293711+0000","last_clean":"2024-04-25T15:25:30.293711+0000","last_became_active":"2024-04-25T15:25:29.299759+0000","last_became_peered":"2024-04-25T15:25:29.299759+0000","last_unstale":"2024-04-25T15:25:30.293711+0000","last_undegraded":"2024-04-25T15:25:30.293711+0000","last_fullsized":"2024-04-25T15:25:30.293711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657884+0000","last_change":"2024-04-25T15:25:29.314325+0000","last_active":"2024-04-25T15:25:30.657884+0000","last_peered":"2024-04-25T15:25:30.657884+0000","last_clean":"2024-04-25T15:25:30.657884+0000","last_became_active":"2024-04-25T15:25:29.311827+0000","last_became_peered":"2024-04-25T15:25:29.311827+0000","last_unstale":"2024-04-25T15:25:30.657884+0000","last_undegraded":"2024-04-25T15:25:30.657884+0000","last_fullsized":"2024-04-25T15:25:30.657884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658676+0000","last_change":"2024-04-25T15:25:29.327856+0000","last_active":"2024-04-25T15:25:30.658676+0000","last_peered":"2024-04-25T15:25:30.658676+0000","last_clean":"2024-04-25T15:25:30.658676+0000","last_became_active":"2024-04-25T15:25:29.327408+0000","last_became_peered":"2024-04-25T15:25:29.327408+0000","last_unstale":"2024-04-25T15:25:30.658676+0000","last_undegraded":"2024-04-25T15:25:30.658676+0000","last_fullsized":"2024-04-25T15:25:30.658676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657624+0000","last_change":"2024-04-25T15:25:29.313009+0000","last_active":"2024-04-25T15:25:30.657624+0000","last_peered":"2024-04-25T15:25:30.657624+0000","last_clean":"2024-04-25T15:25:30.657624+0000","last_became_active":"2024-04-25T15:25:29.312684+0000","last_became_peered":"2024-04-25T15:25:29.312684+0000","last_unstale":"2024-04-25T15:25:30.657624+0000","last_undegraded":"2024-04-25T15:25:30.657624+0000","last_fullsized":"2024-04-25T15:25:30.657624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:26.944400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658764+0000","last_change":"2024-04-25T15:25:29.319894+0000","last_active":"2024-04-25T15:25:30.658764+0000","last_peered":"2024-04-25T15:25:30.658764+0000","last_clean":"2024-04-25T15:25:30.658764+0000","last_became_active":"2024-04-25T15:25:29.319566+0000","last_became_peered":"2024-04-25T15:25:29.319566+0000","last_unstale":"2024-04-25T15:25:30.658764+0000","last_undegraded":"2024-04-25T15:25:30.658764+0000","last_fullsized":"2024-04-25T15:25:30.658764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313203+0000","last_change":"2024-04-25T15:25:29.312046+0000","last_active":"2024-04-25T15:25:30.313203+0000","last_peered":"2024-04-25T15:25:30.313203+0000","last_clean":"2024-04-25T15:25:30.313203+0000","last_became_active":"2024-04-25T15:25:29.310121+0000","last_became_peered":"2024-04-25T15:25:29.310121+0000","last_unstale":"2024-04-25T15:25:30.313203+0000","last_undegraded":"2024-04-25T15:25:30.313203+0000","last_fullsized":"2024-04-25T15:25:30.313203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296808+0000","last_change":"2024-04-25T15:25:29.317661+0000","last_active":"2024-04-25T15:25:30.296808+0000","last_peered":"2024-04-25T15:25:30.296808+0000","last_clean":"2024-04-25T15:25:30.296808+0000","last_became_active":"2024-04-25T15:25:29.317302+0000","last_became_peered":"2024-04-25T15:25:29.317302+0000","last_unstale":"2024-04-25T15:25:30.296808+0000","last_undegraded":"2024-04-25T15:25:30.296808+0000","last_fullsized":"2024-04-25T15:25:30.296808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658934+0000","last_change":"2024-04-25T15:25:29.327012+0000","last_active":"2024-04-25T15:25:30.658934+0000","last_peered":"2024-04-25T15:25:30.658934+0000","last_clean":"2024-04-25T15:25:30.658934+0000","last_became_active":"2024-04-25T15:25:29.326589+0000","last_became_peered":"2024-04-25T15:25:29.326589+0000","last_unstale":"2024-04-25T15:25:30.658934+0000","last_undegraded":"2024-04-25T15:25:30.658934+0000","last_fullsized":"2024-04-25T15:25:30.658934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657788+0000","last_change":"2024-04-25T15:25:29.321180+0000","last_active":"2024-04-25T15:25:30.657788+0000","last_peered":"2024-04-25T15:25:30.657788+0000","last_clean":"2024-04-25T15:25:30.657788+0000","last_became_active":"2024-04-25T15:25:29.320679+0000","last_became_peered":"2024-04-25T15:25:29.320679+0000","last_unstale":"2024-04-25T15:25:30.657788+0000","last_undegraded":"2024-04-25T15:25:30.657788+0000","last_fullsized":"2024-04-25T15:25:30.657788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658226+0000","last_change":"2024-04-25T15:25:29.311398+0000","last_active":"2024-04-25T15:25:30.658226+0000","last_peered":"2024-04-25T15:25:30.658226+0000","last_clean":"2024-04-25T15:25:30.658226+0000","last_became_active":"2024-04-25T15:25:29.311069+0000","last_became_peered":"2024-04-25T15:25:29.311069+0000","last_unstale":"2024-04-25T15:25:30.658226+0000","last_undegraded":"2024-04-25T15:25:30.658226+0000","last_fullsized":"2024-04-25T15:25:30.658226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658383+0000","last_change":"2024-04-25T15:25:29.320926+0000","last_active":"2024-04-25T15:25:30.658383+0000","last_peered":"2024-04-25T15:25:30.658383+0000","last_clean":"2024-04-25T15:25:30.658383+0000","last_became_active":"2024-04-25T15:25:29.320574+0000","last_became_peered":"2024-04-25T15:25:29.320574+0000","last_unstale":"2024-04-25T15:25:30.658383+0000","last_undegraded":"2024-04-25T15:25:30.658383+0000","last_fullsized":"2024-04-25T15:25:30.658383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313448+0000","last_change":"2024-04-25T15:25:29.330048+0000","last_active":"2024-04-25T15:25:30.313448+0000","last_peered":"2024-04-25T15:25:30.313448+0000","last_clean":"2024-04-25T15:25:30.313448+0000","last_became_active":"2024-04-25T15:25:29.329720+0000","last_became_peered":"2024-04-25T15:25:29.329720+0000","last_unstale":"2024-04-25T15:25:30.313448+0000","last_undegraded":"2024-04-25T15:25:30.313448+0000","last_fullsized":"2024-04-25T15:25:30.313448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293796+0000","last_change":"2024-04-25T15:25:29.309985+0000","last_active":"2024-04-25T15:25:30.293796+0000","last_peered":"2024-04-25T15:25:30.293796+0000","last_clean":"2024-04-25T15:25:30.293796+0000","last_became_active":"2024-04-25T15:25:29.309619+0000","last_became_peered":"2024-04-25T15:25:29.309619+0000","last_unstale":"2024-04-25T15:25:30.293796+0000","last_undegraded":"2024-04-25T15:25:30.293796+0000","last_fullsized":"2024-04-25T15:25:30.293796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658375+0000","last_change":"2024-04-25T15:25:29.321037+0000","last_active":"2024-04-25T15:25:30.658375+0000","last_peered":"2024-04-25T15:25:30.658375+0000","last_clean":"2024-04-25T15:25:30.658375+0000","last_became_active":"2024-04-25T15:25:29.320680+0000","last_became_peered":"2024-04-25T15:25:29.320680+0000","last_unstale":"2024-04-25T15:25:30.658375+0000","last_undegraded":"2024-04-25T15:25:30.658375+0000","last_fullsized":"2024-04-25T15:25:30.658375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296855+0000","last_change":"2024-04-25T15:25:29.309805+0000","last_active":"2024-04-25T15:25:30.296855+0000","last_peered":"2024-04-25T15:25:30.296855+0000","last_clean":"2024-04-25T15:25:30.296855+0000","last_became_active":"2024-04-25T15:25:29.309408+0000","last_became_peered":"2024-04-25T15:25:29.309408+0000","last_unstale":"2024-04-25T15:25:30.296855+0000","last_undegraded":"2024-04-25T15:25:30.296855+0000","last_fullsized":"2024-04-25T15:25:30.296855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659006+0000","last_change":"2024-04-25T15:25:29.318254+0000","last_active":"2024-04-25T15:25:30.659006+0000","last_peered":"2024-04-25T15:25:30.659006+0000","last_clean":"2024-04-25T15:25:30.659006+0000","last_became_active":"2024-04-25T15:25:29.317944+0000","last_became_peered":"2024-04-25T15:25:29.317944+0000","last_unstale":"2024-04-25T15:25:30.659006+0000","last_undegraded":"2024-04-25T15:25:30.659006+0000","last_fullsized":"2024-04-25T15:25:30.659006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657620+0000","last_change":"2024-04-25T15:25:29.313901+0000","last_active":"2024-04-25T15:25:30.657620+0000","last_peered":"2024-04-25T15:25:30.657620+0000","last_clean":"2024-04-25T15:25:30.657620+0000","last_became_active":"2024-04-25T15:25:29.313526+0000","last_became_peered":"2024-04-25T15:25:29.313526+0000","last_unstale":"2024-04-25T15:25:30.657620+0000","last_undegraded":"2024-04-25T15:25:30.657620+0000","last_fullsized":"2024-04-25T15:25:30.657620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313041+0000","last_change":"2024-04-25T15:25:29.312724+0000","last_active":"2024-04-25T15:25:30.313041+0000","last_peered":"2024-04-25T15:25:30.313041+0000","last_clean":"2024-04-25T15:25:30.313041+0000","last_became_active":"2024-04-25T15:25:29.312425+0000","last_became_peered":"2024-04-25T15:25:29.312425+0000","last_unstale":"2024-04-25T15:25:30.313041+0000","last_undegraded":"2024-04-25T15:25:30.313041+0000","last_fullsized":"2024-04-25T15:25:30.313041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299747+0000","last_change":"2024-04-25T15:25:29.298478+0000","last_active":"2024-04-25T15:25:30.299747+0000","last_peered":"2024-04-25T15:25:30.299747+0000","last_clean":"2024-04-25T15:25:30.299747+0000","last_became_active":"2024-04-25T15:25:29.298178+0000","last_became_peered":"2024-04-25T15:25:29.298178+0000","last_unstale":"2024-04-25T15:25:30.299747+0000","last_undegraded":"2024-04-25T15:25:30.299747+0000","last_fullsized":"2024-04-25T15:25:30.299747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:53.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658039+0000","last_change":"2024-04-25T15:25:28.301149+0000","last_active":"2024-04-25T15:25:30.658039+0000","last_peered":"2024-04-25T15:25:30.658039+0000","last_clean":"2024-04-25T15:25:30.658039+0000","last_became_active":"2024-04-25T15:25:28.300804+0000","last_became_peered":"2024-04-25T15:25:28.300804+0000","last_unstale":"2024-04-25T15:25:30.658039+0000","last_undegraded":"2024-04-25T15:25:30.658039+0000","last_fullsized":"2024-04-25T15:25:30.658039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296993+0000","last_change":"2024-04-25T15:25:29.306337+0000","last_active":"2024-04-25T15:25:30.296993+0000","last_peered":"2024-04-25T15:25:30.296993+0000","last_clean":"2024-04-25T15:25:30.296993+0000","last_became_active":"2024-04-25T15:25:29.306039+0000","last_became_peered":"2024-04-25T15:25:29.306039+0000","last_unstale":"2024-04-25T15:25:30.296993+0000","last_undegraded":"2024-04-25T15:25:30.296993+0000","last_fullsized":"2024-04-25T15:25:30.296993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313038+0000","last_change":"2024-04-25T15:25:28.301076+0000","last_active":"2024-04-25T15:25:30.313038+0000","last_peered":"2024-04-25T15:25:30.313038+0000","last_clean":"2024-04-25T15:25:30.313038+0000","last_became_active":"2024-04-25T15:25:28.300670+0000","last_became_peered":"2024-04-25T15:25:28.300670+0000","last_unstale":"2024-04-25T15:25:30.313038+0000","last_undegraded":"2024-04-25T15:25:30.313038+0000","last_fullsized":"2024-04-25T15:25:30.313038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657997+0000","last_change":"2024-04-25T15:25:28.305288+0000","last_active":"2024-04-25T15:25:30.657997+0000","last_peered":"2024-04-25T15:25:30.657997+0000","last_clean":"2024-04-25T15:25:30.657997+0000","last_became_active":"2024-04-25T15:25:28.304900+0000","last_became_peered":"2024-04-25T15:25:28.304900+0000","last_unstale":"2024-04-25T15:25:30.657997+0000","last_undegraded":"2024-04-25T15:25:30.657997+0000","last_fullsized":"2024-04-25T15:25:30.657997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:04.261360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658085+0000","last_change":"2024-04-25T15:25:29.310703+0000","last_active":"2024-04-25T15:25:30.658085+0000","last_peered":"2024-04-25T15:25:30.658085+0000","last_clean":"2024-04-25T15:25:30.658085+0000","last_became_active":"2024-04-25T15:25:29.309942+0000","last_became_peered":"2024-04-25T15:25:29.309942+0000","last_unstale":"2024-04-25T15:25:30.658085+0000","last_undegraded":"2024-04-25T15:25:30.658085+0000","last_fullsized":"2024-04-25T15:25:30.658085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658034+0000","last_change":"2024-04-25T15:25:28.292273+0000","last_active":"2024-04-25T15:25:30.658034+0000","last_peered":"2024-04-25T15:25:30.658034+0000","last_clean":"2024-04-25T15:25:30.658034+0000","last_became_active":"2024-04-25T15:25:28.291890+0000","last_became_peered":"2024-04-25T15:25:28.291890+0000","last_unstale":"2024-04-25T15:25:30.658034+0000","last_undegraded":"2024-04-25T15:25:30.658034+0000","last_fullsized":"2024-04-25T15:25:30.658034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293637+0000","last_change":"2024-04-25T15:25:29.310761+0000","last_active":"2024-04-25T15:25:30.293637+0000","last_peered":"2024-04-25T15:25:30.293637+0000","last_clean":"2024-04-25T15:25:30.293637+0000","last_became_active":"2024-04-25T15:25:29.310403+0000","last_became_peered":"2024-04-25T15:25:29.310403+0000","last_unstale":"2024-04-25T15:25:30.293637+0000","last_undegraded":"2024-04-25T15:25:30.293637+0000","last_fullsized":"2024-04-25T15:25:30.293637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313366+0000","last_change":"2024-04-25T15:25:28.306210+0000","last_active":"2024-04-25T15:25:30.313366+0000","last_peered":"2024-04-25T15:25:30.313366+0000","last_clean":"2024-04-25T15:25:30.313366+0000","last_became_active":"2024-04-25T15:25:28.305931+0000","last_became_peered":"2024-04-25T15:25:28.305931+0000","last_unstale":"2024-04-25T15:25:30.313366+0000","last_undegraded":"2024-04-25T15:25:30.313366+0000","last_fullsized":"2024-04-25T15:25:30.313366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658132+0000","last_change":"2024-04-25T15:25:29.328500+0000","last_active":"2024-04-25T15:25:30.658132+0000","last_peered":"2024-04-25T15:25:30.658132+0000","last_clean":"2024-04-25T15:25:30.658132+0000","last_became_active":"2024-04-25T15:25:29.328150+0000","last_became_peered":"2024-04-25T15:25:29.328150+0000","last_unstale":"2024-04-25T15:25:30.658132+0000","last_undegraded":"2024-04-25T15:25:30.658132+0000","last_fullsized":"2024-04-25T15:25:30.658132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658259+0000","last_change":"2024-04-25T15:25:28.305011+0000","last_active":"2024-04-25T15:25:30.658259+0000","last_peered":"2024-04-25T15:25:30.658259+0000","last_clean":"2024-04-25T15:25:30.658259+0000","last_became_active":"2024-04-25T15:25:28.304682+0000","last_became_peered":"2024-04-25T15:25:28.304682+0000","last_unstale":"2024-04-25T15:25:30.658259+0000","last_undegraded":"2024-04-25T15:25:30.658259+0000","last_fullsized":"2024-04-25T15:25:30.658259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297172+0000","last_change":"2024-04-25T15:25:29.318333+0000","last_active":"2024-04-25T15:25:30.297172+0000","last_peered":"2024-04-25T15:25:30.297172+0000","last_clean":"2024-04-25T15:25:30.297172+0000","last_became_active":"2024-04-25T15:25:29.317557+0000","last_became_peered":"2024-04-25T15:25:29.317557+0000","last_unstale":"2024-04-25T15:25:30.297172+0000","last_undegraded":"2024-04-25T15:25:30.297172+0000","last_fullsized":"2024-04-25T15:25:30.297172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293936+0000","last_change":"2024-04-25T15:25:28.297270+0000","last_active":"2024-04-25T15:25:30.293936+0000","last_peered":"2024-04-25T15:25:30.293936+0000","last_clean":"2024-04-25T15:25:30.293936+0000","last_became_active":"2024-04-25T15:25:28.296865+0000","last_became_peered":"2024-04-25T15:25:28.296865+0000","last_unstale":"2024-04-25T15:25:30.293936+0000","last_undegraded":"2024-04-25T15:25:30.293936+0000","last_fullsized":"2024-04-25T15:25:30.293936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300012+0000","last_change":"2024-04-25T15:25:29.324749+0000","last_active":"2024-04-25T15:25:30.300012+0000","last_peered":"2024-04-25T15:25:30.300012+0000","last_clean":"2024-04-25T15:25:30.300012+0000","last_became_active":"2024-04-25T15:25:29.324454+0000","last_became_peered":"2024-04-25T15:25:29.324454+0000","last_unstale":"2024-04-25T15:25:30.300012+0000","last_undegraded":"2024-04-25T15:25:30.300012+0000","last_fullsized":"2024-04-25T15:25:30.300012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313205+0000","last_change":"2024-04-25T15:25:28.302276+0000","last_active":"2024-04-25T15:25:30.313205+0000","last_peered":"2024-04-25T15:25:30.313205+0000","last_clean":"2024-04-25T15:25:30.313205+0000","last_became_active":"2024-04-25T15:25:28.302039+0000","last_became_peered":"2024-04-25T15:25:28.302039+0000","last_unstale":"2024-04-25T15:25:30.313205+0000","last_undegraded":"2024-04-25T15:25:30.313205+0000","last_fullsized":"2024-04-25T15:25:30.313205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658280+0000","last_change":"2024-04-25T15:25:29.324177+0000","last_active":"2024-04-25T15:25:30.658280+0000","last_peered":"2024-04-25T15:25:30.658280+0000","last_clean":"2024-04-25T15:25:30.658280+0000","last_became_active":"2024-04-25T15:25:29.323727+0000","last_became_peered":"2024-04-25T15:25:29.323727+0000","last_unstale":"2024-04-25T15:25:30.658280+0000","last_undegraded":"2024-04-25T15:25:30.658280+0000","last_fullsized":"2024-04-25T15:25:30.658280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296971+0000","last_change":"2024-04-25T15:25:28.288837+0000","last_active":"2024-04-25T15:25:30.296971+0000","last_peered":"2024-04-25T15:25:30.296971+0000","last_clean":"2024-04-25T15:25:30.296971+0000","last_became_active":"2024-04-25T15:25:28.288485+0000","last_became_peered":"2024-04-25T15:25:28.288485+0000","last_unstale":"2024-04-25T15:25:30.296971+0000","last_undegraded":"2024-04-25T15:25:30.296971+0000","last_fullsized":"2024-04-25T15:25:30.296971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:58.179326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296812+0000","last_change":"2024-04-25T15:25:29.323297+0000","last_active":"2024-04-25T15:25:30.296812+0000","last_peered":"2024-04-25T15:25:30.296812+0000","last_clean":"2024-04-25T15:25:30.296812+0000","last_became_active":"2024-04-25T15:25:29.322955+0000","last_became_peered":"2024-04-25T15:25:29.322955+0000","last_unstale":"2024-04-25T15:25:30.296812+0000","last_undegraded":"2024-04-25T15:25:30.296812+0000","last_fullsized":"2024-04-25T15:25:30.296812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659064+0000","last_change":"2024-04-25T15:25:28.300177+0000","last_active":"2024-04-25T15:25:30.659064+0000","last_peered":"2024-04-25T15:25:30.659064+0000","last_clean":"2024-04-25T15:25:30.659064+0000","last_became_active":"2024-04-25T15:25:28.298626+0000","last_became_peered":"2024-04-25T15:25:28.298626+0000","last_unstale":"2024-04-25T15:25:30.659064+0000","last_undegraded":"2024-04-25T15:25:30.659064+0000","last_fullsized":"2024-04-25T15:25:30.659064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657985+0000","last_change":"2024-04-25T15:25:29.316614+0000","last_active":"2024-04-25T15:25:30.657985+0000","last_peered":"2024-04-25T15:25:30.657985+0000","last_clean":"2024-04-25T15:25:30.657985+0000","last_became_active":"2024-04-25T15:25:29.316293+0000","last_became_peered":"2024-04-25T15:25:29.316293+0000","last_unstale":"2024-04-25T15:25:30.657985+0000","last_undegraded":"2024-04-25T15:25:30.657985+0000","last_fullsized":"2024-04-25T15:25:30.657985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312968+0000","last_change":"2024-04-25T15:25:28.304313+0000","last_active":"2024-04-25T15:25:30.312968+0000","last_peered":"2024-04-25T15:25:30.312968+0000","last_clean":"2024-04-25T15:25:30.312968+0000","last_became_active":"2024-04-25T15:25:28.303943+0000","last_became_peered":"2024-04-25T15:25:28.303943+0000","last_unstale":"2024-04-25T15:25:30.312968+0000","last_undegraded":"2024-04-25T15:25:30.312968+0000","last_fullsized":"2024-04-25T15:25:30.312968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313012+0000","last_change":"2024-04-25T15:25:29.310260+0000","last_active":"2024-04-25T15:25:30.313012+0000","last_peered":"2024-04-25T15:25:30.313012+0000","last_clean":"2024-04-25T15:25:30.313012+0000","last_became_active":"2024-04-25T15:25:29.309916+0000","last_became_peered":"2024-04-25T15:25:29.309916+0000","last_unstale":"2024-04-25T15:25:30.313012+0000","last_undegraded":"2024-04-25T15:25:30.313012+0000","last_fullsized":"2024-04-25T15:25:30.313012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657409+0000","last_change":"2024-04-25T15:25:28.293959+0000","last_active":"2024-04-25T15:25:30.657409+0000","last_peered":"2024-04-25T15:25:30.657409+0000","last_clean":"2024-04-25T15:25:30.657409+0000","last_became_active":"2024-04-25T15:25:28.293594+0000","last_became_peered":"2024-04-25T15:25:28.293594+0000","last_unstale":"2024-04-25T15:25:30.657409+0000","last_undegraded":"2024-04-25T15:25:30.657409+0000","last_fullsized":"2024-04-25T15:25:30.657409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658071+0000","last_change":"2024-04-25T15:25:29.318796+0000","last_active":"2024-04-25T15:25:30.658071+0000","last_peered":"2024-04-25T15:25:30.658071+0000","last_clean":"2024-04-25T15:25:30.658071+0000","last_became_active":"2024-04-25T15:25:29.318218+0000","last_became_peered":"2024-04-25T15:25:29.318218+0000","last_unstale":"2024-04-25T15:25:30.658071+0000","last_undegraded":"2024-04-25T15:25:30.658071+0000","last_fullsized":"2024-04-25T15:25:30.658071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300098+0000","last_change":"2024-04-25T15:25:28.293813+0000","last_active":"2024-04-25T15:25:30.300098+0000","last_peered":"2024-04-25T15:25:30.300098+0000","last_clean":"2024-04-25T15:25:30.300098+0000","last_became_active":"2024-04-25T15:25:28.293498+0000","last_became_peered":"2024-04-25T15:25:28.293498+0000","last_unstale":"2024-04-25T15:25:30.300098+0000","last_undegraded":"2024-04-25T15:25:30.300098+0000","last_fullsized":"2024-04-25T15:25:30.300098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:31:00.192290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658957+0000","last_change":"2024-04-25T15:25:29.322896+0000","last_active":"2024-04-25T15:25:30.658957+0000","last_peered":"2024-04-25T15:25:30.658957+0000","last_clean":"2024-04-25T15:25:30.658957+0000","last_became_active":"2024-04-25T15:25:29.321170+0000","last_became_peered":"2024-04-25T15:25:29.321170+0000","last_unstale":"2024-04-25T15:25:30.658957+0000","last_undegraded":"2024-04-25T15:25:30.658957+0000","last_fullsized":"2024-04-25T15:25:30.658957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657752+0000","last_change":"2024-04-25T15:25:28.294831+0000","last_active":"2024-04-25T15:25:30.657752+0000","last_peered":"2024-04-25T15:25:30.657752+0000","last_clean":"2024-04-25T15:25:30.657752+0000","last_became_active":"2024-04-25T15:25:28.294257+0000","last_became_peered":"2024-04-25T15:25:28.294257+0000","last_unstale":"2024-04-25T15:25:30.657752+0000","last_undegraded":"2024-04-25T15:25:30.657752+0000","last_fullsized":"2024-04-25T15:25:30.657752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659232+0000","last_change":"2024-04-25T15:25:29.325425+0000","last_active":"2024-04-25T15:25:30.659232+0000","last_peered":"2024-04-25T15:25:30.659232+0000","last_clean":"2024-04-25T15:25:30.659232+0000","last_became_active":"2024-04-25T15:25:29.324916+0000","last_became_peered":"2024-04-25T15:25:29.324916+0000","last_unstale":"2024-04-25T15:25:30.659232+0000","last_undegraded":"2024-04-25T15:25:30.659232+0000","last_fullsized":"2024-04-25T15:25:30.659232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:46.643371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.294024+0000","last_change":"2024-04-25T15:25:28.297384+0000","last_active":"2024-04-25T15:25:30.294024+0000","last_peered":"2024-04-25T15:25:30.294024+0000","last_clean":"2024-04-25T15:25:30.294024+0000","last_became_active":"2024-04-25T15:25:28.297035+0000","last_became_peered":"2024-04-25T15:25:28.297035+0000","last_unstale":"2024-04-25T15:25:30.294024+0000","last_undegraded":"2024-04-25T15:25:30.294024+0000","last_fullsized":"2024-04-25T15:25:30.294024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658488+0000","last_change":"2024-04-25T15:25:29.323269+0000","last_active":"2024-04-25T15:25:30.658488+0000","last_peered":"2024-04-25T15:25:30.658488+0000","last_clean":"2024-04-25T15:25:30.658488+0000","last_became_active":"2024-04-25T15:25:29.322481+0000","last_became_peered":"2024-04-25T15:25:29.322481+0000","last_unstale":"2024-04-25T15:25:30.658488+0000","last_undegraded":"2024-04-25T15:25:30.658488+0000","last_fullsized":"2024-04-25T15:25:30.658488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657594+0000","last_change":"2024-04-25T15:25:28.293938+0000","last_active":"2024-04-25T15:25:30.657594+0000","last_peered":"2024-04-25T15:25:30.657594+0000","last_clean":"2024-04-25T15:25:30.657594+0000","last_became_active":"2024-04-25T15:25:28.293520+0000","last_became_peered":"2024-04-25T15:25:28.293520+0000","last_unstale":"2024-04-25T15:25:30.657594+0000","last_undegraded":"2024-04-25T15:25:30.657594+0000","last_fullsized":"2024-04-25T15:25:30.657594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658465+0000","last_change":"2024-04-25T15:25:29.325130+0000","last_active":"2024-04-25T15:25:30.658465+0000","last_peered":"2024-04-25T15:25:30.658465+0000","last_clean":"2024-04-25T15:25:30.658465+0000","last_became_active":"2024-04-25T15:25:29.324783+0000","last_became_peered":"2024-04-25T15:25:29.324783+0000","last_unstale":"2024-04-25T15:25:30.658465+0000","last_undegraded":"2024-04-25T15:25:30.658465+0000","last_fullsized":"2024-04-25T15:25:30.658465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657696+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:25:30.657696+0000","last_peered":"2024-04-25T15:25:30.657696+0000","last_clean":"2024-04-25T15:25:30.657696+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:25:30.657696+0000","last_undegraded":"2024-04-25T15:25:30.657696+0000","last_fullsized":"2024-04-25T15:25:30.657696+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657863+0000","last_change":"2024-04-25T15:25:28.293187+0000","last_active":"2024-04-25T15:25:30.657863+0000","last_peered":"2024-04-25T15:25:30.657863+0000","last_clean":"2024-04-25T15:25:30.657863+0000","last_became_active":"2024-04-25T15:25:28.292748+0000","last_became_peered":"2024-04-25T15:25:28.292748+0000","last_unstale":"2024-04-25T15:25:30.657863+0000","last_undegraded":"2024-04-25T15:25:30.657863+0000","last_fullsized":"2024-04-25T15:25:30.657863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657964+0000","last_change":"2024-04-25T15:25:29.323272+0000","last_active":"2024-04-25T15:25:30.657964+0000","last_peered":"2024-04-25T15:25:30.657964+0000","last_clean":"2024-04-25T15:25:30.657964+0000","last_became_active":"2024-04-25T15:25:29.322937+0000","last_became_peered":"2024-04-25T15:25:29.322937+0000","last_unstale":"2024-04-25T15:25:30.657964+0000","last_undegraded":"2024-04-25T15:25:30.657964+0000","last_fullsized":"2024-04-25T15:25:30.657964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657646+0000","last_change":"2024-04-25T15:25:28.301819+0000","last_active":"2024-04-25T15:25:30.657646+0000","last_peered":"2024-04-25T15:25:30.657646+0000","last_clean":"2024-04-25T15:25:30.657646+0000","last_became_active":"2024-04-25T15:25:28.301519+0000","last_became_peered":"2024-04-25T15:25:28.301519+0000","last_unstale":"2024-04-25T15:25:30.657646+0000","last_undegraded":"2024-04-25T15:25:30.657646+0000","last_fullsized":"2024-04-25T15:25:30.657646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296763+0000","last_change":"2024-04-25T15:25:29.306397+0000","last_active":"2024-04-25T15:25:30.296763+0000","last_peered":"2024-04-25T15:25:30.296763+0000","last_clean":"2024-04-25T15:25:30.296763+0000","last_became_active":"2024-04-25T15:25:29.306088+0000","last_became_peered":"2024-04-25T15:25:29.306088+0000","last_unstale":"2024-04-25T15:25:30.296763+0000","last_undegraded":"2024-04-25T15:25:30.296763+0000","last_fullsized":"2024-04-25T15:25:30.296763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:05.665096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657506+0000","last_change":"2024-04-25T15:25:28.295278+0000","last_active":"2024-04-25T15:25:30.657506+0000","last_peered":"2024-04-25T15:25:30.657506+0000","last_clean":"2024-04-25T15:25:30.657506+0000","last_became_active":"2024-04-25T15:25:28.294876+0000","last_became_peered":"2024-04-25T15:25:28.294876+0000","last_unstale":"2024-04-25T15:25:30.657506+0000","last_undegraded":"2024-04-25T15:25:30.657506+0000","last_fullsized":"2024-04-25T15:25:30.657506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293669+0000","last_change":"2024-04-25T15:25:29.307860+0000","last_active":"2024-04-25T15:25:30.293669+0000","last_peered":"2024-04-25T15:25:30.293669+0000","last_clean":"2024-04-25T15:25:30.293669+0000","last_became_active":"2024-04-25T15:25:29.307525+0000","last_became_peered":"2024-04-25T15:25:29.307525+0000","last_unstale":"2024-04-25T15:25:30.293669+0000","last_undegraded":"2024-04-25T15:25:30.293669+0000","last_fullsized":"2024-04-25T15:25:30.293669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659149+0000","last_change":"2024-04-25T15:25:28.298243+0000","last_active":"2024-04-25T15:25:30.659149+0000","last_peered":"2024-04-25T15:25:30.659149+0000","last_clean":"2024-04-25T15:25:30.659149+0000","last_became_active":"2024-04-25T15:25:28.297837+0000","last_became_peered":"2024-04-25T15:25:28.297837+0000","last_unstale":"2024-04-25T15:25:30.659149+0000","last_undegraded":"2024-04-25T15:25:30.659149+0000","last_fullsized":"2024-04-25T15:25:30.659149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299949+0000","last_change":"2024-04-25T15:25:29.308455+0000","last_active":"2024-04-25T15:25:30.299949+0000","last_peered":"2024-04-25T15:25:30.299949+0000","last_clean":"2024-04-25T15:25:30.299949+0000","last_became_active":"2024-04-25T15:25:29.307968+0000","last_became_peered":"2024-04-25T15:25:29.307968+0000","last_unstale":"2024-04-25T15:25:30.299949+0000","last_undegraded":"2024-04-25T15:25:30.299949+0000","last_fullsized":"2024-04-25T15:25:30.299949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:53.177957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293919+0000","last_change":"2024-04-25T15:25:28.298284+0000","last_active":"2024-04-25T15:25:30.293919+0000","last_peered":"2024-04-25T15:25:30.293919+0000","last_clean":"2024-04-25T15:25:30.293919+0000","last_became_active":"2024-04-25T15:25:28.297792+0000","last_became_peered":"2024-04-25T15:25:28.297792+0000","last_unstale":"2024-04-25T15:25:30.293919+0000","last_undegraded":"2024-04-25T15:25:30.293919+0000","last_fullsized":"2024-04-25T15:25:30.293919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658750+0000","last_change":"2024-04-25T15:25:29.313891+0000","last_active":"2024-04-25T15:25:30.658750+0000","last_peered":"2024-04-25T15:25:30.658750+0000","last_clean":"2024-04-25T15:25:30.658750+0000","last_became_active":"2024-04-25T15:25:29.313488+0000","last_became_peered":"2024-04-25T15:25:29.313488+0000","last_unstale":"2024-04-25T15:25:30.658750+0000","last_undegraded":"2024-04-25T15:25:30.658750+0000","last_fullsized":"2024-04-25T15:25:30.658750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658847+0000","last_change":"2024-04-25T15:25:28.301563+0000","last_active":"2024-04-25T15:25:30.658847+0000","last_peered":"2024-04-25T15:25:30.658847+0000","last_clean":"2024-04-25T15:25:30.658847+0000","last_became_active":"2024-04-25T15:25:28.301186+0000","last_became_peered":"2024-04-25T15:25:28.301186+0000","last_unstale":"2024-04-25T15:25:30.658847+0000","last_undegraded":"2024-04-25T15:25:30.658847+0000","last_fullsized":"2024-04-25T15:25:30.658847+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:25.937421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293726+0000","last_change":"2024-04-25T15:25:29.308391+0000","last_active":"2024-04-25T15:25:30.293726+0000","last_peered":"2024-04-25T15:25:30.293726+0000","last_clean":"2024-04-25T15:25:30.293726+0000","last_became_active":"2024-04-25T15:25:29.308010+0000","last_became_peered":"2024-04-25T15:25:29.308010+0000","last_unstale":"2024-04-25T15:25:30.293726+0000","last_undegraded":"2024-04-25T15:25:30.293726+0000","last_fullsized":"2024-04-25T15:25:30.293726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312496+0000","last_change":"2024-04-25T15:25:28.304872+0000","last_active":"2024-04-25T15:25:30.312496+0000","last_peered":"2024-04-25T15:25:30.312496+0000","last_clean":"2024-04-25T15:25:30.312496+0000","last_became_active":"2024-04-25T15:25:28.304443+0000","last_became_peered":"2024-04-25T15:25:28.304443+0000","last_unstale":"2024-04-25T15:25:30.312496+0000","last_undegraded":"2024-04-25T15:25:30.312496+0000","last_fullsized":"2024-04-25T15:25:30.312496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658447+0000","last_change":"2024-04-25T15:25:29.326344+0000","last_active":"2024-04-25T15:25:30.658447+0000","last_peered":"2024-04-25T15:25:30.658447+0000","last_clean":"2024-04-25T15:25:30.658447+0000","last_became_active":"2024-04-25T15:25:29.326016+0000","last_became_peered":"2024-04-25T15:25:29.326016+0000","last_unstale":"2024-04-25T15:25:30.658447+0000","last_undegraded":"2024-04-25T15:25:30.658447+0000","last_fullsized":"2024-04-25T15:25:30.658447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313461+0000","last_change":"2024-04-25T15:25:28.307284+0000","last_active":"2024-04-25T15:25:30.313461+0000","last_peered":"2024-04-25T15:25:30.313461+0000","last_clean":"2024-04-25T15:25:30.313461+0000","last_became_active":"2024-04-25T15:25:28.307013+0000","last_became_peered":"2024-04-25T15:25:28.307013+0000","last_unstale":"2024-04-25T15:25:30.313461+0000","last_undegraded":"2024-04-25T15:25:30.313461+0000","last_fullsized":"2024-04-25T15:25:30.313461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297033+0000","last_change":"2024-04-25T15:25:29.301384+0000","last_active":"2024-04-25T15:25:30.297033+0000","last_peered":"2024-04-25T15:25:30.297033+0000","last_clean":"2024-04-25T15:25:30.297033+0000","last_became_active":"2024-04-25T15:25:29.301021+0000","last_became_peered":"2024-04-25T15:25:29.301021+0000","last_unstale":"2024-04-25T15:25:30.297033+0000","last_undegraded":"2024-04-25T15:25:30.297033+0000","last_fullsized":"2024-04-25T15:25:30.297033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657611+0000","last_change":"2024-04-25T15:25:28.304086+0000","last_active":"2024-04-25T15:25:30.657611+0000","last_peered":"2024-04-25T15:25:30.657611+0000","last_clean":"2024-04-25T15:25:30.657611+0000","last_became_active":"2024-04-25T15:25:28.303771+0000","last_became_peered":"2024-04-25T15:25:28.303771+0000","last_unstale":"2024-04-25T15:25:30.657611+0000","last_undegraded":"2024-04-25T15:25:30.657611+0000","last_fullsized":"2024-04-25T15:25:30.657611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299860+0000","last_change":"2024-04-25T15:25:29.310784+0000","last_active":"2024-04-25T15:25:30.299860+0000","last_peered":"2024-04-25T15:25:30.299860+0000","last_clean":"2024-04-25T15:25:30.299860+0000","last_became_active":"2024-04-25T15:25:29.309711+0000","last_became_peered":"2024-04-25T15:25:29.309711+0000","last_unstale":"2024-04-25T15:25:30.299860+0000","last_undegraded":"2024-04-25T15:25:30.299860+0000","last_fullsized":"2024-04-25T15:25:30.299860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:16.888909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293756+0000","last_change":"2024-04-25T15:25:28.300848+0000","last_active":"2024-04-25T15:25:30.293756+0000","last_peered":"2024-04-25T15:25:30.293756+0000","last_clean":"2024-04-25T15:25:30.293756+0000","last_became_active":"2024-04-25T15:25:28.300218+0000","last_became_peered":"2024-04-25T15:25:28.300218+0000","last_unstale":"2024-04-25T15:25:30.293756+0000","last_undegraded":"2024-04-25T15:25:30.293756+0000","last_fullsized":"2024-04-25T15:25:30.293756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658212+0000","last_change":"2024-04-25T15:25:29.313291+0000","last_active":"2024-04-25T15:25:30.658212+0000","last_peered":"2024-04-25T15:25:30.658212+0000","last_clean":"2024-04-25T15:25:30.658212+0000","last_became_active":"2024-04-25T15:25:29.312960+0000","last_became_peered":"2024-04-25T15:25:29.312960+0000","last_unstale":"2024-04-25T15:25:30.658212+0000","last_undegraded":"2024-04-25T15:25:30.658212+0000","last_fullsized":"2024-04-25T15:25:30.658212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293893+0000","last_change":"2024-04-25T15:25:28.301806+0000","last_active":"2024-04-25T15:25:30.293893+0000","last_peered":"2024-04-25T15:25:30.293893+0000","last_clean":"2024-04-25T15:25:30.293893+0000","last_became_active":"2024-04-25T15:25:28.301447+0000","last_became_peered":"2024-04-25T15:25:28.301447+0000","last_unstale":"2024-04-25T15:25:30.293893+0000","last_undegraded":"2024-04-25T15:25:30.293893+0000","last_fullsized":"2024-04-25T15:25:30.293893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657861+0000","last_change":"2024-04-25T15:25:29.310336+0000","last_active":"2024-04-25T15:25:30.657861+0000","last_peered":"2024-04-25T15:25:30.657861+0000","last_clean":"2024-04-25T15:25:30.657861+0000","last_became_active":"2024-04-25T15:25:29.309914+0000","last_became_peered":"2024-04-25T15:25:29.309914+0000","last_unstale":"2024-04-25T15:25:30.657861+0000","last_undegraded":"2024-04-25T15:25:30.657861+0000","last_fullsized":"2024-04-25T15:25:30.657861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658401+0000","last_change":"2024-04-25T15:25:28.300815+0000","last_active":"2024-04-25T15:25:30.658401+0000","last_peered":"2024-04-25T15:25:30.658401+0000","last_clean":"2024-04-25T15:25:30.658401+0000","last_became_active":"2024-04-25T15:25:28.300487+0000","last_became_peered":"2024-04-25T15:25:28.300487+0000","last_unstale":"2024-04-25T15:25:30.658401+0000","last_undegraded":"2024-04-25T15:25:30.658401+0000","last_fullsized":"2024-04-25T15:25:30.658401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313011+0000","last_change":"2024-04-25T15:25:29.310500+0000","last_active":"2024-04-25T15:25:30.313011+0000","last_peered":"2024-04-25T15:25:30.313011+0000","last_clean":"2024-04-25T15:25:30.313011+0000","last_became_active":"2024-04-25T15:25:29.310111+0000","last_became_peered":"2024-04-25T15:25:29.310111+0000","last_unstale":"2024-04-25T15:25:30.313011+0000","last_undegraded":"2024-04-25T15:25:30.313011+0000","last_fullsized":"2024-04-25T15:25:30.313011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658618+0000","last_change":"2024-04-25T15:25:29.314651+0000","last_active":"2024-04-25T15:25:30.658618+0000","last_peered":"2024-04-25T15:25:30.658618+0000","last_clean":"2024-04-25T15:25:30.658618+0000","last_became_active":"2024-04-25T15:25:29.314239+0000","last_became_peered":"2024-04-25T15:25:29.314239+0000","last_unstale":"2024-04-25T15:25:30.658618+0000","last_undegraded":"2024-04-25T15:25:30.658618+0000","last_fullsized":"2024-04-25T15:25:30.658618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658320+0000","last_change":"2024-04-25T15:25:28.304792+0000","last_active":"2024-04-25T15:25:30.658320+0000","last_peered":"2024-04-25T15:25:30.658320+0000","last_clean":"2024-04-25T15:25:30.658320+0000","last_became_active":"2024-04-25T15:25:28.304489+0000","last_became_peered":"2024-04-25T15:25:28.304489+0000","last_unstale":"2024-04-25T15:25:30.658320+0000","last_undegraded":"2024-04-25T15:25:30.658320+0000","last_fullsized":"2024-04-25T15:25:30.658320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658967+0000","last_change":"2024-04-25T15:25:29.318608+0000","last_active":"2024-04-25T15:25:30.658967+0000","last_peered":"2024-04-25T15:25:30.658967+0000","last_clean":"2024-04-25T15:25:30.658967+0000","last_became_active":"2024-04-25T15:25:29.317330+0000","last_became_peered":"2024-04-25T15:25:29.317330+0000","last_unstale":"2024-04-25T15:25:30.658967+0000","last_undegraded":"2024-04-25T15:25:30.658967+0000","last_fullsized":"2024-04-25T15:25:30.658967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658591+0000","last_change":"2024-04-25T15:25:28.305376+0000","last_active":"2024-04-25T15:25:30.658591+0000","last_peered":"2024-04-25T15:25:30.658591+0000","last_clean":"2024-04-25T15:25:30.658591+0000","last_became_active":"2024-04-25T15:25:28.305083+0000","last_became_peered":"2024-04-25T15:25:28.305083+0000","last_unstale":"2024-04-25T15:25:30.658591+0000","last_undegraded":"2024-04-25T15:25:30.658591+0000","last_fullsized":"2024-04-25T15:25:30.658591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297061+0000","last_change":"2024-04-25T15:25:29.319184+0000","last_active":"2024-04-25T15:25:30.297061+0000","last_peered":"2024-04-25T15:25:30.297061+0000","last_clean":"2024-04-25T15:25:30.297061+0000","last_became_active":"2024-04-25T15:25:29.317757+0000","last_became_peered":"2024-04-25T15:25:29.317757+0000","last_unstale":"2024-04-25T15:25:30.297061+0000","last_undegraded":"2024-04-25T15:25:30.297061+0000","last_fullsized":"2024-04-25T15:25:30.297061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658361+0000","last_change":"2024-04-25T15:25:28.298019+0000","last_active":"2024-04-25T15:25:30.658361+0000","last_peered":"2024-04-25T15:25:30.658361+0000","last_clean":"2024-04-25T15:25:30.658361+0000","last_became_active":"2024-04-25T15:25:28.297570+0000","last_became_peered":"2024-04-25T15:25:28.297570+0000","last_unstale":"2024-04-25T15:25:30.658361+0000","last_undegraded":"2024-04-25T15:25:30.658361+0000","last_fullsized":"2024-04-25T15:25:30.658361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658351+0000","last_change":"2024-04-25T15:25:29.316068+0000","last_active":"2024-04-25T15:25:30.658351+0000","last_peered":"2024-04-25T15:25:30.658351+0000","last_clean":"2024-04-25T15:25:30.658351+0000","last_became_active":"2024-04-25T15:25:29.315705+0000","last_became_peered":"2024-04-25T15:25:29.315705+0000","last_unstale":"2024-04-25T15:25:30.658351+0000","last_undegraded":"2024-04-25T15:25:30.658351+0000","last_fullsized":"2024-04-25T15:25:30.658351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297170+0000","last_change":"2024-04-25T15:25:28.300551+0000","last_active":"2024-04-25T15:25:30.297170+0000","last_peered":"2024-04-25T15:25:30.297170+0000","last_clean":"2024-04-25T15:25:30.297170+0000","last_became_active":"2024-04-25T15:25:28.299984+0000","last_became_peered":"2024-04-25T15:25:28.299984+0000","last_unstale":"2024-04-25T15:25:30.297170+0000","last_undegraded":"2024-04-25T15:25:30.297170+0000","last_fullsized":"2024-04-25T15:25:30.297170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:08:02.904549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:25:33.726 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:25:33.727 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T15:25:33.727 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T15:25:33.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T15:25:33.727 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T15:25:33.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.873+0000 7fcb50b9d640 1 Processor -- start 2024-04-25T15:25:33.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.873+0000 7fcb50b9d640 1 -- start start 2024-04-25T15:25:33.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb50b9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcb4c0b9040 0x7fcb4c06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb50b9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c06e800 0x7fcb4c164670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb50b9d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb4c06edd0 con 0x7fcb4c164bb0 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb50b9d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcb4c0d3260 con 0x7fcb4c0b9040 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb50b9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4c09cd60 con 0x7fcb4c06e800 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb4b7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcb4c0b9040 0x7fcb4c06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb4affd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c06e800 0x7fcb4c164670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb4b7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcb4c0b9040 0x7fcb4c06e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40688/0 (socket says 172.21.15.67:40688) 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb4bfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fcb4c164bb0 0x7fcb4c0d1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:60282/0 (socket says 172.21.15.67:60282) 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.875+0000 7fcb4b7fe640 1 -- 172.21.15.67:0/4092699953 learned_addr learned my addr 172.21.15.67:0/4092699953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.876+0000 7fcb4affd640 1 -- 172.21.15.67:0/4092699953 >> v2:172.21.15.176:3300/0 conn(0x7fcb4c0b9040 msgr2=0x7fcb4c06e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.876+0000 7fcb4affd640 1 --2- 172.21.15.67:0/4092699953 >> v2:172.21.15.176:3300/0 conn(0x7fcb4c0b9040 0x7fcb4c06e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.876+0000 7fcb4affd640 1 -- 172.21.15.67:0/4092699953 >> v1:172.21.15.176:6789/0 conn(0x7fcb4c164bb0 legacy=0x7fcb4c0d1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.876+0000 7fcb4affd640 1 -- 172.21.15.67:0/4092699953 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb4c09d040 con 0x7fcb4c06e800 2024-04-25T15:25:33.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.876+0000 7fcb4a7fc640 1 -- 172.21.15.67:0/4092699953 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 901275230 0 0) 0x7fcb4c06edd0 con 0x7fcb4c164bb0 2024-04-25T15:25:33.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.876+0000 7fcb4b7fe640 1 --2- 172.21.15.67:0/4092699953 >> v2:172.21.15.176:3300/0 conn(0x7fcb4c0b9040 0x7fcb4c06e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.876+0000 7fcb4affd640 1 --2- 172.21.15.67:0/4092699953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c06e800 0x7fcb4c164670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb400097a0 tx=0x7fcb4002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=391bae572c37e9ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.877+0000 7fcb4a7fc640 1 -- 172.21.15.67:0/4092699953 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40037070 con 0x7fcb4c06e800 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.877+0000 7fcb4a7fc640 1 -- 172.21.15.67:0/4092699953 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb40032440 con 0x7fcb4c06e800 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.877+0000 7fcb4a7fc640 1 -- 172.21.15.67:0/4092699953 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40032810 con 0x7fcb4c06e800 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 -- 172.21.15.67:0/4092699953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c06e800 msgr2=0x7fcb4c164670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/4092699953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c06e800 0x7fcb4c164670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcb400097a0 tx=0x7fcb4002a180 comp rx=0 tx=0).stop 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 -- 172.21.15.67:0/4092699953 shutdown_connections 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/4092699953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c06e800 0x7fcb4c164670 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/4092699953 >> v2:172.21.15.176:3300/0 conn(0x7fcb4c0b9040 0x7fcb4c06e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 -- 172.21.15.67:0/4092699953 >> 172.21.15.67:0/4092699953 conn(0x7fcb4c0d03c0 msgr2=0x7fcb4c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 -- 172.21.15.67:0/4092699953 shutdown_connections 2024-04-25T15:25:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.878+0000 7fcb50b9d640 1 -- 172.21.15.67:0/4092699953 wait complete. 2024-04-25T15:25:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.879+0000 7fcb50b9d640 1 Processor -- start 2024-04-25T15:25:33.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.880+0000 7fcb50b9d640 1 -- start start 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb50b9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c06e800 0x7fcb4c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb50b9d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb4c0b9040 0x7fcb4c0ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb50b9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c164bb0 0x7fcb4c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb50b9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c0d1250 con 0x7fcb4c06e800 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb50b9d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c0cf9b0 con 0x7fcb4c0b9040 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb50b9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4c0d0ee0 con 0x7fcb4c164bb0 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c06e800 0x7fcb4c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c06e800 0x7fcb4c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37052/0 (socket says 172.21.15.67:37052) 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 -- 172.21.15.67:0/2791905942 learned_addr learned my addr 172.21.15.67:0/2791905942 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4affd640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb4c0b9040 0x7fcb4c0ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4bfff640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c164bb0 0x7fcb4c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 -- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c164bb0 msgr2=0x7fcb4c15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c164bb0 0x7fcb4c15db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 -- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb4c0b9040 msgr2=0x7fcb4c0ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb4c0b9040 0x7fcb4c0ccaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4b7fe640 1 -- 172.21.15.67:0/2791905942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb4c0cb140 con 0x7fcb4c06e800 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.881+0000 7fcb4affd640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb4c0b9040 0x7fcb4c0ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.882+0000 7fcb4b7fe640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c06e800 0x7fcb4c0cc560 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00e930 tx=0x7fcb3c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.882+0000 7fcb33fff640 1 -- 172.21.15.67:0/2791905942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c017070 con 0x7fcb4c06e800 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.882+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb4c0cb420 con 0x7fcb4c06e800 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.882+0000 7fcb33fff640 1 -- 172.21.15.67:0/2791905942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c00ca00 con 0x7fcb4c06e800 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.882+0000 7fcb33fff640 1 -- 172.21.15.67:0/2791905942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c00cdf0 con 0x7fcb4c06e800 2024-04-25T15:25:33.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.882+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb4c0cb9e0 con 0x7fcb4c06e800 2024-04-25T15:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.883+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb4c0ce450 con 0x7fcb4c06e800 2024-04-25T15:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.884+0000 7fcb33fff640 1 -- 172.21.15.67:0/2791905942 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb3c01a070 con 0x7fcb4c06e800 2024-04-25T15:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.884+0000 7fcb33fff640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb380744b0 0x7fcb380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.884+0000 7fcb33fff640 1 -- 172.21.15.67:0/2791905942 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcb3c095520 con 0x7fcb4c06e800 2024-04-25T15:25:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.884+0000 7fcb4affd640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb380744b0 0x7fcb380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:33.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.885+0000 7fcb4affd640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb380744b0 0x7fcb380769c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb40034000 tx=0x7fcb4002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:33.888+0000 7fcb33fff640 1 -- 172.21.15.67:0/2791905942 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb4c0ce450 con 0x7fcb4c06e800 2024-04-25T15:25:34.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.123+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcb4c050740 con 0x7fcb4c06e800 2024-04-25T15:25:34.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.126+0000 7fcb33fff640 1 -- 172.21.15.67:0/2791905942 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+186750 (secure 0 0 0) 0x7fcb3c05fc30 con 0x7fcb4c06e800 2024-04-25T15:25:34.129 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb380744b0 msgr2=0x7fcb380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb380744b0 0x7fcb380769c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb40034000 tx=0x7fcb4002e000 comp rx=0 tx=0).stop 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c06e800 msgr2=0x7fcb4c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c06e800 0x7fcb4c0cc560 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00e930 tx=0x7fcb3c00edf0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 shutdown_connections 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb380744b0 0x7fcb380769c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb4c164bb0 0x7fcb4c15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb4c0b9040 0x7fcb4c0ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 --2- 172.21.15.67:0/2791905942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c06e800 0x7fcb4c0cc560 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 >> 172.21.15.67:0/2791905942 conn(0x7fcb4c0d03c0 msgr2=0x7fcb4c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 shutdown_connections 2024-04-25T15:25:34.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.129+0000 7fcb50b9d640 1 -- 172.21.15.67:0/2791905942 wait complete. 2024-04-25T15:25:34.141 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":1591852742},{"type":"v1","addr":"172.21.15.176:6833","nonce":1591852742}]},"active_addr":"172.21.15.176:6833/1591852742","active_change":"2024-04-25T15:25:08.783838+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":2502160119}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":1983201899}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":4091588705}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3468315360}]}]} 2024-04-25T15:25:34.145 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T15:25:34.145 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T15:25:34.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.255+0000 7f86ea814640 1 Processor -- start 2024-04-25T15:25:34.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.256+0000 7f86ea814640 1 -- start start 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86ea814640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f86e40d0e00 0x7f86e40d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86ea814640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e40d1850 con 0x7f86e40d4a50 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86ea814640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e4160de0 con 0x7f86e406c5c0 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86ea814640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f86e41610c0 con 0x7f86e40d0e00 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86e8db1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f86e40d4a50 0x7f86e40d4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59400/0 (socket says 172.21.15.67:59400) 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86e8db1640 1 -- 172.21.15.67:0/3627196243 learned_addr learned my addr 172.21.15.67:0/3627196243 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86e3fff640 1 --2- 172.21.15.67:0/3627196243 >> v2:172.21.15.176:3300/0 conn(0x7f86e40d0e00 0x7f86e40d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4037380908 0 0) 0x7f86e40d1850 con 0x7f86e40d4a50 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86d0003520 con 0x7f86e40d4a50 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.257+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2144839085 0 0) 0x7f86d0003520 con 0x7f86e40d4a50 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 >> v2:172.21.15.176:3300/0 conn(0x7f86e40d0e00 msgr2=0x7f86e40d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:34.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 --2- 172.21.15.67:0/3627196243 >> v2:172.21.15.176:3300/0 conn(0x7f86e40d0e00 0x7f86e40d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 >> v1:172.21.15.176:6789/0 conn(0x7f86e406c5c0 legacy=0x7f86e406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e41613a0 con 0x7f86e40d4a50 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f86dc0019f0 con 0x7f86e40d4a50 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 >> v1:172.21.15.67:6789/0 conn(0x7f86e40d4a50 legacy=0x7f86e40d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d00039b0 0x7f86d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86d0004800 0x7f86d0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d0005250 0x7f86d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86e41610c0 con 0x7f86d00039b0 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86d000c1e0 con 0x7f86d0004800 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86d000c3e0 con 0x7f86d0005250 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e3fff640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d00039b0 0x7f86d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e37fe640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86d0004800 0x7f86d0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e8db1640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d0005250 0x7f86d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e3fff640 1 -- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d0005250 msgr2=0x7f86d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e3fff640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d0005250 0x7f86d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e3fff640 1 -- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86d0004800 msgr2=0x7f86d0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e3fff640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86d0004800 0x7f86d0004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e3fff640 1 -- 172.21.15.67:0/3627196243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d0003520 con 0x7f86d00039b0 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.258+0000 7f86e8db1640 1 -- 172.21.15.67:0/3627196243 reap_dead start 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.259+0000 7f86e3fff640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d00039b0 0x7f86d0003e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f86cc009ac0 tx=0x7f86d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2595f48c99aaa3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.259+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc036070 con 0x7f86d00039b0 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.259+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86cc031400 con 0x7f86d00039b0 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.259+0000 7f86e2ffd640 1 -- 172.21.15.67:0/3627196243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc009690 con 0x7f86d00039b0 2024-04-25T15:25:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.260+0000 7f86ea814640 1 -- 172.21.15.67:0/3627196243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d00039b0 msgr2=0x7f86d0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.260+0000 7f86ea814640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d00039b0 0x7f86d0003e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f86cc009ac0 tx=0x7f86d0005830 comp rx=0 tx=0).stop 2024-04-25T15:25:34.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.260+0000 7f86ea814640 1 -- 172.21.15.67:0/3627196243 shutdown_connections 2024-04-25T15:25:34.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.260+0000 7f86ea814640 1 --2- 172.21.15.67:0/3627196243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86d00039b0 0x7f86d0003e20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.260+0000 7f86ea814640 1 -- 172.21.15.67:0/3627196243 >> 172.21.15.67:0/3627196243 conn(0x7f86e40d5d90 msgr2=0x7f86e40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:34.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.260+0000 7f86ea814640 1 -- 172.21.15.67:0/3627196243 shutdown_connections 2024-04-25T15:25:34.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.260+0000 7f86ea814640 1 -- 172.21.15.67:0/3627196243 wait complete. 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.261+0000 7f86ea814640 1 Processor -- start 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.262+0000 7f86ea814640 1 -- start start 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86ea814640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d00039b0 0x7f86e414b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86ea814640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e4162650 0x7f86e41f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86ea814640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86e41f7490 0x7f86e4158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86ea814640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86e4164e00 con 0x7f86e4162650 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86ea814640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86e40b5320 con 0x7f86e41f7490 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86ea814640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86e4164f80 con 0x7f86d00039b0 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e4162650 0x7f86e41f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e4162650 0x7f86e41f6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37070/0 (socket says 172.21.15.67:37070) 2024-04-25T15:25:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 -- 172.21.15.67:0/1726233887 learned_addr learned my addr 172.21.15.67:0/1726233887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e3fff640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d00039b0 0x7f86e414b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e8db1640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86e41f7490 0x7f86e4158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 -- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d00039b0 msgr2=0x7f86e414b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d00039b0 0x7f86e414b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 -- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86e41f7490 msgr2=0x7f86e4158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86e41f7490 0x7f86e4158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e37fe640 1 -- 172.21.15.67:0/1726233887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e414ba60 con 0x7f86e4162650 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e3fff640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d00039b0 0x7f86e414b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.263+0000 7f86e8db1640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86e41f7490 0x7f86e4158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.264+0000 7f86e37fe640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e4162650 0x7f86e41f6f50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f86d400f850 tx=0x7f86d400fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.264+0000 7f86e0ff9640 1 -- 172.21.15.67:0/1726233887 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d400fe40 con 0x7f86e4162650 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.264+0000 7f86e0ff9640 1 -- 172.21.15.67:0/1726233887 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d4018070 con 0x7f86e4162650 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.264+0000 7f86e0ff9640 1 -- 172.21.15.67:0/1726233887 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d400d730 con 0x7f86e4162650 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.264+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e41f7ba0 con 0x7f86e4162650 2024-04-25T15:25:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.264+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e4158370 con 0x7f86e4162650 2024-04-25T15:25:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.265+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e40b8c30 con 0x7f86e4162650 2024-04-25T15:25:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.265+0000 7f86e0ff9640 1 -- 172.21.15.67:0/1726233887 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f86d400d8d0 con 0x7f86e4162650 2024-04-25T15:25:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.266+0000 7f86e0ff9640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f86c40744b0 0x7f86c40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.266+0000 7f86e0ff9640 1 -- 172.21.15.67:0/1726233887 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f86d40950c0 con 0x7f86e4162650 2024-04-25T15:25:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.266+0000 7f86e3fff640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f86c40744b0 0x7f86c40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.267+0000 7f86e3fff640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f86c40744b0 0x7f86c40769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f86cc033000 tx=0x7f86cc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:34.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.270+0000 7f86e0ff9640 1 -- 172.21.15.67:0/1726233887 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f86d405f850 con 0x7f86e4162650 2024-04-25T15:25:34.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.457+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86e41f7930 con 0x7f86e4162650 2024-04-25T15:25:34.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.458+0000 7f86e0ff9640 1 -- 172.21.15.67:0/1726233887 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f86d405f150 con 0x7f86e4162650 2024-04-25T15:25:34.459 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:34.459 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:34.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f86c40744b0 msgr2=0x7f86c40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f86c40744b0 0x7f86c40769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f86cc033000 tx=0x7f86cc008030 comp rx=0 tx=0).stop 2024-04-25T15:25:34.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e4162650 msgr2=0x7f86e41f6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e4162650 0x7f86e41f6f50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f86d400f850 tx=0x7f86d400fd10 comp rx=0 tx=0).stop 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 shutdown_connections 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f86c40744b0 0x7f86c40769c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86e41f7490 0x7f86e4158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86e4162650 0x7f86e41f6f50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 --2- 172.21.15.67:0/1726233887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86d00039b0 0x7f86e414b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 >> 172.21.15.67:0/1726233887 conn(0x7f86e40d5d90 msgr2=0x7f86e40b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 shutdown_connections 2024-04-25T15:25:34.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.460+0000 7f86ea814640 1 -- 172.21.15.67:0/1726233887 wait complete. 2024-04-25T15:25:34.473 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T15:25:34.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:34.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.584+0000 7f4771f0d640 1 Processor -- start 2024-04-25T15:25:34.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.584+0000 7f4771f0d640 1 -- start start 2024-04-25T15:25:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f4771f0d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c1622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f4771f0d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f476c15e940 0x7f476c15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f4771f0d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f476c071c80 con 0x7f476c06c5c0 2024-04-25T15:25:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f4771f0d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f476c0cc7a0 con 0x7f476c15e940 2024-04-25T15:25:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f4771f0d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f476c0cdcc0 con 0x7f476c0c53a0 2024-04-25T15:25:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f476c15e940 0x7f476c15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f476c15e940 0x7f476c15ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37078/0 (socket says 172.21.15.67:37078) 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476bfff640 1 -- 172.21.15.67:0/5289352 learned_addr learned my addr 172.21.15.67:0/5289352 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476affd640 1 --2- 172.21.15.67:0/5289352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c1622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476bfff640 1 -- 172.21.15.67:0/5289352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 msgr2=0x7f476c1622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476bfff640 1 --2- 172.21.15.67:0/5289352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c1622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476bfff640 1 -- 172.21.15.67:0/5289352 >> v1:172.21.15.176:6789/0 conn(0x7f476c06c5c0 legacy=0x7f476c161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476bfff640 1 -- 172.21.15.67:0/5289352 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c0cd670 con 0x7f476c15e940 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.585+0000 7f476affd640 1 --2- 172.21.15.67:0/5289352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c1622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.586+0000 7f476bfff640 1 --2- 172.21.15.67:0/5289352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c15ed50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4764009870 tx=0x7f476402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c88ea80cfe3ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.586+0000 7f476a7fc640 1 -- 172.21.15.67:0/5289352 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764037070 con 0x7f476c15e940 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.586+0000 7f476a7fc640 1 -- 172.21.15.67:0/5289352 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476402ae10 con 0x7f476c15e940 2024-04-25T15:25:34.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.586+0000 7f476a7fc640 1 -- 172.21.15.67:0/5289352 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764032660 con 0x7f476c15e940 2024-04-25T15:25:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 -- 172.21.15.67:0/5289352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 msgr2=0x7f476c15ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 --2- 172.21.15.67:0/5289352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c15ed50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4764009870 tx=0x7f476402a100 comp rx=0 tx=0).stop 2024-04-25T15:25:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 -- 172.21.15.67:0/5289352 shutdown_connections 2024-04-25T15:25:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 --2- 172.21.15.67:0/5289352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c15ed50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 --2- 172.21.15.67:0/5289352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c1622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 -- 172.21.15.67:0/5289352 >> 172.21.15.67:0/5289352 conn(0x7f476c0ce790 msgr2=0x7f476c0ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:34.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 -- 172.21.15.67:0/5289352 shutdown_connections 2024-04-25T15:25:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.587+0000 7f4771f0d640 1 -- 172.21.15.67:0/5289352 wait complete. 2024-04-25T15:25:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.589+0000 7f4771f0d640 1 Processor -- start 2024-04-25T15:25:34.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.589+0000 7f4771f0d640 1 -- start start 2024-04-25T15:25:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f4771f0d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f476c06c5c0 0x7f476c0c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f4771f0d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c162560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f4771f0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c0b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f4771f0d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f476c071f60 con 0x7f476c15e940 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f4771f0d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f476c0b7870 con 0x7f476c06c5c0 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f4771f0d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f476c15e4a0 con 0x7f476c0c53a0 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f476bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c0b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f476bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c0b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37086/0 (socket says 172.21.15.67:37086) 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.590+0000 7f476bfff640 1 -- 172.21.15.67:0/2171003533 learned_addr learned my addr 172.21.15.67:0/2171003533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476b7fe640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f476c06c5c0 0x7f476c0c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476affd640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476bfff640 1 -- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 msgr2=0x7f476c162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476bfff640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476bfff640 1 -- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f476c06c5c0 msgr2=0x7f476c0c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476bfff640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f476c06c5c0 0x7f476c0c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476bfff640 1 -- 172.21.15.67:0/2171003533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c0c8290 con 0x7f476c15e940 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476affd640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f476bfff640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c0b15a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4764009870 tx=0x7f47640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f4753fff640 1 -- 172.21.15.67:0/2171003533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764041070 con 0x7f476c15e940 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f4753fff640 1 -- 172.21.15.67:0/2171003533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476403c670 con 0x7f476c15e940 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476c162aa0 con 0x7f476c15e940 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f4753fff640 1 -- 172.21.15.67:0/2171003533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764037040 con 0x7f476c15e940 2024-04-25T15:25:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.591+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f476c0a27a0 con 0x7f476c15e940 2024-04-25T15:25:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.592+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f476c050740 con 0x7f476c15e940 2024-04-25T15:25:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.593+0000 7f4753fff640 1 -- 172.21.15.67:0/2171003533 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4764044070 con 0x7f476c15e940 2024-04-25T15:25:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.593+0000 7f4753fff640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47580744b0 0x7f47580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.593+0000 7f4753fff640 1 -- 172.21.15.67:0/2171003533 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f47640b5940 con 0x7f476c15e940 2024-04-25T15:25:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.594+0000 7f476b7fe640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47580744b0 0x7f47580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.594+0000 7f476b7fe640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47580744b0 0x7f47580769c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f476c0ce760 tx=0x7f475c00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:34.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.598+0000 7f4753fff640 1 -- 172.21.15.67:0/2171003533 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f476407fb10 con 0x7f476c15e940 2024-04-25T15:25:34.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.785+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f476c06fcf0 con 0x7f476c15e940 2024-04-25T15:25:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.786+0000 7f4753fff640 1 -- 172.21.15.67:0/2171003533 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f4764083e10 con 0x7f476c15e940 2024-04-25T15:25:34.787 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:34.787 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47580744b0 msgr2=0x7f47580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47580744b0 0x7f47580769c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f476c0ce760 tx=0x7f475c00d040 comp rx=0 tx=0).stop 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 msgr2=0x7f476c0b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c0b15a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4764009870 tx=0x7f47640046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 shutdown_connections 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47580744b0 0x7f47580769c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f476c15e940 0x7f476c0b15a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f476c0c53a0 0x7f476c162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 --2- 172.21.15.67:0/2171003533 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f476c06c5c0 0x7f476c0c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 >> 172.21.15.67:0/2171003533 conn(0x7f476c0ce790 msgr2=0x7f476c15ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 shutdown_connections 2024-04-25T15:25:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:34.788+0000 7f4771f0d640 1 -- 172.21.15.67:0/2171003533 wait complete. 2024-04-25T15:25:34.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T15:25:34.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T15:25:34.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T15:25:34.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T15:25:34.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T15:25:34.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T15:25:34.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T15:25:34.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T15:25:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.028+0000 7f91a339f640 1 Processor -- start 2024-04-25T15:25:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.028+0000 7f91a339f640 1 -- start start 2024-04-25T15:25:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.029+0000 7f2aeda3c640 1 Processor -- start 2024-04-25T15:25:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.029+0000 7fd26e19d640 1 Processor -- start 2024-04-25T15:25:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.029+0000 7f7d2a034640 1 Processor -- start 2024-04-25T15:25:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.029+0000 7f7d2a034640 1 -- start start 2024-04-25T15:25:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7f5544934640 1 Processor -- start 2024-04-25T15:25:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7fcc009ff640 1 Processor -- start 2024-04-25T15:25:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7f9431d23640 1 Processor -- start 2024-04-25T15:25:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.029+0000 7f2aeda3c640 1 -- start start 2024-04-25T15:25:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7f10dd480640 1 Processor -- start 2024-04-25T15:25:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a339f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c0d1f80 0x7f919c071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a339f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f919c071cc0 0x7f919c165070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a339f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f919c0720d0 con 0x7f919c0b17d0 2024-04-25T15:25:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a339f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f919c0d0dd0 con 0x7f919c071cc0 2024-04-25T15:25:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a339f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f919c0c8f70 con 0x7f919c0d1f80 2024-04-25T15:25:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f2aeda3c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ae80d0950 0x7f2ae816f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f2ae6ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2ae81655d0 0x7f2ae806f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:60332/0 (socket says 172.21.15.67:60332) 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f2ae6ffd640 1 -- 172.21.15.67:0/1348727019 learned_addr learned my addr 172.21.15.67:0/1348727019 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7fcc009ff640 1 -- start start 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.029+0000 7fd26e19d640 1 -- start start 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd26e19d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd268164ae0 0x7fd2680c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd2677fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2681646d0 0x7fd2680b3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59402/0 (socket says 172.21.15.67:59402) 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd2677fe640 1 -- 172.21.15.67:0/1273367934 learned_addr learned my addr 172.21.15.67:0/1273367934 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd26e19d640 1 --2- 172.21.15.67:0/1273367934 >> v2:172.21.15.67:3300/0 conn(0x7fd2680c5f10 0x7fd2680b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd266ffd640 1 --2- 172.21.15.67:0/1273367934 >> v2:172.21.15.176:3300/0 conn(0x7fd268164ae0 0x7fd2680c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2680b3b60 con 0x7fd2681646d0 2024-04-25T15:25:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd2680cf280 con 0x7fd2680c5f10 2024-04-25T15:25:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd2680d00e0 con 0x7fd268164ae0 2024-04-25T15:25:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7f9431d23640 1 -- start start 2024-04-25T15:25:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9431d23640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9424047db0 0x7f94240e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9431d23640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424033950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9431d23640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9424006130 con 0x7f9424033e90 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9431d23640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f942404f690 con 0x7f9424047db0 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9431d23640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f94240342a0 con 0x7f9424048300 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9430d21640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9424047db0 0x7f94240e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9430d21640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9424047db0 0x7f94240e7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37102/0 (socket says 172.21.15.67:37102) 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f9430d21640 1 -- 172.21.15.67:0/409921882 learned_addr learned my addr 172.21.15.67:0/409921882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae806f890 con 0x7f2ae8073250 2024-04-25T15:25:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a2b9e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f919c071cc0 0x7f919c165070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f942bfff640 1 --2- 172.21.15.67:0/409921882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424033950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a2b9e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f919c071cc0 0x7f919c165070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37098/0 (socket says 172.21.15.67:37098) 2024-04-25T15:25:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae806c5c0 con 0x7f2ae81655d0 2024-04-25T15:25:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7f10dd480640 1 -- start start 2024-04-25T15:25:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a2b9e640 1 -- 172.21.15.67:0/2468255608 learned_addr learned my addr 172.21.15.67:0/2468255608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f10dd480640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f10dd480640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10d806c910 0x7f10d806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f10dd480640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10d806d260 con 0x7f10d815c7c0 2024-04-25T15:25:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f10dd480640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10d815d060 con 0x7f10d806c910 2024-04-25T15:25:35.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f10dd480640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f10d815d340 con 0x7f10d80b7370 2024-04-25T15:25:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.031+0000 7f91a1b9c640 1 --2- 172.21.15.67:0/2468255608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c0d1f80 0x7f919c071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f91a1b9c640 1 -- 172.21.15.67:0/2468255608 >> v2:172.21.15.67:3300/0 conn(0x7f919c071cc0 msgr2=0x7f919c165070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f91a1b9c640 1 --2- 172.21.15.67:0/2468255608 >> v2:172.21.15.67:3300/0 conn(0x7f919c071cc0 0x7f919c165070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f91a1b9c640 1 -- 172.21.15.67:0/2468255608 >> v1:172.21.15.176:6789/0 conn(0x7f919c0b17d0 legacy=0x7f919c0d06c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:25:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae806c8a0 con 0x7f2ae80d0950 2024-04-25T15:25:35.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.032+0000 7f91a1b9c640 1 -- 172.21.15.67:0/2468255608 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c164b70 con 0x7f919c0d1f80 2024-04-25T15:25:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.030+0000 7f5544934640 1 -- start start 2024-04-25T15:25:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fcc009ff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf40e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fcc009ff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf4006130 con 0x7fcbf4034060 2024-04-25T15:25:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fcc009ff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf404ffa0 con 0x7fcbf4047db0 2024-04-25T15:25:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fcc009ff640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf404f690 con 0x7fcbf4048300 2024-04-25T15:25:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fcbfaffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcbf4034060 0x7fcbf40065e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59420/0 (socket says 172.21.15.67:59420) 2024-04-25T15:25:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fcbfaffd640 1 -- 172.21.15.67:0/3823027478 learned_addr learned my addr 172.21.15.67:0/3823027478 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f5544934640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 0x7f5540164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f5544934640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f55400d1960 0x7f55400b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f5544934640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5540165d90 con 0x7f55400c6a30 2024-04-25T15:25:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f5544934640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f55401654f0 con 0x7f55400d1960 2024-04-25T15:25:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f5544934640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f55401657d0 con 0x7f55400b9320 2024-04-25T15:25:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f91a1b9c640 1 --2- 172.21.15.67:0/2468255608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c0d1f80 0x7f919c071780 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9194009d90 tx=0x7f919402a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c207df3a1d848c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f2ae5ffb640 1 -- 172.21.15.67:0/1348727019 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231719667 0 0) 0x7f2ae806c5c0 con 0x7f2ae81655d0 2024-04-25T15:25:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fcbfbfff640 1 --2- 172.21.15.67:0/3823027478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf40e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fd267fff640 1 --2- 172.21.15.67:0/1273367934 >> v2:172.21.15.67:3300/0 conn(0x7fd2680c5f10 0x7fd2680b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f2ae5ffb640 1 -- 172.21.15.67:0/1348727019 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ad0003520 con 0x7f2ae81655d0 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f7d2a034640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7d2406f150 0x7f7d241605b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f7d2a034640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d24161290 0x7f7d240aff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f7d2a034640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d2406d5d0 0x7f7d2406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f7d2a034640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7d240aeb10 con 0x7f7d24161290 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f7d2a034640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7d2406ff80 con 0x7f7d2406f150 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f7d2a034640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7d2406df20 con 0x7f7d2406d5d0 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fd2667fc640 1 -- 172.21.15.67:0/1273367934 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628206019 0 0) 0x7fd25c001110 con 0x7fd2681646d0 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fd2667fc640 1 -- 172.21.15.67:0/1273367934 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd250003520 con 0x7fd2681646d0 2024-04-25T15:25:35.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f91a139b640 1 -- 172.21.15.67:0/2468255608 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194031600 con 0x7f919c0d1f80 2024-04-25T15:25:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f553dd9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f55400d1960 0x7f55400b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f553dd9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f55400d1960 0x7f55400b68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40732/0 (socket says 172.21.15.67:40732) 2024-04-25T15:25:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f553dd9b640 1 -- 172.21.15.67:0/4086722083 learned_addr learned my addr 172.21.15.67:0/4086722083 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f553e59c640 1 --2- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 0x7f5540164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f91a139b640 1 -- 172.21.15.67:0/2468255608 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91940317a0 con 0x7f919c0d1f80 2024-04-25T15:25:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f2ae5ffb640 1 -- 172.21.15.67:0/1348727019 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1852506351 0 0) 0x7f2ae806f890 con 0x7f2ae8073250 2024-04-25T15:25:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fd266ffd640 1 -- 172.21.15.67:0/1273367934 >> v2:172.21.15.67:3300/0 conn(0x7fd2680c5f10 msgr2=0x7fd2680b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7fd266ffd640 1 --2- 172.21.15.67:0/1273367934 >> v2:172.21.15.67:3300/0 conn(0x7fd2680c5f10 0x7fd2680b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7fd266ffd640 1 -- 172.21.15.67:0/1273367934 >> v1:172.21.15.67:6789/0 conn(0x7fd2681646d0 legacy=0x7fd2680b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7fd266ffd640 1 -- 172.21.15.67:0/1273367934 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2680c6320 con 0x7fd268164ae0 2024-04-25T15:25:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f2ae5ffb640 1 -- 172.21.15.67:0/1348727019 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ae806c5c0 con 0x7f2ae8073250 2024-04-25T15:25:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f7d23fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d2406d5d0 0x7f7d2406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.033+0000 7f10d7fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f942bfff640 1 -- 172.21.15.67:0/409921882 >> v2:172.21.15.67:3300/0 conn(0x7f9424047db0 msgr2=0x7f94240e7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f10dcc7f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10d806c910 0x7f10d806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f942bfff640 1 --2- 172.21.15.67:0/409921882 >> v2:172.21.15.67:3300/0 conn(0x7f9424047db0 0x7f94240e7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f942bfff640 1 -- 172.21.15.67:0/409921882 >> v1:172.21.15.67:6789/0 conn(0x7f9424033e90 legacy=0x7f94240531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f10dcc7f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10d806c910 0x7f10d806cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37118/0 (socket says 172.21.15.67:37118) 2024-04-25T15:25:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f10d77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f10d815c7c0 0x7f10d81605b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59436/0 (socket says 172.21.15.67:59436) 2024-04-25T15:25:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f10d77fe640 1 -- 172.21.15.67:0/3026011382 learned_addr learned my addr 172.21.15.67:0/3026011382 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f942bfff640 1 -- 172.21.15.67:0/409921882 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942404ffa0 con 0x7f9424048300 2024-04-25T15:25:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f7d237fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7d2406f150 0x7f7d241605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f7d23fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d2406d5d0 0x7f7d2406d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36718/0 (socket says 172.21.15.67:36718) 2024-04-25T15:25:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f10dcc7f640 1 -- 172.21.15.67:0/3026011382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 msgr2=0x7f10d8070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f10dcc7f640 1 --2- 172.21.15.67:0/3026011382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8070f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f7d23fff640 1 -- 172.21.15.67:0/2632215558 learned_addr learned my addr 172.21.15.67:0/2632215558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f7d22ffd640 1 --2- 172.21.15.67:0/2632215558 >> v2:172.21.15.67:3300/0 conn(0x7f7d24161290 0x7f7d240aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f10dcc7f640 1 -- 172.21.15.67:0/3026011382 >> v1:172.21.15.67:6789/0 conn(0x7f10d815c7c0 legacy=0x7f10d81605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7fcbfbfff640 1 -- 172.21.15.67:0/3823027478 >> v1:172.21.15.176:6789/0 conn(0x7fcbf4047db0 legacy=0x7fcbf4033950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7fcbfbfff640 1 -- 172.21.15.67:0/3823027478 >> v1:172.21.15.67:6789/0 conn(0x7fcbf4034060 legacy=0x7fcbf40065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7fcbfbfff640 1 -- 172.21.15.67:0/3823027478 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf4050280 con 0x7fcbf4048300 2024-04-25T15:25:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f553dd9b640 1 -- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 msgr2=0x7f5540164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f553dd9b640 1 --2- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 0x7f5540164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f553dd9b640 1 -- 172.21.15.67:0/4086722083 >> v1:172.21.15.67:6789/0 conn(0x7f55400c6a30 legacy=0x7f55400c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f553dd9b640 1 -- 172.21.15.67:0/4086722083 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5540165ab0 con 0x7f55400d1960 2024-04-25T15:25:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7fd266ffd640 1 --2- 172.21.15.67:0/1273367934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd268164ae0 0x7fd2680c59d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd26809c3a0 tx=0x7fd26002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ad43c4d7c35c622 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f91a339f640 1 -- 172.21.15.67:0/2468255608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c0d1f80 msgr2=0x7f919c071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f91a339f640 1 --2- 172.21.15.67:0/2468255608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c0d1f80 0x7f919c071780 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9194009d90 tx=0x7f919402a610 comp rx=0 tx=0).stop 2024-04-25T15:25:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.034+0000 7f2ae7fff640 1 --2- 172.21.15.67:0/1348727019 >> v2:172.21.15.67:3300/0 conn(0x7f2ae80d0950 0x7f2ae816f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f91a339f640 1 -- 172.21.15.67:0/2468255608 shutdown_connections 2024-04-25T15:25:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f91a339f640 1 --2- 172.21.15.67:0/2468255608 >> v2:172.21.15.67:3300/0 conn(0x7f919c071cc0 0x7f919c165070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f91a339f640 1 --2- 172.21.15.67:0/2468255608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c0d1f80 0x7f919c071780 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f7d23fff640 1 -- 172.21.15.67:0/2632215558 >> v2:172.21.15.176:3300/0 conn(0x7f7d2406f150 msgr2=0x7f7d241605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f7d23fff640 1 --2- 172.21.15.67:0/2632215558 >> v2:172.21.15.176:3300/0 conn(0x7f7d2406f150 0x7f7d241605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f91a339f640 1 -- 172.21.15.67:0/2468255608 >> 172.21.15.67:0/2468255608 conn(0x7f919c0cf240 msgr2=0x7f919c0cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f10d6ffd640 1 -- 172.21.15.67:0/3026011382 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3625545442 0 0) 0x7f10cc001110 con 0x7f10d815c7c0 2024-04-25T15:25:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f7d23fff640 1 -- 172.21.15.67:0/2632215558 >> v2:172.21.15.67:3300/0 conn(0x7f7d24161290 msgr2=0x7f7d240aff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f7d23fff640 1 --2- 172.21.15.67:0/2632215558 >> v2:172.21.15.67:3300/0 conn(0x7f7d24161290 0x7f7d240aff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f7d23fff640 1 -- 172.21.15.67:0/2632215558 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d240b0460 con 0x7f7d2406d5d0 2024-04-25T15:25:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f7d237fe640 1 --2- 172.21.15.67:0/2632215558 >> v2:172.21.15.176:3300/0 conn(0x7f7d2406f150 0x7f7d241605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f10dcc7f640 1 -- 172.21.15.67:0/3026011382 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d815d620 con 0x7f10d806c910 2024-04-25T15:25:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f942bfff640 1 --2- 172.21.15.67:0/409921882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424033950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94200097a0 tx=0x7f942002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=948b01fb12346777 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f2ae7fff640 1 -- 172.21.15.67:0/1348727019 >> v1:172.21.15.176:6789/0 conn(0x7f2ae81655d0 legacy=0x7f2ae806f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7fcbfbfff640 1 --2- 172.21.15.67:0/3823027478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf40e7d50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80097d0 tx=0x7fcbe802a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608f7d6656a71cc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f2ae7fff640 1 -- 172.21.15.67:0/1348727019 >> v1:172.21.15.67:6789/0 conn(0x7f2ae8073250 legacy=0x7f2ae8073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7f2ae7fff640 1 -- 172.21.15.67:0/1348727019 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae806cb80 con 0x7f2ae80d0950 2024-04-25T15:25:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7fd2667fc640 1 -- 172.21.15.67:0/1273367934 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260007740 con 0x7fd268164ae0 2024-04-25T15:25:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7fd2667fc640 1 -- 172.21.15.67:0/1273367934 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2600078e0 con 0x7fd268164ae0 2024-04-25T15:25:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.035+0000 7fd2667fc640 1 -- 172.21.15.67:0/1273367934 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260007cb0 con 0x7fd268164ae0 2024-04-25T15:25:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f942b7fe640 1 -- 172.21.15.67:0/409921882 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420037070 con 0x7f9424048300 2024-04-25T15:25:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f942b7fe640 1 -- 172.21.15.67:0/409921882 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9420032440 con 0x7f9424048300 2024-04-25T15:25:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f942b7fe640 1 -- 172.21.15.67:0/409921882 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420032810 con 0x7f9424048300 2024-04-25T15:25:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7fcbfa7fc640 1 -- 172.21.15.67:0/3823027478 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8037070 con 0x7fcbf4048300 2024-04-25T15:25:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7fcbfa7fc640 1 -- 172.21.15.67:0/3823027478 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe8032440 con 0x7fcbf4048300 2024-04-25T15:25:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7fcbfa7fc640 1 -- 172.21.15.67:0/3823027478 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8032850 con 0x7fcbf4048300 2024-04-25T15:25:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f91a339f640 1 -- 172.21.15.67:0/2468255608 shutdown_connections 2024-04-25T15:25:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f553dd9b640 1 --2- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400d1960 0x7f55400b68b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5530009870 tx=0x7f553002a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1d8606cc1e6bc30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f7d23fff640 1 --2- 172.21.15.67:0/2632215558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d2406d5d0 0x7f7d2406d9e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7d240729b0 tx=0x7f7d1c02a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=470ac8888dbd206d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f7d227fc640 1 -- 172.21.15.67:0/2632215558 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c036070 con 0x7f7d2406d5d0 2024-04-25T15:25:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f7d227fc640 1 -- 172.21.15.67:0/2632215558 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d1c031600 con 0x7f7d2406d5d0 2024-04-25T15:25:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f7d227fc640 1 -- 172.21.15.67:0/2632215558 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0319d0 con 0x7f7d2406d5d0 2024-04-25T15:25:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd268164ae0 msgr2=0x7fd2680c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fd26e19d640 1 --2- 172.21.15.67:0/1273367934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd268164ae0 0x7fd2680c59d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd26809c3a0 tx=0x7fd26002a680 comp rx=0 tx=0).stop 2024-04-25T15:25:35.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f2ae7fff640 1 --2- 172.21.15.67:0/1348727019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae80d0950 0x7f2ae816f5f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0097a0 tx=0x7f2adc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cf9d01a95f39ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f10dcc7f640 1 --2- 172.21.15.67:0/3026011382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d806c910 0x7f10d806cd20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009d90 tx=0x7f10d002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9ce831302ce239 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f10d6ffd640 1 -- 172.21.15.67:0/3026011382 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0007740 con 0x7f10d806c910 2024-04-25T15:25:35.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f10d6ffd640 1 -- 172.21.15.67:0/3026011382 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d00078e0 con 0x7f10d806c910 2024-04-25T15:25:35.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 -- 172.21.15.67:0/409921882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 msgr2=0x7f9424033950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 --2- 172.21.15.67:0/409921882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424033950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94200097a0 tx=0x7f942002a180 comp rx=0 tx=0).stop 2024-04-25T15:25:35.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f2ae5ffb640 1 -- 172.21.15.67:0/1348727019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc037070 con 0x7f2ae80d0950 2024-04-25T15:25:35.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.036+0000 7f91a339f640 1 -- 172.21.15.67:0/2468255608 wait complete. 2024-04-25T15:25:35.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f553d59a640 1 -- 172.21.15.67:0/4086722083 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530037070 con 0x7f55400d1960 2024-04-25T15:25:35.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f553d59a640 1 -- 172.21.15.67:0/4086722083 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f553002ae00 con 0x7f55400d1960 2024-04-25T15:25:35.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f553d59a640 1 -- 172.21.15.67:0/4086722083 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530032660 con 0x7f55400d1960 2024-04-25T15:25:35.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f2ae5ffb640 1 -- 172.21.15.67:0/1348727019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2adc02aba0 con 0x7f2ae80d0950 2024-04-25T15:25:35.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f2ae5ffb640 1 -- 172.21.15.67:0/1348727019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0001950 con 0x7f2ae80d0950 2024-04-25T15:25:35.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fcc009ff640 1 -- 172.21.15.67:0/3823027478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 msgr2=0x7fcbf40e7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fcc009ff640 1 --2- 172.21.15.67:0/3823027478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf40e7d50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80097d0 tx=0x7fcbe802a230 comp rx=0 tx=0).stop 2024-04-25T15:25:35.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 shutdown_connections 2024-04-25T15:25:35.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 -- 172.21.15.67:0/409921882 shutdown_connections 2024-04-25T15:25:35.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 --2- 172.21.15.67:0/409921882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424033950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 --2- 172.21.15.67:0/409921882 >> v2:172.21.15.67:3300/0 conn(0x7f9424047db0 0x7f94240e7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 -- 172.21.15.67:0/409921882 >> 172.21.15.67:0/409921882 conn(0x7f9424054cf0 msgr2=0x7f9424055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fd26e19d640 1 --2- 172.21.15.67:0/1273367934 >> v2:172.21.15.67:3300/0 conn(0x7fd2680c5f10 0x7fd2680b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fd26e19d640 1 --2- 172.21.15.67:0/1273367934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd268164ae0 0x7fd2680c59d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 >> 172.21.15.67:0/1273367934 conn(0x7fd26809b7d0 msgr2=0x7fd26809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 -- 172.21.15.67:0/409921882 shutdown_connections 2024-04-25T15:25:35.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f9431d23640 1 -- 172.21.15.67:0/409921882 wait complete. 2024-04-25T15:25:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 -- 172.21.15.67:0/2632215558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d2406d5d0 msgr2=0x7f7d2406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.037+0000 7f10d6ffd640 1 -- 172.21.15.67:0/3026011382 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0007cb0 con 0x7f10d806c910 2024-04-25T15:25:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 -- 172.21.15.67:0/3026011382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d806c910 msgr2=0x7f10d806cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 --2- 172.21.15.67:0/3026011382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d806c910 0x7f10d806cd20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009d90 tx=0x7f10d002a660 comp rx=0 tx=0).stop 2024-04-25T15:25:35.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 --2- 172.21.15.67:0/2632215558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d2406d5d0 0x7f7d2406d9e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7d240729b0 tx=0x7f7d1c02a610 comp rx=0 tx=0).stop 2024-04-25T15:25:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 shutdown_connections 2024-04-25T15:25:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7fd26e19d640 1 -- 172.21.15.67:0/1273367934 wait complete. 2024-04-25T15:25:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 -- 172.21.15.67:0/3026011382 shutdown_connections 2024-04-25T15:25:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 --2- 172.21.15.67:0/3026011382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d806c910 0x7f10d806cd20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 --2- 172.21.15.67:0/3026011382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8070f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 -- 172.21.15.67:0/3026011382 >> 172.21.15.67:0/3026011382 conn(0x7f10d80d1d50 msgr2=0x7f10d8072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 -- 172.21.15.67:0/2632215558 shutdown_connections 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 --2- 172.21.15.67:0/2632215558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d2406d5d0 0x7f7d2406d9e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 --2- 172.21.15.67:0/2632215558 >> v2:172.21.15.67:3300/0 conn(0x7f7d24161290 0x7f7d240aff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 --2- 172.21.15.67:0/2632215558 >> v2:172.21.15.176:3300/0 conn(0x7f7d2406f150 0x7f7d241605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 -- 172.21.15.67:0/2632215558 >> 172.21.15.67:0/2632215558 conn(0x7f7d24072f00 msgr2=0x7f7d24073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f5544934640 1 -- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400d1960 msgr2=0x7f55400b68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f5544934640 1 --2- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400d1960 0x7f55400b68b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5530009870 tx=0x7f553002a0f0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 -- 172.21.15.67:0/2632215558 shutdown_connections 2024-04-25T15:25:35.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 -- 172.21.15.67:0/3026011382 shutdown_connections 2024-04-25T15:25:35.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae80d0950 msgr2=0x7f2ae816f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f2aeda3c640 1 --2- 172.21.15.67:0/1348727019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae80d0950 0x7f2ae816f5f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0097a0 tx=0x7f2adc02a110 comp rx=0 tx=0).stop 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 shutdown_connections 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f2aeda3c640 1 --2- 172.21.15.67:0/1348727019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae80d0950 0x7f2ae816f5f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 >> 172.21.15.67:0/1348727019 conn(0x7f2ae80d1c20 msgr2=0x7f2ae80d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7fcc009ff640 1 -- 172.21.15.67:0/3823027478 shutdown_connections 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7fcc009ff640 1 --2- 172.21.15.67:0/3823027478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf40e7d50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7fcc009ff640 1 -- 172.21.15.67:0/3823027478 >> 172.21.15.67:0/3823027478 conn(0x7fcbf4054cf0 msgr2=0x7fcbf4055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f91a339f640 1 Processor -- start 2024-04-25T15:25:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7fcc009ff640 1 -- 172.21.15.67:0/3823027478 shutdown_connections 2024-04-25T15:25:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7fcc009ff640 1 -- 172.21.15.67:0/3823027478 wait complete. 2024-04-25T15:25:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f5544934640 1 -- 172.21.15.67:0/4086722083 shutdown_connections 2024-04-25T15:25:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f5544934640 1 --2- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400d1960 0x7f55400b68b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f5544934640 1 --2- 172.21.15.67:0/4086722083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 0x7f5540164de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f5544934640 1 -- 172.21.15.67:0/4086722083 >> 172.21.15.67:0/4086722083 conn(0x7f55400d3a60 msgr2=0x7f55400d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 shutdown_connections 2024-04-25T15:25:35.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f91a339f640 1 -- start start 2024-04-25T15:25:35.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f10dd480640 1 -- 172.21.15.67:0/3026011382 wait complete. 2024-04-25T15:25:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.040+0000 7f9431d23640 1 Processor -- start 2024-04-25T15:25:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a339f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c071cc0 0x7f919c159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a339f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0b17d0 0x7f919c1fab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a339f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f919c0d1f80 0x7f919c1634b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a339f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f919c0d2560 con 0x7f919c0b17d0 2024-04-25T15:25:35.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a339f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f919c0c94f0 con 0x7f919c0d1f80 2024-04-25T15:25:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7fd26e19d640 1 Processor -- start 2024-04-25T15:25:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a339f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f919c0cee50 con 0x7f919c071cc0 2024-04-25T15:25:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f9431d23640 1 -- start start 2024-04-25T15:25:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a1b9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0b17d0 0x7f919c1fab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a1b9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0b17d0 0x7f919c1fab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37158/0 (socket says 172.21.15.67:37158) 2024-04-25T15:25:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a1b9c640 1 -- 172.21.15.67:0/2929303251 learned_addr learned my addr 172.21.15.67:0/2929303251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7fd26e19d640 1 -- start start 2024-04-25T15:25:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a239d640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c071cc0 0x7f919c159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.038+0000 7f7d2a034640 1 -- 172.21.15.67:0/2632215558 wait complete. 2024-04-25T15:25:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f7d2a034640 1 Processor -- start 2024-04-25T15:25:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.039+0000 7f2aeda3c640 1 -- 172.21.15.67:0/1348727019 wait complete. 2024-04-25T15:25:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f7d2a034640 1 -- start start 2024-04-25T15:25:35.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a2b9e640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f919c0d1f80 0x7f919c1634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a239d640 1 -- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f919c0d1f80 msgr2=0x7f919c1634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a239d640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f919c0d1f80 0x7f919c1634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a239d640 1 -- 172.21.15.67:0/2929303251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0b17d0 msgr2=0x7f919c1fab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a239d640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919c0b17d0 0x7f919c1fab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a239d640 1 -- 172.21.15.67:0/2929303251 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c1fbcd0 con 0x7f919c071cc0 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.041+0000 7f91a2b9e640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f919c0d1f80 0x7f919c1634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f5544934640 1 -- 172.21.15.67:0/4086722083 shutdown_connections 2024-04-25T15:25:35.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f5544934640 1 -- 172.21.15.67:0/4086722083 wait complete. 2024-04-25T15:25:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f2aeda3c640 1 Processor -- start 2024-04-25T15:25:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f10dd480640 1 Processor -- start 2024-04-25T15:25:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f91a239d640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c071cc0 0x7f919c159f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f918c00b3e0 tx=0x7f918c00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f2aeda3c640 1 -- start start 2024-04-25T15:25:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f10dd480640 1 -- start start 2024-04-25T15:25:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.043+0000 7fcc009ff640 1 Processor -- start 2024-04-25T15:25:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.043+0000 7fcc009ff640 1 -- start start 2024-04-25T15:25:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.042+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c015020 con 0x7f919c071cc0 2024-04-25T15:25:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.043+0000 7f91a339f640 1 -- 172.21.15.67:0/2929303251 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919c1fb640 con 0x7f919c071cc0 2024-04-25T15:25:35.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.043+0000 7f91a339f640 1 -- 172.21.15.67:0/2929303251 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f919c0cafd0 con 0x7f919c071cc0 2024-04-25T15:25:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.043+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918c011830 con 0x7f919c071cc0 2024-04-25T15:25:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.043+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c015020 con 0x7f919c071cc0 2024-04-25T15:25:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.043+0000 7fd26e19d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2680c5f10 0x7fd268163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd26e19d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2681646d0 0x7fd268166ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd26e19d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd268164ae0 0x7fd2681522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f9431d23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9424033e90 0x7f942404b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f9431d23640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9424047db0 0x7f94240ddb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f7d2a034640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2406d5d0 0x7f7d2415e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd266ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2681646d0 0x7fd268166ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd266ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2681646d0 0x7fd268166ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40772/0 (socket says 172.21.15.67:40772) 2024-04-25T15:25:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd266ffd640 1 -- 172.21.15.67:0/997332851 learned_addr learned my addr 172.21.15.67:0/997332851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f9431d23640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd267fff640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd268164ae0 0x7fd2681522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f7d237fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2406d5d0 0x7f7d2415e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2680cfa90 con 0x7fd2680c5f10 2024-04-25T15:25:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2680cf7b0 con 0x7fd2681646d0 2024-04-25T15:25:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f9431d23640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9424050840 con 0x7f9424033e90 2024-04-25T15:25:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2680cfd70 con 0x7fd268164ae0 2024-04-25T15:25:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f7d237fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2406d5d0 0x7f7d2415e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37174/0 (socket says 172.21.15.67:37174) 2024-04-25T15:25:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.044+0000 7f9431522640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431522640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424039d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36734/0 (socket says 172.21.15.67:36734) 2024-04-25T15:25:35.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431522640 1 -- 172.21.15.67:0/2070391206 learned_addr learned my addr 172.21.15.67:0/2070391206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9424050560 con 0x7f9424047db0 2024-04-25T15:25:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f94240509c0 con 0x7f9424048300 2024-04-25T15:25:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2aeda3c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae8073250 0x7f2ae80b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7fd2677fe640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2680c5f10 0x7fd268163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f5544934640 1 Processor -- start 2024-04-25T15:25:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f5544934640 1 -- start start 2024-04-25T15:25:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10dd480640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d806c910 0x7f10d81ef310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10dd480640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2aeda3c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae80d0950 0x7f2ae81f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2aeda3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae81655d0 0x7f2ae8148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2aeda3c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae81659e0 con 0x7f2ae81655d0 2024-04-25T15:25:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2aeda3c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae806d140 con 0x7f2ae8073250 2024-04-25T15:25:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2aeda3c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae8165b60 con 0x7f2ae80d0950 2024-04-25T15:25:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10dd480640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d815c7c0 0x7f10d815e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f91a339f640 1 -- 172.21.15.67:0/2929303251 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9184001030 con 0x7f919c071cc0 2024-04-25T15:25:35.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10dd480640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10d8160cc0 con 0x7f10d815c7c0 2024-04-25T15:25:35.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431522640 1 -- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9424047db0 msgr2=0x7f94240ddb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:25:35.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431522640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9424047db0 0x7f94240ddb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10dd480640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10d8160090 con 0x7f10d806c910 2024-04-25T15:25:35.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431522640 1 -- 172.21.15.67:0/2070391206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9424033e90 msgr2=0x7f942404b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9430d21640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9424033e90 0x7f942404b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431522640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9424033e90 0x7f942404b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10dd480640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f10d815cbd0 con 0x7f10d80b7370 2024-04-25T15:25:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10d77fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2ae6ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae8073250 0x7f2ae80b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10d77fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8156e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36748/0 (socket says 172.21.15.67:36748) 2024-04-25T15:25:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f10d77fe640 1 -- 172.21.15.67:0/466229863 learned_addr learned my addr 172.21.15.67:0/466229863 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7fcc009ff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbf4047db0 0x7fcbf404eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f7d2a034640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d2406f150 0x7f7d2415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f7d2a034640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d24161290 0x7f7d2415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f7d2a034640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2406fc10 con 0x7f7d2406d5d0 2024-04-25T15:25:35.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f7d2a034640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2406f930 con 0x7f7d2406f150 2024-04-25T15:25:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f7d2a034640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7d24070c30 con 0x7f7d24161290 2024-04-25T15:25:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2ae6ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae8073250 0x7f2ae80b6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40780/0 (socket says 172.21.15.67:40780) 2024-04-25T15:25:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2ae6ffd640 1 -- 172.21.15.67:0/3125457262 learned_addr learned my addr 172.21.15.67:0/3125457262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f2ae67fc640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae80d0950 0x7f2ae81f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.045+0000 7f9431522640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942404a690 con 0x7f9424048300 2024-04-25T15:25:35.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f9430d21640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9424033e90 0x7f942404b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d237fe640 1 -- 172.21.15.67:0/1096697937 learned_addr learned my addr 172.21.15.67:0/1096697937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcc009ff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf4047360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fd266ffd640 1 -- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd268164ae0 msgr2=0x7fd2681522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fd266ffd640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd268164ae0 0x7fd2681522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbf4047db0 0x7fcbf404eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d23fff640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d24161290 0x7f7d2415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f9431522640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424039d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f941800c3f0 tx=0x7f941800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d22ffd640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d2406f150 0x7f7d2415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf4047360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fd266ffd640 1 -- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2680c5f10 msgr2=0x7fd268163cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fd266ffd640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2680c5f10 0x7fd268163cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fd266ffd640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd268164210 con 0x7fd2681646d0 2024-04-25T15:25:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcbfaffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf4047360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36758/0 (socket says 172.21.15.67:36758) 2024-04-25T15:25:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f10d7fff640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d806c910 0x7f10d81ef310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fd266ffd640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2681646d0 0x7fd268166ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd26002ac70 tx=0x7fd260004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcc009ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf404a170 0x7fcbf4181460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcc009ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf40509c0 con 0x7fcbf404a170 2024-04-25T15:25:35.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcc009ff640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf4050840 con 0x7fcbf4047db0 2024-04-25T15:25:35.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcbfb7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbf4047db0 0x7fcbf404eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40788/0 (socket says 172.21.15.67:40788) 2024-04-25T15:25:35.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180052d0 con 0x7f9424048300 2024-04-25T15:25:35.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcbfb7fe640 1 -- 172.21.15.67:0/28698932 learned_addr learned my addr 172.21.15.67:0/28698932 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf4006cf0 con 0x7fcbf4048300 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f2ae6ffd640 1 -- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae80d0950 msgr2=0x7f2ae81f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f2ae6ffd640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae80d0950 0x7f2ae81f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f2ae6ffd640 1 -- 172.21.15.67:0/3125457262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae81655d0 msgr2=0x7f2ae8148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f2ae6ffd640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae81655d0 0x7f2ae8148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f2ae6ffd640 1 -- 172.21.15.67:0/3125457262 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae8166350 con 0x7f2ae8073250 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f10d77fe640 1 -- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d806c910 msgr2=0x7f10d81ef310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f10d77fe640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d806c910 0x7f10d81ef310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f10dcc7f640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d815c7c0 0x7f10d815e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9418005470 con 0x7f9424048300 2024-04-25T15:25:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418007260 con 0x7f9424048300 2024-04-25T15:25:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f10d77fe640 1 -- 172.21.15.67:0/466229863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d815c7c0 msgr2=0x7f10d815e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d22ffd640 1 -- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d24161290 msgr2=0x7f7d2415a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d22ffd640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d24161290 0x7f7d2415a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d22ffd640 1 -- 172.21.15.67:0/1096697937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2406d5d0 msgr2=0x7f7d2415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d22ffd640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d2406d5d0 0x7f7d2415e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f7d22ffd640 1 -- 172.21.15.67:0/1096697937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2415b2a0 con 0x7f7d2406f150 2024-04-25T15:25:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942404ab90 con 0x7f9424048300 2024-04-25T15:25:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94240e8580 con 0x7f9424048300 2024-04-25T15:25:35.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f2ae6ffd640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae8073250 0x7f2ae80b6720 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800c410 tx=0x7f2ad800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.046+0000 7f10d77fe640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d815c7c0 0x7f10d815e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f10d77fe640 1 -- 172.21.15.67:0/466229863 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d8157e00 con 0x7f10d80b7370 2024-04-25T15:25:35.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f10dcc7f640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10d815c7c0 0x7f10d815e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:35.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fcbfb7fe640 1 -- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 msgr2=0x7fcbf4047360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f7d22ffd640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d2406f150 0x7f7d2415a2b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e8d0 tx=0x7f7d1800ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26003f070 con 0x7fd2681646d0 2024-04-25T15:25:35.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fcbfb7fe640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf4047360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fcbfb7fe640 1 -- 172.21.15.67:0/28698932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf404a170 msgr2=0x7fcbf4181460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f10d77fe640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8156e80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c3c0 tx=0x7f10cc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26814c300 con 0x7fd2681646d0 2024-04-25T15:25:35.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd26814c8c0 con 0x7fd2681646d0 2024-04-25T15:25:35.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18017070 con 0x7f7d2406f150 2024-04-25T15:25:35.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fcbfb7fe640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbf404a170 0x7fcbf4181460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d1800c940 con 0x7f7d2406f150 2024-04-25T15:25:35.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f7d2a034640 1 -- 172.21.15.67:0/1096697937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2415afc0 con 0x7f7d2406f150 2024-04-25T15:25:35.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7fcbfb7fe640 1 -- 172.21.15.67:0/28698932 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf40478a0 con 0x7fcbf4047db0 2024-04-25T15:25:35.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f5544934640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 0x7f554014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f7d2a034640 1 -- 172.21.15.67:0/1096697937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d2415bf80 con 0x7f7d2406f150 2024-04-25T15:25:35.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8017070 con 0x7f2ae8073250 2024-04-25T15:25:35.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad8005ce0 con 0x7f2ae8073250 2024-04-25T15:25:35.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad8007310 con 0x7f2ae8073250 2024-04-25T15:25:35.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae8166630 con 0x7f2ae8073250 2024-04-25T15:25:35.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f5544934640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400c6a30 0x7f5540153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f5544934640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55400d1960 0x7f55401535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae8166bf0 con 0x7f2ae8073250 2024-04-25T15:25:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.047+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f918c018020 con 0x7f919c071cc0 2024-04-25T15:25:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f9192ffd640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91800748b0 0x7f9180076dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f94240e7d50 con 0x7f9424048300 2024-04-25T15:25:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7fcbfaffd640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbf4048300 0x7fcbf4047360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f553ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55400d1960 0x7f55401535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f553ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55400d1960 0x7f55401535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37238/0 (socket says 172.21.15.67:37238) 2024-04-25T15:25:35.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7fcbfb7fe640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbf4047db0 0x7fcbf404eb70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcbf000c470 tx=0x7fcbf000c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f553ed9d640 1 -- 172.21.15.67:0/1799610375 learned_addr learned my addr 172.21.15.67:0/1799610375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f5544934640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55400b6f70 con 0x7f55400d1960 2024-04-25T15:25:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f5544934640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f55400b6df0 con 0x7f55400c6a30 2024-04-25T15:25:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd260031bc0 con 0x7fd2681646d0 2024-04-25T15:25:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f918c011630 con 0x7f919c071cc0 2024-04-25T15:25:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f9192ffd640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7f918007d630 0x7f918007fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 --> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91800801f0 con 0x7f918007d630 2024-04-25T15:25:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f5544934640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f55401689e0 con 0x7f55400b9320 2024-04-25T15:25:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd268050740 con 0x7fd2681646d0 2024-04-25T15:25:35.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f553e59c640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 0x7f554014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18009e60 con 0x7f7d2406f150 2024-04-25T15:25:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f91a2b9e640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7f918007d630 0x7f918007fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.048+0000 7f553dd9b640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400c6a30 0x7f5540153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf00052d0 con 0x7fcbf4047db0 2024-04-25T15:25:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf404a8c0 con 0x7fcbf4047db0 2024-04-25T15:25:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f553ed9d640 1 -- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 msgr2=0x7f554014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f553ed9d640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55400b9320 0x7f554014ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f553ed9d640 1 -- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400c6a30 msgr2=0x7f5540153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f553ed9d640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55400c6a30 0x7f5540153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f553ed9d640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55401505f0 con 0x7f55400d1960 2024-04-25T15:25:35.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f918c05b1c0 con 0x7f919c071cc0 2024-04-25T15:25:35.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf404ae80 con 0x7fcbf4047db0 2024-04-25T15:25:35.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf0005490 con 0x7fcbf4047db0 2024-04-25T15:25:35.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0007260 con 0x7fcbf4047db0 2024-04-25T15:25:35.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f91a2b9e640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7f918007d630 0x7f918007fb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f553ed9d640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55400d1960 0x7f55401535a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f553800b9c0 tx=0x7f553800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260044460 con 0x7fd2681646d0 2024-04-25T15:25:35.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538019070 con 0x7f55400d1960 2024-04-25T15:25:35.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f553800cba0 con 0x7f55400d1960 2024-04-25T15:25:35.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5538014440 con 0x7f55400d1960 2024-04-25T15:25:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0074e0 con 0x7f10d80b7370 2024-04-25T15:25:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f10dd480640 1 -- 172.21.15.67:0/466229863 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d8157b20 con 0x7f10d80b7370 2024-04-25T15:25:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f10dd480640 1 -- 172.21.15.67:0/466229863 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d81fad30 con 0x7f10d80b7370 2024-04-25T15:25:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10cc0047e0 con 0x7f10d80b7370 2024-04-25T15:25:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f5544934640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55401508d0 con 0x7f55400d1960 2024-04-25T15:25:35.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f5544934640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5540151220 con 0x7f55400d1960 2024-04-25T15:25:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.049+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7d04001030 con 0x7f7d2406f150 2024-04-25T15:25:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.051+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7d1801a070 con 0x7f7d2406f150 2024-04-25T15:25:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc004bd0 con 0x7f10d80b7370 2024-04-25T15:25:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.050+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10b8001030 con 0x7f10d80b7370 2024-04-25T15:25:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.051+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== osd.5 v2:172.21.15.176:6824/960414228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91800801f0 con 0x7f918007d630 2024-04-25T15:25:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.051+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5518001030 con 0x7f55400d1960 2024-04-25T15:25:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcbdc001030 con 0x7fcbf4047db0 2024-04-25T15:25:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.051+0000 7f7d0bfff640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d100745b0 0x7f7d10076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7d180952a0 con 0x7f7d2406f150 2024-04-25T15:25:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f7d0bfff640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7f7d1007c880 0x7f7d1007ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 --> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d18016ec0 con 0x7f7d1007c880 2024-04-25T15:25:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7d18095690 con 0x7f7d2406f150 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.051+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2ad801a070 con 0x7f2ae8073250 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f2acffff640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ad4074580 0x7f2ad4076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ac8001030 con 0x7f2ae8073250 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2ad80945e0 con 0x7f2ae8073250 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f2acffff640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7f2ad407c7d0 0x7f2ad407ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 --> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ad407f2f0 con 0x7f2ad407c7d0 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f2ae67fc640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ad4074580 0x7f2ad4076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2ad805ae10 con 0x7f2ae8073250 2024-04-25T15:25:35.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f7d23fff640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7f7d1007c880 0x7f7d1007ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f10cc011070 con 0x7f10d80b7370 2024-04-25T15:25:35.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.052+0000 7f91a1b9c640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91800748b0 0x7f9180076dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7f7d237fe640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d100745b0 0x7f7d10076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcbf0007400 con 0x7fcbf4047db0 2024-04-25T15:25:35.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7f7d23fff640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7f7d1007c880 0x7f7d1007ecd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7f10d4ff9640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10c40745b0 0x7f10c4076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f10cc094da0 con 0x7f10d80b7370 2024-04-25T15:25:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5538007370 con 0x7f55400d1960 2024-04-25T15:25:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7fcbe7fff640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbe00744b0 0x7fcbe00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7f7d237fe640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d100745b0 0x7f7d10076ac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7d14006fd0 tx=0x7f7d14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f10d4ff9640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f10c407c820 0x7f10c407ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f10d7fff640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10c40745b0 0x7f10c4076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcbf0094510 con 0x7fcbf4047db0 2024-04-25T15:25:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7fcbe7fff640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fcbe007c6a0 0x7fcbe007eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd26003b410 con 0x7fd2681646d0 2024-04-25T15:25:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 --> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10c407f340 con 0x7f10c407c820 2024-04-25T15:25:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f10cc097400 con 0x7f10d80b7370 2024-04-25T15:25:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 --> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbe007f1c0 con 0x7fcbe007c6a0 2024-04-25T15:25:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fcbf009b050 con 0x7fcbf4047db0 2024-04-25T15:25:35.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7fcbfaffd640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbe00744b0 0x7fcbe00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f10dcc7f640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f10c407c820 0x7f10c407ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f552effd640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55200744e0 0x7f55200769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7fcbfbfff640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fcbe007c6a0 0x7fcbe007eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== osd.2 v2:172.21.15.67:6817/2881232327 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d18016ec0 con 0x7f7d1007c880 2024-04-25T15:25:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9418007400 con 0x7f9424048300 2024-04-25T15:25:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5538095050 con 0x7f55400d1960 2024-04-25T15:25:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fcbfbfff640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fcbe007c6a0 0x7fcbe007eaf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f552effd640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f552007c730 0x7f552007eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fd24ffff640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2540745b0 0x7fd254076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5538013ec0 con 0x7f552007c730 2024-04-25T15:25:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f10d7fff640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10c40745b0 0x7f10c4076ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f10c8006fd0 tx=0x7f10c800c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f553dd9b640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f552007c730 0x7f552007eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.053+0000 7f2ae7fff640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7f2ad407c7d0 0x7f2ad407ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f2ae67fc640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ad4074580 0x7f2ad4076a90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8167c80 tx=0x7f2ae0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fd2677fe640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2540745b0 0x7fd254076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.054+0000 7f2ae7fff640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7f2ad407c7d0 0x7f2ad407ec20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f55380954b0 con 0x7f55400d1960 2024-04-25T15:25:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f10dcc7f640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f10c407c820 0x7f10c407ec70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f94297fa640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94100745b0 0x7f9410076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9418095150 con 0x7f9424048300 2024-04-25T15:25:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd2600b6a30 con 0x7fd2681646d0 2024-04-25T15:25:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f553e59c640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55200744e0 0x7f55200769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f94297fa640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f941007c880 0x7f941007ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9418006e70 con 0x7f941007c880 2024-04-25T15:25:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f9418095540 con 0x7f9424048300 2024-04-25T15:25:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fcbfaffd640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbe00744b0 0x7fcbe00769c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0514f0 tx=0x7fcbfc06a5c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== osd.4 v2:172.21.15.176:6816/614570349 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2ad407f2f0 con 0x7f2ad407c7d0 2024-04-25T15:25:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f91a1b9c640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91800748b0 0x7f9180076dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f919c164830 tx=0x7f919402f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f553dd9b640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f552007c730 0x7f552007eb80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fd24ffff640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7fd25407c880 0x7fd25407ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd260042e50 con 0x7fd25407c880 2024-04-25T15:25:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fd2600b6e20 con 0x7fd2681646d0 2024-04-25T15:25:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.055+0000 7f9430d21640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94100745b0 0x7f9410076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.056+0000 7f942bfff640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f941007c880 0x7f941007ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.056+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== osd.0 v2:172.21.15.67:6808/3594090151 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcbe007f1c0 con 0x7fcbe007c6a0 2024-04-25T15:25:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.056+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== osd.6 v2:172.21.15.176:6800/744880207 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10c407f340 con 0x7f10c407c820 2024-04-25T15:25:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.056+0000 7fd267fff640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7fd25407c880 0x7fd25407ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.056+0000 7f553e59c640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55200744e0 0x7f55200769f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55340096d0 tx=0x7f5534008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.056+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== osd.1 v2:172.21.15.67:6800/1421304389 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5538013ec0 con 0x7f552007c730 2024-04-25T15:25:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.058+0000 7f9430d21640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94100745b0 0x7f9410076ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f942c050c60 tx=0x7f942c070040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.057+0000 7fd267fff640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7fd25407c880 0x7fd25407ecd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.058+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== osd.3 v2:172.21.15.67:6816/1127217835 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd260042e50 con 0x7fd25407c880 2024-04-25T15:25:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.058+0000 7fd2677fe640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2540745b0 0x7fd254076ac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0012b0 tx=0x7fd25c007b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.058+0000 7f942bfff640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f941007c880 0x7f941007ecd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.059+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== osd.7 v2:172.21.15.176:6808/592477191 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9418006e70 con 0x7f941007c880 2024-04-25T15:25:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.084+0000 7f91a339f640 1 -- 172.21.15.67:0/2929303251 --> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9184002dd0 con 0x7f918007d630 2024-04-25T15:25:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.085+0000 7f9192ffd640 1 -- 172.21.15.67:0/2929303251 <== osd.5 v2:172.21.15.176:6824/960414228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9184002dd0 con 0x7f918007d630 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.086+0000 7f9190ff9640 1 -- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7f918007d630 msgr2=0x7f918007fb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.086+0000 7f9190ff9640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7f918007d630 0x7f918007fb20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.086+0000 7f9190ff9640 1 -- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91800748b0 msgr2=0x7f9180076dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.086+0000 7f9190ff9640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91800748b0 0x7f9180076dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f919c164830 tx=0x7f919402f000 comp rx=0 tx=0).stop 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.086+0000 7f9190ff9640 1 -- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c071cc0 msgr2=0x7f919c159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.086+0000 7f9190ff9640 1 --2- 172.21.15.67:0/2929303251 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f919c071cc0 0x7f919c159f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f918c00b3e0 tx=0x7f918c00b8a0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.086+0000 7f91a2b9e640 1 -- 172.21.15.67:0/2929303251 reap_dead start 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.087+0000 7f9190ff9640 1 -- 172.21.15.67:0/2929303251 shutdown_connections 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.087+0000 7f9190ff9640 1 -- 172.21.15.67:0/2929303251 >> 172.21.15.67:0/2929303251 conn(0x7f919c0cf240 msgr2=0x7f919c0c8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.087+0000 7f9190ff9640 1 -- 172.21.15.67:0/2929303251 shutdown_connections 2024-04-25T15:25:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.087+0000 7f9190ff9640 1 -- 172.21.15.67:0/2929303251 wait complete. 2024-04-25T15:25:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.087+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 --> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f10b8002dd0 con 0x7f10c407c820 2024-04-25T15:25:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.088+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 --> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7d04002dd0 con 0x7f7d1007c880 2024-04-25T15:25:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.088+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 --> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ac8002dd0 con 0x7f2ad407c7d0 2024-04-25T15:25:35.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.088+0000 7f10d4ff9640 1 -- 172.21.15.67:0/466229863 <== osd.6 v2:172.21.15.176:6800/744880207 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f10b8002dd0 con 0x7f10c407c820 2024-04-25T15:25:35.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.088+0000 7f2acffff640 1 -- 172.21.15.67:0/3125457262 <== osd.4 v2:172.21.15.176:6816/614570349 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2ac8002dd0 con 0x7f2ad407c7d0 2024-04-25T15:25:35.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f7d0bfff640 1 -- 172.21.15.67:0/1096697937 <== osd.2 v2:172.21.15.67:6817/2881232327 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7d04002dd0 con 0x7f7d1007c880 2024-04-25T15:25:35.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f10c407c820 msgr2=0x7f10c407ec70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f10c407c820 0x7f10c407ec70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10c40745b0 msgr2=0x7f10c4076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10c40745b0 0x7f10c4076ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f10c8006fd0 tx=0x7f10c800c040 comp rx=0 tx=0).stop 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 msgr2=0x7f10d8156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 --2- 172.21.15.67:0/466229863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10d80b7370 0x7f10d8156e80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c3c0 tx=0x7f10cc00c880 comp rx=0 tx=0).stop 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10dcc7f640 1 -- 172.21.15.67:0/466229863 reap_dead start 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 shutdown_connections 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 >> 172.21.15.67:0/466229863 conn(0x7f10d80d1d50 msgr2=0x7f10d80684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7f7d1007c880 msgr2=0x7f7d1007ecd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.089+0000 7f7d09ffb640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7f7d1007c880 0x7f7d1007ecd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d100745b0 msgr2=0x7f7d10076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d100745b0 0x7f7d10076ac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7d14006fd0 tx=0x7f7d14008040 comp rx=0 tx=0).stop 2024-04-25T15:25:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d2406f150 msgr2=0x7f7d2415a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 --2- 172.21.15.67:0/1096697937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d2406f150 0x7f7d2415a2b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800e8d0 tx=0x7f7d1800ed90 comp rx=0 tx=0).stop 2024-04-25T15:25:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 shutdown_connections 2024-04-25T15:25:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f10be7fc640 1 -- 172.21.15.67:0/466229863 wait complete. 2024-04-25T15:25:35.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d23fff640 1 -- 172.21.15.67:0/1096697937 reap_dead start 2024-04-25T15:25:35.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 shutdown_connections 2024-04-25T15:25:35.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 >> 172.21.15.67:0/1096697937 conn(0x7f7d24072f00 msgr2=0x7f7d24073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7f2ad407c7d0 msgr2=0x7f2ad407ec20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7f2ad407c7d0 0x7f2ad407ec20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 shutdown_connections 2024-04-25T15:25:35.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ad4074580 msgr2=0x7f2ad4076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ad4074580 0x7f2ad4076a90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8167c80 tx=0x7f2ae0009040 comp rx=0 tx=0).stop 2024-04-25T15:25:35.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae8073250 msgr2=0x7f2ae80b6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 --2- 172.21.15.67:0/3125457262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae8073250 0x7f2ae80b6720 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800c410 tx=0x7f2ad800c8d0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f7d09ffb640 1 -- 172.21.15.67:0/1096697937 wait complete. 2024-04-25T15:25:35.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2ae7fff640 1 -- 172.21.15.67:0/3125457262 reap_dead start 2024-04-25T15:25:35.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 --> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5518002dd0 con 0x7f552007c730 2024-04-25T15:25:35.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 shutdown_connections 2024-04-25T15:25:35.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.090+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 >> 172.21.15.67:0/3125457262 conn(0x7f2ae80d1c20 msgr2=0x7f2ae80d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.091+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 --> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcbdc002dd0 con 0x7fcbe007c6a0 2024-04-25T15:25:35.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.091+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 shutdown_connections 2024-04-25T15:25:35.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.091+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 --> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd2680b7be0 con 0x7fd25407c880 2024-04-25T15:25:35.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.091+0000 7f2aeda3c640 1 -- 172.21.15.67:0/3125457262 wait complete. 2024-04-25T15:25:35.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.091+0000 7f552effd640 1 -- 172.21.15.67:0/1799610375 <== osd.1 v2:172.21.15.67:6800/1421304389 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5518002dd0 con 0x7f552007c730 2024-04-25T15:25:35.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.091+0000 7fcbe7fff640 1 -- 172.21.15.67:0/28698932 <== osd.0 v2:172.21.15.67:6808/3594090151 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcbdc002dd0 con 0x7fcbe007c6a0 2024-04-25T15:25:35.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fd24ffff640 1 -- 172.21.15.67:0/997332851 <== osd.3 v2:172.21.15.67:6816/1127217835 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd2680b7be0 con 0x7fd25407c880 2024-04-25T15:25:35.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f552007c730 msgr2=0x7f552007eb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f552007c730 0x7f552007eb80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55200744e0 msgr2=0x7f55200769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55200744e0 0x7f55200769f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55340096d0 tx=0x7f5534008040 comp rx=0 tx=0).stop 2024-04-25T15:25:35.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55400d1960 msgr2=0x7f55401535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 --2- 172.21.15.67:0/1799610375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55400d1960 0x7f55401535a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f553800b9c0 tx=0x7f553800be80 comp rx=0 tx=0).stop 2024-04-25T15:25:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f553ed9d640 1 -- 172.21.15.67:0/1799610375 reap_dead start 2024-04-25T15:25:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fcbe007c6a0 msgr2=0x7fcbe007eaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fcc009ff640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fcbe007c6a0 0x7fcbe007eaf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 shutdown_connections 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 >> 172.21.15.67:0/1799610375 conn(0x7f55400d3a60 msgr2=0x7f55400ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbe00744b0 msgr2=0x7fcbe00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fcc009ff640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbe00744b0 0x7fcbe00769c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0514f0 tx=0x7fcbfc06a5c0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbf4047db0 msgr2=0x7fcbf404eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fcc009ff640 1 --2- 172.21.15.67:0/28698932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbf4047db0 0x7fcbf404eb70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcbf000c470 tx=0x7fcbf000c930 comp rx=0 tx=0).stop 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.092+0000 7fcbfbfff640 1 -- 172.21.15.67:0/28698932 reap_dead start 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 shutdown_connections 2024-04-25T15:25:35.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 >> 172.21.15.67:0/28698932 conn(0x7fcbf4054cf0 msgr2=0x7fcbf40369d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 shutdown_connections 2024-04-25T15:25:35.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7fd25407c880 msgr2=0x7fd25407ecd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7fd25407c880 0x7fd25407ecd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7f552cff9640 1 -- 172.21.15.67:0/1799610375 wait complete. 2024-04-25T15:25:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2540745b0 msgr2=0x7fd254076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2540745b0 0x7fd254076ac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0012b0 tx=0x7fd25c007b70 comp rx=0 tx=0).stop 2024-04-25T15:25:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2681646d0 msgr2=0x7fd268166ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 --2- 172.21.15.67:0/997332851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2681646d0 0x7fd268166ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd26002ac70 tx=0x7fd260004a40 comp rx=0 tx=0).stop 2024-04-25T15:25:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd267fff640 1 -- 172.21.15.67:0/997332851 reap_dead start 2024-04-25T15:25:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 shutdown_connections 2024-04-25T15:25:35.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 shutdown_connections 2024-04-25T15:25:35.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 >> 172.21.15.67:0/997332851 conn(0x7fd26809b7d0 msgr2=0x7fd2680cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 --> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9424004860 con 0x7f941007c880 2024-04-25T15:25:35.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fcc009ff640 1 -- 172.21.15.67:0/28698932 wait complete. 2024-04-25T15:25:35.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 shutdown_connections 2024-04-25T15:25:35.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.093+0000 7fd26e19d640 1 -- 172.21.15.67:0/997332851 wait complete. 2024-04-25T15:25:35.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.094+0000 7f94297fa640 1 -- 172.21.15.67:0/2070391206 <== osd.7 v2:172.21.15.176:6808/592477191 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9424004860 con 0x7f941007c880 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.094+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f941007c880 msgr2=0x7f941007ecd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.094+0000 7f9431d23640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f941007c880 0x7f941007ecd0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.094+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94100745b0 msgr2=0x7f9410076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.094+0000 7f9431d23640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94100745b0 0x7f9410076ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f942c050c60 tx=0x7f942c070040 comp rx=0 tx=0).stop 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.094+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 msgr2=0x7f9424039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.094+0000 7f9431d23640 1 --2- 172.21.15.67:0/2070391206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9424048300 0x7f9424039d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f941800c3f0 tx=0x7f941800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.095+0000 7f9431522640 1 -- 172.21.15.67:0/2070391206 reap_dead start 2024-04-25T15:25:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.095+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 shutdown_connections 2024-04-25T15:25:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.095+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 >> 172.21.15.67:0/2070391206 conn(0x7f9424054cf0 msgr2=0x7f94240e5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.098+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 shutdown_connections 2024-04-25T15:25:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.098+0000 7f9431d23640 1 -- 172.21.15.67:0/2070391206 wait complete. 2024-04-25T15:25:35.278 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:25:35.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:25:35.280 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:25:35.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:25:35.281 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:25:35.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:25:35.282 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:25:35.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:25:35.283 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:25:35.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:25:35.284 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:25:35.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:25:35.285 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-04-25T15:25:35.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:25:35.286 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-04-25T15:25:35.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:25:35.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.535+0000 7f8599308640 1 Processor -- start 2024-04-25T15:25:35.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.536+0000 7f96f359e640 1 Processor -- start 2024-04-25T15:25:35.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.535+0000 7f8599308640 1 -- start start 2024-04-25T15:25:35.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.537+0000 7f96f359e640 1 -- start start 2024-04-25T15:25:35.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.537+0000 7f8592ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f858c047de0 0x7f858c033980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59444/0 (socket says 172.21.15.67:59444) 2024-04-25T15:25:35.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.537+0000 7f8592ffd640 1 -- 172.21.15.67:0/3702604806 learned_addr learned my addr 172.21.15.67:0/3702604806 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:35.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.538+0000 7f8599308640 1 --2- 172.21.15.67:0/3702604806 >> v2:172.21.15.176:3300/0 conn(0x7f858c034090 0x7f858c0e7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.538+0000 7f8599308640 1 --2- 172.21.15.67:0/3702604806 >> v2:172.21.15.67:3300/0 conn(0x7f858c048330 0x7f858c006610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.538+0000 7f85937fe640 1 --2- 172.21.15.67:0/3702604806 >> v2:172.21.15.67:3300/0 conn(0x7f858c048330 0x7f858c006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.538+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f858c006160 con 0x7f858c047de0 2024-04-25T15:25:35.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f858c04f6c0 con 0x7f858c048330 2024-04-25T15:25:35.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f858c04ffd0 con 0x7f858c034090 2024-04-25T15:25:35.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f85927fc640 1 --2- 172.21.15.67:0/3702604806 >> v2:172.21.15.176:3300/0 conn(0x7f858c034090 0x7f858c0e7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f8591ffb640 1 -- 172.21.15.67:0/3702604806 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 87117037 0 0) 0x7f858c006160 con 0x7f858c047de0 2024-04-25T15:25:35.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f35af59e640 1 Processor -- start 2024-04-25T15:25:35.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f8591ffb640 1 -- 172.21.15.67:0/3702604806 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f857c003520 con 0x7f858c047de0 2024-04-25T15:25:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f96f359e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f96ec048300 0x7f96ec0e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f96f359e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96ec006130 con 0x7f96ec047db0 2024-04-25T15:25:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f96f359e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96ec04ffa0 con 0x7f96ec034060 2024-04-25T15:25:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f96f359e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f96ec04f690 con 0x7f96ec048300 2024-04-25T15:25:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.540+0000 7f96f1d9b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f96ec034060 0x7f96ec0065e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:60346/0 (socket says 172.21.15.67:60346) 2024-04-25T15:25:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.540+0000 7f96f1d9b640 1 -- 172.21.15.67:0/1182397475 learned_addr learned my addr 172.21.15.67:0/1182397475 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:35.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f35af59e640 1 -- start start 2024-04-25T15:25:35.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.540+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589789050 0 0) 0x7f96ec006130 con 0x7f96ec047db0 2024-04-25T15:25:35.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.540+0000 7f96f2d9d640 1 --2- 172.21.15.67:0/1182397475 >> v2:172.21.15.176:3300/0 conn(0x7f96ec048300 0x7f96ec0e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f85937fe640 1 -- 172.21.15.67:0/3702604806 >> v2:172.21.15.176:3300/0 conn(0x7f858c034090 msgr2=0x7f858c0e7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f85937fe640 1 --2- 172.21.15.67:0/3702604806 >> v2:172.21.15.176:3300/0 conn(0x7f858c034090 0x7f858c0e7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f85937fe640 1 -- 172.21.15.67:0/3702604806 >> v1:172.21.15.67:6789/0 conn(0x7f858c047de0 legacy=0x7f858c033980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.539+0000 7f85937fe640 1 -- 172.21.15.67:0/3702604806 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858c0502b0 con 0x7f858c048330 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.540+0000 7f85937fe640 1 --2- 172.21.15.67:0/3702604806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048330 0x7f858c006610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85940514f0 tx=0x7f85940695d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d9a2851913b498 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f8591ffb640 1 -- 172.21.15.67:0/3702604806 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85940b4070 con 0x7f858c048330 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f8591ffb640 1 -- 172.21.15.67:0/3702604806 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85940a7da0 con 0x7f858c048330 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f8591ffb640 1 -- 172.21.15.67:0/3702604806 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85940af660 con 0x7f858c048330 2024-04-25T15:25:35.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35af59e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f35a80547b0 0x7f35a8006d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35af59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a803d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96dc003520 con 0x7f96ec047db0 2024-04-25T15:25:35.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35af59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a80e3ca0 con 0x7f35a80e3890 2024-04-25T15:25:35.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35af59e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f35a8007990 con 0x7f35a80547b0 2024-04-25T15:25:35.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57465300 0 0) 0x7f96ec04ffa0 con 0x7f96ec034060 2024-04-25T15:25:35.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7fd4b57d7640 1 Processor -- start 2024-04-25T15:25:35.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96ec006130 con 0x7f96ec034060 2024-04-25T15:25:35.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35af59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f35a8007c70 con 0x7f35a8007290 2024-04-25T15:25:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048330 msgr2=0x7f858c006610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 --2- 172.21.15.67:0/3702604806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048330 0x7f858c006610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85940514f0 tx=0x7f85940695d0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2945103382 0 0) 0x7f96dc003520 con 0x7f96ec047db0 2024-04-25T15:25:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 >> v2:172.21.15.176:3300/0 conn(0x7f96ec048300 msgr2=0x7f96ec0e7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f96f159a640 1 --2- 172.21.15.67:0/1182397475 >> v2:172.21.15.176:3300/0 conn(0x7f96ec048300 0x7f96ec0e7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35ae59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f35a80e3890 0x7f35a80540a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59458/0 (socket says 172.21.15.67:59458) 2024-04-25T15:25:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35ae59c640 1 -- 172.21.15.67:0/750452351 learned_addr learned my addr 172.21.15.67:0/750452351 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7f35add9b640 1 --2- 172.21.15.67:0/750452351 >> v2:172.21.15.176:3300/0 conn(0x7f35a80547b0 0x7f35a8006d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.542+0000 7fd4b57d7640 1 -- start start 2024-04-25T15:25:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 >> v1:172.21.15.176:6789/0 conn(0x7f96ec034060 legacy=0x7f96ec0065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ec050280 con 0x7f96ec047db0 2024-04-25T15:25:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f35ad59a640 1 -- 172.21.15.67:0/750452351 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554480598 0 0) 0x7f35a80e3ca0 con 0x7f35a80e3890 2024-04-25T15:25:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 shutdown_connections 2024-04-25T15:25:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 --2- 172.21.15.67:0/3702604806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048330 0x7f858c006610 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 --2- 172.21.15.67:0/3702604806 >> v2:172.21.15.176:3300/0 conn(0x7f858c034090 0x7f858c0e7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 >> 172.21.15.67:0/3702604806 conn(0x7f858c054d20 msgr2=0x7f858c055130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f35ad59a640 1 -- 172.21.15.67:0/750452351 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3598003520 con 0x7f35a80e3890 2024-04-25T15:25:35.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f35aed9d640 1 --2- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a803d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f96e8001670 con 0x7f96ec047db0 2024-04-25T15:25:35.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 shutdown_connections 2024-04-25T15:25:35.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f8599308640 1 -- 172.21.15.67:0/3702604806 wait complete. 2024-04-25T15:25:35.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.543+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 >> v1:172.21.15.67:6789/0 conn(0x7f96ec047db0 legacy=0x7f96ec033950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7fd6b8805640 1 Processor -- start 2024-04-25T15:25:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7fd6b8805640 1 -- start start 2024-04-25T15:25:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f40313ef640 1 Processor -- start 2024-04-25T15:25:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f40313ef640 1 -- start start 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f96f159a640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96dc004ba0 0x7f96dc005010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f96f159a640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96dc0037a0 0x7f96dc003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f96f159a640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004160 0x7f96dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec04f690 con 0x7f96dc004160 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f96dc00c160 con 0x7f96dc004ba0 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f35add9b640 1 -- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 msgr2=0x7f35a803d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f35add9b640 1 --2- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a803d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f35add9b640 1 -- 172.21.15.67:0/750452351 >> v1:172.21.15.67:6789/0 conn(0x7f35a80e3890 legacy=0x7f35a80540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f35add9b640 1 -- 172.21.15.67:0/750452351 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a803c950 con 0x7f35a80547b0 2024-04-25T15:25:35.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f35ad59a640 1 -- 172.21.15.67:0/750452351 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3610090937 0 0) 0x7f3598003520 con 0x7f35a80e3890 2024-04-25T15:25:35.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7fd4b57d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7fd4b57d7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4b00cf940 0x7fd4b00cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7fd4b57d7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4b006f890 con 0x7fd4b01655d0 2024-04-25T15:25:35.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7fd4b57d7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd4b0073ba0 con 0x7fd4b00cf940 2024-04-25T15:25:35.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7fd4b57d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b006c5c0 con 0x7fd4b0073250 2024-04-25T15:25:35.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.544+0000 7f96f1d9b640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96dc0037a0 0x7f96dc003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96dc00c360 con 0x7f96dc0037a0 2024-04-25T15:25:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4ae7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4ae7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36790/0 (socket says 172.21.15.67:36790) 2024-04-25T15:25:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4ae7fc640 1 -- 172.21.15.67:0/407618669 learned_addr learned my addr 172.21.15.67:0/407618669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4affff640 1 --2- 172.21.15.67:0/407618669 >> v2:172.21.15.67:3300/0 conn(0x7fd4b00cf940 0x7fd4b00cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7ff34cbce640 1 Processor -- start 2024-04-25T15:25:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7f96f2d9d640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004160 0x7f96dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4affff640 1 -- 172.21.15.67:0/407618669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 msgr2=0x7fd4b0073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4affff640 1 --2- 172.21.15.67:0/407618669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0073660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4affff640 1 -- 172.21.15.67:0/407618669 >> v1:172.21.15.67:6789/0 conn(0x7fd4b01655d0 legacy=0x7fd4b006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7fd4affff640 1 -- 172.21.15.67:0/407618669 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b006c8a0 con 0x7fd4b00cf940 2024-04-25T15:25:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7ff34cbce640 1 -- start start 2024-04-25T15:25:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f35add9b640 1 --2- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80547b0 0x7f35a8006d50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f35a40097a0 tx=0x7f35a402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cabb6df0839554c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f98b5051640 1 Processor -- start 2024-04-25T15:25:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f35ad59a640 1 -- 172.21.15.67:0/750452351 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a4037070 con 0x7f35a80547b0 2024-04-25T15:25:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f35ad59a640 1 -- 172.21.15.67:0/750452351 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a402aba0 con 0x7f35a80547b0 2024-04-25T15:25:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f35ad59a640 1 -- 172.21.15.67:0/750452351 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0001690 con 0x7f35a80547b0 2024-04-25T15:25:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7fd6b8805640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6ac006dc0 0x7fd6ac0dfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7fd4affff640 1 --2- 172.21.15.67:0/407618669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b00cf940 0x7fd4b00cfd50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009d90 tx=0x7fd4a802a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54eb5930fae72d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7f96f2d9d640 1 -- 172.21.15.67:0/1182397475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96dc0037a0 msgr2=0x7f96dc003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7f96f2d9d640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96dc0037a0 0x7f96dc003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7f96f2d9d640 1 -- 172.21.15.67:0/1182397475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96dc004ba0 msgr2=0x7f96dc005010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:25:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.545+0000 7f96f2d9d640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96dc004ba0 0x7f96dc005010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f40313ef640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4024038b40 0x7f4024007b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f96f2d9d640 1 -- 172.21.15.67:0/1182397475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96dc003520 con 0x7f96dc004160 2024-04-25T15:25:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f98b5051640 1 -- start start 2024-04-25T15:25:35.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f402bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4024006d20 0x7f40240dfba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59472/0 (socket says 172.21.15.67:59472) 2024-04-25T15:25:35.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd4adffb640 1 -- 172.21.15.67:0/407618669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8007740 con 0x7fd4b00cf940 2024-04-25T15:25:35.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b8805640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd6ac038f50 0x7fd6ac050430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd4adffb640 1 -- 172.21.15.67:0/407618669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a80078e0 con 0x7fd4b00cf940 2024-04-25T15:25:35.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd4adffb640 1 -- 172.21.15.67:0/407618669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8007cb0 con 0x7fd4b00cf940 2024-04-25T15:25:35.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f402b7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4024038b40 0x7f4024007b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.546+0000 7f402b7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4024038b40 0x7f4024007b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40854/0 (socket says 172.21.15.67:40854) 2024-04-25T15:25:35.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f402b7fe640 1 -- 172.21.15.67:0/991941421 learned_addr learned my addr 172.21.15.67:0/991941421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f40313ef640 1 --2- 172.21.15.67:0/991941421 >> v2:172.21.15.67:3300/0 conn(0x7f4024038f50 0x7f4024006420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f8599308640 1 Processor -- start 2024-04-25T15:25:35.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b8805640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ac0e00e0 con 0x7fd6ac038b40 2024-04-25T15:25:35.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b8805640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ac008330 con 0x7fd6ac006dc0 2024-04-25T15:25:35.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f96f2d9d640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004160 0x7f96dc00b9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f96e400ba00 tx=0x7f96dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc18c20cca783103 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b3fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd6ac038f50 0x7fd6ac050430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b3fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd6ac038f50 0x7fd6ac050430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40860/0 (socket says 172.21.15.67:40860) 2024-04-25T15:25:35.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b3fff640 1 -- 172.21.15.67:0/3927334966 learned_addr learned my addr 172.21.15.67:0/3927334966 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b8805640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ac0541c0 con 0x7fd6ac038f50 2024-04-25T15:25:35.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f96f2d9d640 1 -- 172.21.15.67:0/1182397475 reap_dead start 2024-04-25T15:25:35.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f8599308640 1 -- start start 2024-04-25T15:25:35.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7ff34cbce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348162060 0x7ff34806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40240084b0 con 0x7f4024006d20 2024-04-25T15:25:35.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f40240541c0 con 0x7f4024038f50 2024-04-25T15:25:35.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4024053890 con 0x7f4024038b40 2024-04-25T15:25:35.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7f4030bee640 1 --2- 172.21.15.67:0/991941421 >> v2:172.21.15.67:3300/0 conn(0x7f4024038f50 0x7f4024006420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b37fe640 1 --2- 172.21.15.67:0/3927334966 >> v2:172.21.15.67:3300/0 conn(0x7fd6ac006dc0 0x7fd6ac0dfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7ff34cbce640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff34806db10 0x7ff348168230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff3477fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348162060 0x7ff34806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff3477fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348162060 0x7ff34806d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36802/0 (socket says 172.21.15.67:36802) 2024-04-25T15:25:35.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff346ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff34806db10 0x7ff348168230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff346ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff34806db10 0x7ff348168230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37280/0 (socket says 172.21.15.67:37280) 2024-04-25T15:25:35.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 -- 172.21.15.67:0/407618669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b00cf940 msgr2=0x7fd4b00cfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/407618669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b00cf940 0x7fd4b00cfd50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009d90 tx=0x7fd4a802a570 comp rx=0 tx=0).stop 2024-04-25T15:25:35.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff3477fe640 1 -- 172.21.15.67:0/4277728614 learned_addr learned my addr 172.21.15.67:0/4277728614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 -- 172.21.15.67:0/407618669 shutdown_connections 2024-04-25T15:25:35.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/407618669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b00cf940 0x7fd4b00cfd50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/407618669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 -- 172.21.15.67:0/407618669 >> 172.21.15.67:0/407618669 conn(0x7fd4b00d1c20 msgr2=0x7fd4b00d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 -- 172.21.15.67:0/407618669 shutdown_connections 2024-04-25T15:25:35.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b3fff640 1 -- 172.21.15.67:0/3927334966 >> v2:172.21.15.67:3300/0 conn(0x7fd6ac006dc0 msgr2=0x7fd6ac0dfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f35af59e640 1 -- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80547b0 msgr2=0x7f35a8006d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f35af59e640 1 --2- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80547b0 0x7f35a8006d50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f35a40097a0 tx=0x7f35a402a110 comp rx=0 tx=0).stop 2024-04-25T15:25:35.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3481659e0 con 0x7ff348168770 2024-04-25T15:25:35.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b3fff640 1 --2- 172.21.15.67:0/3927334966 >> v2:172.21.15.67:3300/0 conn(0x7fd6ac006dc0 0x7fd6ac0dfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4030070 con 0x7f96dc004160 2024-04-25T15:25:35.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff348162470 con 0x7ff34806db10 2024-04-25T15:25:35.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff348168f60 con 0x7ff348162060 2024-04-25T15:25:35.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b3fff640 1 -- 172.21.15.67:0/3927334966 >> v1:172.21.15.67:6789/0 conn(0x7fd6ac038b40 legacy=0x7fd6ac007c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f96f159a640 1 -- 172.21.15.67:0/1182397475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e402ade0 con 0x7f96dc004160 2024-04-25T15:25:35.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7fd4b57d7640 1 -- 172.21.15.67:0/407618669 wait complete. 2024-04-25T15:25:35.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f35af59e640 1 -- 172.21.15.67:0/750452351 shutdown_connections 2024-04-25T15:25:35.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f35af59e640 1 --2- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a803d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f35af59e640 1 --2- 172.21.15.67:0/750452351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80547b0 0x7f35a8006d50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.547+0000 7fd6b3fff640 1 -- 172.21.15.67:0/3927334966 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac008550 con 0x7fd6ac038f50 2024-04-25T15:25:35.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f35af59e640 1 -- 172.21.15.67:0/750452351 >> 172.21.15.67:0/750452351 conn(0x7f35a803ac50 msgr2=0x7f35a803b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7f35af59e640 1 -- 172.21.15.67:0/750452351 shutdown_connections 2024-04-25T15:25:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f402b7fe640 1 -- 172.21.15.67:0/991941421 >> v2:172.21.15.67:3300/0 conn(0x7f4024038f50 msgr2=0x7f4024006420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f402b7fe640 1 --2- 172.21.15.67:0/991941421 >> v2:172.21.15.67:3300/0 conn(0x7f4024038f50 0x7f4024006420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f402b7fe640 1 -- 172.21.15.67:0/991941421 >> v1:172.21.15.67:6789/0 conn(0x7f4024006d20 legacy=0x7f40240dfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.548+0000 7f402b7fe640 1 -- 172.21.15.67:0/991941421 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4024053b70 con 0x7f4024038b40 2024-04-25T15:25:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7fd6b3fff640 1 --2- 172.21.15.67:0/3927334966 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 0x7fd6ac050430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4074f10 tx=0x7fd6b406a5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aad14d88fb304102 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7ff3477fe640 1 -- 172.21.15.67:0/4277728614 >> v2:172.21.15.67:3300/0 conn(0x7ff34806db10 msgr2=0x7ff348168230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:35.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7ff3477fe640 1 --2- 172.21.15.67:0/4277728614 >> v2:172.21.15.67:3300/0 conn(0x7ff34806db10 0x7ff348168230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7ff3477fe640 1 -- 172.21.15.67:0/4277728614 >> v1:172.21.15.176:6789/0 conn(0x7ff348168770 legacy=0x7ff34806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7ff3477fe640 1 -- 172.21.15.67:0/4277728614 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34806df20 con 0x7ff348162060 2024-04-25T15:25:35.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7fd6b27fc640 1 -- 172.21.15.67:0/3927334966 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4073040 con 0x7fd6ac038f50 2024-04-25T15:25:35.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7fd6b27fc640 1 -- 172.21.15.67:0/3927334966 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6b40ae600 con 0x7fd6ac038f50 2024-04-25T15:25:35.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7fd6b27fc640 1 -- 172.21.15.67:0/3927334966 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40aea70 con 0x7fd6ac038f50 2024-04-25T15:25:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7f8599308640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f858c047de0 0x7f858c0e9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7f8599308640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f858c048330 0x7f858c04abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f8592ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f858c047de0 0x7f858c0e9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f8592ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f858c047de0 0x7f858c0e9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40872/0 (socket says 172.21.15.67:40872) 2024-04-25T15:25:35.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f8592ffd640 1 -- 172.21.15.67:0/3078540722 learned_addr learned my addr 172.21.15.67:0/3078540722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7f402b7fe640 1 --2- 172.21.15.67:0/991941421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024038b40 0x7f4024007b80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4024008790 tx=0x7f401802c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2e307bbc298665 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7f402affd640 1 -- 172.21.15.67:0/991941421 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401800b040 con 0x7f4024038b40 2024-04-25T15:25:35.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f402affd640 1 -- 172.21.15.67:0/991941421 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4018031710 con 0x7f4024038b40 2024-04-25T15:25:35.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f402affd640 1 -- 172.21.15.67:0/991941421 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4018031ae0 con 0x7f4024038b40 2024-04-25T15:25:35.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f85927fc640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f858c048330 0x7f858c04abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f8599308640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048fe0 0x7f858c0493f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f96f359e640 1 -- 172.21.15.67:0/1182397475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004160 msgr2=0x7f96dc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f96f359e640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004160 0x7f96dc00b9c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f96e400ba00 tx=0x7f96dc005830 comp rx=0 tx=0).stop 2024-04-25T15:25:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.549+0000 7f35af59e640 1 -- 172.21.15.67:0/750452351 wait complete. 2024-04-25T15:25:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7ff3477fe640 1 --2- 172.21.15.67:0/4277728614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348162060 0x7ff34806d5d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff33802a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=674bb19417c1c2fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7ff3467fc640 1 -- 172.21.15.67:0/4277728614 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338037070 con 0x7ff348162060 2024-04-25T15:25:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7ff3467fc640 1 -- 172.21.15.67:0/4277728614 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff338032440 con 0x7ff348162060 2024-04-25T15:25:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7ff3467fc640 1 -- 172.21.15.67:0/4277728614 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338032810 con 0x7ff348162060 2024-04-25T15:25:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f858c0509f0 con 0x7f858c048fe0 2024-04-25T15:25:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f858c050870 con 0x7f858c047de0 2024-04-25T15:25:35.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7fd6b8805640 1 -- 172.21.15.67:0/3927334966 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 msgr2=0x7fd6ac050430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f858c006b50 con 0x7f858c048330 2024-04-25T15:25:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7fd6b8805640 1 --2- 172.21.15.67:0/3927334966 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 0x7fd6ac050430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4074f10 tx=0x7fd6b406a5c0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7fd6b8805640 1 -- 172.21.15.67:0/3927334966 shutdown_connections 2024-04-25T15:25:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7fd6b8805640 1 --2- 172.21.15.67:0/3927334966 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 0x7fd6ac050430 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f96f359e640 1 -- 172.21.15.67:0/1182397475 shutdown_connections 2024-04-25T15:25:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f96f359e640 1 --2- 172.21.15.67:0/1182397475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96dc004160 0x7f96dc00b9c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f96f359e640 1 -- 172.21.15.67:0/1182397475 >> 172.21.15.67:0/1182397475 conn(0x7f96ec054cf0 msgr2=0x7f96ec055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7fd6b8805640 1 --2- 172.21.15.67:0/3927334966 >> v2:172.21.15.67:3300/0 conn(0x7fd6ac006dc0 0x7fd6ac0dfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7fd6b8805640 1 -- 172.21.15.67:0/3927334966 >> 172.21.15.67:0/3927334966 conn(0x7fd6ac00aa70 msgr2=0x7fd6ac00ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98b5051640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98a800c2d0 0x7f98a80340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98aed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98a800c2d0 0x7f98a80340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98aed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98a800c2d0 0x7f98a80340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37284/0 (socket says 172.21.15.67:37284) 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98aed9d640 1 -- 172.21.15.67:0/3731075006 learned_addr learned my addr 172.21.15.67:0/3731075006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98b5051640 1 --2- 172.21.15.67:0/3731075006 >> v2:172.21.15.176:3300/0 conn(0x7f98a8006aa0 0x7f98a80dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98a80e8700 con 0x7f98a80e02d0 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f98a80071d0 con 0x7f98a800c2d0 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.550+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f98a80074b0 con 0x7f98a8006aa0 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f98aed9d640 1 -- 172.21.15.67:0/3731075006 >> v2:172.21.15.176:3300/0 conn(0x7f98a8006aa0 msgr2=0x7f98a80dfd90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f98aed9d640 1 --2- 172.21.15.67:0/3731075006 >> v2:172.21.15.176:3300/0 conn(0x7f98a8006aa0 0x7f98a80dfd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f98aed9d640 1 -- 172.21.15.67:0/3731075006 >> v1:172.21.15.67:6789/0 conn(0x7f98a80e02d0 legacy=0x7f98a80e06e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:25:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f98aed9d640 1 -- 172.21.15.67:0/3731075006 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a8007790 con 0x7f98a800c2d0 2024-04-25T15:25:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024038b40 msgr2=0x7f4024007b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f40313ef640 1 --2- 172.21.15.67:0/991941421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024038b40 0x7f4024007b80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4024008790 tx=0x7f401802c6f0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f85937fe640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048fe0 0x7f858c0493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f85937fe640 1 -- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f858c048330 msgr2=0x7f858c04abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f85937fe640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f858c048330 0x7f858c04abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f85937fe640 1 -- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f858c047de0 msgr2=0x7f858c0e9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f85937fe640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f858c047de0 0x7f858c0e9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f85937fe640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858c0ea170 con 0x7f858c048fe0 2024-04-25T15:25:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f8592ffd640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f858c047de0 0x7f858c0e9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f96f359e640 1 -- 172.21.15.67:0/1182397475 shutdown_connections 2024-04-25T15:25:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7fd4b57d7640 1 Processor -- start 2024-04-25T15:25:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7fd4b57d7640 1 -- start start 2024-04-25T15:25:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f85937fe640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048fe0 0x7f858c0493f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f859406bc20 tx=0x7f8594075af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 shutdown_connections 2024-04-25T15:25:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f40313ef640 1 --2- 172.21.15.67:0/991941421 >> v2:172.21.15.67:3300/0 conn(0x7f4024038f50 0x7f4024006420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f40313ef640 1 --2- 172.21.15.67:0/991941421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024038b40 0x7f4024007b80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 >> 172.21.15.67:0/991941421 conn(0x7f402400aa70 msgr2=0x7f402400ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7fd6b8805640 1 -- 172.21.15.67:0/3927334966 shutdown_connections 2024-04-25T15:25:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f35af59e640 1 Processor -- start 2024-04-25T15:25:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348162060 msgr2=0x7ff34806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7ff34cbce640 1 --2- 172.21.15.67:0/4277728614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348162060 0x7ff34806d5d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff33802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 shutdown_connections 2024-04-25T15:25:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7ff34cbce640 1 --2- 172.21.15.67:0/4277728614 >> v2:172.21.15.67:3300/0 conn(0x7ff34806db10 0x7ff348168230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7ff34cbce640 1 --2- 172.21.15.67:0/4277728614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348162060 0x7ff34806d5d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 >> 172.21.15.67:0/4277728614 conn(0x7ff348072ee0 msgr2=0x7ff3480732f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7fd6b8805640 1 -- 172.21.15.67:0/3927334966 wait complete. 2024-04-25T15:25:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 shutdown_connections 2024-04-25T15:25:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f40313ef640 1 -- 172.21.15.67:0/991941421 wait complete. 2024-04-25T15:25:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f35af59e640 1 -- start start 2024-04-25T15:25:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 shutdown_connections 2024-04-25T15:25:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7ff34cbce640 1 -- 172.21.15.67:0/4277728614 wait complete. 2024-04-25T15:25:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f98aed9d640 1 --2- 172.21.15.67:0/3731075006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a800c2d0 0x7f98a80340c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98a800d9b0 tx=0x7f98a003b190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8d31c77f4c3b6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f857b7fe640 1 -- 172.21.15.67:0/3078540722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85940b4050 con 0x7f858c048fe0 2024-04-25T15:25:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f857b7fe640 1 -- 172.21.15.67:0/3078540722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85940b5040 con 0x7f858c048fe0 2024-04-25T15:25:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f857b7fe640 1 -- 172.21.15.67:0/3078540722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85940b9610 con 0x7f858c048fe0 2024-04-25T15:25:35.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.552+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f858c0ea680 con 0x7f858c048fe0 2024-04-25T15:25:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.553+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f858c0487e0 con 0x7f858c048fe0 2024-04-25T15:25:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.553+0000 7f98add9b640 1 -- 172.21.15.67:0/3731075006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a0048070 con 0x7f98a800c2d0 2024-04-25T15:25:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.553+0000 7f98add9b640 1 -- 172.21.15.67:0/3731075006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a0043440 con 0x7f98a800c2d0 2024-04-25T15:25:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.553+0000 7f98add9b640 1 -- 172.21.15.67:0/3731075006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a0043810 con 0x7f98a800c2d0 2024-04-25T15:25:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.554+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a800c2d0 msgr2=0x7f98a80340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.554+0000 7f98b5051640 1 --2- 172.21.15.67:0/3731075006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a800c2d0 0x7f98a80340c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98a800d9b0 tx=0x7f98a003b190 comp rx=0 tx=0).stop 2024-04-25T15:25:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.554+0000 7f35af59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a80511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.554+0000 7fd4b57d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.551+0000 7f96f359e640 1 -- 172.21.15.67:0/1182397475 wait complete. 2024-04-25T15:25:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35af59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80e3890 0x7f35a8051710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4b57d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4b00cf940 0x7fd4b0163b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4b57d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b01655d0 0x7fd4b014ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4b57d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b006d140 con 0x7fd4b01655d0 2024-04-25T15:25:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4b57d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b006ce60 con 0x7fd4b00cf940 2024-04-25T15:25:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4b57d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd4b01659e0 con 0x7fd4b0073250 2024-04-25T15:25:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7ff34cbce640 1 Processor -- start 2024-04-25T15:25:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b01655d0 0x7fd4b014ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b01655d0 0x7fd4b014ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37302/0 (socket says 172.21.15.67:37302) 2024-04-25T15:25:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd6b8805640 1 Processor -- start 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4affff640 1 -- 172.21.15.67:0/1931847124 learned_addr learned my addr 172.21.15.67:0/1931847124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4aeffd640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4ae7fc640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4b00cf940 0x7fd4b0163b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35af59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a80d50c0 0x7f35a80d54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35af59e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35a80076a0 con 0x7f35a80d50c0 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35af59e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f35a803cf10 con 0x7f35a80e3890 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35af59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f35a803d950 con 0x7f35a8007290 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35ae59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a80511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35ae59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a80511d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36822/0 (socket says 172.21.15.67:36822) 2024-04-25T15:25:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35ae59c640 1 -- 172.21.15.67:0/4205126298 learned_addr learned my addr 172.21.15.67:0/4205126298 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7ff34cbce640 1 -- start start 2024-04-25T15:25:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f96f359e640 1 Processor -- start 2024-04-25T15:25:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f35add9b640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80e3890 0x7f35a8051710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd6b8805640 1 -- start start 2024-04-25T15:25:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 shutdown_connections 2024-04-25T15:25:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f98b5051640 1 --2- 172.21.15.67:0/3731075006 >> v2:172.21.15.176:3300/0 conn(0x7f98a8006aa0 0x7f98a80dfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f98b5051640 1 --2- 172.21.15.67:0/3731075006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a800c2d0 0x7f98a80340c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 >> 172.21.15.67:0/3731075006 conn(0x7f98a8054cf0 msgr2=0x7f98a8055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7f85797fa640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f858c0ea3f0 con 0x7f858c048fe0 2024-04-25T15:25:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4ae7fc640 1 -- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 msgr2=0x7fd4b0149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4ae7fc640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0149a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4ae7fc640 1 -- 172.21.15.67:0/1931847124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b01655d0 msgr2=0x7fd4b014ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.555+0000 7fd4ae7fc640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b01655d0 0x7fd4b014ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4ae7fc640 1 -- 172.21.15.67:0/1931847124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b01489c0 con 0x7fd4b00cf940 2024-04-25T15:25:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 shutdown_connections 2024-04-25T15:25:35.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4aeffd640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:35.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f35ae59c640 1 -- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80e3890 msgr2=0x7f35a8051710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f98b5051640 1 -- 172.21.15.67:0/3731075006 wait complete. 2024-04-25T15:25:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f35aed9d640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a80d50c0 0x7f35a80d54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f35ae59c640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80e3890 0x7f35a8051710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f35ae59c640 1 -- 172.21.15.67:0/4205126298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a80d50c0 msgr2=0x7f35a80d54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f35ae59c640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a80d50c0 0x7f35a80d54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f35ae59c640 1 -- 172.21.15.67:0/4205126298 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a802fad0 con 0x7f35a8007290 2024-04-25T15:25:35.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f35aed9d640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a80d50c0 0x7f35a80d54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4ae7fc640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4b00cf940 0x7fd4b0163b40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400d5d0 tx=0x7fd4a400da90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4977fe640 1 -- 172.21.15.67:0/1931847124 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4018020 con 0x7fd4b00cf940 2024-04-25T15:25:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4977fe640 1 -- 172.21.15.67:0/1931847124 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a4014620 con 0x7fd4b00cf940 2024-04-25T15:25:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4977fe640 1 -- 172.21.15.67:0/1931847124 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a40149f0 con 0x7fd4b00cf940 2024-04-25T15:25:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b0148ca0 con 0x7fd4b00cf940 2024-04-25T15:25:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b0149260 con 0x7fd4b00cf940 2024-04-25T15:25:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.556+0000 7f96f359e640 1 -- start start 2024-04-25T15:25:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f40313ef640 1 Processor -- start 2024-04-25T15:25:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f40313ef640 1 -- start start 2024-04-25T15:25:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b8805640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6ac006dc0 0x7fd6ac1796a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f857b7fe640 1 -- 172.21.15.67:0/3078540722 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f85940c2030 con 0x7f858c048fe0 2024-04-25T15:25:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7ff34cbce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff34806db10 0x7ff348148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b8805640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6ac038b40 0x7fd6ac0e2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7ff34cbce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff348162060 0x7ff3481fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7f857b7fe640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f85800745b0 0x7f8580076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7f857b7fe640 1 -- 172.21.15.67:0/3078540722 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8594131c20 con 0x7f858c048fe0 2024-04-25T15:25:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f96f359e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96ec17d4b0 0x7f96ec0d8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f96f359e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96ec04ca70 0x7f96ec0e63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f96f359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96ec0e6920 0x7f96ec039430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f96f359e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec0509c0 con 0x7f96ec0e6920 2024-04-25T15:25:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f96f359e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec050840 con 0x7f96ec04ca70 2024-04-25T15:25:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f96f359e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96ec0e7d50 con 0x7f96ec17d4b0 2024-04-25T15:25:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f35ae59c640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a80511d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c700 tx=0x7f35a000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f359effd640 1 -- 172.21.15.67:0/4205126298 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0005ad0 con 0x7f35a8007290 2024-04-25T15:25:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f359effd640 1 -- 172.21.15.67:0/4205126298 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a0005c70 con 0x7f35a8007290 2024-04-25T15:25:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f359effd640 1 -- 172.21.15.67:0/4205126298 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0007270 con 0x7f35a8007290 2024-04-25T15:25:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a802ffd0 con 0x7f35a8007290 2024-04-25T15:25:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.557+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a8030590 con 0x7f35a8007290 2024-04-25T15:25:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7ff3477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff34806db10 0x7ff348148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7ff3477fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff34806db10 0x7ff348148f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37346/0 (socket says 172.21.15.67:37346) 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff34cbce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348168770 0x7ff3481592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff34cbce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3481651e0 con 0x7ff34806db10 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff34cbce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff348164f00 con 0x7ff348162060 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff34cbce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3481654c0 con 0x7ff348168770 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff3477fe640 1 -- 172.21.15.67:0/3483136105 learned_addr learned my addr 172.21.15.67:0/3483136105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff3477fe640 1 -- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348168770 msgr2=0x7ff3481592b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff3477fe640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348168770 0x7ff3481592b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff3477fe640 1 -- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff348162060 msgr2=0x7ff3481fe8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff3477fe640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff348162060 0x7ff3481fe8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6ac038b40 0x7fd6ac0e2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6ac038b40 0x7fd6ac0e2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37330/0 (socket says 172.21.15.67:37330) 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b2ffd640 1 -- 172.21.15.67:0/2282802206 learned_addr learned my addr 172.21.15.67:0/2282802206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b8805640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 0x7fd6ac0e3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac053e50 con 0x7fd6ac038b40 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac053b70 con 0x7fd6ac038f50 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac050970 con 0x7fd6ac006dc0 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7fd6b3fff640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 0x7fd6ac0e3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.558+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4b0148730 con 0x7fd4b00cf940 2024-04-25T15:25:35.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff3477fe640 1 -- 172.21.15.67:0/3483136105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3481494b0 con 0x7ff34806db10 2024-04-25T15:25:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f98b5051640 1 Processor -- start 2024-04-25T15:25:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f8592ffd640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f85800745b0 0x7f8580076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7fd6b37fe640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6ac006dc0 0x7fd6ac1796a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f98b5051640 1 -- start start 2024-04-25T15:25:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3594005290 con 0x7f35a8007290 2024-04-25T15:25:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.559+0000 7ff3477fe640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff34806db10 0x7ff348148f70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff3380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd4977fe640 1 -- 172.21.15.67:0/1931847124 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd4a401b020 con 0x7fd4b00cf940 2024-04-25T15:25:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f96f259c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96ec17d4b0 0x7f96ec0d8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f96f259c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96ec17d4b0 0x7f96ec0d8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36856/0 (socket says 172.21.15.67:36856) 2024-04-25T15:25:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f96f259c640 1 -- 172.21.15.67:0/1178449058 learned_addr learned my addr 172.21.15.67:0/1178449058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f359effd640 1 -- 172.21.15.67:0/4205126298 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f35a0007410 con 0x7f35a8007290 2024-04-25T15:25:35.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd6b2ffd640 1 -- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6ac006dc0 msgr2=0x7fd6ac1796a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd6b2ffd640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6ac006dc0 0x7fd6ac1796a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f8592ffd640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f85800745b0 0x7f8580076ac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8584005880 tx=0x7f8584009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd4977fe640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd49c0744e0 0x7fd49c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f96f1d9b640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96ec04ca70 0x7f96ec0e63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f96f2d9d640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96ec0e6920 0x7f96ec039430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd4977fe640 1 -- 172.21.15.67:0/1931847124 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd4a40955c0 con 0x7fd4b00cf940 2024-04-25T15:25:35.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7f359effd640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3590074530 0x7f3590076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f96f1d9b640 1 -- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96ec17d4b0 msgr2=0x7f96ec0d8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f96f1d9b640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96ec17d4b0 0x7f96ec0d8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f96f1d9b640 1 -- 172.21.15.67:0/1178449058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96ec0e6920 msgr2=0x7f96ec039430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f96f1d9b640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96ec0e6920 0x7f96ec039430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f96f1d9b640 1 -- 172.21.15.67:0/1178449058 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ec0e6d30 con 0x7f96ec04ca70 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f96f259c640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96ec17d4b0 0x7f96ec0d8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7fd4aeffd640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd49c0744e0 0x7fd49c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f359effd640 1 -- 172.21.15.67:0/4205126298 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f35a0094ca0 con 0x7f35a8007290 2024-04-25T15:25:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f35add9b640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3590074530 0x7f3590076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd6b2ffd640 1 -- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 msgr2=0x7fd6ac0e3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd6b2ffd640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 0x7fd6ac0e3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.560+0000 7fd6b2ffd640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac04abb0 con 0x7fd6ac038b40 2024-04-25T15:25:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7fd6b2ffd640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6ac038b40 0x7fd6ac0e2b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000b780 tx=0x7fd6a000bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f96f1d9b640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96ec04ca70 0x7f96ec0e63e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f96f406b700 tx=0x7f96f4069d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f35add9b640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3590074530 0x7f3590076a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35a4034000 tx=0x7f35a402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7fd4aeffd640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd49c0744e0 0x7fd49c0769f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0006fd0 tx=0x7fd4a0009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.561+0000 7f40313ef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024006d20 0x7f40240e1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f40313ef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4024038b40 0x7f40240dac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f402bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024006d20 0x7f40240e1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f40313ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4024038f50 0x7f40240db1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f402bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024006d20 0x7f40240e1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40922/0 (socket says 172.21.15.67:40922) 2024-04-25T15:25:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f402bfff640 1 -- 172.21.15.67:0/1461216367 learned_addr learned my addr 172.21.15.67:0/1461216367 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98b5051640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98a8006aa0 0x7f98a80dbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4024008240 con 0x7f4024038f50 2024-04-25T15:25:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f40240080c0 con 0x7f4024006d20 2024-04-25T15:25:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4024006960 con 0x7f4024038b40 2024-04-25T15:25:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f4030bee640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4024038f50 0x7f40240db1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f402b7fe640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4024038b40 0x7f40240dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98b5051640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98a800c2d0 0x7f98a80d5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98aed9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98a8006aa0 0x7f98a80dbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98aed9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98a8006aa0 0x7f98a80dbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36884/0 (socket says 172.21.15.67:36884) 2024-04-25T15:25:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98b5051640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a80e02d0 0x7f98a8034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f4030bee640 1 -- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4024038b40 msgr2=0x7f40240dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98ae59c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98a800c2d0 0x7f98a80d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98ae59c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98a800c2d0 0x7f98a80d5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40936/0 (socket says 172.21.15.67:40936) 2024-04-25T15:25:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f4030bee640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4024038b40 0x7f40240dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f4030bee640 1 -- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024006d20 msgr2=0x7f40240e1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f4030bee640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024006d20 0x7f40240e1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f4030bee640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4024178fb0 con 0x7f4024038f50 2024-04-25T15:25:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98aed9d640 1 -- 172.21.15.67:0/4289862629 learned_addr learned my addr 172.21.15.67:0/4289862629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98af59e640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a80e02d0 0x7f98a8034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98a804fb80 con 0x7f98a80e02d0 2024-04-25T15:25:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f98a804fa00 con 0x7f98a800c2d0 2024-04-25T15:25:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f98a804fd00 con 0x7f98a8006aa0 2024-04-25T15:25:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.562+0000 7f4030bee640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4024038f50 0x7f40240db1d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f402c06b700 tx=0x7f402c06eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.563+0000 7f98af59e640 1 -- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98a8006aa0 msgr2=0x7f98a80dbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.563+0000 7f98af59e640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98a8006aa0 0x7f98a80dbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.563+0000 7f98af59e640 1 -- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98a800c2d0 msgr2=0x7f98a80d5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.563+0000 7f98af59e640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98a800c2d0 0x7f98a80d5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.563+0000 7f98af59e640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a80dcd30 con 0x7f98a80e02d0 2024-04-25T15:25:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.563+0000 7f98af59e640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a80e02d0 0x7f98a8034630 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f98b006b530 tx=0x7f98b006eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7ff32ffff640 1 -- 172.21.15.67:0/3483136105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338037040 con 0x7ff34806db10 2024-04-25T15:25:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f4028ff9640 1 -- 172.21.15.67:0/1461216367 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c09a070 con 0x7f4024038f50 2024-04-25T15:25:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7fd69ffff640 1 -- 172.21.15.67:0/2282802206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0019070 con 0x7fd6ac038b40 2024-04-25T15:25:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f98977fe640 1 -- 172.21.15.67:0/4289862629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b006ee50 con 0x7f98a80e02d0 2024-04-25T15:25:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f857b7fe640 1 -- 172.21.15.67:0/3078540722 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85940c9020 con 0x7f858c048fe0 2024-04-25T15:25:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f4028ff9640 1 -- 172.21.15.67:0/1461216367 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c06ee50 con 0x7f4024038f50 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4024179290 con 0x7f4024038f50 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f4028ff9640 1 -- 172.21.15.67:0/1461216367 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c076650 con 0x7f4024038f50 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3481499c0 con 0x7ff34806db10 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7ff32ffff640 1 -- 172.21.15.67:0/3483136105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff338042070 con 0x7ff34806db10 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3481fe0c0 con 0x7ff34806db10 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7ff32ffff640 1 -- 172.21.15.67:0/3483136105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff338030b70 con 0x7ff34806db10 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ac04ae90 con 0x7fd6ac038b40 2024-04-25T15:25:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ac04b450 con 0x7fd6ac038b40 2024-04-25T15:25:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a80dd010 con 0x7f98a80e02d0 2024-04-25T15:25:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a80359c0 con 0x7f98a80e02d0 2024-04-25T15:25:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f98977fe640 1 -- 172.21.15.67:0/4289862629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b00763d0 con 0x7f98a80e02d0 2024-04-25T15:25:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.566+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4024179850 con 0x7f4024038f50 2024-04-25T15:25:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.567+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9890005290 con 0x7f98a80e02d0 2024-04-25T15:25:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.568+0000 7f98977fe640 1 -- 172.21.15.67:0/4289862629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b009a060 con 0x7f98a80e02d0 2024-04-25T15:25:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.568+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3480ce440 con 0x7ff34806db10 2024-04-25T15:25:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.568+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40240cb200 con 0x7f4024038f50 2024-04-25T15:25:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.568+0000 7fd69ffff640 1 -- 172.21.15.67:0/2282802206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a000c720 con 0x7fd6ac038b40 2024-04-25T15:25:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.568+0000 7fd69ffff640 1 -- 172.21.15.67:0/2282802206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0023060 con 0x7fd6ac038b40 2024-04-25T15:25:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.569+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd694005290 con 0x7fd6ac038b40 2024-04-25T15:25:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.569+0000 7ff32ffff640 1 -- 172.21.15.67:0/3483136105 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff338030d10 con 0x7ff34806db10 2024-04-25T15:25:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.570+0000 7ff32ffff640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3340744b0 0x7ff3340769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.570+0000 7f4028ff9640 1 -- 172.21.15.67:0/1461216367 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f402c0953f0 con 0x7f4024038f50 2024-04-25T15:25:35.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.570+0000 7ff32ffff640 1 -- 172.21.15.67:0/3483136105 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff33807c0f0 con 0x7ff34806db10 2024-04-25T15:25:35.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.570+0000 7f98977fe640 1 -- 172.21.15.67:0/4289862629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f98b0095420 con 0x7f98a80e02d0 2024-04-25T15:25:35.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.570+0000 7ff346ffd640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3340744b0 0x7ff3340769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f4028ff9640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40100744b0 0x7f40100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f98977fe640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f989c074650 0x7f989c076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7ff346ffd640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3340744b0 0x7ff3340769c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff3480d0f60 tx=0x7ff33c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f98977fe640 1 -- 172.21.15.67:0/4289862629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f98b0111c40 con 0x7f98a80e02d0 2024-04-25T15:25:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f4028ff9640 1 -- 172.21.15.67:0/1461216367 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f402c112220 con 0x7f4024038f50 2024-04-25T15:25:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f96e2ffd640 1 -- 172.21.15.67:0/1178449058 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f406c890 con 0x7f96ec04ca70 2024-04-25T15:25:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f96e2ffd640 1 -- 172.21.15.67:0/1178449058 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f406ca30 con 0x7f96ec04ca70 2024-04-25T15:25:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f402bfff640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40100744b0 0x7f40100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f96f359e640 1 -- 172.21.15.67:0/1178449058 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ec04d260 con 0x7f96ec04ca70 2024-04-25T15:25:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f96f359e640 1 -- 172.21.15.67:0/1178449058 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ec17ddc0 con 0x7f96ec04ca70 2024-04-25T15:25:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f98aed9d640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f989c074650 0x7f989c076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7fd69ffff640 1 -- 172.21.15.67:0/2282802206 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd6a001c070 con 0x7fd6ac038b40 2024-04-25T15:25:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7f96e2ffd640 1 -- 172.21.15.67:0/1178449058 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f406cde0 con 0x7f96ec04ca70 2024-04-25T15:25:35.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.571+0000 7fd4977fe640 1 -- 172.21.15.67:0/1931847124 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4a4022020 con 0x7fd4b00cf940 2024-04-25T15:25:35.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.572+0000 7fd69ffff640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd6980744e0 0x7fd6980769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.572+0000 7fd6b37fe640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd6980744e0 0x7fd6980769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.572+0000 7fd69ffff640 1 -- 172.21.15.67:0/2282802206 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd6a0094bb0 con 0x7fd6ac038b40 2024-04-25T15:25:35.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.572+0000 7f359effd640 1 -- 172.21.15.67:0/4205126298 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35a0099070 con 0x7f35a8007290 2024-04-25T15:25:35.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.573+0000 7f402bfff640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40100744b0 0x7f40100769c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4020000f60 tx=0x7f4020007f00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.573+0000 7f98aed9d640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f989c074650 0x7f989c076b60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009630 tx=0x7f98a003f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.573+0000 7fd6b37fe640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd6980744e0 0x7fd6980769f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80045f0 tx=0x7fd6a8013040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.573+0000 7f96f359e640 1 -- 172.21.15.67:0/1178449058 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d8005290 con 0x7f96ec04ca70 2024-04-25T15:25:35.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.573+0000 7f96e2ffd640 1 -- 172.21.15.67:0/1178449058 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f96f4094070 con 0x7f96ec04ca70 2024-04-25T15:25:35.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.574+0000 7f96e2ffd640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96d4074500 0x7f96d4076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:35.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.574+0000 7f96e2ffd640 1 -- 172.21.15.67:0/1178449058 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f96f4110ee0 con 0x7f96ec04ca70 2024-04-25T15:25:35.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.574+0000 7f96f259c640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96d4074500 0x7f96d4076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:35.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.575+0000 7f96f259c640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96d4074500 0x7f96d4076a10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f96ec17b0f0 tx=0x7f96e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:35.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.579+0000 7f96e2ffd640 1 -- 172.21.15.67:0/1178449058 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f96ec0e6d30 con 0x7f96ec04ca70 2024-04-25T15:25:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.580+0000 7ff32ffff640 1 -- 172.21.15.67:0/3483136105 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff3480ce440 con 0x7ff34806db10 2024-04-25T15:25:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.590+0000 7f4028ff9640 1 -- 172.21.15.67:0/1461216367 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40240cb200 con 0x7f4024038f50 2024-04-25T15:25:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.600+0000 7f98977fe640 1 -- 172.21.15.67:0/4289862629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98b00dbd80 con 0x7f98a80e02d0 2024-04-25T15:25:35.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.611+0000 7fd69ffff640 1 -- 172.21.15.67:0/2282802206 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6a005ecd0 con 0x7fd6ac038b40 2024-04-25T15:25:35.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.976+0000 7f85797fa640 1 -- 172.21.15.67:0/3078540722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f858c04f180 con 0x7f858c048fe0 2024-04-25T15:25:35.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.977+0000 7f857b7fe640 1 -- 172.21.15.67:0/3078540722 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f85940fbe60 con 0x7f858c048fe0 2024-04-25T15:25:35.978 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-04-25T15:25:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.980+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f85800745b0 msgr2=0x7f8580076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.980+0000 7f8599308640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f85800745b0 0x7f8580076ac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8584005880 tx=0x7f8584009040 comp rx=0 tx=0).stop 2024-04-25T15:25:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.980+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048fe0 msgr2=0x7f858c0493f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.980+0000 7f8599308640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048fe0 0x7f858c0493f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f859406bc20 tx=0x7f8594075af0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.981+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 shutdown_connections 2024-04-25T15:25:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.981+0000 7f8599308640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f85800745b0 0x7f8580076ac0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.981+0000 7f8599308640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f858c048fe0 0x7f858c0493f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.981+0000 7f8599308640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f858c048330 0x7f858c04abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.981+0000 7f8599308640 1 --2- 172.21.15.67:0/3078540722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f858c047de0 0x7f858c0e9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.981+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 >> 172.21.15.67:0/3078540722 conn(0x7f858c054d20 msgr2=0x7f858c035e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.981+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 shutdown_connections 2024-04-25T15:25:35.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.982+0000 7f8599308640 1 -- 172.21.15.67:0/3078540722 wait complete. 2024-04-25T15:25:35.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.990+0000 7f96f359e640 1 -- 172.21.15.67:0/1178449058 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f96d8005850 con 0x7f96ec04ca70 2024-04-25T15:25:35.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.991+0000 7f96e2ffd640 1 -- 172.21.15.67:0/1178449058 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f96f40db190 con 0x7f96ec04ca70 2024-04-25T15:25:35.992 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:25:35.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 -- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96d4074500 msgr2=0x7f96d4076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96d4074500 0x7f96d4076a10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f96ec17b0f0 tx=0x7f96e8009040 comp rx=0 tx=0).stop 2024-04-25T15:25:35.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 -- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96ec04ca70 msgr2=0x7f96ec0e63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:35.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96ec04ca70 0x7f96ec0e63e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f96f406b700 tx=0x7f96f4069d50 comp rx=0 tx=0).stop 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 -- 172.21.15.67:0/1178449058 shutdown_connections 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96d4074500 0x7f96d4076a10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96ec0e6920 0x7f96ec039430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96ec04ca70 0x7f96ec0e63e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 --2- 172.21.15.67:0/1178449058 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96ec17d4b0 0x7f96ec0d8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.995+0000 7f96e0ff9640 1 -- 172.21.15.67:0/1178449058 >> 172.21.15.67:0/1178449058 conn(0x7f96ec054cf0 msgr2=0x7f96ec005990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.996+0000 7f96e0ff9640 1 -- 172.21.15.67:0/1178449058 shutdown_connections 2024-04-25T15:25:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.996+0000 7f96e0ff9640 1 -- 172.21.15.67:0/1178449058 wait complete. 2024-04-25T15:25:35.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.998+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd4b006a160 con 0x7fd4b00cf940 2024-04-25T15:25:36.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-04-25T15:25:36.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:35.999+0000 7fd4977fe640 1 -- 172.21.15.67:0/1931847124 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd4a405f6f0 con 0x7fd4b00cf940 2024-04-25T15:25:36.000 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-04-25T15:25:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd49c0744e0 msgr2=0x7fd49c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd49c0744e0 0x7fd49c0769f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0006fd0 tx=0x7fd4a0009060 comp rx=0 tx=0).stop 2024-04-25T15:25:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4b00cf940 msgr2=0x7fd4b0163b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4b00cf940 0x7fd4b0163b40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400d5d0 tx=0x7fd4a400da90 comp rx=0 tx=0).stop 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 shutdown_connections 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd49c0744e0 0x7fd49c0769f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4b01655d0 0x7fd4b014ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4b00cf940 0x7fd4b0163b40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 --2- 172.21.15.67:0/1931847124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4b0073250 0x7fd4b0149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.002+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 >> 172.21.15.67:0/1931847124 conn(0x7fd4b00d1c20 msgr2=0x7fd4b00d2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.003+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 shutdown_connections 2024-04-25T15:25:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.003+0000 7fd4b57d7640 1 -- 172.21.15.67:0/1931847124 wait complete. 2024-04-25T15:25:36.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.007+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3594005580 con 0x7f35a8007290 2024-04-25T15:25:36.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.007+0000 7f359effd640 1 -- 172.21.15.67:0/4205126298 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f35a0005e20 con 0x7f35a8007290 2024-04-25T15:25:36.008 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:25:36.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.010+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3590074530 msgr2=0x7f3590076a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.010+0000 7f35af59e640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3590074530 0x7f3590076a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35a4034000 tx=0x7f35a402e000 comp rx=0 tx=0).stop 2024-04-25T15:25:36.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 msgr2=0x7f35a80511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a80511d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c700 tx=0x7f35a000cbc0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 shutdown_connections 2024-04-25T15:25:36.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3590074530 0x7f3590076a40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a80d50c0 0x7f35a80d54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35a80e3890 0x7f35a8051710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 --2- 172.21.15.67:0/4205126298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35a8007290 0x7f35a80511d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 >> 172.21.15.67:0/4205126298 conn(0x7f35a803ac50 msgr2=0x7f35a8057df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:36.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 shutdown_connections 2024-04-25T15:25:36.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.011+0000 7f35af59e640 1 -- 172.21.15.67:0/4205126298 wait complete. 2024-04-25T15:25:36.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2024-04-25T15:25:36.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.017+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff3481616e0 con 0x7ff34806db10 2024-04-25T15:25:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.017+0000 7ff32ffff640 1 -- 172.21.15.67:0/3483136105 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff34806df20 con 0x7ff34806db10 2024-04-25T15:25:36.018 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:25:36.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.018+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4024048de0 con 0x7f4024038f50 2024-04-25T15:25:36.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.018+0000 7f4028ff9640 1 -- 172.21.15.67:0/1461216367 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4024048de0 con 0x7f4024038f50 2024-04-25T15:25:36.019 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:25:36.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2024-04-25T15:25:36.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3340744b0 msgr2=0x7ff3340769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3340744b0 0x7ff3340769c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff3480d0f60 tx=0x7ff33c009190 comp rx=0 tx=0).stop 2024-04-25T15:25:36.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff34806db10 msgr2=0x7ff348148f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff34806db10 0x7ff348148f70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff338009870 tx=0x7ff3380046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 shutdown_connections 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff3340744b0 0x7ff3340769c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff348168770 0x7ff3481592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff348162060 0x7ff3481fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 --2- 172.21.15.67:0/3483136105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff34806db10 0x7ff348148f70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 >> 172.21.15.67:0/3483136105 conn(0x7ff348072ee0 msgr2=0x7ff348160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 shutdown_connections 2024-04-25T15:25:36.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7ff34cbce640 1 -- 172.21.15.67:0/3483136105 wait complete. 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40100744b0 msgr2=0x7f40100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7f40313ef640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40100744b0 0x7f40100769c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4020000f60 tx=0x7f4020007f00 comp rx=0 tx=0).stop 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4024038f50 msgr2=0x7f40240db1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.021+0000 7f40313ef640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4024038f50 0x7f40240db1d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f402c06b700 tx=0x7f402c06eb20 comp rx=0 tx=0).stop 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 shutdown_connections 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40100744b0 0x7f40100769c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4024038f50 0x7f40240db1d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4024038b40 0x7f40240dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 --2- 172.21.15.67:0/1461216367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4024006d20 0x7f40240e1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 >> 172.21.15.67:0/1461216367 conn(0x7f402400aa70 msgr2=0x7f402404ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 shutdown_connections 2024-04-25T15:25:36.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.022+0000 7f40313ef640 1 -- 172.21.15.67:0/1461216367 wait complete. 2024-04-25T15:25:36.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.025+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd694005850 con 0x7fd6ac038b40 2024-04-25T15:25:36.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.026+0000 7fd69ffff640 1 -- 172.21.15.67:0/2282802206 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd6a0062fd0 con 0x7fd6ac038b40 2024-04-25T15:25:36.026 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-04-25T15:25:36.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.027+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f9890005850 con 0x7f98a80e02d0 2024-04-25T15:25:36.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.027+0000 7f98977fe640 1 -- 172.21.15.67:0/4289862629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f98b00e0080 con 0x7f98a80e02d0 2024-04-25T15:25:36.028 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-04-25T15:25:36.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd6980744e0 msgr2=0x7fd6980769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd6980744e0 0x7fd6980769f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd6a80045f0 tx=0x7fd6a8013040 comp rx=0 tx=0).stop 2024-04-25T15:25:36.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6ac038b40 msgr2=0x7fd6ac0e2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6ac038b40 0x7fd6ac0e2b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000b780 tx=0x7fd6a000bc40 comp rx=0 tx=0).stop 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 shutdown_connections 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd6980744e0 0x7fd6980769f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6ac038f50 0x7fd6ac0e3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6ac038b40 0x7fd6ac0e2b70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 --2- 172.21.15.67:0/2282802206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6ac006dc0 0x7fd6ac1796a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.028+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 >> 172.21.15.67:0/2282802206 conn(0x7fd6ac00aa70 msgr2=0x7fd6ac0d9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.029+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 shutdown_connections 2024-04-25T15:25:36.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.029+0000 7fd6b8805640 1 -- 172.21.15.67:0/2282802206 wait complete. 2024-04-25T15:25:36.031 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f989c074650 msgr2=0x7f989c076b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f989c074650 0x7f989c076b60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009630 tx=0x7f98a003f000 comp rx=0 tx=0).stop 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a80e02d0 msgr2=0x7f98a8034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a80e02d0 0x7f98a8034630 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f98b006b530 tx=0x7f98b006eb20 comp rx=0 tx=0).stop 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 shutdown_connections 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f989c074650 0x7f989c076b60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98a80e02d0 0x7f98a8034630 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98a800c2d0 0x7f98a80d5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 --2- 172.21.15.67:0/4289862629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98a8006aa0 0x7f98a80dbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 >> 172.21.15.67:0/4289862629 conn(0x7f98a8054cf0 msgr2=0x7f98a8005990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 shutdown_connections 2024-04-25T15:25:36.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:36.030+0000 7f98b5051640 1 -- 172.21.15.67:0/4289862629 wait complete. 2024-04-25T15:25:36.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-04-25T15:25:36.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-04-25T15:25:36.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-04-25T15:25:36.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-04-25T15:25:37.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:25:37.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:25:37.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:25:37.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:25:37.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:25:37.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:25:37.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:25:37.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:25:37.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.180+0000 7f008e34d640 1 Processor -- start 2024-04-25T15:25:37.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.181+0000 7f008e34d640 1 -- start start 2024-04-25T15:25:37.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f008e34d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0088073b70 0x7f008815fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f008e34d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00880b6990 0x7f00880cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f008e34d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880cf5b0 0x7f00880cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f008e34d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f008815c9b0 con 0x7f00880b6990 2024-04-25T15:25:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f008e34d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0088160450 con 0x7f0088073b70 2024-04-25T15:25:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f008e34d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0088160730 con 0x7f00880cf5b0 2024-04-25T15:25:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f00877fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00880b6990 0x7f00880cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f00877fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00880b6990 0x7f00880cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37374/0 (socket says 172.21.15.67:37374) 2024-04-25T15:25:37.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.185+0000 7f00877fe640 1 -- 172.21.15.67:0/2719660636 learned_addr learned my addr 172.21.15.67:0/2719660636 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.186+0000 7f00877fe640 1 -- 172.21.15.67:0/2719660636 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880cf5b0 msgr2=0x7f00880cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.186+0000 7f00877fe640 1 --2- 172.21.15.67:0/2719660636 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880cf5b0 0x7f00880cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.186+0000 7f00877fe640 1 -- 172.21.15.67:0/2719660636 >> v2:172.21.15.176:3300/0 conn(0x7f0088073b70 msgr2=0x7f008815fe00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:25:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.186+0000 7f00877fe640 1 --2- 172.21.15.67:0/2719660636 >> v2:172.21.15.176:3300/0 conn(0x7f0088073b70 0x7f008815fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.186+0000 7f00877fe640 1 -- 172.21.15.67:0/2719660636 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0088160a10 con 0x7f00880b6990 2024-04-25T15:25:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.187+0000 7f00877fe640 1 --2- 172.21.15.67:0/2719660636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880b6990 0x7f00880cdd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0088072820 tx=0x7f008002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d38cbcae2d29d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.188+0000 7f0086ffd640 1 -- 172.21.15.67:0/2719660636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080036070 con 0x7f00880b6990 2024-04-25T15:25:37.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.188+0000 7f0086ffd640 1 -- 172.21.15.67:0/2719660636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0080031600 con 0x7f00880b6990 2024-04-25T15:25:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 -- 172.21.15.67:0/2719660636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880b6990 msgr2=0x7f00880cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 --2- 172.21.15.67:0/2719660636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880b6990 0x7f00880cdd60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0088072820 tx=0x7f008002a6d0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 -- 172.21.15.67:0/2719660636 shutdown_connections 2024-04-25T15:25:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 --2- 172.21.15.67:0/2719660636 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880cf5b0 0x7f00880cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 --2- 172.21.15.67:0/2719660636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880b6990 0x7f00880cdd60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 --2- 172.21.15.67:0/2719660636 >> v2:172.21.15.176:3300/0 conn(0x7f0088073b70 0x7f008815fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 -- 172.21.15.67:0/2719660636 >> 172.21.15.67:0/2719660636 conn(0x7f0088073050 msgr2=0x7f0088073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 -- 172.21.15.67:0/2719660636 shutdown_connections 2024-04-25T15:25:37.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.190+0000 7f008e34d640 1 -- 172.21.15.67:0/2719660636 wait complete. 2024-04-25T15:25:37.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.194+0000 7f008e34d640 1 Processor -- start 2024-04-25T15:25:37.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.194+0000 7f008e34d640 1 -- start start 2024-04-25T15:25:37.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.197+0000 7f008e34d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0088073b70 0x7f008814a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.197+0000 7f008e34d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880b6990 0x7f0088137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.197+0000 7f008e34d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880cf5b0 0x7f0088137d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.197+0000 7f00877fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880b6990 0x7f0088137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.197+0000 7f00877fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880b6990 0x7f0088137810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36902/0 (socket says 172.21.15.67:36902) 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.198+0000 7f008c8ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880cf5b0 0x7f0088137d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.198+0000 7f008c8ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880cf5b0 0x7f0088137d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37390/0 (socket says 172.21.15.67:37390) 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.198+0000 7f00877fe640 1 -- 172.21.15.67:0/3944265453 learned_addr learned my addr 172.21.15.67:0/3944265453 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.198+0000 7f008e34d640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00880ce2a0 con 0x7f00880cf5b0 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.198+0000 7f008e34d640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f00880b6da0 con 0x7f0088073b70 2024-04-25T15:25:37.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.198+0000 7f008e34d640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f00880ce420 con 0x7f00880b6990 2024-04-25T15:25:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.199+0000 7f008c8ea640 1 -- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880b6990 msgr2=0x7f0088137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.199+0000 7f008c8ea640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880b6990 0x7f0088137810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.199+0000 7f008c8ea640 1 -- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0088073b70 msgr2=0x7f008814a650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:25:37.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.199+0000 7f008c8ea640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0088073b70 0x7f008814a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.199+0000 7f008c8ea640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008814d540 con 0x7f00880cf5b0 2024-04-25T15:25:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.200+0000 7f008c8ea640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880cf5b0 0x7f0088137d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f007800b700 tx=0x7f007800bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.201+0000 7f0084ff9640 1 -- 172.21.15.67:0/3944265453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0078019070 con 0x7f00880cf5b0 2024-04-25T15:25:37.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.201+0000 7f0084ff9640 1 -- 172.21.15.67:0/3944265453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f007800c720 con 0x7f00880cf5b0 2024-04-25T15:25:37.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.201+0000 7f008e34d640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008814d820 con 0x7f00880cf5b0 2024-04-25T15:25:37.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.204+0000 7f008e34d640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00881372f0 con 0x7f00880cf5b0 2024-04-25T15:25:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.206+0000 7f0084ff9640 1 -- 172.21.15.67:0/3944265453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007800cb10 con 0x7f00880cf5b0 2024-04-25T15:25:37.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.206+0000 7f0084ff9640 1 -- 172.21.15.67:0/3944265453 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f007801c070 con 0x7f00880cf5b0 2024-04-25T15:25:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.207+0000 7f0084ff9640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f00740745e0 0x7f0074076af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.207+0000 7f0084ff9640 1 -- 172.21.15.67:0/3944265453 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0078099060 con 0x7f00880cf5b0 2024-04-25T15:25:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.209+0000 7f008e34d640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0068005290 con 0x7f00880cf5b0 2024-04-25T15:25:37.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.211+0000 7f0087fff640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f00740745e0 0x7f0074076af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.219+0000 7f0084ff9640 1 -- 172.21.15.67:0/3944265453 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f007805e4d0 con 0x7f00880cf5b0 2024-04-25T15:25:37.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.233+0000 7f0087fff640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f00740745e0 0x7f0074076af0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f00880d0b50 tx=0x7f007c006b30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.239+0000 7fdd0de92640 1 Processor -- start 2024-04-25T15:25:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.240+0000 7fdd0de92640 1 -- start start 2024-04-25T15:25:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd0de92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd080cd050 0x7fdd080b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd0de92640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd0815d400 0x7fdd080b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd0de92640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd080b80b0 con 0x7fdd080b7ca0 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd0de92640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdd08163e40 con 0x7fdd0815d400 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd0de92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd080d0010 con 0x7fdd080cd050 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd06ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd0815d400 0x7fdd080b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd06ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd0815d400 0x7fdd080b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40958/0 (socket says 172.21.15.67:40958) 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.242+0000 7fdd06ffd640 1 -- 172.21.15.67:0/2538085027 learned_addr learned my addr 172.21.15.67:0/2538085027 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.243+0000 7fdd077fe640 1 --2- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd080cd050 0x7fdd080b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.243+0000 7fdd06ffd640 1 -- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd080cd050 msgr2=0x7fdd080b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.243+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd080cd050 0x7fdd080b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.243+0000 7fdd06ffd640 1 -- 172.21.15.67:0/2538085027 >> v1:172.21.15.176:6789/0 conn(0x7fdd080b7ca0 legacy=0x7fdd0806e250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:25:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.243+0000 7fdd06ffd640 1 -- 172.21.15.67:0/2538085027 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd080d02f0 con 0x7fdd0815d400 2024-04-25T15:25:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.244+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd0815d400 0x7fdd080b7760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd080b8ab0 tx=0x7fdd0002a6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9494ff6331c42137 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.244+0000 7fdd067fc640 1 -- 172.21.15.67:0/2538085027 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00036070 con 0x7fdd0815d400 2024-04-25T15:25:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.245+0000 7fdd067fc640 1 -- 172.21.15.67:0/2538085027 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd00031600 con 0x7fdd0815d400 2024-04-25T15:25:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.245+0000 7fdd067fc640 1 -- 172.21.15.67:0/2538085027 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd000319b0 con 0x7fdd0815d400 2024-04-25T15:25:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.246+0000 7fdd0de92640 1 -- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd0815d400 msgr2=0x7fdd080b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.246+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd0815d400 0x7fdd080b7760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd080b8ab0 tx=0x7fdd0002a6d0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.246+0000 7fdd0de92640 1 -- 172.21.15.67:0/2538085027 shutdown_connections 2024-04-25T15:25:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.246+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd0815d400 0x7fdd080b7760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.246+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2538085027 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd080cd050 0x7fdd080b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.246+0000 7fdd0de92640 1 -- 172.21.15.67:0/2538085027 >> 172.21.15.67:0/2538085027 conn(0x7fdd0809b7d0 msgr2=0x7fdd0809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.246+0000 7fdd0de92640 1 -- 172.21.15.67:0/2538085027 shutdown_connections 2024-04-25T15:25:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.247+0000 7fdd0de92640 1 -- 172.21.15.67:0/2538085027 wait complete. 2024-04-25T15:25:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.249+0000 7fdd0de92640 1 Processor -- start 2024-04-25T15:25:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.250+0000 7fdd0de92640 1 -- start start 2024-04-25T15:25:37.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.250+0000 7f8894e61640 1 Processor -- start 2024-04-25T15:25:37.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.250+0000 7f8894e61640 1 -- start start 2024-04-25T15:25:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7fdd0de92640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd080b7ca0 0x7fdd080d2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7fdd0de92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd080cd050 0x7fdd080d3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7fdd0de92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0815d400 0x7fdd080b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7fdd0de92640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0806dd30 con 0x7fdd080b7ca0 2024-04-25T15:25:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7fdd0de92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0806da50 con 0x7fdd080cd050 2024-04-25T15:25:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7fdd0de92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd080b2920 con 0x7fdd0815d400 2024-04-25T15:25:37.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7efeac420640 1 Processor -- start 2024-04-25T15:25:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd080b7ca0 0x7fdd080d2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd080b7ca0 0x7fdd080d2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37396/0 (socket says 172.21.15.67:37396) 2024-04-25T15:25:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 -- 172.21.15.67:0/2160905634 learned_addr learned my addr 172.21.15.67:0/2160905634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7f8894e61640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88900701b0 0x7f889015c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd080cd050 0x7fdd080d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7f8894e61640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f889006c5c0 0x7f88900cd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7f888e59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88900701b0 0x7f889015c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7f888e59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88900701b0 0x7f889015c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36930/0 (socket says 172.21.15.67:36930) 2024-04-25T15:25:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7f888e59c640 1 -- 172.21.15.67:0/3991328599 learned_addr learned my addr 172.21.15.67:0/3991328599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd07fff640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0815d400 0x7fdd080b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 -- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0815d400 msgr2=0x7fdd080b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0815d400 0x7fdd080b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 -- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd080cd050 msgr2=0x7fdd080d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd080cd050 0x7fdd080d3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.253+0000 7fdd077fe640 1 -- 172.21.15.67:0/2160905634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd08163350 con 0x7fdd080b7ca0 2024-04-25T15:25:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.254+0000 7f8887fff640 1 --2- 172.21.15.67:0/3991328599 >> v2:172.21.15.176:3300/0 conn(0x7f889006c5c0 0x7f88900cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.254+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8890070e60 con 0x7f8890069820 2024-04-25T15:25:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.254+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f88900cd4e0 con 0x7f889006c5c0 2024-04-25T15:25:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.254+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8890070990 con 0x7f88900701b0 2024-04-25T15:25:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.252+0000 7efeac420640 1 -- start start 2024-04-25T15:25:37.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.254+0000 7f888dd9b640 1 -- 172.21.15.67:0/3991328599 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783726942 0 0) 0x7f8890070e60 con 0x7f8890069820 2024-04-25T15:25:37.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.254+0000 7fdd077fe640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd080b7ca0 0x7fdd080d2ad0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c7f0 tx=0x7fdcf800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.254+0000 7f888dd9b640 1 -- 172.21.15.67:0/3991328599 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f887c005730 con 0x7f8890069820 2024-04-25T15:25:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.255+0000 7f888e59c640 1 -- 172.21.15.67:0/3991328599 >> v2:172.21.15.176:3300/0 conn(0x7f889006c5c0 msgr2=0x7f88900cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.255+0000 7f888e59c640 1 --2- 172.21.15.67:0/3991328599 >> v2:172.21.15.176:3300/0 conn(0x7f889006c5c0 0x7f88900cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.255+0000 7f888e59c640 1 -- 172.21.15.67:0/3991328599 >> v1:172.21.15.67:6789/0 conn(0x7f8890069820 legacy=0x7f88900cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.255+0000 7f8887fff640 1 --2- 172.21.15.67:0/3991328599 >> v2:172.21.15.176:3300/0 conn(0x7f889006c5c0 0x7f88900cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.255+0000 7f888e59c640 1 -- 172.21.15.67:0/3991328599 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88900706b0 con 0x7f88900701b0 2024-04-25T15:25:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.255+0000 7fdceffff640 1 -- 172.21.15.67:0/2160905634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8004230 con 0x7fdd080b7ca0 2024-04-25T15:25:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd08163630 con 0x7fdd080b7ca0 2024-04-25T15:25:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7f888e59c640 1 --2- 172.21.15.67:0/3991328599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88900701b0 0x7f889015c230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f888000af00 tx=0x7f888000abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46bb7ad8620f5996 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0815fbc0 con 0x7fdd080b7ca0 2024-04-25T15:25:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7efeac420640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7efea41692b0 0x7efea40b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7efeac420640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea40d2bb0 con 0x7efea406d8e0 2024-04-25T15:25:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7efeac420640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea40d28d0 con 0x7efea40cf5f0 2024-04-25T15:25:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7efeac420640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7efea40ceb10 con 0x7efea41692b0 2024-04-25T15:25:37.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7f2fa37c5640 1 Processor -- start 2024-04-25T15:25:37.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7f10b9f06640 1 Processor -- start 2024-04-25T15:25:37.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7efea99bb640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efea406d8e0 0x7efea406dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59486/0 (socket says 172.21.15.67:59486) 2024-04-25T15:25:37.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7efea99bb640 1 -- 172.21.15.67:0/1201750428 learned_addr learned my addr 172.21.15.67:0/1201750428 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:37.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7efeaa1bc640 1 --2- 172.21.15.67:0/1201750428 >> v2:172.21.15.176:3300/0 conn(0x7efea41692b0 0x7efea40b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7f888dd9b640 1 -- 172.21.15.67:0/3991328599 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888003a070 con 0x7f88900701b0 2024-04-25T15:25:37.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7f888dd9b640 1 -- 172.21.15.67:0/3991328599 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8880008df0 con 0x7f88900701b0 2024-04-25T15:25:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7f617da42640 1 Processor -- start 2024-04-25T15:25:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7f888dd9b640 1 -- 172.21.15.67:0/3991328599 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888000a2f0 con 0x7f88900701b0 2024-04-25T15:25:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7f10b9f06640 1 -- start start 2024-04-25T15:25:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.258+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88900701b0 msgr2=0x7f889015c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.258+0000 7f8894e61640 1 --2- 172.21.15.67:0/3991328599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88900701b0 0x7f889015c230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f888000af00 tx=0x7f888000abe0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.258+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467731216 0 0) 0x7efea40d2bb0 con 0x7efea406d8e0 2024-04-25T15:25:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.258+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe90003520 con 0x7efea406d8e0 2024-04-25T15:25:37.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7fdceffff640 1 -- 172.21.15.67:0/2160905634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf80043d0 con 0x7fdd080b7ca0 2024-04-25T15:25:37.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7fdceffff640 1 -- 172.21.15.67:0/2160905634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8007820 con 0x7fdd080b7ca0 2024-04-25T15:25:37.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.257+0000 7f617da42640 1 -- start start 2024-04-25T15:25:37.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 557289847 0 0) 0x7efe90003520 con 0x7efea406d8e0 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 >> v2:172.21.15.176:3300/0 conn(0x7efea41692b0 msgr2=0x7efea40b68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 --2- 172.21.15.67:0/1201750428 >> v2:172.21.15.176:3300/0 conn(0x7efea41692b0 0x7efea40b68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 >> v1:172.21.15.176:6789/0 conn(0x7efea40cf5f0 legacy=0x7efea40cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea40cedf0 con 0x7efea406d8e0 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 shutdown_connections 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f8894e61640 1 --2- 172.21.15.67:0/3991328599 >> v2:172.21.15.176:3300/0 conn(0x7f889006c5c0 0x7f88900cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f8894e61640 1 --2- 172.21.15.67:0/3991328599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88900701b0 0x7f889015c230 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 >> 172.21.15.67:0/3991328599 conn(0x7f88900ce5e0 msgr2=0x7f88900ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 shutdown_connections 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7efe9c0019f0 con 0x7efea406d8e0 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 >> v1:172.21.15.67:6789/0 conn(0x7efea406d8e0 legacy=0x7efea406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe90003a00 0x7efe90003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7efea91ba640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efe90004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7efea91ba640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe90005270 0x7efe9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7efea99bb640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efe90004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efea40ceb10 con 0x7efe90003a00 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efe9000c190 con 0x7efe90004800 2024-04-25T15:25:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7efeaa1bc640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe90003a00 0x7efe90003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efe9000c390 con 0x7efe90005270 2024-04-25T15:25:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7efeaa9bd640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe90005270 0x7efe9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f617da42640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61780cd060 0x7f61780cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f617da42640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f61780719b0 0x7f6178071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f617da42640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6178072300 con 0x7f61780ce920 2024-04-25T15:25:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f617da42640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6178165a80 con 0x7f61780cd060 2024-04-25T15:25:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f617da42640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6178164e90 con 0x7f61780719b0 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.256+0000 7f2fa37c5640 1 -- start start 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa37c5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f9c15d400 0x7f2f9c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa37c5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2f9c0b7ca0 0x7f2f9c06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa1561640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2f9c0cd050 0x7f2f9c0b23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:60386/0 (socket says 172.21.15.67:60386) 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa1561640 1 -- 172.21.15.67:0/2335836176 learned_addr learned my addr 172.21.15.67:0/2335836176 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa1d62640 1 --2- 172.21.15.67:0/2335836176 >> v2:172.21.15.176:3300/0 conn(0x7f2f9c0b7ca0 0x7f2f9c06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f9c0b80b0 con 0x7f2f9c0cd050 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2f9c0d0010 con 0x7f2f9c15d400 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2f9c0d02f0 con 0x7f2f9c0b7ca0 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f2f93fff640 1 -- 172.21.15.67:0/2335836176 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269415323 0 0) 0x7f2f98001110 con 0x7f2f9c0cd050 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f2f93fff640 1 -- 172.21.15.67:0/2335836176 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f88003520 con 0x7f2f9c0cd050 2024-04-25T15:25:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f2fa1d62640 1 -- 172.21.15.67:0/2335836176 >> v2:172.21.15.67:3300/0 conn(0x7f2f9c15d400 msgr2=0x7f2f9c0b7760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f10b9f06640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10b40b5e40 0x7f10b40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f10b9f06640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f10b406c5c0 0x7f10b406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f10b9f06640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b40b6790 con 0x7f10b414ad60 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f10b9f06640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10b40b4af0 con 0x7f10b40b5e40 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.259+0000 7f10b9f06640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f10b40b4dd0 con 0x7f10b406c5c0 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f10b2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10b40b5e40 0x7f10b40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f10b2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10b40b5e40 0x7f10b40b6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37406/0 (socket says 172.21.15.67:37406) 2024-04-25T15:25:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f10b2ffd640 1 -- 172.21.15.67:0/930843574 learned_addr learned my addr 172.21.15.67:0/930843574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f8894e61640 1 -- 172.21.15.67:0/3991328599 wait complete. 2024-04-25T15:25:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f6176ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61780cd060 0x7f61780cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7fdceffff640 1 -- 172.21.15.67:0/2160905634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdcf8007c10 con 0x7fdd080b7ca0 2024-04-25T15:25:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7fdceffff640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdcf4074580 0x7fdcf4076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7fdceffff640 1 -- 172.21.15.67:0/2160905634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdcf8095310 con 0x7fdd080b7ca0 2024-04-25T15:25:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f2fa1d62640 1 --2- 172.21.15.67:0/2335836176 >> v2:172.21.15.67:3300/0 conn(0x7f2f9c15d400 0x7f2f9c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f2fa1d62640 1 -- 172.21.15.67:0/2335836176 >> v1:172.21.15.176:6789/0 conn(0x7f2f9c0cd050 legacy=0x7f2f9c0b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f2fa1d62640 1 -- 172.21.15.67:0/2335836176 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c06d770 con 0x7f2f9c0b7ca0 2024-04-25T15:25:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f6176ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61780cd060 0x7f61780cdda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37440/0 (socket says 172.21.15.67:37440) 2024-04-25T15:25:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f61777fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f61780ce920 0x7f61780ced30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59496/0 (socket says 172.21.15.67:59496) 2024-04-25T15:25:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f61777fe640 1 -- 172.21.15.67:0/4169196417 learned_addr learned my addr 172.21.15.67:0/4169196417 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7efea99bb640 1 -- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe90005270 msgr2=0x7efe9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7efea99bb640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe90005270 0x7efe9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7efea99bb640 1 -- 172.21.15.67:0/1201750428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe90003a00 msgr2=0x7efe90003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7efea99bb640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe90003a00 0x7efe90003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7efea99bb640 1 -- 172.21.15.67:0/1201750428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe90003520 con 0x7efe90004800 2024-04-25T15:25:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdcf4074580 0x7fdcf4076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.260+0000 7f10b3fff640 1 --2- 172.21.15.67:0/930843574 >> v2:172.21.15.176:3300/0 conn(0x7f10b406c5c0 0x7f10b406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f10b2ffd640 1 -- 172.21.15.67:0/930843574 >> v2:172.21.15.176:3300/0 conn(0x7f10b406c5c0 msgr2=0x7f10b406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f10b2ffd640 1 --2- 172.21.15.67:0/930843574 >> v2:172.21.15.176:3300/0 conn(0x7f10b406c5c0 0x7f10b406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f10b2ffd640 1 -- 172.21.15.67:0/930843574 >> v1:172.21.15.176:6789/0 conn(0x7f10b414ad60 legacy=0x7f10b4070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f10b2ffd640 1 -- 172.21.15.67:0/930843574 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b40b50b0 con 0x7f10b40b5e40 2024-04-25T15:25:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.261+0000 7f616e7fc640 1 --2- 172.21.15.67:0/4169196417 >> v2:172.21.15.176:3300/0 conn(0x7f61780719b0 0x7f6178071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f2fa1d62640 1 --2- 172.21.15.67:0/2335836176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0b7ca0 0x7f2f9c06e250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c06fbf0 tx=0x7f2f9402a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1cefcb23126d49e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdcf4074580 0x7fdcf4076a90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd00033000 tx=0x7fdd00009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce8005290 con 0x7fdd080b7ca0 2024-04-25T15:25:37.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7efeaa9bd640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe90005270 0x7efe9000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:37.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7efeaa9bd640 1 -- 172.21.15.67:0/1201750428 reap_dead start 2024-04-25T15:25:37.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f10b2ffd640 1 --2- 172.21.15.67:0/930843574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40b5e40 0x7f10b40b6250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f10b40ce320 tx=0x7f10a802a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9967cdc478d7b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f61767fc640 1 -- 172.21.15.67:0/4169196417 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129239141 0 0) 0x7f6170001490 con 0x7f61780ce920 2024-04-25T15:25:37.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f61767fc640 1 -- 172.21.15.67:0/4169196417 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6158003520 con 0x7f61780ce920 2024-04-25T15:25:37.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f2f93fff640 1 -- 172.21.15.67:0/2335836176 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94007740 con 0x7f2f9c0b7ca0 2024-04-25T15:25:37.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.263+0000 7efea99bb640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efe90004c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00a7b0 tx=0x7efe90005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7386923f41be839a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f6176ffd640 1 -- 172.21.15.67:0/4169196417 >> v2:172.21.15.176:3300/0 conn(0x7f61780719b0 msgr2=0x7f6178071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f6176ffd640 1 --2- 172.21.15.67:0/4169196417 >> v2:172.21.15.176:3300/0 conn(0x7f61780719b0 0x7f6178071dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.263+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00a3d0 con 0x7efe90004800 2024-04-25T15:25:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.263+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9c02f070 con 0x7efe90004800 2024-04-25T15:25:37.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f6176ffd640 1 -- 172.21.15.67:0/4169196417 >> v1:172.21.15.67:6789/0 conn(0x7f61780ce920 legacy=0x7f61780ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.262+0000 7f6176ffd640 1 -- 172.21.15.67:0/4169196417 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6178165150 con 0x7f61780cd060 2024-04-25T15:25:37.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f6176ffd640 1 --2- 172.21.15.67:0/4169196417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cd060 0x7f61780cdda0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f61780d3650 tx=0x7f616802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3d917aa141d267 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7efea91ba640 1 -- 172.21.15.67:0/1201750428 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c02bcd0 con 0x7efe90004800 2024-04-25T15:25:37.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.263+0000 7f2f93fff640 1 -- 172.21.15.67:0/2335836176 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f940078e0 con 0x7f2f9c0b7ca0 2024-04-25T15:25:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.263+0000 7f2f93fff640 1 -- 172.21.15.67:0/2335836176 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94007cf0 con 0x7f2f9c0b7ca0 2024-04-25T15:25:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f8894e61640 1 Processor -- start 2024-04-25T15:25:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0b7ca0 msgr2=0x7f2f9c06e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/2335836176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0b7ca0 0x7f2f9c06e250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c06fbf0 tx=0x7f2f9402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f8894e61640 1 -- start start 2024-04-25T15:25:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7efeac420640 1 -- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 msgr2=0x7efe90004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7efeac420640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efe90004c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00a7b0 tx=0x7efe90005830 comp rx=0 tx=0).stop 2024-04-25T15:25:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 shutdown_connections 2024-04-25T15:25:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/2335836176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0b7ca0 0x7f2f9c06e250 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/2335836176 >> v2:172.21.15.67:3300/0 conn(0x7f2f9c15d400 0x7f2f9c0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 >> 172.21.15.67:0/2335836176 conn(0x7f2f9c09b7d0 msgr2=0x7f2f9c09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7efeac420640 1 -- 172.21.15.67:0/1201750428 shutdown_connections 2024-04-25T15:25:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7efeac420640 1 --2- 172.21.15.67:0/1201750428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efe90004c10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7efeac420640 1 -- 172.21.15.67:0/1201750428 >> 172.21.15.67:0/1201750428 conn(0x7efea40d3aa0 msgr2=0x7efea40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 shutdown_connections 2024-04-25T15:25:37.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.264+0000 7f2fa37c5640 1 -- 172.21.15.67:0/2335836176 wait complete. 2024-04-25T15:25:37.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.266+0000 7efeac420640 1 -- 172.21.15.67:0/1201750428 shutdown_connections 2024-04-25T15:25:37.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.266+0000 7efeac420640 1 -- 172.21.15.67:0/1201750428 wait complete. 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.266+0000 7f8894e61640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8890069820 0x7f88901f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.266+0000 7f8894e61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889006c5c0 0x7f88900c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.266+0000 7f8894e61640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88900701b0 0x7f88900c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f8887fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889006c5c0 0x7f88900c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f8894e61640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f889006c9d0 con 0x7f889006c5c0 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f888e59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8890069820 0x7f88901f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f888e59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8890069820 0x7f88901f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36956/0 (socket says 172.21.15.67:36956) 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f888e59c640 1 -- 172.21.15.67:0/2123438274 learned_addr learned my addr 172.21.15.67:0/2123438274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f88900cd170 con 0x7f88900701b0 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f888ed9d640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88900701b0 0x7f88900c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f889006cb50 con 0x7f8890069820 2024-04-25T15:25:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f2fa37c5640 1 Processor -- start 2024-04-25T15:25:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.267+0000 7f2fa37c5640 1 -- start start 2024-04-25T15:25:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7efeac420640 1 Processor -- start 2024-04-25T15:25:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7efeac420640 1 -- start start 2024-04-25T15:25:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f888ed9d640 1 -- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8890069820 msgr2=0x7f88901f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f888ed9d640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8890069820 0x7f88901f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f888ed9d640 1 -- 172.21.15.67:0/2123438274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889006c5c0 msgr2=0x7f88900c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f888ed9d640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889006c5c0 0x7f88900c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f888ed9d640 1 -- 172.21.15.67:0/2123438274 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88900ca340 con 0x7f88900701b0 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f888e59c640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8890069820 0x7f88901f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f8887fff640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889006c5c0 0x7f88900c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f888ed9d640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88900701b0 0x7f88900c7de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88900c8be0 tx=0x7f888800cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f8886ffd640 1 -- 172.21.15.67:0/2123438274 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888003b70 con 0x7f88900701b0 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f8886ffd640 1 -- 172.21.15.67:0/2123438274 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8888003d30 con 0x7f88900701b0 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.268+0000 7f8886ffd640 1 -- 172.21.15.67:0/2123438274 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888005310 con 0x7f88900701b0 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.269+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88900ca620 con 0x7f88900701b0 2024-04-25T15:25:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.269+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88900cabe0 con 0x7f88900701b0 2024-04-25T15:25:37.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.269+0000 7efeac420640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efea4154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.269+0000 7efeac420640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea4163630 0x7efea41651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.269+0000 7f2fa37c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9c0b7ca0 0x7f2f9c0d3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efeac420640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efea4163a40 0x7efea4165720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa37c5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0cd050 0x7f2f9c0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efeac420640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efea40c7350 con 0x7efea4163630 2024-04-25T15:25:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efeac420640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efea40c71d0 con 0x7efe90004800 2024-04-25T15:25:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efeac420640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efea40d3350 con 0x7efea4163a40 2024-04-25T15:25:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea4163630 0x7efea41651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa1561640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9c0b7ca0 0x7f2f9c0d3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa1561640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9c0b7ca0 0x7f2f9c0d3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36970/0 (socket says 172.21.15.67:36970) 2024-04-25T15:25:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea4163630 0x7efea41651e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37448/0 (socket says 172.21.15.67:37448) 2024-04-25T15:25:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 -- 172.21.15.67:0/1794756113 learned_addr learned my addr 172.21.15.67:0/1794756113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa37c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c15d400 0x7f2f9c14cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa0d60640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0cd050 0x7f2f9c0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efeaa9bd640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efea4163a40 0x7efea4165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa0d60640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0cd050 0x7f2f9c0d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:41032/0 (socket says 172.21.15.67:41032) 2024-04-25T15:25:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa1561640 1 -- 172.21.15.67:0/3828576436 learned_addr learned my addr 172.21.15.67:0/3828576436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7f2fa1d62640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c15d400 0x7f2f9c14cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 -- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efea4163a40 msgr2=0x7efea4165720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efea4163a40 0x7efea4165720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 -- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 msgr2=0x7efea4154da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:25:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efea4154da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.270+0000 7efea99bb640 1 -- 172.21.15.67:0/1794756113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea40cd430 con 0x7efea4163630 2024-04-25T15:25:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c0cfc90 con 0x7f2f9c15d400 2024-04-25T15:25:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c06dd30 con 0x7f2f9c0cd050 2024-04-25T15:25:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c0b40d0 con 0x7f2f9c0b7ca0 2024-04-25T15:25:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f889006fd60 con 0x7f88900701b0 2024-04-25T15:25:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7efea99bb640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea4163630 0x7efea41651e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00a8d0 tx=0x7efe9c001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f8886ffd640 1 -- 172.21.15.67:0/2123438274 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f88880054b0 con 0x7f88900701b0 2024-04-25T15:25:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa0d60640 1 -- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9c0b7ca0 msgr2=0x7f2f9c0d3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa0d60640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9c0b7ca0 0x7f2f9c0d3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa0d60640 1 -- 172.21.15.67:0/3828576436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c15d400 msgr2=0x7f2f9c14cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa0d60640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c15d400 0x7f2f9c14cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa0d60640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c0d1620 con 0x7f2f9c0cd050 2024-04-25T15:25:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa1d62640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c15d400 0x7f2f9c14cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.271+0000 7f2fa1561640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9c0b7ca0 0x7f2f9c0d3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:37.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7f2fa0d60640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0cd050 0x7f2f9c0d26c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00b3e0 tx=0x7f2f8c00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7f8886ffd640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88700744e0 0x7f88700769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7f8886ffd640 1 -- 172.21.15.67:0/2123438274 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8888094ea0 con 0x7f88900701b0 2024-04-25T15:25:37.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7fdceffff640 1 -- 172.21.15.67:0/2160905634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdcf8015020 con 0x7fdd080b7ca0 2024-04-25T15:25:37.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7f10b27fc640 1 -- 172.21.15.67:0/930843574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8037070 con 0x7f10b40b5e40 2024-04-25T15:25:37.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7f61767fc640 1 -- 172.21.15.67:0/4169196417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6168037070 con 0x7f61780cd060 2024-04-25T15:25:37.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7efe9affd640 1 -- 172.21.15.67:0/1794756113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c044070 con 0x7efea4163630 2024-04-25T15:25:37.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea41632c0 con 0x7efea4163630 2024-04-25T15:25:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7f888e59c640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88700744e0 0x7f88700769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efea40cc670 con 0x7efea4163630 2024-04-25T15:25:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.272+0000 7f10b27fc640 1 -- 172.21.15.67:0/930843574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a8032440 con 0x7f10b40b5e40 2024-04-25T15:25:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.273+0000 7f10b27fc640 1 -- 172.21.15.67:0/930843574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8032810 con 0x7f10b40b5e40 2024-04-25T15:25:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.273+0000 7f61767fc640 1 -- 172.21.15.67:0/4169196417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6168032440 con 0x7f61780cd060 2024-04-25T15:25:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.273+0000 7f888e59c640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88700744e0 0x7f88700769f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f88800081b0 tx=0x7f888000a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.273+0000 7f10b9f06640 1 -- 172.21.15.67:0/930843574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40b5e40 msgr2=0x7f10b40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.273+0000 7efe9affd640 1 -- 172.21.15.67:0/1794756113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9c02f020 con 0x7efea4163630 2024-04-25T15:25:37.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.273+0000 7f10b9f06640 1 --2- 172.21.15.67:0/930843574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40b5e40 0x7f10b40b6250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f10b40ce320 tx=0x7f10a802a1f0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f10b9f06640 1 -- 172.21.15.67:0/930843574 shutdown_connections 2024-04-25T15:25:37.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f10b9f06640 1 --2- 172.21.15.67:0/930843574 >> v2:172.21.15.176:3300/0 conn(0x7f10b406c5c0 0x7f10b406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f10b9f06640 1 --2- 172.21.15.67:0/930843574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40b5e40 0x7f10b40b6250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f10b9f06640 1 -- 172.21.15.67:0/930843574 >> 172.21.15.67:0/930843574 conn(0x7f10b40d1ae0 msgr2=0x7f10b40d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f10b9f06640 1 -- 172.21.15.67:0/930843574 shutdown_connections 2024-04-25T15:25:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7efe9affd640 1 -- 172.21.15.67:0/1794756113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00a3d0 con 0x7efea4163630 2024-04-25T15:25:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe8c005290 con 0x7efea4163630 2024-04-25T15:25:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.273+0000 7f61767fc640 1 -- 172.21.15.67:0/4169196417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6168037070 con 0x7f61780cd060 2024-04-25T15:25:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 -- 172.21.15.67:0/4169196417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cd060 msgr2=0x7f61780cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 --2- 172.21.15.67:0/4169196417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cd060 0x7f61780cdda0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f61780d3650 tx=0x7f616802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 -- 172.21.15.67:0/4169196417 shutdown_connections 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 --2- 172.21.15.67:0/4169196417 >> v2:172.21.15.176:3300/0 conn(0x7f61780719b0 0x7f6178071dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 --2- 172.21.15.67:0/4169196417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cd060 0x7f61780cdda0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 -- 172.21.15.67:0/4169196417 >> 172.21.15.67:0/4169196417 conn(0x7f61780d2a10 msgr2=0x7f61780d2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 -- 172.21.15.67:0/4169196417 shutdown_connections 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f617da42640 1 -- 172.21.15.67:0/4169196417 wait complete. 2024-04-25T15:25:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.274+0000 7f10b9f06640 1 -- 172.21.15.67:0/930843574 wait complete. 2024-04-25T15:25:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.276+0000 7efe9affd640 1 -- 172.21.15.67:0/1794756113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efe9c047070 con 0x7efea4163630 2024-04-25T15:25:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.277+0000 7efe9affd640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efe880744e0 0x7efe880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.277+0000 7efe9affd640 1 -- 172.21.15.67:0/1794756113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efe9c0b5ff0 con 0x7efea4163630 2024-04-25T15:25:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.277+0000 7efeaa1bc640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efe880744e0 0x7efe880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.277+0000 7f617da42640 1 Processor -- start 2024-04-25T15:25:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.277+0000 7f10b9f06640 1 Processor -- start 2024-04-25T15:25:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.278+0000 7efeaa1bc640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efe880744e0 0x7efe880769f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efea4074aa0 tx=0x7efea0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.277+0000 7f617da42640 1 -- start start 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.278+0000 7f10b9f06640 1 -- start start 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f10b9f06640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10b406c5c0 0x7f10b41fda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f617da42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61780719b0 0x7f617815b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f617da42640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61780cd060 0x7f6178166f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f617da42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780ce920 0x7f61781674d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f617da42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6178168390 con 0x7f61780ce920 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f617da42640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6178165710 con 0x7f61780cd060 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f617da42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6178168510 con 0x7f61780719b0 2024-04-25T15:25:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f6176ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61780719b0 0x7f617815b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f6176ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61780719b0 0x7f617815b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36980/0 (socket says 172.21.15.67:36980) 2024-04-25T15:25:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f6176ffd640 1 -- 172.21.15.67:0/1872558396 learned_addr learned my addr 172.21.15.67:0/1872558396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.277+0000 7f2f91ffb640 1 -- 172.21.15.67:0/3828576436 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c015020 con 0x7f2f9c0cd050 2024-04-25T15:25:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.278+0000 7f2f91ffb640 1 -- 172.21.15.67:0/3828576436 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8c011830 con 0x7f2f9c0cd050 2024-04-25T15:25:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.278+0000 7f2f91ffb640 1 -- 172.21.15.67:0/3828576436 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c011c20 con 0x7f2f9c0cd050 2024-04-25T15:25:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.279+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9c0d1900 con 0x7f2f9c0cd050 2024-04-25T15:25:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.279+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9c0d1be0 con 0x7f2f9c0cd050 2024-04-25T15:25:37.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f61777fe640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780ce920 0x7f61781674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.280+0000 7f10b9f06640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10b40b5e40 0x7f10b4148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10b406c5c0 0x7f10b41fda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10b406c5c0 0x7f10b41fda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:36984/0 (socket says 172.21.15.67:36984) 2024-04-25T15:25:37.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f10b37fe640 1 -- 172.21.15.67:0/2960586008 learned_addr learned my addr 172.21.15.67:0/2960586008 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f10b9f06640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b414ad60 0x7f10b41491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f616e7fc640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61780cd060 0x7f6178166f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f61777fe640 1 -- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61780719b0 msgr2=0x7f617815b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f61777fe640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61780719b0 0x7f617815b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f61777fe640 1 -- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61780cd060 msgr2=0x7f6178166f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10b406fb80 con 0x7f10b414ad60 2024-04-25T15:25:37.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10b40b5670 con 0x7f10b40b5e40 2024-04-25T15:25:37.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f10b406fd00 con 0x7f10b406c5c0 2024-04-25T15:25:37.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.281+0000 7f61777fe640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61780cd060 0x7f6178166f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.282+0000 7f61777fe640 1 -- 172.21.15.67:0/1872558396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617815a470 con 0x7f61780ce920 2024-04-25T15:25:37.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.282+0000 7f616e7fc640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61780cd060 0x7f6178166f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:37.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.282+0000 7f61777fe640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780ce920 0x7f61781674d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6178073fc0 tx=0x7f617000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.282+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f9c050740 con 0x7f2f9c0cd050 2024-04-25T15:25:37.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.279+0000 7f8886ffd640 1 -- 172.21.15.67:0/2123438274 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f888805f6b0 con 0x7f88900701b0 2024-04-25T15:25:37.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.282+0000 7f10b3fff640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b414ad60 0x7f10b41491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.282+0000 7f10b2ffd640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10b40b5e40 0x7f10b4148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f2f91ffb640 1 -- 172.21.15.67:0/3828576436 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2f8c018020 con 0x7f2f9c0cd050 2024-04-25T15:25:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b37fe640 1 -- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10b40b5e40 msgr2=0x7f10b4148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b37fe640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10b40b5e40 0x7f10b4148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b37fe640 1 -- 172.21.15.67:0/2960586008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b414ad60 msgr2=0x7f10b41491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b37fe640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b414ad60 0x7f10b41491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b37fe640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b41fe0f0 con 0x7f10b406c5c0 2024-04-25T15:25:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b3fff640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b414ad60 0x7f10b41491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b37fe640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10b406c5c0 0x7f10b41fda40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00c760 tx=0x7f10ac00cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f2f91ffb640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800745b0 0x7f2f80076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f109bfff640 1 -- 172.21.15.67:0/2960586008 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac003b90 con 0x7f10b406c5c0 2024-04-25T15:25:37.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f2f91ffb640 1 -- 172.21.15.67:0/3828576436 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2f8c095030 con 0x7f2f9c0cd050 2024-04-25T15:25:37.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f2fa1561640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800745b0 0x7f2f80076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f109bfff640 1 -- 172.21.15.67:0/2960586008 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ac003d30 con 0x7f10b406c5c0 2024-04-25T15:25:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.283+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b41ed0c0 con 0x7f10b406c5c0 2024-04-25T15:25:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.284+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b41ed680 con 0x7f10b406c5c0 2024-04-25T15:25:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.284+0000 7f2fa1561640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800745b0 0x7f2f80076ac0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f98005880 tx=0x7f2f98007b90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.284+0000 7f6174ff9640 1 -- 172.21.15.67:0/1872558396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170017070 con 0x7f61780ce920 2024-04-25T15:25:37.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.285+0000 7f6174ff9640 1 -- 172.21.15.67:0/1872558396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617000ab10 con 0x7f61780ce920 2024-04-25T15:25:37.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.285+0000 7f109bfff640 1 -- 172.21.15.67:0/2960586008 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00aa90 con 0x7f10b406c5c0 2024-04-25T15:25:37.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.285+0000 7f6174ff9640 1 -- 172.21.15.67:0/1872558396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170012400 con 0x7f61780ce920 2024-04-25T15:25:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.285+0000 7efe9affd640 1 -- 172.21.15.67:0/1794756113 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe9c04e070 con 0x7efea4163630 2024-04-25T15:25:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.285+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617815a750 con 0x7f61780ce920 2024-04-25T15:25:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.285+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f617815ad10 con 0x7f61780ce920 2024-04-25T15:25:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.287+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6178050740 con 0x7f61780ce920 2024-04-25T15:25:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.287+0000 7f109bfff640 1 -- 172.21.15.67:0/2960586008 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f10ac00ac30 con 0x7f10b406c5c0 2024-04-25T15:25:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.287+0000 7f109bfff640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10a00744b0 0x7f10a00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.287+0000 7f109bfff640 1 -- 172.21.15.67:0/2960586008 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f10ac095010 con 0x7f10b406c5c0 2024-04-25T15:25:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.288+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1094005290 con 0x7f10b406c5c0 2024-04-25T15:25:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.288+0000 7f10b2ffd640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10a00744b0 0x7f10a00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.288+0000 7f2f91ffb640 1 -- 172.21.15.67:0/3828576436 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f8c01f020 con 0x7f2f9c0cd050 2024-04-25T15:25:37.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.288+0000 7f10b2ffd640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10a00744b0 0x7f10a00769c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f10a8034000 tx=0x7f10a802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.289+0000 7f6174ff9640 1 -- 172.21.15.67:0/1872558396 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f617001a070 con 0x7f61780ce920 2024-04-25T15:25:37.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.289+0000 7f6174ff9640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f615c0744b0 0x7f615c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.289+0000 7f6176ffd640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f615c0744b0 0x7f615c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.290+0000 7f6174ff9640 1 -- 172.21.15.67:0/1872558396 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6170094e20 con 0x7f61780ce920 2024-04-25T15:25:37.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.290+0000 7f6176ffd640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f615c0744b0 0x7f615c0769c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6168009560 tx=0x7f616802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.293+0000 7f109bfff640 1 -- 172.21.15.67:0/2960586008 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10b40b6790 con 0x7f10b406c5c0 2024-04-25T15:25:37.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.295+0000 7f9ae9a34640 1 Processor -- start 2024-04-25T15:25:37.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.296+0000 7f9ae9a34640 1 -- start start 2024-04-25T15:25:37.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.298+0000 7f9ae9a34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae4164ae0 0x7f9ae40c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.298+0000 7f9ae9a34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ae40c5f10 0x7f9ae40b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.298+0000 7f9ae9a34640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae40b3b60 con 0x7f9ae41646d0 2024-04-25T15:25:37.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.298+0000 7f9ae9a34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9ae40cf280 con 0x7f9ae40c5f10 2024-04-25T15:25:37.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.298+0000 7f9ae9a34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae40d00e0 con 0x7f9ae4164ae0 2024-04-25T15:25:37.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.299+0000 7f6174ff9640 1 -- 172.21.15.67:0/1872558396 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6178072300 con 0x7f61780ce920 2024-04-25T15:25:37.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.299+0000 7f9ae37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ae40c5f10 0x7f9ae40b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.299+0000 7f9ae37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ae40c5f10 0x7f9ae40b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37474/0 (socket says 172.21.15.67:37474) 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.299+0000 7f9ae37fe640 1 -- 172.21.15.67:0/2043657561 learned_addr learned my addr 172.21.15.67:0/2043657561 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.300+0000 7f9ae27fc640 1 --2- 172.21.15.67:0/2043657561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae4164ae0 0x7f9ae40c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.300+0000 7f9ae37fe640 1 -- 172.21.15.67:0/2043657561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae4164ae0 msgr2=0x7f9ae40c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.300+0000 7f9ae37fe640 1 --2- 172.21.15.67:0/2043657561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae4164ae0 0x7f9ae40c59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.300+0000 7f9ae37fe640 1 -- 172.21.15.67:0/2043657561 >> v1:172.21.15.176:6789/0 conn(0x7f9ae41646d0 legacy=0x7f9ae40b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.300+0000 7f9ae37fe640 1 -- 172.21.15.67:0/2043657561 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae40c6320 con 0x7f9ae40c5f10 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.302+0000 7f9ae27fc640 1 --2- 172.21.15.67:0/2043657561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae4164ae0 0x7f9ae40c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.302+0000 7f9ae37fe640 1 --2- 172.21.15.67:0/2043657561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae40c5f10 0x7f9ae40b1870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009e50 tx=0x7f9adc02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcbcba806229606 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.302+0000 7f9ae1ffb640 1 -- 172.21.15.67:0/2043657561 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc036070 con 0x7f9ae40c5f10 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.302+0000 7f9ae1ffb640 1 -- 172.21.15.67:0/2043657561 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc031600 con 0x7f9ae40c5f10 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.302+0000 7f9ae1ffb640 1 -- 172.21.15.67:0/2043657561 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc0319b0 con 0x7f9ae40c5f10 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.304+0000 7f9ae9a34640 1 -- 172.21.15.67:0/2043657561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae40c5f10 msgr2=0x7f9ae40b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.304+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/2043657561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae40c5f10 0x7f9ae40b1870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009e50 tx=0x7f9adc02a730 comp rx=0 tx=0).stop 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.304+0000 7f9ae9a34640 1 -- 172.21.15.67:0/2043657561 shutdown_connections 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.304+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/2043657561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae40c5f10 0x7f9ae40b1870 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.304+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/2043657561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae4164ae0 0x7f9ae40c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.304+0000 7f9ae9a34640 1 -- 172.21.15.67:0/2043657561 >> 172.21.15.67:0/2043657561 conn(0x7f9ae409b7d0 msgr2=0x7f9ae409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.305+0000 7f9ae9a34640 1 -- 172.21.15.67:0/2043657561 shutdown_connections 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.305+0000 7f9ae9a34640 1 -- 172.21.15.67:0/2043657561 wait complete. 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.308+0000 7f9ae9a34640 1 Processor -- start 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.308+0000 7f9ae9a34640 1 -- start start 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.311+0000 7f9ae9a34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c5f10 0x7f9ae4203260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.311+0000 7f9ae9a34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae41646d0 0x7f9ae40c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.311+0000 7f9ae9a34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae4164ae0 0x7f9ae4162f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.311+0000 7f9ae9a34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae40cfa90 con 0x7f9ae4164ae0 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.311+0000 7f9ae37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae4164ae0 0x7f9ae4162f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.312+0000 7f9ae37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae4164ae0 0x7f9ae4162f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37480/0 (socket says 172.21.15.67:37480) 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.312+0000 7f9ae37fe640 1 -- 172.21.15.67:0/696880197 learned_addr learned my addr 172.21.15.67:0/696880197 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.312+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae40cf7b0 con 0x7f9ae40c5f10 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.312+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae40cfd70 con 0x7f9ae41646d0 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.312+0000 7f9ae2ffd640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c5f10 0x7f9ae4203260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.312+0000 7f9ae27fc640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae41646d0 0x7f9ae40c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.313+0000 7f9ae37fe640 1 -- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae41646d0 msgr2=0x7f9ae40c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.313+0000 7f9ae37fe640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae41646d0 0x7f9ae40c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.313+0000 7f9ae37fe640 1 -- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c5f10 msgr2=0x7f9ae4203260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.313+0000 7f9ae37fe640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c5f10 0x7f9ae4203260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.313+0000 7f9ae37fe640 1 -- 172.21.15.67:0/696880197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae40c91c0 con 0x7f9ae4164ae0 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.314+0000 7f9ae37fe640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae4164ae0 0x7f9ae4162f10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009f70 tx=0x7f9adc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.314+0000 7f9ae2ffd640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c5f10 0x7f9ae4203260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.315+0000 7f9acbfff640 1 -- 172.21.15.67:0/696880197 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc036050 con 0x7f9ae4164ae0 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.315+0000 7f9acbfff640 1 -- 172.21.15.67:0/696880197 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc042070 con 0x7f9ae4164ae0 2024-04-25T15:25:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.315+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae40c6750 con 0x7f9ae4164ae0 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.315+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae40c6d10 con 0x7f9ae4164ae0 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.315+0000 7f9acbfff640 1 -- 172.21.15.67:0/696880197 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc009510 con 0x7f9ae4164ae0 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.319+0000 7f9acbfff640 1 -- 172.21.15.67:0/696880197 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9adc004450 con 0x7f9ae4164ae0 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.319+0000 7f9acbfff640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ad0074580 0x7f9ad0076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.319+0000 7f9acbfff640 1 -- 172.21.15.67:0/696880197 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9adc0b5f60 con 0x7f9ae4164ae0 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.320+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac4005290 con 0x7f9ae4164ae0 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.324+0000 7f9ae2ffd640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ad0074580 0x7f9ad0076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.331+0000 7f9acbfff640 1 -- 172.21.15.67:0/696880197 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9adc0e4b80 con 0x7f9ae4164ae0 2024-04-25T15:25:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.346+0000 7f9ae2ffd640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ad0074580 0x7f9ad0076a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8000f60 tx=0x7f9ad8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.646+0000 7f008e34d640 1 -- 172.21.15.67:0/3944265453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0068005850 con 0x7f00880cf5b0 2024-04-25T15:25:37.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.647+0000 7f0084ff9640 1 -- 172.21.15.67:0/3944265453 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f00780627d0 con 0x7f00880cf5b0 2024-04-25T15:25:37.648 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:25:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.650+0000 7f006e7fc640 1 -- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f00740745e0 msgr2=0x7f0074076af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.651+0000 7f006e7fc640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f00740745e0 0x7f0074076af0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f00880d0b50 tx=0x7f007c006b30 comp rx=0 tx=0).stop 2024-04-25T15:25:37.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.651+0000 7f006e7fc640 1 -- 172.21.15.67:0/3944265453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880cf5b0 msgr2=0x7f0088137d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.651+0000 7f006e7fc640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880cf5b0 0x7f0088137d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f007800b700 tx=0x7f007800bbc0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.651+0000 7f006e7fc640 1 -- 172.21.15.67:0/3944265453 shutdown_connections 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.651+0000 7f006e7fc640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f00740745e0 0x7f0074076af0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.651+0000 7f006e7fc640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00880cf5b0 0x7f0088137d50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.651+0000 7f006e7fc640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00880b6990 0x7f0088137810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.652+0000 7f006e7fc640 1 --2- 172.21.15.67:0/3944265453 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0088073b70 0x7f008814a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.652+0000 7f006e7fc640 1 -- 172.21.15.67:0/3944265453 >> 172.21.15.67:0/3944265453 conn(0x7f0088073050 msgr2=0x7f008815f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.652+0000 7f006e7fc640 1 -- 172.21.15.67:0/3944265453 shutdown_connections 2024-04-25T15:25:37.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.652+0000 7f006e7fc640 1 -- 172.21.15.67:0/3944265453 wait complete. 2024-04-25T15:25:37.671 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-04-25T15:25:37.671 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.686+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fdce8005580 con 0x7fdd080b7ca0 2024-04-25T15:25:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.687+0000 7fdceffff640 1 -- 172.21.15.67:0/2160905634 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdcf805f430 con 0x7fdd080b7ca0 2024-04-25T15:25:37.688 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-04-25T15:25:37.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdcf4074580 msgr2=0x7fdcf4076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdcf4074580 0x7fdcf4076a90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd00033000 tx=0x7fdd00009000 comp rx=0 tx=0).stop 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd080b7ca0 msgr2=0x7fdd080d2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd080b7ca0 0x7fdd080d2ad0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c7f0 tx=0x7fdcf800ccb0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 shutdown_connections 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdcf4074580 0x7fdcf4076a90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0815d400 0x7fdd080b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd080cd050 0x7fdd080d3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 --2- 172.21.15.67:0/2160905634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd080b7ca0 0x7fdd080d2ad0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.689+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 >> 172.21.15.67:0/2160905634 conn(0x7fdd0809b7d0 msgr2=0x7fdd080b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.690+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 shutdown_connections 2024-04-25T15:25:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.690+0000 7fdd0de92640 1 -- 172.21.15.67:0/2160905634 wait complete. 2024-04-25T15:25:37.702 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-04-25T15:25:37.702 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.714+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f88900b07d0 con 0x7f88900701b0 2024-04-25T15:25:37.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.715+0000 7f8886ffd640 1 -- 172.21.15.67:0/2123438274 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f888805efb0 con 0x7f88900701b0 2024-04-25T15:25:37.716 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:25:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88700744e0 msgr2=0x7f88700769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88700744e0 0x7f88700769f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f88800081b0 tx=0x7f888000a000 comp rx=0 tx=0).stop 2024-04-25T15:25:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88900701b0 msgr2=0x7f88900c7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88900701b0 0x7f88900c7de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88900c8be0 tx=0x7f888800cb90 comp rx=0 tx=0).stop 2024-04-25T15:25:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 shutdown_connections 2024-04-25T15:25:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88700744e0 0x7f88700769f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88900701b0 0x7f88900c7de0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f889006c5c0 0x7f88900c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 --2- 172.21.15.67:0/2123438274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8890069820 0x7f88901f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.717+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 >> 172.21.15.67:0/2123438274 conn(0x7f88900ce5e0 msgr2=0x7f88900b0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.719+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 shutdown_connections 2024-04-25T15:25:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.719+0000 7f8894e61640 1 -- 172.21.15.67:0/2123438274 wait complete. 2024-04-25T15:25:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.719+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7efe8c005850 con 0x7efea4163630 2024-04-25T15:25:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.720+0000 7efe9affd640 1 -- 172.21.15.67:0/1794756113 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efe9c080770 con 0x7efea4163630 2024-04-25T15:25:37.721 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efe880744e0 msgr2=0x7efe880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efe880744e0 0x7efe880769f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efea4074aa0 tx=0x7efea0009190 comp rx=0 tx=0).stop 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea4163630 msgr2=0x7efea41651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea4163630 0x7efea41651e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00a8d0 tx=0x7efe9c001cf0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 shutdown_connections 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efe880744e0 0x7efe880769f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efea4163a40 0x7efea4165720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efea4163630 0x7efea41651e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 --2- 172.21.15.67:0/1794756113 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe90004800 0x7efea4154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.722+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 >> 172.21.15.67:0/1794756113 conn(0x7efea40d3aa0 msgr2=0x7efea40b8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.723+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 shutdown_connections 2024-04-25T15:25:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.723+0000 7efeac420640 1 -- 172.21.15.67:0/1794756113 wait complete. 2024-04-25T15:25:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.727+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f1094005580 con 0x7f10b406c5c0 2024-04-25T15:25:37.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.728+0000 7f109bfff640 1 -- 172.21.15.67:0/2960586008 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10ac05f0b0 con 0x7f10b406c5c0 2024-04-25T15:25:37.729 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:25:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.728+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2f9c0b6dc0 con 0x7f2f9c0cd050 2024-04-25T15:25:37.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.728+0000 7f2f91ffb640 1 -- 172.21.15.67:0/3828576436 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2f8c05f150 con 0x7f2f9c0cd050 2024-04-25T15:25:37.729 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:25:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.731+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10a00744b0 msgr2=0x7f10a00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.731+0000 7f10b9f06640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10a00744b0 0x7f10a00769c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f10a8034000 tx=0x7f10a802e000 comp rx=0 tx=0).stop 2024-04-25T15:25:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.731+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10b406c5c0 msgr2=0x7f10b41fda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.731+0000 7f10b9f06640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10b406c5c0 0x7f10b41fda40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00c760 tx=0x7f10ac00cc20 comp rx=0 tx=0).stop 2024-04-25T15:25:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.731+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 shutdown_connections 2024-04-25T15:25:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800745b0 msgr2=0x7f2f80076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800745b0 0x7f2f80076ac0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f98005880 tx=0x7f2f98007b90 comp rx=0 tx=0).stop 2024-04-25T15:25:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f10b9f06640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10a00744b0 0x7f10a00769c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0cd050 msgr2=0x7f2f9c0d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0cd050 0x7f2f9c0d26c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00b3e0 tx=0x7f2f8c00b8a0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f10b9f06640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b414ad60 0x7f10b41491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 shutdown_connections 2024-04-25T15:25:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800745b0 0x7f2f80076ac0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9c15d400 0x7f2f9c14cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f9c0cd050 0x7f2f9c0d26c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 --2- 172.21.15.67:0/3828576436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9c0b7ca0 0x7f2f9c0d3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 >> 172.21.15.67:0/3828576436 conn(0x7f2f9c09b7d0 msgr2=0x7f2f9c06c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f10b9f06640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10b40b5e40 0x7f10b4148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 shutdown_connections 2024-04-25T15:25:37.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f2fa37c5640 1 -- 172.21.15.67:0/3828576436 wait complete. 2024-04-25T15:25:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f10b9f06640 1 --2- 172.21.15.67:0/2960586008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10b406c5c0 0x7f10b41fda40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.732+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 >> 172.21.15.67:0/2960586008 conn(0x7f10b40d1ae0 msgr2=0x7f10b40b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.733+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 shutdown_connections 2024-04-25T15:25:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.733+0000 7f10b9f06640 1 -- 172.21.15.67:0/2960586008 wait complete. 2024-04-25T15:25:37.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2024-04-25T15:25:37.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.738+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f617806ef70 con 0x7f61780ce920 2024-04-25T15:25:37.739 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-04-25T15:25:37.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.738+0000 7f6174ff9640 1 -- 172.21.15.67:0/1872558396 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f617005ee40 con 0x7f61780ce920 2024-04-25T15:25:37.740 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:25:37.740 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.741+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f615c0744b0 msgr2=0x7f615c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.741+0000 7f617da42640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f615c0744b0 0x7f615c0769c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6168009560 tx=0x7f616802e000 comp rx=0 tx=0).stop 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780ce920 msgr2=0x7f61781674d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780ce920 0x7f61781674d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6178073fc0 tx=0x7f617000cf00 comp rx=0 tx=0).stop 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 shutdown_connections 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f615c0744b0 0x7f615c0769c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780ce920 0x7f61781674d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61780cd060 0x7f6178166f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 --2- 172.21.15.67:0/1872558396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61780719b0 0x7f617815b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 >> 172.21.15.67:0/1872558396 conn(0x7f61780d2a10 msgr2=0x7f6178161600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 shutdown_connections 2024-04-25T15:25:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.742+0000 7f617da42640 1 -- 172.21.15.67:0/1872558396 wait complete. 2024-04-25T15:25:37.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.750+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f9ac4005580 con 0x7f9ae4164ae0 2024-04-25T15:25:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.750+0000 7f9acbfff640 1 -- 172.21.15.67:0/696880197 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9ae40c6320 con 0x7f9ae4164ae0 2024-04-25T15:25:37.751 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-04-25T15:25:37.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-04-25T15:25:37.753 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-04-25T15:25:37.753 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ad0074580 msgr2=0x7f9ad0076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ad0074580 0x7f9ad0076a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8000f60 tx=0x7f9ad8009040 comp rx=0 tx=0).stop 2024-04-25T15:25:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae4164ae0 msgr2=0x7f9ae4162f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae4164ae0 0x7f9ae4162f10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009f70 tx=0x7f9adc004a40 comp rx=0 tx=0).stop 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 shutdown_connections 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ad0074580 0x7f9ad0076a90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ae4164ae0 0x7f9ae4162f10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae41646d0 0x7f9ae40c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 --2- 172.21.15.67:0/696880197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c5f10 0x7f9ae4203260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 >> 172.21.15.67:0/696880197 conn(0x7f9ae409b7d0 msgr2=0x7f9ae40d1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 shutdown_connections 2024-04-25T15:25:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.753+0000 7f9ae9a34640 1 -- 172.21.15.67:0/696880197 wait complete. 2024-04-25T15:25:37.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-04-25T15:25:37.756 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.765 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-04-25T15:25:37.765 DEBUG:teuthology.parallel:result is None 2024-04-25T15:25:37.765 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T15:25:37.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:25:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.917+0000 7f1e0d772640 1 Processor -- start 2024-04-25T15:25:37.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.917+0000 7f1e0d772640 1 -- start start 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e0d772640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1e08160870 0x7f1e080b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e0d772640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080cf300 0x7f1e080d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e0d772640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e080cfb30 con 0x7f1e080d2240 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e0d772640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1e080b3910 con 0x7f1e08160870 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e0d772640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1e080d00f0 con 0x7f1e080cf300 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e06ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1e08160870 0x7f1e080b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e067fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080cf300 0x7f1e080d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e06ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1e08160870 0x7f1e080b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39558/0 (socket says 172.21.15.67:39558) 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e06ffd640 1 -- 172.21.15.67:0/1400597421 learned_addr learned my addr 172.21.15.67:0/1400597421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.918+0000 7f1e077fe640 1 --1- 172.21.15.67:0/1400597421 >> v1:172.21.15.176:6789/0 conn(0x7f1e080d2240 0x7f1e080d2990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56086/0 (socket says 172.21.15.67:56086) 2024-04-25T15:25:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.919+0000 7f1e067fc640 1 -- 172.21.15.67:0/1400597421 >> v2:172.21.15.176:3300/0 conn(0x7f1e08160870 msgr2=0x7f1e080b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.919+0000 7f1e067fc640 1 --2- 172.21.15.67:0/1400597421 >> v2:172.21.15.176:3300/0 conn(0x7f1e08160870 0x7f1e080b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.919+0000 7f1e067fc640 1 -- 172.21.15.67:0/1400597421 >> v1:172.21.15.176:6789/0 conn(0x7f1e080d2240 legacy=0x7f1e080d2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.919+0000 7f1e067fc640 1 -- 172.21.15.67:0/1400597421 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e080cfe10 con 0x7f1e080cf300 2024-04-25T15:25:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.919+0000 7f1e06ffd640 1 --2- 172.21.15.67:0/1400597421 >> v2:172.21.15.176:3300/0 conn(0x7f1e08160870 0x7f1e080b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.919+0000 7f1e067fc640 1 --2- 172.21.15.67:0/1400597421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080cf300 0x7f1e080d1d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0097a0 tx=0x7f1dfc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247fda92f9cf103b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e05ffb640 1 -- 172.21.15.67:0/1400597421 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc037070 con 0x7f1e080cf300 2024-04-25T15:25:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e05ffb640 1 -- 172.21.15.67:0/1400597421 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dfc032440 con 0x7f1e080cf300 2024-04-25T15:25:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e05ffb640 1 -- 172.21.15.67:0/1400597421 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc032810 con 0x7f1e080cf300 2024-04-25T15:25:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 -- 172.21.15.67:0/1400597421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080cf300 msgr2=0x7f1e080d1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 --2- 172.21.15.67:0/1400597421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080cf300 0x7f1e080d1d00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0097a0 tx=0x7f1dfc02a180 comp rx=0 tx=0).stop 2024-04-25T15:25:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 -- 172.21.15.67:0/1400597421 shutdown_connections 2024-04-25T15:25:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 --2- 172.21.15.67:0/1400597421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080cf300 0x7f1e080d1d00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 --2- 172.21.15.67:0/1400597421 >> v2:172.21.15.176:3300/0 conn(0x7f1e08160870 0x7f1e080b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 -- 172.21.15.67:0/1400597421 >> 172.21.15.67:0/1400597421 conn(0x7f1e0809b7d0 msgr2=0x7f1e0809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 -- 172.21.15.67:0/1400597421 shutdown_connections 2024-04-25T15:25:37.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.920+0000 7f1e0d772640 1 -- 172.21.15.67:0/1400597421 wait complete. 2024-04-25T15:25:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.922+0000 7f1e0d772640 1 Processor -- start 2024-04-25T15:25:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.922+0000 7f1e0d772640 1 -- start start 2024-04-25T15:25:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e0d772640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e080cf300 0x7f1e080c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e0d772640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080d2240 0x7f1e080c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e0d772640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e08160870 0x7f1e080c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e0d772640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e080d2650 con 0x7f1e080cf300 2024-04-25T15:25:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e0d772640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1e080b2470 con 0x7f1e08160870 2024-04-25T15:25:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e0d772640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1e080af850 con 0x7f1e080d2240 2024-04-25T15:25:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e080cf300 0x7f1e080c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e080cf300 0x7f1e080c9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38050/0 (socket says 172.21.15.67:38050) 2024-04-25T15:25:37.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 -- 172.21.15.67:0/8985886 learned_addr learned my addr 172.21.15.67:0/8985886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e077fe640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e08160870 0x7f1e080c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e067fc640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080d2240 0x7f1e080c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 -- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080d2240 msgr2=0x7f1e080c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080d2240 0x7f1e080c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 -- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e08160870 msgr2=0x7f1e080c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:37.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e08160870 0x7f1e080c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 -- 172.21.15.67:0/8985886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e080cc950 con 0x7f1e080cf300 2024-04-25T15:25:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1e06ffd640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e080cf300 0x7f1e080c9ac0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000efc0 tx=0x7f1e0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.924+0000 7f1def7fe640 1 -- 172.21.15.67:0/8985886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0000ce00 con 0x7f1e080cf300 2024-04-25T15:25:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.925+0000 7f1def7fe640 1 -- 172.21.15.67:0/8985886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e00019070 con 0x7f1e080cf300 2024-04-25T15:25:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.925+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e081fae60 con 0x7f1e080cf300 2024-04-25T15:25:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.925+0000 7f1def7fe640 1 -- 172.21.15.67:0/8985886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e000146a0 con 0x7f1e080cf300 2024-04-25T15:25:37.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.925+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e08162780 con 0x7f1e080cf300 2024-04-25T15:25:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.925+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e08050740 con 0x7f1e080cf300 2024-04-25T15:25:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.926+0000 7f1def7fe640 1 -- 172.21.15.67:0/8985886 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1e00005d10 con 0x7f1e080cf300 2024-04-25T15:25:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.927+0000 7f1def7fe640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1df40744b0 0x7f1df40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.927+0000 7f1def7fe640 1 -- 172.21.15.67:0/8985886 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1e00095570 con 0x7f1e080cf300 2024-04-25T15:25:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.927+0000 7f1e067fc640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1df40744b0 0x7f1df40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.927+0000 7f1e067fc640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1df40744b0 0x7f1df40769c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc034000 tx=0x7f1dfc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:37.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:37.931+0000 7f1def7fe640 1 -- 172.21.15.67:0/8985886 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e00010040 con 0x7f1e080cf300 2024-04-25T15:25:38.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.112+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1e080d2d00 con 0x7f1df40744b0 2024-04-25T15:25:38.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.119+0000 7f1def7fe640 1 -- 172.21.15.67:0/8985886 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f1e080d2d00 con 0x7f1df40744b0 2024-04-25T15:25:38.122 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:38.123 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:25:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1df40744b0 msgr2=0x7f1df40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1df40744b0 0x7f1df40769c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc034000 tx=0x7f1dfc02e000 comp rx=0 tx=0).stop 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e080cf300 msgr2=0x7f1e080c9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e080cf300 0x7f1e080c9ac0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000efc0 tx=0x7f1e0000c490 comp rx=0 tx=0).stop 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 shutdown_connections 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1df40744b0 0x7f1df40769c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e08160870 0x7f1e080c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e080d2240 0x7f1e080c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 --2- 172.21.15.67:0/8985886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e080cf300 0x7f1e080c9ac0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 >> 172.21.15.67:0/8985886 conn(0x7f1e0809b7d0 msgr2=0x7f1e0806e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 shutdown_connections 2024-04-25T15:25:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.122+0000 7f1e0d772640 1 -- 172.21.15.67:0/8985886 wait complete. 2024-04-25T15:25:38.134 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-25T15:25:36.796901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002224"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659051+0000","last_change":"2024-04-25T15:25:29.325672+0000","last_active":"2024-04-25T15:25:30.659051+0000","last_peered":"2024-04-25T15:25:30.659051+0000","last_clean":"2024-04-25T15:25:30.659051+0000","last_became_active":"2024-04-25T15:25:29.324999+0000","last_became_peered":"2024-04-25T15:25:29.324999+0000","last_unstale":"2024-04-25T15:25:30.659051+0000","last_undegraded":"2024-04-25T15:25:30.659051+0000","last_fullsized":"2024-04-25T15:25:30.659051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293966+0000","last_change":"2024-04-25T15:25:29.303032+0000","last_active":"2024-04-25T15:25:30.293966+0000","last_peered":"2024-04-25T15:25:30.293966+0000","last_clean":"2024-04-25T15:25:30.293966+0000","last_became_active":"2024-04-25T15:25:29.302027+0000","last_became_peered":"2024-04-25T15:25:29.302027+0000","last_unstale":"2024-04-25T15:25:30.293966+0000","last_undegraded":"2024-04-25T15:25:30.293966+0000","last_fullsized":"2024-04-25T15:25:30.293966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297058+0000","last_change":"2024-04-25T15:25:29.307526+0000","last_active":"2024-04-25T15:25:30.297058+0000","last_peered":"2024-04-25T15:25:30.297058+0000","last_clean":"2024-04-25T15:25:30.297058+0000","last_became_active":"2024-04-25T15:25:29.307109+0000","last_became_peered":"2024-04-25T15:25:29.307109+0000","last_unstale":"2024-04-25T15:25:30.297058+0000","last_undegraded":"2024-04-25T15:25:30.297058+0000","last_fullsized":"2024-04-25T15:25:30.297058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323540+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322901+0000","last_became_peered":"2024-04-25T15:25:29.322901+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658248+0000","last_change":"2024-04-25T15:25:29.313182+0000","last_active":"2024-04-25T15:25:30.658248+0000","last_peered":"2024-04-25T15:25:30.658248+0000","last_clean":"2024-04-25T15:25:30.658248+0000","last_became_active":"2024-04-25T15:25:29.312850+0000","last_became_peered":"2024-04-25T15:25:29.312850+0000","last_unstale":"2024-04-25T15:25:30.658248+0000","last_undegraded":"2024-04-25T15:25:30.658248+0000","last_fullsized":"2024-04-25T15:25:30.658248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658795+0000","last_change":"2024-04-25T15:25:29.319442+0000","last_active":"2024-04-25T15:25:30.658795+0000","last_peered":"2024-04-25T15:25:30.658795+0000","last_clean":"2024-04-25T15:25:30.658795+0000","last_became_active":"2024-04-25T15:25:29.319141+0000","last_became_peered":"2024-04-25T15:25:29.319141+0000","last_unstale":"2024-04-25T15:25:30.658795+0000","last_undegraded":"2024-04-25T15:25:30.658795+0000","last_fullsized":"2024-04-25T15:25:30.658795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657979+0000","last_change":"2024-04-25T15:25:29.322616+0000","last_active":"2024-04-25T15:25:30.657979+0000","last_peered":"2024-04-25T15:25:30.657979+0000","last_clean":"2024-04-25T15:25:30.657979+0000","last_became_active":"2024-04-25T15:25:29.322311+0000","last_became_peered":"2024-04-25T15:25:29.322311+0000","last_unstale":"2024-04-25T15:25:30.657979+0000","last_undegraded":"2024-04-25T15:25:30.657979+0000","last_fullsized":"2024-04-25T15:25:30.657979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323286+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322911+0000","last_became_peered":"2024-04-25T15:25:29.322911+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659143+0000","last_change":"2024-04-25T15:25:29.325686+0000","last_active":"2024-04-25T15:25:30.659143+0000","last_peered":"2024-04-25T15:25:30.659143+0000","last_clean":"2024-04-25T15:25:30.659143+0000","last_became_active":"2024-04-25T15:25:29.325332+0000","last_became_peered":"2024-04-25T15:25:29.325332+0000","last_unstale":"2024-04-25T15:25:30.659143+0000","last_undegraded":"2024-04-25T15:25:30.659143+0000","last_fullsized":"2024-04-25T15:25:30.659143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:20.087185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313378+0000","last_change":"2024-04-25T15:25:29.323852+0000","last_active":"2024-04-25T15:25:30.313378+0000","last_peered":"2024-04-25T15:25:30.313378+0000","last_clean":"2024-04-25T15:25:30.313378+0000","last_became_active":"2024-04-25T15:25:29.323524+0000","last_became_peered":"2024-04-25T15:25:29.323524+0000","last_unstale":"2024-04-25T15:25:30.313378+0000","last_undegraded":"2024-04-25T15:25:30.313378+0000","last_fullsized":"2024-04-25T15:25:30.313378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313308+0000","last_change":"2024-04-25T15:25:29.325815+0000","last_active":"2024-04-25T15:25:30.313308+0000","last_peered":"2024-04-25T15:25:30.313308+0000","last_clean":"2024-04-25T15:25:30.313308+0000","last_became_active":"2024-04-25T15:25:29.325436+0000","last_became_peered":"2024-04-25T15:25:29.325436+0000","last_unstale":"2024-04-25T15:25:30.313308+0000","last_undegraded":"2024-04-25T15:25:30.313308+0000","last_fullsized":"2024-04-25T15:25:30.313308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300011+0000","last_change":"2024-04-25T15:25:29.311301+0000","last_active":"2024-04-25T15:25:30.300011+0000","last_peered":"2024-04-25T15:25:30.300011+0000","last_clean":"2024-04-25T15:25:30.300011+0000","last_became_active":"2024-04-25T15:25:29.310966+0000","last_became_peered":"2024-04-25T15:25:29.310966+0000","last_unstale":"2024-04-25T15:25:30.300011+0000","last_undegraded":"2024-04-25T15:25:30.300011+0000","last_fullsized":"2024-04-25T15:25:30.300011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:44.511850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657661+0000","last_change":"2024-04-25T15:25:29.315008+0000","last_active":"2024-04-25T15:25:30.657661+0000","last_peered":"2024-04-25T15:25:30.657661+0000","last_clean":"2024-04-25T15:25:30.657661+0000","last_became_active":"2024-04-25T15:25:29.314675+0000","last_became_peered":"2024-04-25T15:25:29.314675+0000","last_unstale":"2024-04-25T15:25:30.657661+0000","last_undegraded":"2024-04-25T15:25:30.657661+0000","last_fullsized":"2024-04-25T15:25:30.657661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296694+0000","last_change":"2024-04-25T15:25:29.301255+0000","last_active":"2024-04-25T15:25:30.296694+0000","last_peered":"2024-04-25T15:25:30.296694+0000","last_clean":"2024-04-25T15:25:30.296694+0000","last_became_active":"2024-04-25T15:25:29.300877+0000","last_became_peered":"2024-04-25T15:25:29.300877+0000","last_unstale":"2024-04-25T15:25:30.296694+0000","last_undegraded":"2024-04-25T15:25:30.296694+0000","last_fullsized":"2024-04-25T15:25:30.296694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:40.562364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293711+0000","last_change":"2024-04-25T15:25:29.300064+0000","last_active":"2024-04-25T15:25:30.293711+0000","last_peered":"2024-04-25T15:25:30.293711+0000","last_clean":"2024-04-25T15:25:30.293711+0000","last_became_active":"2024-04-25T15:25:29.299759+0000","last_became_peered":"2024-04-25T15:25:29.299759+0000","last_unstale":"2024-04-25T15:25:30.293711+0000","last_undegraded":"2024-04-25T15:25:30.293711+0000","last_fullsized":"2024-04-25T15:25:30.293711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657884+0000","last_change":"2024-04-25T15:25:29.314325+0000","last_active":"2024-04-25T15:25:30.657884+0000","last_peered":"2024-04-25T15:25:30.657884+0000","last_clean":"2024-04-25T15:25:30.657884+0000","last_became_active":"2024-04-25T15:25:29.311827+0000","last_became_peered":"2024-04-25T15:25:29.311827+0000","last_unstale":"2024-04-25T15:25:30.657884+0000","last_undegraded":"2024-04-25T15:25:30.657884+0000","last_fullsized":"2024-04-25T15:25:30.657884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658676+0000","last_change":"2024-04-25T15:25:29.327856+0000","last_active":"2024-04-25T15:25:30.658676+0000","last_peered":"2024-04-25T15:25:30.658676+0000","last_clean":"2024-04-25T15:25:30.658676+0000","last_became_active":"2024-04-25T15:25:29.327408+0000","last_became_peered":"2024-04-25T15:25:29.327408+0000","last_unstale":"2024-04-25T15:25:30.658676+0000","last_undegraded":"2024-04-25T15:25:30.658676+0000","last_fullsized":"2024-04-25T15:25:30.658676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657624+0000","last_change":"2024-04-25T15:25:29.313009+0000","last_active":"2024-04-25T15:25:30.657624+0000","last_peered":"2024-04-25T15:25:30.657624+0000","last_clean":"2024-04-25T15:25:30.657624+0000","last_became_active":"2024-04-25T15:25:29.312684+0000","last_became_peered":"2024-04-25T15:25:29.312684+0000","last_unstale":"2024-04-25T15:25:30.657624+0000","last_undegraded":"2024-04-25T15:25:30.657624+0000","last_fullsized":"2024-04-25T15:25:30.657624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:26.944400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658764+0000","last_change":"2024-04-25T15:25:29.319894+0000","last_active":"2024-04-25T15:25:30.658764+0000","last_peered":"2024-04-25T15:25:30.658764+0000","last_clean":"2024-04-25T15:25:30.658764+0000","last_became_active":"2024-04-25T15:25:29.319566+0000","last_became_peered":"2024-04-25T15:25:29.319566+0000","last_unstale":"2024-04-25T15:25:30.658764+0000","last_undegraded":"2024-04-25T15:25:30.658764+0000","last_fullsized":"2024-04-25T15:25:30.658764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313203+0000","last_change":"2024-04-25T15:25:29.312046+0000","last_active":"2024-04-25T15:25:30.313203+0000","last_peered":"2024-04-25T15:25:30.313203+0000","last_clean":"2024-04-25T15:25:30.313203+0000","last_became_active":"2024-04-25T15:25:29.310121+0000","last_became_peered":"2024-04-25T15:25:29.310121+0000","last_unstale":"2024-04-25T15:25:30.313203+0000","last_undegraded":"2024-04-25T15:25:30.313203+0000","last_fullsized":"2024-04-25T15:25:30.313203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296808+0000","last_change":"2024-04-25T15:25:29.317661+0000","last_active":"2024-04-25T15:25:30.296808+0000","last_peered":"2024-04-25T15:25:30.296808+0000","last_clean":"2024-04-25T15:25:30.296808+0000","last_became_active":"2024-04-25T15:25:29.317302+0000","last_became_peered":"2024-04-25T15:25:29.317302+0000","last_unstale":"2024-04-25T15:25:30.296808+0000","last_undegraded":"2024-04-25T15:25:30.296808+0000","last_fullsized":"2024-04-25T15:25:30.296808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658934+0000","last_change":"2024-04-25T15:25:29.327012+0000","last_active":"2024-04-25T15:25:30.658934+0000","last_peered":"2024-04-25T15:25:30.658934+0000","last_clean":"2024-04-25T15:25:30.658934+0000","last_became_active":"2024-04-25T15:25:29.326589+0000","last_became_peered":"2024-04-25T15:25:29.326589+0000","last_unstale":"2024-04-25T15:25:30.658934+0000","last_undegraded":"2024-04-25T15:25:30.658934+0000","last_fullsized":"2024-04-25T15:25:30.658934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657788+0000","last_change":"2024-04-25T15:25:29.321180+0000","last_active":"2024-04-25T15:25:30.657788+0000","last_peered":"2024-04-25T15:25:30.657788+0000","last_clean":"2024-04-25T15:25:30.657788+0000","last_became_active":"2024-04-25T15:25:29.320679+0000","last_became_peered":"2024-04-25T15:25:29.320679+0000","last_unstale":"2024-04-25T15:25:30.657788+0000","last_undegraded":"2024-04-25T15:25:30.657788+0000","last_fullsized":"2024-04-25T15:25:30.657788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658226+0000","last_change":"2024-04-25T15:25:29.311398+0000","last_active":"2024-04-25T15:25:30.658226+0000","last_peered":"2024-04-25T15:25:30.658226+0000","last_clean":"2024-04-25T15:25:30.658226+0000","last_became_active":"2024-04-25T15:25:29.311069+0000","last_became_peered":"2024-04-25T15:25:29.311069+0000","last_unstale":"2024-04-25T15:25:30.658226+0000","last_undegraded":"2024-04-25T15:25:30.658226+0000","last_fullsized":"2024-04-25T15:25:30.658226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658383+0000","last_change":"2024-04-25T15:25:29.320926+0000","last_active":"2024-04-25T15:25:30.658383+0000","last_peered":"2024-04-25T15:25:30.658383+0000","last_clean":"2024-04-25T15:25:30.658383+0000","last_became_active":"2024-04-25T15:25:29.320574+0000","last_became_peered":"2024-04-25T15:25:29.320574+0000","last_unstale":"2024-04-25T15:25:30.658383+0000","last_undegraded":"2024-04-25T15:25:30.658383+0000","last_fullsized":"2024-04-25T15:25:30.658383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313448+0000","last_change":"2024-04-25T15:25:29.330048+0000","last_active":"2024-04-25T15:25:30.313448+0000","last_peered":"2024-04-25T15:25:30.313448+0000","last_clean":"2024-04-25T15:25:30.313448+0000","last_became_active":"2024-04-25T15:25:29.329720+0000","last_became_peered":"2024-04-25T15:25:29.329720+0000","last_unstale":"2024-04-25T15:25:30.313448+0000","last_undegraded":"2024-04-25T15:25:30.313448+0000","last_fullsized":"2024-04-25T15:25:30.313448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293796+0000","last_change":"2024-04-25T15:25:29.309985+0000","last_active":"2024-04-25T15:25:30.293796+0000","last_peered":"2024-04-25T15:25:30.293796+0000","last_clean":"2024-04-25T15:25:30.293796+0000","last_became_active":"2024-04-25T15:25:29.309619+0000","last_became_peered":"2024-04-25T15:25:29.309619+0000","last_unstale":"2024-04-25T15:25:30.293796+0000","last_undegraded":"2024-04-25T15:25:30.293796+0000","last_fullsized":"2024-04-25T15:25:30.293796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658375+0000","last_change":"2024-04-25T15:25:29.321037+0000","last_active":"2024-04-25T15:25:30.658375+0000","last_peered":"2024-04-25T15:25:30.658375+0000","last_clean":"2024-04-25T15:25:30.658375+0000","last_became_active":"2024-04-25T15:25:29.320680+0000","last_became_peered":"2024-04-25T15:25:29.320680+0000","last_unstale":"2024-04-25T15:25:30.658375+0000","last_undegraded":"2024-04-25T15:25:30.658375+0000","last_fullsized":"2024-04-25T15:25:30.658375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296855+0000","last_change":"2024-04-25T15:25:29.309805+0000","last_active":"2024-04-25T15:25:30.296855+0000","last_peered":"2024-04-25T15:25:30.296855+0000","last_clean":"2024-04-25T15:25:30.296855+0000","last_became_active":"2024-04-25T15:25:29.309408+0000","last_became_peered":"2024-04-25T15:25:29.309408+0000","last_unstale":"2024-04-25T15:25:30.296855+0000","last_undegraded":"2024-04-25T15:25:30.296855+0000","last_fullsized":"2024-04-25T15:25:30.296855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659006+0000","last_change":"2024-04-25T15:25:29.318254+0000","last_active":"2024-04-25T15:25:30.659006+0000","last_peered":"2024-04-25T15:25:30.659006+0000","last_clean":"2024-04-25T15:25:30.659006+0000","last_became_active":"2024-04-25T15:25:29.317944+0000","last_became_peered":"2024-04-25T15:25:29.317944+0000","last_unstale":"2024-04-25T15:25:30.659006+0000","last_undegraded":"2024-04-25T15:25:30.659006+0000","last_fullsized":"2024-04-25T15:25:30.659006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657620+0000","last_change":"2024-04-25T15:25:29.313901+0000","last_active":"2024-04-25T15:25:30.657620+0000","last_peered":"2024-04-25T15:25:30.657620+0000","last_clean":"2024-04-25T15:25:30.657620+0000","last_became_active":"2024-04-25T15:25:29.313526+0000","last_became_peered":"2024-04-25T15:25:29.313526+0000","last_unstale":"2024-04-25T15:25:30.657620+0000","last_undegraded":"2024-04-25T15:25:30.657620+0000","last_fullsized":"2024-04-25T15:25:30.657620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313041+0000","last_change":"2024-04-25T15:25:29.312724+0000","last_active":"2024-04-25T15:25:30.313041+0000","last_peered":"2024-04-25T15:25:30.313041+0000","last_clean":"2024-04-25T15:25:30.313041+0000","last_became_active":"2024-04-25T15:25:29.312425+0000","last_became_peered":"2024-04-25T15:25:29.312425+0000","last_unstale":"2024-04-25T15:25:30.313041+0000","last_undegraded":"2024-04-25T15:25:30.313041+0000","last_fullsized":"2024-04-25T15:25:30.313041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299747+0000","last_change":"2024-04-25T15:25:29.298478+0000","last_active":"2024-04-25T15:25:30.299747+0000","last_peered":"2024-04-25T15:25:30.299747+0000","last_clean":"2024-04-25T15:25:30.299747+0000","last_became_active":"2024-04-25T15:25:29.298178+0000","last_became_peered":"2024-04-25T15:25:29.298178+0000","last_unstale":"2024-04-25T15:25:30.299747+0000","last_undegraded":"2024-04-25T15:25:30.299747+0000","last_fullsized":"2024-04-25T15:25:30.299747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:53.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658039+0000","last_change":"2024-04-25T15:25:28.301149+0000","last_active":"2024-04-25T15:25:30.658039+0000","last_peered":"2024-04-25T15:25:30.658039+0000","last_clean":"2024-04-25T15:25:30.658039+0000","last_became_active":"2024-04-25T15:25:28.300804+0000","last_became_peered":"2024-04-25T15:25:28.300804+0000","last_unstale":"2024-04-25T15:25:30.658039+0000","last_undegraded":"2024-04-25T15:25:30.658039+0000","last_fullsized":"2024-04-25T15:25:30.658039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296993+0000","last_change":"2024-04-25T15:25:29.306337+0000","last_active":"2024-04-25T15:25:30.296993+0000","last_peered":"2024-04-25T15:25:30.296993+0000","last_clean":"2024-04-25T15:25:30.296993+0000","last_became_active":"2024-04-25T15:25:29.306039+0000","last_became_peered":"2024-04-25T15:25:29.306039+0000","last_unstale":"2024-04-25T15:25:30.296993+0000","last_undegraded":"2024-04-25T15:25:30.296993+0000","last_fullsized":"2024-04-25T15:25:30.296993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313038+0000","last_change":"2024-04-25T15:25:28.301076+0000","last_active":"2024-04-25T15:25:30.313038+0000","last_peered":"2024-04-25T15:25:30.313038+0000","last_clean":"2024-04-25T15:25:30.313038+0000","last_became_active":"2024-04-25T15:25:28.300670+0000","last_became_peered":"2024-04-25T15:25:28.300670+0000","last_unstale":"2024-04-25T15:25:30.313038+0000","last_undegraded":"2024-04-25T15:25:30.313038+0000","last_fullsized":"2024-04-25T15:25:30.313038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657997+0000","last_change":"2024-04-25T15:25:28.305288+0000","last_active":"2024-04-25T15:25:30.657997+0000","last_peered":"2024-04-25T15:25:30.657997+0000","last_clean":"2024-04-25T15:25:30.657997+0000","last_became_active":"2024-04-25T15:25:28.304900+0000","last_became_peered":"2024-04-25T15:25:28.304900+0000","last_unstale":"2024-04-25T15:25:30.657997+0000","last_undegraded":"2024-04-25T15:25:30.657997+0000","last_fullsized":"2024-04-25T15:25:30.657997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:04.261360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658085+0000","last_change":"2024-04-25T15:25:29.310703+0000","last_active":"2024-04-25T15:25:30.658085+0000","last_peered":"2024-04-25T15:25:30.658085+0000","last_clean":"2024-04-25T15:25:30.658085+0000","last_became_active":"2024-04-25T15:25:29.309942+0000","last_became_peered":"2024-04-25T15:25:29.309942+0000","last_unstale":"2024-04-25T15:25:30.658085+0000","last_undegraded":"2024-04-25T15:25:30.658085+0000","last_fullsized":"2024-04-25T15:25:30.658085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658034+0000","last_change":"2024-04-25T15:25:28.292273+0000","last_active":"2024-04-25T15:25:30.658034+0000","last_peered":"2024-04-25T15:25:30.658034+0000","last_clean":"2024-04-25T15:25:30.658034+0000","last_became_active":"2024-04-25T15:25:28.291890+0000","last_became_peered":"2024-04-25T15:25:28.291890+0000","last_unstale":"2024-04-25T15:25:30.658034+0000","last_undegraded":"2024-04-25T15:25:30.658034+0000","last_fullsized":"2024-04-25T15:25:30.658034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293637+0000","last_change":"2024-04-25T15:25:29.310761+0000","last_active":"2024-04-25T15:25:30.293637+0000","last_peered":"2024-04-25T15:25:30.293637+0000","last_clean":"2024-04-25T15:25:30.293637+0000","last_became_active":"2024-04-25T15:25:29.310403+0000","last_became_peered":"2024-04-25T15:25:29.310403+0000","last_unstale":"2024-04-25T15:25:30.293637+0000","last_undegraded":"2024-04-25T15:25:30.293637+0000","last_fullsized":"2024-04-25T15:25:30.293637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313366+0000","last_change":"2024-04-25T15:25:28.306210+0000","last_active":"2024-04-25T15:25:30.313366+0000","last_peered":"2024-04-25T15:25:30.313366+0000","last_clean":"2024-04-25T15:25:30.313366+0000","last_became_active":"2024-04-25T15:25:28.305931+0000","last_became_peered":"2024-04-25T15:25:28.305931+0000","last_unstale":"2024-04-25T15:25:30.313366+0000","last_undegraded":"2024-04-25T15:25:30.313366+0000","last_fullsized":"2024-04-25T15:25:30.313366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658132+0000","last_change":"2024-04-25T15:25:29.328500+0000","last_active":"2024-04-25T15:25:30.658132+0000","last_peered":"2024-04-25T15:25:30.658132+0000","last_clean":"2024-04-25T15:25:30.658132+0000","last_became_active":"2024-04-25T15:25:29.328150+0000","last_became_peered":"2024-04-25T15:25:29.328150+0000","last_unstale":"2024-04-25T15:25:30.658132+0000","last_undegraded":"2024-04-25T15:25:30.658132+0000","last_fullsized":"2024-04-25T15:25:30.658132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658259+0000","last_change":"2024-04-25T15:25:28.305011+0000","last_active":"2024-04-25T15:25:30.658259+0000","last_peered":"2024-04-25T15:25:30.658259+0000","last_clean":"2024-04-25T15:25:30.658259+0000","last_became_active":"2024-04-25T15:25:28.304682+0000","last_became_peered":"2024-04-25T15:25:28.304682+0000","last_unstale":"2024-04-25T15:25:30.658259+0000","last_undegraded":"2024-04-25T15:25:30.658259+0000","last_fullsized":"2024-04-25T15:25:30.658259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297172+0000","last_change":"2024-04-25T15:25:29.318333+0000","last_active":"2024-04-25T15:25:30.297172+0000","last_peered":"2024-04-25T15:25:30.297172+0000","last_clean":"2024-04-25T15:25:30.297172+0000","last_became_active":"2024-04-25T15:25:29.317557+0000","last_became_peered":"2024-04-25T15:25:29.317557+0000","last_unstale":"2024-04-25T15:25:30.297172+0000","last_undegraded":"2024-04-25T15:25:30.297172+0000","last_fullsized":"2024-04-25T15:25:30.297172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293936+0000","last_change":"2024-04-25T15:25:28.297270+0000","last_active":"2024-04-25T15:25:30.293936+0000","last_peered":"2024-04-25T15:25:30.293936+0000","last_clean":"2024-04-25T15:25:30.293936+0000","last_became_active":"2024-04-25T15:25:28.296865+0000","last_became_peered":"2024-04-25T15:25:28.296865+0000","last_unstale":"2024-04-25T15:25:30.293936+0000","last_undegraded":"2024-04-25T15:25:30.293936+0000","last_fullsized":"2024-04-25T15:25:30.293936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300012+0000","last_change":"2024-04-25T15:25:29.324749+0000","last_active":"2024-04-25T15:25:30.300012+0000","last_peered":"2024-04-25T15:25:30.300012+0000","last_clean":"2024-04-25T15:25:30.300012+0000","last_became_active":"2024-04-25T15:25:29.324454+0000","last_became_peered":"2024-04-25T15:25:29.324454+0000","last_unstale":"2024-04-25T15:25:30.300012+0000","last_undegraded":"2024-04-25T15:25:30.300012+0000","last_fullsized":"2024-04-25T15:25:30.300012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313205+0000","last_change":"2024-04-25T15:25:28.302276+0000","last_active":"2024-04-25T15:25:30.313205+0000","last_peered":"2024-04-25T15:25:30.313205+0000","last_clean":"2024-04-25T15:25:30.313205+0000","last_became_active":"2024-04-25T15:25:28.302039+0000","last_became_peered":"2024-04-25T15:25:28.302039+0000","last_unstale":"2024-04-25T15:25:30.313205+0000","last_undegraded":"2024-04-25T15:25:30.313205+0000","last_fullsized":"2024-04-25T15:25:30.313205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658280+0000","last_change":"2024-04-25T15:25:29.324177+0000","last_active":"2024-04-25T15:25:30.658280+0000","last_peered":"2024-04-25T15:25:30.658280+0000","last_clean":"2024-04-25T15:25:30.658280+0000","last_became_active":"2024-04-25T15:25:29.323727+0000","last_became_peered":"2024-04-25T15:25:29.323727+0000","last_unstale":"2024-04-25T15:25:30.658280+0000","last_undegraded":"2024-04-25T15:25:30.658280+0000","last_fullsized":"2024-04-25T15:25:30.658280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296971+0000","last_change":"2024-04-25T15:25:28.288837+0000","last_active":"2024-04-25T15:25:30.296971+0000","last_peered":"2024-04-25T15:25:30.296971+0000","last_clean":"2024-04-25T15:25:30.296971+0000","last_became_active":"2024-04-25T15:25:28.288485+0000","last_became_peered":"2024-04-25T15:25:28.288485+0000","last_unstale":"2024-04-25T15:25:30.296971+0000","last_undegraded":"2024-04-25T15:25:30.296971+0000","last_fullsized":"2024-04-25T15:25:30.296971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:58.179326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296812+0000","last_change":"2024-04-25T15:25:29.323297+0000","last_active":"2024-04-25T15:25:30.296812+0000","last_peered":"2024-04-25T15:25:30.296812+0000","last_clean":"2024-04-25T15:25:30.296812+0000","last_became_active":"2024-04-25T15:25:29.322955+0000","last_became_peered":"2024-04-25T15:25:29.322955+0000","last_unstale":"2024-04-25T15:25:30.296812+0000","last_undegraded":"2024-04-25T15:25:30.296812+0000","last_fullsized":"2024-04-25T15:25:30.296812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659064+0000","last_change":"2024-04-25T15:25:28.300177+0000","last_active":"2024-04-25T15:25:30.659064+0000","last_peered":"2024-04-25T15:25:30.659064+0000","last_clean":"2024-04-25T15:25:30.659064+0000","last_became_active":"2024-04-25T15:25:28.298626+0000","last_became_peered":"2024-04-25T15:25:28.298626+0000","last_unstale":"2024-04-25T15:25:30.659064+0000","last_undegraded":"2024-04-25T15:25:30.659064+0000","last_fullsized":"2024-04-25T15:25:30.659064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657985+0000","last_change":"2024-04-25T15:25:29.316614+0000","last_active":"2024-04-25T15:25:30.657985+0000","last_peered":"2024-04-25T15:25:30.657985+0000","last_clean":"2024-04-25T15:25:30.657985+0000","last_became_active":"2024-04-25T15:25:29.316293+0000","last_became_peered":"2024-04-25T15:25:29.316293+0000","last_unstale":"2024-04-25T15:25:30.657985+0000","last_undegraded":"2024-04-25T15:25:30.657985+0000","last_fullsized":"2024-04-25T15:25:30.657985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312968+0000","last_change":"2024-04-25T15:25:28.304313+0000","last_active":"2024-04-25T15:25:30.312968+0000","last_peered":"2024-04-25T15:25:30.312968+0000","last_clean":"2024-04-25T15:25:30.312968+0000","last_became_active":"2024-04-25T15:25:28.303943+0000","last_became_peered":"2024-04-25T15:25:28.303943+0000","last_unstale":"2024-04-25T15:25:30.312968+0000","last_undegraded":"2024-04-25T15:25:30.312968+0000","last_fullsized":"2024-04-25T15:25:30.312968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313012+0000","last_change":"2024-04-25T15:25:29.310260+0000","last_active":"2024-04-25T15:25:30.313012+0000","last_peered":"2024-04-25T15:25:30.313012+0000","last_clean":"2024-04-25T15:25:30.313012+0000","last_became_active":"2024-04-25T15:25:29.309916+0000","last_became_peered":"2024-04-25T15:25:29.309916+0000","last_unstale":"2024-04-25T15:25:30.313012+0000","last_undegraded":"2024-04-25T15:25:30.313012+0000","last_fullsized":"2024-04-25T15:25:30.313012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657409+0000","last_change":"2024-04-25T15:25:28.293959+0000","last_active":"2024-04-25T15:25:30.657409+0000","last_peered":"2024-04-25T15:25:30.657409+0000","last_clean":"2024-04-25T15:25:30.657409+0000","last_became_active":"2024-04-25T15:25:28.293594+0000","last_became_peered":"2024-04-25T15:25:28.293594+0000","last_unstale":"2024-04-25T15:25:30.657409+0000","last_undegraded":"2024-04-25T15:25:30.657409+0000","last_fullsized":"2024-04-25T15:25:30.657409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658071+0000","last_change":"2024-04-25T15:25:29.318796+0000","last_active":"2024-04-25T15:25:30.658071+0000","last_peered":"2024-04-25T15:25:30.658071+0000","last_clean":"2024-04-25T15:25:30.658071+0000","last_became_active":"2024-04-25T15:25:29.318218+0000","last_became_peered":"2024-04-25T15:25:29.318218+0000","last_unstale":"2024-04-25T15:25:30.658071+0000","last_undegraded":"2024-04-25T15:25:30.658071+0000","last_fullsized":"2024-04-25T15:25:30.658071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300098+0000","last_change":"2024-04-25T15:25:28.293813+0000","last_active":"2024-04-25T15:25:30.300098+0000","last_peered":"2024-04-25T15:25:30.300098+0000","last_clean":"2024-04-25T15:25:30.300098+0000","last_became_active":"2024-04-25T15:25:28.293498+0000","last_became_peered":"2024-04-25T15:25:28.293498+0000","last_unstale":"2024-04-25T15:25:30.300098+0000","last_undegraded":"2024-04-25T15:25:30.300098+0000","last_fullsized":"2024-04-25T15:25:30.300098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:31:00.192290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658957+0000","last_change":"2024-04-25T15:25:29.322896+0000","last_active":"2024-04-25T15:25:30.658957+0000","last_peered":"2024-04-25T15:25:30.658957+0000","last_clean":"2024-04-25T15:25:30.658957+0000","last_became_active":"2024-04-25T15:25:29.321170+0000","last_became_peered":"2024-04-25T15:25:29.321170+0000","last_unstale":"2024-04-25T15:25:30.658957+0000","last_undegraded":"2024-04-25T15:25:30.658957+0000","last_fullsized":"2024-04-25T15:25:30.658957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657752+0000","last_change":"2024-04-25T15:25:28.294831+0000","last_active":"2024-04-25T15:25:30.657752+0000","last_peered":"2024-04-25T15:25:30.657752+0000","last_clean":"2024-04-25T15:25:30.657752+0000","last_became_active":"2024-04-25T15:25:28.294257+0000","last_became_peered":"2024-04-25T15:25:28.294257+0000","last_unstale":"2024-04-25T15:25:30.657752+0000","last_undegraded":"2024-04-25T15:25:30.657752+0000","last_fullsized":"2024-04-25T15:25:30.657752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659232+0000","last_change":"2024-04-25T15:25:29.325425+0000","last_active":"2024-04-25T15:25:30.659232+0000","last_peered":"2024-04-25T15:25:30.659232+0000","last_clean":"2024-04-25T15:25:30.659232+0000","last_became_active":"2024-04-25T15:25:29.324916+0000","last_became_peered":"2024-04-25T15:25:29.324916+0000","last_unstale":"2024-04-25T15:25:30.659232+0000","last_undegraded":"2024-04-25T15:25:30.659232+0000","last_fullsized":"2024-04-25T15:25:30.659232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:46.643371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.294024+0000","last_change":"2024-04-25T15:25:28.297384+0000","last_active":"2024-04-25T15:25:30.294024+0000","last_peered":"2024-04-25T15:25:30.294024+0000","last_clean":"2024-04-25T15:25:30.294024+0000","last_became_active":"2024-04-25T15:25:28.297035+0000","last_became_peered":"2024-04-25T15:25:28.297035+0000","last_unstale":"2024-04-25T15:25:30.294024+0000","last_undegraded":"2024-04-25T15:25:30.294024+0000","last_fullsized":"2024-04-25T15:25:30.294024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658488+0000","last_change":"2024-04-25T15:25:29.323269+0000","last_active":"2024-04-25T15:25:30.658488+0000","last_peered":"2024-04-25T15:25:30.658488+0000","last_clean":"2024-04-25T15:25:30.658488+0000","last_became_active":"2024-04-25T15:25:29.322481+0000","last_became_peered":"2024-04-25T15:25:29.322481+0000","last_unstale":"2024-04-25T15:25:30.658488+0000","last_undegraded":"2024-04-25T15:25:30.658488+0000","last_fullsized":"2024-04-25T15:25:30.658488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657594+0000","last_change":"2024-04-25T15:25:28.293938+0000","last_active":"2024-04-25T15:25:30.657594+0000","last_peered":"2024-04-25T15:25:30.657594+0000","last_clean":"2024-04-25T15:25:30.657594+0000","last_became_active":"2024-04-25T15:25:28.293520+0000","last_became_peered":"2024-04-25T15:25:28.293520+0000","last_unstale":"2024-04-25T15:25:30.657594+0000","last_undegraded":"2024-04-25T15:25:30.657594+0000","last_fullsized":"2024-04-25T15:25:30.657594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658465+0000","last_change":"2024-04-25T15:25:29.325130+0000","last_active":"2024-04-25T15:25:30.658465+0000","last_peered":"2024-04-25T15:25:30.658465+0000","last_clean":"2024-04-25T15:25:30.658465+0000","last_became_active":"2024-04-25T15:25:29.324783+0000","last_became_peered":"2024-04-25T15:25:29.324783+0000","last_unstale":"2024-04-25T15:25:30.658465+0000","last_undegraded":"2024-04-25T15:25:30.658465+0000","last_fullsized":"2024-04-25T15:25:30.658465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657696+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:25:30.657696+0000","last_peered":"2024-04-25T15:25:30.657696+0000","last_clean":"2024-04-25T15:25:30.657696+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:25:30.657696+0000","last_undegraded":"2024-04-25T15:25:30.657696+0000","last_fullsized":"2024-04-25T15:25:30.657696+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657863+0000","last_change":"2024-04-25T15:25:28.293187+0000","last_active":"2024-04-25T15:25:30.657863+0000","last_peered":"2024-04-25T15:25:30.657863+0000","last_clean":"2024-04-25T15:25:30.657863+0000","last_became_active":"2024-04-25T15:25:28.292748+0000","last_became_peered":"2024-04-25T15:25:28.292748+0000","last_unstale":"2024-04-25T15:25:30.657863+0000","last_undegraded":"2024-04-25T15:25:30.657863+0000","last_fullsized":"2024-04-25T15:25:30.657863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657964+0000","last_change":"2024-04-25T15:25:29.323272+0000","last_active":"2024-04-25T15:25:30.657964+0000","last_peered":"2024-04-25T15:25:30.657964+0000","last_clean":"2024-04-25T15:25:30.657964+0000","last_became_active":"2024-04-25T15:25:29.322937+0000","last_became_peered":"2024-04-25T15:25:29.322937+0000","last_unstale":"2024-04-25T15:25:30.657964+0000","last_undegraded":"2024-04-25T15:25:30.657964+0000","last_fullsized":"2024-04-25T15:25:30.657964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657646+0000","last_change":"2024-04-25T15:25:28.301819+0000","last_active":"2024-04-25T15:25:30.657646+0000","last_peered":"2024-04-25T15:25:30.657646+0000","last_clean":"2024-04-25T15:25:30.657646+0000","last_became_active":"2024-04-25T15:25:28.301519+0000","last_became_peered":"2024-04-25T15:25:28.301519+0000","last_unstale":"2024-04-25T15:25:30.657646+0000","last_undegraded":"2024-04-25T15:25:30.657646+0000","last_fullsized":"2024-04-25T15:25:30.657646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296763+0000","last_change":"2024-04-25T15:25:29.306397+0000","last_active":"2024-04-25T15:25:30.296763+0000","last_peered":"2024-04-25T15:25:30.296763+0000","last_clean":"2024-04-25T15:25:30.296763+0000","last_became_active":"2024-04-25T15:25:29.306088+0000","last_became_peered":"2024-04-25T15:25:29.306088+0000","last_unstale":"2024-04-25T15:25:30.296763+0000","last_undegraded":"2024-04-25T15:25:30.296763+0000","last_fullsized":"2024-04-25T15:25:30.296763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:05.665096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657506+0000","last_change":"2024-04-25T15:25:28.295278+0000","last_active":"2024-04-25T15:25:30.657506+0000","last_peered":"2024-04-25T15:25:30.657506+0000","last_clean":"2024-04-25T15:25:30.657506+0000","last_became_active":"2024-04-25T15:25:28.294876+0000","last_became_peered":"2024-04-25T15:25:28.294876+0000","last_unstale":"2024-04-25T15:25:30.657506+0000","last_undegraded":"2024-04-25T15:25:30.657506+0000","last_fullsized":"2024-04-25T15:25:30.657506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293669+0000","last_change":"2024-04-25T15:25:29.307860+0000","last_active":"2024-04-25T15:25:30.293669+0000","last_peered":"2024-04-25T15:25:30.293669+0000","last_clean":"2024-04-25T15:25:30.293669+0000","last_became_active":"2024-04-25T15:25:29.307525+0000","last_became_peered":"2024-04-25T15:25:29.307525+0000","last_unstale":"2024-04-25T15:25:30.293669+0000","last_undegraded":"2024-04-25T15:25:30.293669+0000","last_fullsized":"2024-04-25T15:25:30.293669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659149+0000","last_change":"2024-04-25T15:25:28.298243+0000","last_active":"2024-04-25T15:25:30.659149+0000","last_peered":"2024-04-25T15:25:30.659149+0000","last_clean":"2024-04-25T15:25:30.659149+0000","last_became_active":"2024-04-25T15:25:28.297837+0000","last_became_peered":"2024-04-25T15:25:28.297837+0000","last_unstale":"2024-04-25T15:25:30.659149+0000","last_undegraded":"2024-04-25T15:25:30.659149+0000","last_fullsized":"2024-04-25T15:25:30.659149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299949+0000","last_change":"2024-04-25T15:25:29.308455+0000","last_active":"2024-04-25T15:25:30.299949+0000","last_peered":"2024-04-25T15:25:30.299949+0000","last_clean":"2024-04-25T15:25:30.299949+0000","last_became_active":"2024-04-25T15:25:29.307968+0000","last_became_peered":"2024-04-25T15:25:29.307968+0000","last_unstale":"2024-04-25T15:25:30.299949+0000","last_undegraded":"2024-04-25T15:25:30.299949+0000","last_fullsized":"2024-04-25T15:25:30.299949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:53.177957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293919+0000","last_change":"2024-04-25T15:25:28.298284+0000","last_active":"2024-04-25T15:25:30.293919+0000","last_peered":"2024-04-25T15:25:30.293919+0000","last_clean":"2024-04-25T15:25:30.293919+0000","last_became_active":"2024-04-25T15:25:28.297792+0000","last_became_peered":"2024-04-25T15:25:28.297792+0000","last_unstale":"2024-04-25T15:25:30.293919+0000","last_undegraded":"2024-04-25T15:25:30.293919+0000","last_fullsized":"2024-04-25T15:25:30.293919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658750+0000","last_change":"2024-04-25T15:25:29.313891+0000","last_active":"2024-04-25T15:25:30.658750+0000","last_peered":"2024-04-25T15:25:30.658750+0000","last_clean":"2024-04-25T15:25:30.658750+0000","last_became_active":"2024-04-25T15:25:29.313488+0000","last_became_peered":"2024-04-25T15:25:29.313488+0000","last_unstale":"2024-04-25T15:25:30.658750+0000","last_undegraded":"2024-04-25T15:25:30.658750+0000","last_fullsized":"2024-04-25T15:25:30.658750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658847+0000","last_change":"2024-04-25T15:25:28.301563+0000","last_active":"2024-04-25T15:25:30.658847+0000","last_peered":"2024-04-25T15:25:30.658847+0000","last_clean":"2024-04-25T15:25:30.658847+0000","last_became_active":"2024-04-25T15:25:28.301186+0000","last_became_peered":"2024-04-25T15:25:28.301186+0000","last_unstale":"2024-04-25T15:25:30.658847+0000","last_undegraded":"2024-04-25T15:25:30.658847+0000","last_fullsized":"2024-04-25T15:25:30.658847+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:25.937421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293726+0000","last_change":"2024-04-25T15:25:29.308391+0000","last_active":"2024-04-25T15:25:30.293726+0000","last_peered":"2024-04-25T15:25:30.293726+0000","last_clean":"2024-04-25T15:25:30.293726+0000","last_became_active":"2024-04-25T15:25:29.308010+0000","last_became_peered":"2024-04-25T15:25:29.308010+0000","last_unstale":"2024-04-25T15:25:30.293726+0000","last_undegraded":"2024-04-25T15:25:30.293726+0000","last_fullsized":"2024-04-25T15:25:30.293726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312496+0000","last_change":"2024-04-25T15:25:28.304872+0000","last_active":"2024-04-25T15:25:30.312496+0000","last_peered":"2024-04-25T15:25:30.312496+0000","last_clean":"2024-04-25T15:25:30.312496+0000","last_became_active":"2024-04-25T15:25:28.304443+0000","last_became_peered":"2024-04-25T15:25:28.304443+0000","last_unstale":"2024-04-25T15:25:30.312496+0000","last_undegraded":"2024-04-25T15:25:30.312496+0000","last_fullsized":"2024-04-25T15:25:30.312496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658447+0000","last_change":"2024-04-25T15:25:29.326344+0000","last_active":"2024-04-25T15:25:30.658447+0000","last_peered":"2024-04-25T15:25:30.658447+0000","last_clean":"2024-04-25T15:25:30.658447+0000","last_became_active":"2024-04-25T15:25:29.326016+0000","last_became_peered":"2024-04-25T15:25:29.326016+0000","last_unstale":"2024-04-25T15:25:30.658447+0000","last_undegraded":"2024-04-25T15:25:30.658447+0000","last_fullsized":"2024-04-25T15:25:30.658447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313461+0000","last_change":"2024-04-25T15:25:28.307284+0000","last_active":"2024-04-25T15:25:30.313461+0000","last_peered":"2024-04-25T15:25:30.313461+0000","last_clean":"2024-04-25T15:25:30.313461+0000","last_became_active":"2024-04-25T15:25:28.307013+0000","last_became_peered":"2024-04-25T15:25:28.307013+0000","last_unstale":"2024-04-25T15:25:30.313461+0000","last_undegraded":"2024-04-25T15:25:30.313461+0000","last_fullsized":"2024-04-25T15:25:30.313461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297033+0000","last_change":"2024-04-25T15:25:29.301384+0000","last_active":"2024-04-25T15:25:30.297033+0000","last_peered":"2024-04-25T15:25:30.297033+0000","last_clean":"2024-04-25T15:25:30.297033+0000","last_became_active":"2024-04-25T15:25:29.301021+0000","last_became_peered":"2024-04-25T15:25:29.301021+0000","last_unstale":"2024-04-25T15:25:30.297033+0000","last_undegraded":"2024-04-25T15:25:30.297033+0000","last_fullsized":"2024-04-25T15:25:30.297033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657611+0000","last_change":"2024-04-25T15:25:28.304086+0000","last_active":"2024-04-25T15:25:30.657611+0000","last_peered":"2024-04-25T15:25:30.657611+0000","last_clean":"2024-04-25T15:25:30.657611+0000","last_became_active":"2024-04-25T15:25:28.303771+0000","last_became_peered":"2024-04-25T15:25:28.303771+0000","last_unstale":"2024-04-25T15:25:30.657611+0000","last_undegraded":"2024-04-25T15:25:30.657611+0000","last_fullsized":"2024-04-25T15:25:30.657611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299860+0000","last_change":"2024-04-25T15:25:29.310784+0000","last_active":"2024-04-25T15:25:30.299860+0000","last_peered":"2024-04-25T15:25:30.299860+0000","last_clean":"2024-04-25T15:25:30.299860+0000","last_became_active":"2024-04-25T15:25:29.309711+0000","last_became_peered":"2024-04-25T15:25:29.309711+0000","last_unstale":"2024-04-25T15:25:30.299860+0000","last_undegraded":"2024-04-25T15:25:30.299860+0000","last_fullsized":"2024-04-25T15:25:30.299860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:16.888909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293756+0000","last_change":"2024-04-25T15:25:28.300848+0000","last_active":"2024-04-25T15:25:30.293756+0000","last_peered":"2024-04-25T15:25:30.293756+0000","last_clean":"2024-04-25T15:25:30.293756+0000","last_became_active":"2024-04-25T15:25:28.300218+0000","last_became_peered":"2024-04-25T15:25:28.300218+0000","last_unstale":"2024-04-25T15:25:30.293756+0000","last_undegraded":"2024-04-25T15:25:30.293756+0000","last_fullsized":"2024-04-25T15:25:30.293756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658212+0000","last_change":"2024-04-25T15:25:29.313291+0000","last_active":"2024-04-25T15:25:30.658212+0000","last_peered":"2024-04-25T15:25:30.658212+0000","last_clean":"2024-04-25T15:25:30.658212+0000","last_became_active":"2024-04-25T15:25:29.312960+0000","last_became_peered":"2024-04-25T15:25:29.312960+0000","last_unstale":"2024-04-25T15:25:30.658212+0000","last_undegraded":"2024-04-25T15:25:30.658212+0000","last_fullsized":"2024-04-25T15:25:30.658212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293893+0000","last_change":"2024-04-25T15:25:28.301806+0000","last_active":"2024-04-25T15:25:30.293893+0000","last_peered":"2024-04-25T15:25:30.293893+0000","last_clean":"2024-04-25T15:25:30.293893+0000","last_became_active":"2024-04-25T15:25:28.301447+0000","last_became_peered":"2024-04-25T15:25:28.301447+0000","last_unstale":"2024-04-25T15:25:30.293893+0000","last_undegraded":"2024-04-25T15:25:30.293893+0000","last_fullsized":"2024-04-25T15:25:30.293893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657861+0000","last_change":"2024-04-25T15:25:29.310336+0000","last_active":"2024-04-25T15:25:30.657861+0000","last_peered":"2024-04-25T15:25:30.657861+0000","last_clean":"2024-04-25T15:25:30.657861+0000","last_became_active":"2024-04-25T15:25:29.309914+0000","last_became_peered":"2024-04-25T15:25:29.309914+0000","last_unstale":"2024-04-25T15:25:30.657861+0000","last_undegraded":"2024-04-25T15:25:30.657861+0000","last_fullsized":"2024-04-25T15:25:30.657861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658401+0000","last_change":"2024-04-25T15:25:28.300815+0000","last_active":"2024-04-25T15:25:30.658401+0000","last_peered":"2024-04-25T15:25:30.658401+0000","last_clean":"2024-04-25T15:25:30.658401+0000","last_became_active":"2024-04-25T15:25:28.300487+0000","last_became_peered":"2024-04-25T15:25:28.300487+0000","last_unstale":"2024-04-25T15:25:30.658401+0000","last_undegraded":"2024-04-25T15:25:30.658401+0000","last_fullsized":"2024-04-25T15:25:30.658401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313011+0000","last_change":"2024-04-25T15:25:29.310500+0000","last_active":"2024-04-25T15:25:30.313011+0000","last_peered":"2024-04-25T15:25:30.313011+0000","last_clean":"2024-04-25T15:25:30.313011+0000","last_became_active":"2024-04-25T15:25:29.310111+0000","last_became_peered":"2024-04-25T15:25:29.310111+0000","last_unstale":"2024-04-25T15:25:30.313011+0000","last_undegraded":"2024-04-25T15:25:30.313011+0000","last_fullsized":"2024-04-25T15:25:30.313011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658618+0000","last_change":"2024-04-25T15:25:29.314651+0000","last_active":"2024-04-25T15:25:30.658618+0000","last_peered":"2024-04-25T15:25:30.658618+0000","last_clean":"2024-04-25T15:25:30.658618+0000","last_became_active":"2024-04-25T15:25:29.314239+0000","last_became_peered":"2024-04-25T15:25:29.314239+0000","last_unstale":"2024-04-25T15:25:30.658618+0000","last_undegraded":"2024-04-25T15:25:30.658618+0000","last_fullsized":"2024-04-25T15:25:30.658618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658320+0000","last_change":"2024-04-25T15:25:28.304792+0000","last_active":"2024-04-25T15:25:30.658320+0000","last_peered":"2024-04-25T15:25:30.658320+0000","last_clean":"2024-04-25T15:25:30.658320+0000","last_became_active":"2024-04-25T15:25:28.304489+0000","last_became_peered":"2024-04-25T15:25:28.304489+0000","last_unstale":"2024-04-25T15:25:30.658320+0000","last_undegraded":"2024-04-25T15:25:30.658320+0000","last_fullsized":"2024-04-25T15:25:30.658320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658967+0000","last_change":"2024-04-25T15:25:29.318608+0000","last_active":"2024-04-25T15:25:30.658967+0000","last_peered":"2024-04-25T15:25:30.658967+0000","last_clean":"2024-04-25T15:25:30.658967+0000","last_became_active":"2024-04-25T15:25:29.317330+0000","last_became_peered":"2024-04-25T15:25:29.317330+0000","last_unstale":"2024-04-25T15:25:30.658967+0000","last_undegraded":"2024-04-25T15:25:30.658967+0000","last_fullsized":"2024-04-25T15:25:30.658967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658591+0000","last_change":"2024-04-25T15:25:28.305376+0000","last_active":"2024-04-25T15:25:30.658591+0000","last_peered":"2024-04-25T15:25:30.658591+0000","last_clean":"2024-04-25T15:25:30.658591+0000","last_became_active":"2024-04-25T15:25:28.305083+0000","last_became_peered":"2024-04-25T15:25:28.305083+0000","last_unstale":"2024-04-25T15:25:30.658591+0000","last_undegraded":"2024-04-25T15:25:30.658591+0000","last_fullsized":"2024-04-25T15:25:30.658591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297061+0000","last_change":"2024-04-25T15:25:29.319184+0000","last_active":"2024-04-25T15:25:30.297061+0000","last_peered":"2024-04-25T15:25:30.297061+0000","last_clean":"2024-04-25T15:25:30.297061+0000","last_became_active":"2024-04-25T15:25:29.317757+0000","last_became_peered":"2024-04-25T15:25:29.317757+0000","last_unstale":"2024-04-25T15:25:30.297061+0000","last_undegraded":"2024-04-25T15:25:30.297061+0000","last_fullsized":"2024-04-25T15:25:30.297061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658361+0000","last_change":"2024-04-25T15:25:28.298019+0000","last_active":"2024-04-25T15:25:30.658361+0000","last_peered":"2024-04-25T15:25:30.658361+0000","last_clean":"2024-04-25T15:25:30.658361+0000","last_became_active":"2024-04-25T15:25:28.297570+0000","last_became_peered":"2024-04-25T15:25:28.297570+0000","last_unstale":"2024-04-25T15:25:30.658361+0000","last_undegraded":"2024-04-25T15:25:30.658361+0000","last_fullsized":"2024-04-25T15:25:30.658361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658351+0000","last_change":"2024-04-25T15:25:29.316068+0000","last_active":"2024-04-25T15:25:30.658351+0000","last_peered":"2024-04-25T15:25:30.658351+0000","last_clean":"2024-04-25T15:25:30.658351+0000","last_became_active":"2024-04-25T15:25:29.315705+0000","last_became_peered":"2024-04-25T15:25:29.315705+0000","last_unstale":"2024-04-25T15:25:30.658351+0000","last_undegraded":"2024-04-25T15:25:30.658351+0000","last_fullsized":"2024-04-25T15:25:30.658351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297170+0000","last_change":"2024-04-25T15:25:28.300551+0000","last_active":"2024-04-25T15:25:30.297170+0000","last_peered":"2024-04-25T15:25:30.297170+0000","last_clean":"2024-04-25T15:25:30.297170+0000","last_became_active":"2024-04-25T15:25:28.299984+0000","last_became_peered":"2024-04-25T15:25:28.299984+0000","last_unstale":"2024-04-25T15:25:30.297170+0000","last_undegraded":"2024-04-25T15:25:30.297170+0000","last_fullsized":"2024-04-25T15:25:30.297170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:08:02.904549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:25:38.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:25:38.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.286+0000 7f336283c640 1 Processor -- start 2024-04-25T15:25:38.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.286+0000 7f336283c640 1 -- start start 2024-04-25T15:25:38.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.287+0000 7f336283c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f335c164b00 0x7f335c0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.287+0000 7f336283c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.287+0000 7f336283c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f335c1615f0 con 0x7f335c160890 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.287+0000 7f336283c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f335c1613b0 con 0x7f335c164b00 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.287+0000 7f336283c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f335c164670 con 0x7f335c15df80 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f335bfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f335c164b00 0x7f335c0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f335b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f3360dd9640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f335c160890 0x7f335c160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56100/0 (socket says 172.21.15.67:56100) 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f3360dd9640 1 -- 172.21.15.67:0/871984625 learned_addr learned my addr 172.21.15.67:0/871984625 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f335b7fe640 1 -- 172.21.15.67:0/871984625 >> v2:172.21.15.176:3300/0 conn(0x7f335c164b00 msgr2=0x7f335c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f335b7fe640 1 --2- 172.21.15.67:0/871984625 >> v2:172.21.15.176:3300/0 conn(0x7f335c164b00 0x7f335c0b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f335b7fe640 1 -- 172.21.15.67:0/871984625 >> v1:172.21.15.176:6789/0 conn(0x7f335c160890 legacy=0x7f335c160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.288+0000 7f335b7fe640 1 -- 172.21.15.67:0/871984625 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c164020 con 0x7f335c15df80 2024-04-25T15:25:38.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.289+0000 7f335b7fe640 1 --2- 172.21.15.67:0/871984625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c0b80c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f334c0097a0 tx=0x7f334c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=552797d7602688bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.289+0000 7f335affd640 1 -- 172.21.15.67:0/871984625 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c037070 con 0x7f335c15df80 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.289+0000 7f335affd640 1 -- 172.21.15.67:0/871984625 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f334c032440 con 0x7f335c15df80 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.289+0000 7f335affd640 1 -- 172.21.15.67:0/871984625 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334c032810 con 0x7f335c15df80 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.289+0000 7f336283c640 1 -- 172.21.15.67:0/871984625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 msgr2=0x7f335c0b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.289+0000 7f336283c640 1 --2- 172.21.15.67:0/871984625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c0b80c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f334c0097a0 tx=0x7f334c02a180 comp rx=0 tx=0).stop 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.290+0000 7f336283c640 1 -- 172.21.15.67:0/871984625 shutdown_connections 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.290+0000 7f336283c640 1 --2- 172.21.15.67:0/871984625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c0b80c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.290+0000 7f336283c640 1 --2- 172.21.15.67:0/871984625 >> v2:172.21.15.176:3300/0 conn(0x7f335c164b00 0x7f335c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.290+0000 7f336283c640 1 -- 172.21.15.67:0/871984625 >> 172.21.15.67:0/871984625 conn(0x7f335c0d2bc0 msgr2=0x7f335c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.290+0000 7f336283c640 1 -- 172.21.15.67:0/871984625 shutdown_connections 2024-04-25T15:25:38.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.290+0000 7f336283c640 1 -- 172.21.15.67:0/871984625 wait complete. 2024-04-25T15:25:38.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.291+0000 7f336283c640 1 Processor -- start 2024-04-25T15:25:38.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.292+0000 7f336283c640 1 -- start start 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f336283c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f336283c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c160890 0x7f335c1eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f336283c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f335c164b00 0x7f335c0cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f336283c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f335c0b4a00 con 0x7f335c164b00 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f336283c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f335c0b4880 con 0x7f335c160890 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f336283c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f335c0b4b80 con 0x7f335c15df80 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f335c164b00 0x7f335c0cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f335c164b00 0x7f335c0cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38054/0 (socket says 172.21.15.67:38054) 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 -- 172.21.15.67:0/2599704367 learned_addr learned my addr 172.21.15.67:0/2599704367 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f335bfff640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f335b7fe640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c160890 0x7f335c1eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 -- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 msgr2=0x7f335c159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 -- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c160890 msgr2=0x7f335c1eb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c160890 0x7f335c1eb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f3360dd9640 1 -- 172.21.15.67:0/2599704367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c159d80 con 0x7f335c164b00 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.293+0000 7f335bfff640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.294+0000 7f3360dd9640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f335c164b00 0x7f335c0cd360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f335400ca20 tx=0x7f335400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.294+0000 7f3358ff9640 1 -- 172.21.15.67:0/2599704367 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354017070 con 0x7f335c164b00 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.294+0000 7f3358ff9640 1 -- 172.21.15.67:0/2599704367 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335400abd0 con 0x7f335c164b00 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.294+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c1e9fe0 con 0x7f335c164b00 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.294+0000 7f3358ff9640 1 -- 172.21.15.67:0/2599704367 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354012400 con 0x7f335c164b00 2024-04-25T15:25:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.294+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c1ea5a0 con 0x7f335c164b00 2024-04-25T15:25:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.295+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f335c050740 con 0x7f335c164b00 2024-04-25T15:25:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.295+0000 7f3358ff9640 1 -- 172.21.15.67:0/2599704367 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f335401a070 con 0x7f335c164b00 2024-04-25T15:25:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.296+0000 7f3358ff9640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f33480744b0 0x7f33480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.296+0000 7f3358ff9640 1 -- 172.21.15.67:0/2599704367 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3354094db0 con 0x7f335c164b00 2024-04-25T15:25:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.296+0000 7f335bfff640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f33480744b0 0x7f33480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.297+0000 7f335bfff640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f33480744b0 0x7f33480769c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3350009af0 tx=0x7f3350008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:38.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.301+0000 7f3358ff9640 1 -- 172.21.15.67:0/2599704367 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f335405ef00 con 0x7f335c164b00 2024-04-25T15:25:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.484+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f335c0d4170 con 0x7f33480744b0 2024-04-25T15:25:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.490+0000 7f3358ff9640 1 -- 172.21.15.67:0/2599704367 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f335c0d4170 con 0x7f33480744b0 2024-04-25T15:25:38.493 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:38.495 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:25:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f33480744b0 msgr2=0x7f33480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f33480744b0 0x7f33480769c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3350009af0 tx=0x7f3350008040 comp rx=0 tx=0).stop 2024-04-25T15:25:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f335c164b00 msgr2=0x7f335c0cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f335c164b00 0x7f335c0cd360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f335400ca20 tx=0x7f335400cee0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 shutdown_connections 2024-04-25T15:25:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f33480744b0 0x7f33480769c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f335c164b00 0x7f335c0cd360 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335c160890 0x7f335c1eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 --2- 172.21.15.67:0/2599704367 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f335c15df80 0x7f335c159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.493+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 >> 172.21.15.67:0/2599704367 conn(0x7f335c0d2bc0 msgr2=0x7f335c15f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.494+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 shutdown_connections 2024-04-25T15:25:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.494+0000 7f336283c640 1 -- 172.21.15.67:0/2599704367 wait complete. 2024-04-25T15:25:38.505 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-25T15:25:36.796901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002224"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659051+0000","last_change":"2024-04-25T15:25:29.325672+0000","last_active":"2024-04-25T15:25:30.659051+0000","last_peered":"2024-04-25T15:25:30.659051+0000","last_clean":"2024-04-25T15:25:30.659051+0000","last_became_active":"2024-04-25T15:25:29.324999+0000","last_became_peered":"2024-04-25T15:25:29.324999+0000","last_unstale":"2024-04-25T15:25:30.659051+0000","last_undegraded":"2024-04-25T15:25:30.659051+0000","last_fullsized":"2024-04-25T15:25:30.659051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293966+0000","last_change":"2024-04-25T15:25:29.303032+0000","last_active":"2024-04-25T15:25:30.293966+0000","last_peered":"2024-04-25T15:25:30.293966+0000","last_clean":"2024-04-25T15:25:30.293966+0000","last_became_active":"2024-04-25T15:25:29.302027+0000","last_became_peered":"2024-04-25T15:25:29.302027+0000","last_unstale":"2024-04-25T15:25:30.293966+0000","last_undegraded":"2024-04-25T15:25:30.293966+0000","last_fullsized":"2024-04-25T15:25:30.293966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297058+0000","last_change":"2024-04-25T15:25:29.307526+0000","last_active":"2024-04-25T15:25:30.297058+0000","last_peered":"2024-04-25T15:25:30.297058+0000","last_clean":"2024-04-25T15:25:30.297058+0000","last_became_active":"2024-04-25T15:25:29.307109+0000","last_became_peered":"2024-04-25T15:25:29.307109+0000","last_unstale":"2024-04-25T15:25:30.297058+0000","last_undegraded":"2024-04-25T15:25:30.297058+0000","last_fullsized":"2024-04-25T15:25:30.297058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323540+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322901+0000","last_became_peered":"2024-04-25T15:25:29.322901+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658248+0000","last_change":"2024-04-25T15:25:29.313182+0000","last_active":"2024-04-25T15:25:30.658248+0000","last_peered":"2024-04-25T15:25:30.658248+0000","last_clean":"2024-04-25T15:25:30.658248+0000","last_became_active":"2024-04-25T15:25:29.312850+0000","last_became_peered":"2024-04-25T15:25:29.312850+0000","last_unstale":"2024-04-25T15:25:30.658248+0000","last_undegraded":"2024-04-25T15:25:30.658248+0000","last_fullsized":"2024-04-25T15:25:30.658248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658795+0000","last_change":"2024-04-25T15:25:29.319442+0000","last_active":"2024-04-25T15:25:30.658795+0000","last_peered":"2024-04-25T15:25:30.658795+0000","last_clean":"2024-04-25T15:25:30.658795+0000","last_became_active":"2024-04-25T15:25:29.319141+0000","last_became_peered":"2024-04-25T15:25:29.319141+0000","last_unstale":"2024-04-25T15:25:30.658795+0000","last_undegraded":"2024-04-25T15:25:30.658795+0000","last_fullsized":"2024-04-25T15:25:30.658795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657979+0000","last_change":"2024-04-25T15:25:29.322616+0000","last_active":"2024-04-25T15:25:30.657979+0000","last_peered":"2024-04-25T15:25:30.657979+0000","last_clean":"2024-04-25T15:25:30.657979+0000","last_became_active":"2024-04-25T15:25:29.322311+0000","last_became_peered":"2024-04-25T15:25:29.322311+0000","last_unstale":"2024-04-25T15:25:30.657979+0000","last_undegraded":"2024-04-25T15:25:30.657979+0000","last_fullsized":"2024-04-25T15:25:30.657979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658002+0000","last_change":"2024-04-25T15:25:29.323286+0000","last_active":"2024-04-25T15:25:30.658002+0000","last_peered":"2024-04-25T15:25:30.658002+0000","last_clean":"2024-04-25T15:25:30.658002+0000","last_became_active":"2024-04-25T15:25:29.322911+0000","last_became_peered":"2024-04-25T15:25:29.322911+0000","last_unstale":"2024-04-25T15:25:30.658002+0000","last_undegraded":"2024-04-25T15:25:30.658002+0000","last_fullsized":"2024-04-25T15:25:30.658002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659143+0000","last_change":"2024-04-25T15:25:29.325686+0000","last_active":"2024-04-25T15:25:30.659143+0000","last_peered":"2024-04-25T15:25:30.659143+0000","last_clean":"2024-04-25T15:25:30.659143+0000","last_became_active":"2024-04-25T15:25:29.325332+0000","last_became_peered":"2024-04-25T15:25:29.325332+0000","last_unstale":"2024-04-25T15:25:30.659143+0000","last_undegraded":"2024-04-25T15:25:30.659143+0000","last_fullsized":"2024-04-25T15:25:30.659143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:20.087185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313378+0000","last_change":"2024-04-25T15:25:29.323852+0000","last_active":"2024-04-25T15:25:30.313378+0000","last_peered":"2024-04-25T15:25:30.313378+0000","last_clean":"2024-04-25T15:25:30.313378+0000","last_became_active":"2024-04-25T15:25:29.323524+0000","last_became_peered":"2024-04-25T15:25:29.323524+0000","last_unstale":"2024-04-25T15:25:30.313378+0000","last_undegraded":"2024-04-25T15:25:30.313378+0000","last_fullsized":"2024-04-25T15:25:30.313378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313308+0000","last_change":"2024-04-25T15:25:29.325815+0000","last_active":"2024-04-25T15:25:30.313308+0000","last_peered":"2024-04-25T15:25:30.313308+0000","last_clean":"2024-04-25T15:25:30.313308+0000","last_became_active":"2024-04-25T15:25:29.325436+0000","last_became_peered":"2024-04-25T15:25:29.325436+0000","last_unstale":"2024-04-25T15:25:30.313308+0000","last_undegraded":"2024-04-25T15:25:30.313308+0000","last_fullsized":"2024-04-25T15:25:30.313308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300011+0000","last_change":"2024-04-25T15:25:29.311301+0000","last_active":"2024-04-25T15:25:30.300011+0000","last_peered":"2024-04-25T15:25:30.300011+0000","last_clean":"2024-04-25T15:25:30.300011+0000","last_became_active":"2024-04-25T15:25:29.310966+0000","last_became_peered":"2024-04-25T15:25:29.310966+0000","last_unstale":"2024-04-25T15:25:30.300011+0000","last_undegraded":"2024-04-25T15:25:30.300011+0000","last_fullsized":"2024-04-25T15:25:30.300011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:44.511850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657661+0000","last_change":"2024-04-25T15:25:29.315008+0000","last_active":"2024-04-25T15:25:30.657661+0000","last_peered":"2024-04-25T15:25:30.657661+0000","last_clean":"2024-04-25T15:25:30.657661+0000","last_became_active":"2024-04-25T15:25:29.314675+0000","last_became_peered":"2024-04-25T15:25:29.314675+0000","last_unstale":"2024-04-25T15:25:30.657661+0000","last_undegraded":"2024-04-25T15:25:30.657661+0000","last_fullsized":"2024-04-25T15:25:30.657661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296694+0000","last_change":"2024-04-25T15:25:29.301255+0000","last_active":"2024-04-25T15:25:30.296694+0000","last_peered":"2024-04-25T15:25:30.296694+0000","last_clean":"2024-04-25T15:25:30.296694+0000","last_became_active":"2024-04-25T15:25:29.300877+0000","last_became_peered":"2024-04-25T15:25:29.300877+0000","last_unstale":"2024-04-25T15:25:30.296694+0000","last_undegraded":"2024-04-25T15:25:30.296694+0000","last_fullsized":"2024-04-25T15:25:30.296694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:40.562364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293711+0000","last_change":"2024-04-25T15:25:29.300064+0000","last_active":"2024-04-25T15:25:30.293711+0000","last_peered":"2024-04-25T15:25:30.293711+0000","last_clean":"2024-04-25T15:25:30.293711+0000","last_became_active":"2024-04-25T15:25:29.299759+0000","last_became_peered":"2024-04-25T15:25:29.299759+0000","last_unstale":"2024-04-25T15:25:30.293711+0000","last_undegraded":"2024-04-25T15:25:30.293711+0000","last_fullsized":"2024-04-25T15:25:30.293711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657884+0000","last_change":"2024-04-25T15:25:29.314325+0000","last_active":"2024-04-25T15:25:30.657884+0000","last_peered":"2024-04-25T15:25:30.657884+0000","last_clean":"2024-04-25T15:25:30.657884+0000","last_became_active":"2024-04-25T15:25:29.311827+0000","last_became_peered":"2024-04-25T15:25:29.311827+0000","last_unstale":"2024-04-25T15:25:30.657884+0000","last_undegraded":"2024-04-25T15:25:30.657884+0000","last_fullsized":"2024-04-25T15:25:30.657884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658676+0000","last_change":"2024-04-25T15:25:29.327856+0000","last_active":"2024-04-25T15:25:30.658676+0000","last_peered":"2024-04-25T15:25:30.658676+0000","last_clean":"2024-04-25T15:25:30.658676+0000","last_became_active":"2024-04-25T15:25:29.327408+0000","last_became_peered":"2024-04-25T15:25:29.327408+0000","last_unstale":"2024-04-25T15:25:30.658676+0000","last_undegraded":"2024-04-25T15:25:30.658676+0000","last_fullsized":"2024-04-25T15:25:30.658676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657624+0000","last_change":"2024-04-25T15:25:29.313009+0000","last_active":"2024-04-25T15:25:30.657624+0000","last_peered":"2024-04-25T15:25:30.657624+0000","last_clean":"2024-04-25T15:25:30.657624+0000","last_became_active":"2024-04-25T15:25:29.312684+0000","last_became_peered":"2024-04-25T15:25:29.312684+0000","last_unstale":"2024-04-25T15:25:30.657624+0000","last_undegraded":"2024-04-25T15:25:30.657624+0000","last_fullsized":"2024-04-25T15:25:30.657624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:26.944400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658764+0000","last_change":"2024-04-25T15:25:29.319894+0000","last_active":"2024-04-25T15:25:30.658764+0000","last_peered":"2024-04-25T15:25:30.658764+0000","last_clean":"2024-04-25T15:25:30.658764+0000","last_became_active":"2024-04-25T15:25:29.319566+0000","last_became_peered":"2024-04-25T15:25:29.319566+0000","last_unstale":"2024-04-25T15:25:30.658764+0000","last_undegraded":"2024-04-25T15:25:30.658764+0000","last_fullsized":"2024-04-25T15:25:30.658764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313203+0000","last_change":"2024-04-25T15:25:29.312046+0000","last_active":"2024-04-25T15:25:30.313203+0000","last_peered":"2024-04-25T15:25:30.313203+0000","last_clean":"2024-04-25T15:25:30.313203+0000","last_became_active":"2024-04-25T15:25:29.310121+0000","last_became_peered":"2024-04-25T15:25:29.310121+0000","last_unstale":"2024-04-25T15:25:30.313203+0000","last_undegraded":"2024-04-25T15:25:30.313203+0000","last_fullsized":"2024-04-25T15:25:30.313203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296808+0000","last_change":"2024-04-25T15:25:29.317661+0000","last_active":"2024-04-25T15:25:30.296808+0000","last_peered":"2024-04-25T15:25:30.296808+0000","last_clean":"2024-04-25T15:25:30.296808+0000","last_became_active":"2024-04-25T15:25:29.317302+0000","last_became_peered":"2024-04-25T15:25:29.317302+0000","last_unstale":"2024-04-25T15:25:30.296808+0000","last_undegraded":"2024-04-25T15:25:30.296808+0000","last_fullsized":"2024-04-25T15:25:30.296808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658934+0000","last_change":"2024-04-25T15:25:29.327012+0000","last_active":"2024-04-25T15:25:30.658934+0000","last_peered":"2024-04-25T15:25:30.658934+0000","last_clean":"2024-04-25T15:25:30.658934+0000","last_became_active":"2024-04-25T15:25:29.326589+0000","last_became_peered":"2024-04-25T15:25:29.326589+0000","last_unstale":"2024-04-25T15:25:30.658934+0000","last_undegraded":"2024-04-25T15:25:30.658934+0000","last_fullsized":"2024-04-25T15:25:30.658934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:49.176404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657788+0000","last_change":"2024-04-25T15:25:29.321180+0000","last_active":"2024-04-25T15:25:30.657788+0000","last_peered":"2024-04-25T15:25:30.657788+0000","last_clean":"2024-04-25T15:25:30.657788+0000","last_became_active":"2024-04-25T15:25:29.320679+0000","last_became_peered":"2024-04-25T15:25:29.320679+0000","last_unstale":"2024-04-25T15:25:30.657788+0000","last_undegraded":"2024-04-25T15:25:30.657788+0000","last_fullsized":"2024-04-25T15:25:30.657788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658226+0000","last_change":"2024-04-25T15:25:29.311398+0000","last_active":"2024-04-25T15:25:30.658226+0000","last_peered":"2024-04-25T15:25:30.658226+0000","last_clean":"2024-04-25T15:25:30.658226+0000","last_became_active":"2024-04-25T15:25:29.311069+0000","last_became_peered":"2024-04-25T15:25:29.311069+0000","last_unstale":"2024-04-25T15:25:30.658226+0000","last_undegraded":"2024-04-25T15:25:30.658226+0000","last_fullsized":"2024-04-25T15:25:30.658226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658383+0000","last_change":"2024-04-25T15:25:29.320926+0000","last_active":"2024-04-25T15:25:30.658383+0000","last_peered":"2024-04-25T15:25:30.658383+0000","last_clean":"2024-04-25T15:25:30.658383+0000","last_became_active":"2024-04-25T15:25:29.320574+0000","last_became_peered":"2024-04-25T15:25:29.320574+0000","last_unstale":"2024-04-25T15:25:30.658383+0000","last_undegraded":"2024-04-25T15:25:30.658383+0000","last_fullsized":"2024-04-25T15:25:30.658383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:57:54.790042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313448+0000","last_change":"2024-04-25T15:25:29.330048+0000","last_active":"2024-04-25T15:25:30.313448+0000","last_peered":"2024-04-25T15:25:30.313448+0000","last_clean":"2024-04-25T15:25:30.313448+0000","last_became_active":"2024-04-25T15:25:29.329720+0000","last_became_peered":"2024-04-25T15:25:29.329720+0000","last_unstale":"2024-04-25T15:25:30.313448+0000","last_undegraded":"2024-04-25T15:25:30.313448+0000","last_fullsized":"2024-04-25T15:25:30.313448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293796+0000","last_change":"2024-04-25T15:25:29.309985+0000","last_active":"2024-04-25T15:25:30.293796+0000","last_peered":"2024-04-25T15:25:30.293796+0000","last_clean":"2024-04-25T15:25:30.293796+0000","last_became_active":"2024-04-25T15:25:29.309619+0000","last_became_peered":"2024-04-25T15:25:29.309619+0000","last_unstale":"2024-04-25T15:25:30.293796+0000","last_undegraded":"2024-04-25T15:25:30.293796+0000","last_fullsized":"2024-04-25T15:25:30.293796+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658375+0000","last_change":"2024-04-25T15:25:29.321037+0000","last_active":"2024-04-25T15:25:30.658375+0000","last_peered":"2024-04-25T15:25:30.658375+0000","last_clean":"2024-04-25T15:25:30.658375+0000","last_became_active":"2024-04-25T15:25:29.320680+0000","last_became_peered":"2024-04-25T15:25:29.320680+0000","last_unstale":"2024-04-25T15:25:30.658375+0000","last_undegraded":"2024-04-25T15:25:30.658375+0000","last_fullsized":"2024-04-25T15:25:30.658375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296855+0000","last_change":"2024-04-25T15:25:29.309805+0000","last_active":"2024-04-25T15:25:30.296855+0000","last_peered":"2024-04-25T15:25:30.296855+0000","last_clean":"2024-04-25T15:25:30.296855+0000","last_became_active":"2024-04-25T15:25:29.309408+0000","last_became_peered":"2024-04-25T15:25:29.309408+0000","last_unstale":"2024-04-25T15:25:30.296855+0000","last_undegraded":"2024-04-25T15:25:30.296855+0000","last_fullsized":"2024-04-25T15:25:30.296855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659006+0000","last_change":"2024-04-25T15:25:29.318254+0000","last_active":"2024-04-25T15:25:30.659006+0000","last_peered":"2024-04-25T15:25:30.659006+0000","last_clean":"2024-04-25T15:25:30.659006+0000","last_became_active":"2024-04-25T15:25:29.317944+0000","last_became_peered":"2024-04-25T15:25:29.317944+0000","last_unstale":"2024-04-25T15:25:30.659006+0000","last_undegraded":"2024-04-25T15:25:30.659006+0000","last_fullsized":"2024-04-25T15:25:30.659006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:50.457796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657620+0000","last_change":"2024-04-25T15:25:29.313901+0000","last_active":"2024-04-25T15:25:30.657620+0000","last_peered":"2024-04-25T15:25:30.657620+0000","last_clean":"2024-04-25T15:25:30.657620+0000","last_became_active":"2024-04-25T15:25:29.313526+0000","last_became_peered":"2024-04-25T15:25:29.313526+0000","last_unstale":"2024-04-25T15:25:30.657620+0000","last_undegraded":"2024-04-25T15:25:30.657620+0000","last_fullsized":"2024-04-25T15:25:30.657620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313041+0000","last_change":"2024-04-25T15:25:29.312724+0000","last_active":"2024-04-25T15:25:30.313041+0000","last_peered":"2024-04-25T15:25:30.313041+0000","last_clean":"2024-04-25T15:25:30.313041+0000","last_became_active":"2024-04-25T15:25:29.312425+0000","last_became_peered":"2024-04-25T15:25:29.312425+0000","last_unstale":"2024-04-25T15:25:30.313041+0000","last_undegraded":"2024-04-25T15:25:30.313041+0000","last_fullsized":"2024-04-25T15:25:30.313041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299747+0000","last_change":"2024-04-25T15:25:29.298478+0000","last_active":"2024-04-25T15:25:30.299747+0000","last_peered":"2024-04-25T15:25:30.299747+0000","last_clean":"2024-04-25T15:25:30.299747+0000","last_became_active":"2024-04-25T15:25:29.298178+0000","last_became_peered":"2024-04-25T15:25:29.298178+0000","last_unstale":"2024-04-25T15:25:30.299747+0000","last_undegraded":"2024-04-25T15:25:30.299747+0000","last_fullsized":"2024-04-25T15:25:30.299747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:53.290039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658039+0000","last_change":"2024-04-25T15:25:28.301149+0000","last_active":"2024-04-25T15:25:30.658039+0000","last_peered":"2024-04-25T15:25:30.658039+0000","last_clean":"2024-04-25T15:25:30.658039+0000","last_became_active":"2024-04-25T15:25:28.300804+0000","last_became_peered":"2024-04-25T15:25:28.300804+0000","last_unstale":"2024-04-25T15:25:30.658039+0000","last_undegraded":"2024-04-25T15:25:30.658039+0000","last_fullsized":"2024-04-25T15:25:30.658039+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296993+0000","last_change":"2024-04-25T15:25:29.306337+0000","last_active":"2024-04-25T15:25:30.296993+0000","last_peered":"2024-04-25T15:25:30.296993+0000","last_clean":"2024-04-25T15:25:30.296993+0000","last_became_active":"2024-04-25T15:25:29.306039+0000","last_became_peered":"2024-04-25T15:25:29.306039+0000","last_unstale":"2024-04-25T15:25:30.296993+0000","last_undegraded":"2024-04-25T15:25:30.296993+0000","last_fullsized":"2024-04-25T15:25:30.296993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:28.889412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313038+0000","last_change":"2024-04-25T15:25:28.301076+0000","last_active":"2024-04-25T15:25:30.313038+0000","last_peered":"2024-04-25T15:25:30.313038+0000","last_clean":"2024-04-25T15:25:30.313038+0000","last_became_active":"2024-04-25T15:25:28.300670+0000","last_became_peered":"2024-04-25T15:25:28.300670+0000","last_unstale":"2024-04-25T15:25:30.313038+0000","last_undegraded":"2024-04-25T15:25:30.313038+0000","last_fullsized":"2024-04-25T15:25:30.313038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657997+0000","last_change":"2024-04-25T15:25:28.305288+0000","last_active":"2024-04-25T15:25:30.657997+0000","last_peered":"2024-04-25T15:25:30.657997+0000","last_clean":"2024-04-25T15:25:30.657997+0000","last_became_active":"2024-04-25T15:25:28.304900+0000","last_became_peered":"2024-04-25T15:25:28.304900+0000","last_unstale":"2024-04-25T15:25:30.657997+0000","last_undegraded":"2024-04-25T15:25:30.657997+0000","last_fullsized":"2024-04-25T15:25:30.657997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:04.261360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658085+0000","last_change":"2024-04-25T15:25:29.310703+0000","last_active":"2024-04-25T15:25:30.658085+0000","last_peered":"2024-04-25T15:25:30.658085+0000","last_clean":"2024-04-25T15:25:30.658085+0000","last_became_active":"2024-04-25T15:25:29.309942+0000","last_became_peered":"2024-04-25T15:25:29.309942+0000","last_unstale":"2024-04-25T15:25:30.658085+0000","last_undegraded":"2024-04-25T15:25:30.658085+0000","last_fullsized":"2024-04-25T15:25:30.658085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658034+0000","last_change":"2024-04-25T15:25:28.292273+0000","last_active":"2024-04-25T15:25:30.658034+0000","last_peered":"2024-04-25T15:25:30.658034+0000","last_clean":"2024-04-25T15:25:30.658034+0000","last_became_active":"2024-04-25T15:25:28.291890+0000","last_became_peered":"2024-04-25T15:25:28.291890+0000","last_unstale":"2024-04-25T15:25:30.658034+0000","last_undegraded":"2024-04-25T15:25:30.658034+0000","last_fullsized":"2024-04-25T15:25:30.658034+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293637+0000","last_change":"2024-04-25T15:25:29.310761+0000","last_active":"2024-04-25T15:25:30.293637+0000","last_peered":"2024-04-25T15:25:30.293637+0000","last_clean":"2024-04-25T15:25:30.293637+0000","last_became_active":"2024-04-25T15:25:29.310403+0000","last_became_peered":"2024-04-25T15:25:29.310403+0000","last_unstale":"2024-04-25T15:25:30.293637+0000","last_undegraded":"2024-04-25T15:25:30.293637+0000","last_fullsized":"2024-04-25T15:25:30.293637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313366+0000","last_change":"2024-04-25T15:25:28.306210+0000","last_active":"2024-04-25T15:25:30.313366+0000","last_peered":"2024-04-25T15:25:30.313366+0000","last_clean":"2024-04-25T15:25:30.313366+0000","last_became_active":"2024-04-25T15:25:28.305931+0000","last_became_peered":"2024-04-25T15:25:28.305931+0000","last_unstale":"2024-04-25T15:25:30.313366+0000","last_undegraded":"2024-04-25T15:25:30.313366+0000","last_fullsized":"2024-04-25T15:25:30.313366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658132+0000","last_change":"2024-04-25T15:25:29.328500+0000","last_active":"2024-04-25T15:25:30.658132+0000","last_peered":"2024-04-25T15:25:30.658132+0000","last_clean":"2024-04-25T15:25:30.658132+0000","last_became_active":"2024-04-25T15:25:29.328150+0000","last_became_peered":"2024-04-25T15:25:29.328150+0000","last_unstale":"2024-04-25T15:25:30.658132+0000","last_undegraded":"2024-04-25T15:25:30.658132+0000","last_fullsized":"2024-04-25T15:25:30.658132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658259+0000","last_change":"2024-04-25T15:25:28.305011+0000","last_active":"2024-04-25T15:25:30.658259+0000","last_peered":"2024-04-25T15:25:30.658259+0000","last_clean":"2024-04-25T15:25:30.658259+0000","last_became_active":"2024-04-25T15:25:28.304682+0000","last_became_peered":"2024-04-25T15:25:28.304682+0000","last_unstale":"2024-04-25T15:25:30.658259+0000","last_undegraded":"2024-04-25T15:25:30.658259+0000","last_fullsized":"2024-04-25T15:25:30.658259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297172+0000","last_change":"2024-04-25T15:25:29.318333+0000","last_active":"2024-04-25T15:25:30.297172+0000","last_peered":"2024-04-25T15:25:30.297172+0000","last_clean":"2024-04-25T15:25:30.297172+0000","last_became_active":"2024-04-25T15:25:29.317557+0000","last_became_peered":"2024-04-25T15:25:29.317557+0000","last_unstale":"2024-04-25T15:25:30.297172+0000","last_undegraded":"2024-04-25T15:25:30.297172+0000","last_fullsized":"2024-04-25T15:25:30.297172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:41.313185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293936+0000","last_change":"2024-04-25T15:25:28.297270+0000","last_active":"2024-04-25T15:25:30.293936+0000","last_peered":"2024-04-25T15:25:30.293936+0000","last_clean":"2024-04-25T15:25:30.293936+0000","last_became_active":"2024-04-25T15:25:28.296865+0000","last_became_peered":"2024-04-25T15:25:28.296865+0000","last_unstale":"2024-04-25T15:25:30.293936+0000","last_undegraded":"2024-04-25T15:25:30.293936+0000","last_fullsized":"2024-04-25T15:25:30.293936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300012+0000","last_change":"2024-04-25T15:25:29.324749+0000","last_active":"2024-04-25T15:25:30.300012+0000","last_peered":"2024-04-25T15:25:30.300012+0000","last_clean":"2024-04-25T15:25:30.300012+0000","last_became_active":"2024-04-25T15:25:29.324454+0000","last_became_peered":"2024-04-25T15:25:29.324454+0000","last_unstale":"2024-04-25T15:25:30.300012+0000","last_undegraded":"2024-04-25T15:25:30.300012+0000","last_fullsized":"2024-04-25T15:25:30.300012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:42.718665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313205+0000","last_change":"2024-04-25T15:25:28.302276+0000","last_active":"2024-04-25T15:25:30.313205+0000","last_peered":"2024-04-25T15:25:30.313205+0000","last_clean":"2024-04-25T15:25:30.313205+0000","last_became_active":"2024-04-25T15:25:28.302039+0000","last_became_peered":"2024-04-25T15:25:28.302039+0000","last_unstale":"2024-04-25T15:25:30.313205+0000","last_undegraded":"2024-04-25T15:25:30.313205+0000","last_fullsized":"2024-04-25T15:25:30.313205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658280+0000","last_change":"2024-04-25T15:25:29.324177+0000","last_active":"2024-04-25T15:25:30.658280+0000","last_peered":"2024-04-25T15:25:30.658280+0000","last_clean":"2024-04-25T15:25:30.658280+0000","last_became_active":"2024-04-25T15:25:29.323727+0000","last_became_peered":"2024-04-25T15:25:29.323727+0000","last_unstale":"2024-04-25T15:25:30.658280+0000","last_undegraded":"2024-04-25T15:25:30.658280+0000","last_fullsized":"2024-04-25T15:25:30.658280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296971+0000","last_change":"2024-04-25T15:25:28.288837+0000","last_active":"2024-04-25T15:25:30.296971+0000","last_peered":"2024-04-25T15:25:30.296971+0000","last_clean":"2024-04-25T15:25:30.296971+0000","last_became_active":"2024-04-25T15:25:28.288485+0000","last_became_peered":"2024-04-25T15:25:28.288485+0000","last_unstale":"2024-04-25T15:25:30.296971+0000","last_undegraded":"2024-04-25T15:25:30.296971+0000","last_fullsized":"2024-04-25T15:25:30.296971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:58.179326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296812+0000","last_change":"2024-04-25T15:25:29.323297+0000","last_active":"2024-04-25T15:25:30.296812+0000","last_peered":"2024-04-25T15:25:30.296812+0000","last_clean":"2024-04-25T15:25:30.296812+0000","last_became_active":"2024-04-25T15:25:29.322955+0000","last_became_peered":"2024-04-25T15:25:29.322955+0000","last_unstale":"2024-04-25T15:25:30.296812+0000","last_undegraded":"2024-04-25T15:25:30.296812+0000","last_fullsized":"2024-04-25T15:25:30.296812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659064+0000","last_change":"2024-04-25T15:25:28.300177+0000","last_active":"2024-04-25T15:25:30.659064+0000","last_peered":"2024-04-25T15:25:30.659064+0000","last_clean":"2024-04-25T15:25:30.659064+0000","last_became_active":"2024-04-25T15:25:28.298626+0000","last_became_peered":"2024-04-25T15:25:28.298626+0000","last_unstale":"2024-04-25T15:25:30.659064+0000","last_undegraded":"2024-04-25T15:25:30.659064+0000","last_fullsized":"2024-04-25T15:25:30.659064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657985+0000","last_change":"2024-04-25T15:25:29.316614+0000","last_active":"2024-04-25T15:25:30.657985+0000","last_peered":"2024-04-25T15:25:30.657985+0000","last_clean":"2024-04-25T15:25:30.657985+0000","last_became_active":"2024-04-25T15:25:29.316293+0000","last_became_peered":"2024-04-25T15:25:29.316293+0000","last_unstale":"2024-04-25T15:25:30.657985+0000","last_undegraded":"2024-04-25T15:25:30.657985+0000","last_fullsized":"2024-04-25T15:25:30.657985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312968+0000","last_change":"2024-04-25T15:25:28.304313+0000","last_active":"2024-04-25T15:25:30.312968+0000","last_peered":"2024-04-25T15:25:30.312968+0000","last_clean":"2024-04-25T15:25:30.312968+0000","last_became_active":"2024-04-25T15:25:28.303943+0000","last_became_peered":"2024-04-25T15:25:28.303943+0000","last_unstale":"2024-04-25T15:25:30.312968+0000","last_undegraded":"2024-04-25T15:25:30.312968+0000","last_fullsized":"2024-04-25T15:25:30.312968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313012+0000","last_change":"2024-04-25T15:25:29.310260+0000","last_active":"2024-04-25T15:25:30.313012+0000","last_peered":"2024-04-25T15:25:30.313012+0000","last_clean":"2024-04-25T15:25:30.313012+0000","last_became_active":"2024-04-25T15:25:29.309916+0000","last_became_peered":"2024-04-25T15:25:29.309916+0000","last_unstale":"2024-04-25T15:25:30.313012+0000","last_undegraded":"2024-04-25T15:25:30.313012+0000","last_fullsized":"2024-04-25T15:25:30.313012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:27.467948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657409+0000","last_change":"2024-04-25T15:25:28.293959+0000","last_active":"2024-04-25T15:25:30.657409+0000","last_peered":"2024-04-25T15:25:30.657409+0000","last_clean":"2024-04-25T15:25:30.657409+0000","last_became_active":"2024-04-25T15:25:28.293594+0000","last_became_peered":"2024-04-25T15:25:28.293594+0000","last_unstale":"2024-04-25T15:25:30.657409+0000","last_undegraded":"2024-04-25T15:25:30.657409+0000","last_fullsized":"2024-04-25T15:25:30.657409+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658071+0000","last_change":"2024-04-25T15:25:29.318796+0000","last_active":"2024-04-25T15:25:30.658071+0000","last_peered":"2024-04-25T15:25:30.658071+0000","last_clean":"2024-04-25T15:25:30.658071+0000","last_became_active":"2024-04-25T15:25:29.318218+0000","last_became_peered":"2024-04-25T15:25:29.318218+0000","last_unstale":"2024-04-25T15:25:30.658071+0000","last_undegraded":"2024-04-25T15:25:30.658071+0000","last_fullsized":"2024-04-25T15:25:30.658071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.300098+0000","last_change":"2024-04-25T15:25:28.293813+0000","last_active":"2024-04-25T15:25:30.300098+0000","last_peered":"2024-04-25T15:25:30.300098+0000","last_clean":"2024-04-25T15:25:30.300098+0000","last_became_active":"2024-04-25T15:25:28.293498+0000","last_became_peered":"2024-04-25T15:25:28.293498+0000","last_unstale":"2024-04-25T15:25:30.300098+0000","last_undegraded":"2024-04-25T15:25:30.300098+0000","last_fullsized":"2024-04-25T15:25:30.300098+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:31:00.192290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658957+0000","last_change":"2024-04-25T15:25:29.322896+0000","last_active":"2024-04-25T15:25:30.658957+0000","last_peered":"2024-04-25T15:25:30.658957+0000","last_clean":"2024-04-25T15:25:30.658957+0000","last_became_active":"2024-04-25T15:25:29.321170+0000","last_became_peered":"2024-04-25T15:25:29.321170+0000","last_unstale":"2024-04-25T15:25:30.658957+0000","last_undegraded":"2024-04-25T15:25:30.658957+0000","last_fullsized":"2024-04-25T15:25:30.658957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657752+0000","last_change":"2024-04-25T15:25:28.294831+0000","last_active":"2024-04-25T15:25:30.657752+0000","last_peered":"2024-04-25T15:25:30.657752+0000","last_clean":"2024-04-25T15:25:30.657752+0000","last_became_active":"2024-04-25T15:25:28.294257+0000","last_became_peered":"2024-04-25T15:25:28.294257+0000","last_unstale":"2024-04-25T15:25:30.657752+0000","last_undegraded":"2024-04-25T15:25:30.657752+0000","last_fullsized":"2024-04-25T15:25:30.657752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659232+0000","last_change":"2024-04-25T15:25:29.325425+0000","last_active":"2024-04-25T15:25:30.659232+0000","last_peered":"2024-04-25T15:25:30.659232+0000","last_clean":"2024-04-25T15:25:30.659232+0000","last_became_active":"2024-04-25T15:25:29.324916+0000","last_became_peered":"2024-04-25T15:25:29.324916+0000","last_unstale":"2024-04-25T15:25:30.659232+0000","last_undegraded":"2024-04-25T15:25:30.659232+0000","last_fullsized":"2024-04-25T15:25:30.659232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:36:46.643371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.294024+0000","last_change":"2024-04-25T15:25:28.297384+0000","last_active":"2024-04-25T15:25:30.294024+0000","last_peered":"2024-04-25T15:25:30.294024+0000","last_clean":"2024-04-25T15:25:30.294024+0000","last_became_active":"2024-04-25T15:25:28.297035+0000","last_became_peered":"2024-04-25T15:25:28.297035+0000","last_unstale":"2024-04-25T15:25:30.294024+0000","last_undegraded":"2024-04-25T15:25:30.294024+0000","last_fullsized":"2024-04-25T15:25:30.294024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658488+0000","last_change":"2024-04-25T15:25:29.323269+0000","last_active":"2024-04-25T15:25:30.658488+0000","last_peered":"2024-04-25T15:25:30.658488+0000","last_clean":"2024-04-25T15:25:30.658488+0000","last_became_active":"2024-04-25T15:25:29.322481+0000","last_became_peered":"2024-04-25T15:25:29.322481+0000","last_unstale":"2024-04-25T15:25:30.658488+0000","last_undegraded":"2024-04-25T15:25:30.658488+0000","last_fullsized":"2024-04-25T15:25:30.658488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657594+0000","last_change":"2024-04-25T15:25:28.293938+0000","last_active":"2024-04-25T15:25:30.657594+0000","last_peered":"2024-04-25T15:25:30.657594+0000","last_clean":"2024-04-25T15:25:30.657594+0000","last_became_active":"2024-04-25T15:25:28.293520+0000","last_became_peered":"2024-04-25T15:25:28.293520+0000","last_unstale":"2024-04-25T15:25:30.657594+0000","last_undegraded":"2024-04-25T15:25:30.657594+0000","last_fullsized":"2024-04-25T15:25:30.657594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658465+0000","last_change":"2024-04-25T15:25:29.325130+0000","last_active":"2024-04-25T15:25:30.658465+0000","last_peered":"2024-04-25T15:25:30.658465+0000","last_clean":"2024-04-25T15:25:30.658465+0000","last_became_active":"2024-04-25T15:25:29.324783+0000","last_became_peered":"2024-04-25T15:25:29.324783+0000","last_unstale":"2024-04-25T15:25:30.658465+0000","last_undegraded":"2024-04-25T15:25:30.658465+0000","last_fullsized":"2024-04-25T15:25:30.658465+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:14.821793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657696+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:25:30.657696+0000","last_peered":"2024-04-25T15:25:30.657696+0000","last_clean":"2024-04-25T15:25:30.657696+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:25:30.657696+0000","last_undegraded":"2024-04-25T15:25:30.657696+0000","last_fullsized":"2024-04-25T15:25:30.657696+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657863+0000","last_change":"2024-04-25T15:25:28.293187+0000","last_active":"2024-04-25T15:25:30.657863+0000","last_peered":"2024-04-25T15:25:30.657863+0000","last_clean":"2024-04-25T15:25:30.657863+0000","last_became_active":"2024-04-25T15:25:28.292748+0000","last_became_peered":"2024-04-25T15:25:28.292748+0000","last_unstale":"2024-04-25T15:25:30.657863+0000","last_undegraded":"2024-04-25T15:25:30.657863+0000","last_fullsized":"2024-04-25T15:25:30.657863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657964+0000","last_change":"2024-04-25T15:25:29.323272+0000","last_active":"2024-04-25T15:25:30.657964+0000","last_peered":"2024-04-25T15:25:30.657964+0000","last_clean":"2024-04-25T15:25:30.657964+0000","last_became_active":"2024-04-25T15:25:29.322937+0000","last_became_peered":"2024-04-25T15:25:29.322937+0000","last_unstale":"2024-04-25T15:25:30.657964+0000","last_undegraded":"2024-04-25T15:25:30.657964+0000","last_fullsized":"2024-04-25T15:25:30.657964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:42.499651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657646+0000","last_change":"2024-04-25T15:25:28.301819+0000","last_active":"2024-04-25T15:25:30.657646+0000","last_peered":"2024-04-25T15:25:30.657646+0000","last_clean":"2024-04-25T15:25:30.657646+0000","last_became_active":"2024-04-25T15:25:28.301519+0000","last_became_peered":"2024-04-25T15:25:28.301519+0000","last_unstale":"2024-04-25T15:25:30.657646+0000","last_undegraded":"2024-04-25T15:25:30.657646+0000","last_fullsized":"2024-04-25T15:25:30.657646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.504871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.296763+0000","last_change":"2024-04-25T15:25:29.306397+0000","last_active":"2024-04-25T15:25:30.296763+0000","last_peered":"2024-04-25T15:25:30.296763+0000","last_clean":"2024-04-25T15:25:30.296763+0000","last_became_active":"2024-04-25T15:25:29.306088+0000","last_became_peered":"2024-04-25T15:25:29.306088+0000","last_unstale":"2024-04-25T15:25:30.296763+0000","last_undegraded":"2024-04-25T15:25:30.296763+0000","last_fullsized":"2024-04-25T15:25:30.296763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:05.665096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657506+0000","last_change":"2024-04-25T15:25:28.295278+0000","last_active":"2024-04-25T15:25:30.657506+0000","last_peered":"2024-04-25T15:25:30.657506+0000","last_clean":"2024-04-25T15:25:30.657506+0000","last_became_active":"2024-04-25T15:25:28.294876+0000","last_became_peered":"2024-04-25T15:25:28.294876+0000","last_unstale":"2024-04-25T15:25:30.657506+0000","last_undegraded":"2024-04-25T15:25:30.657506+0000","last_fullsized":"2024-04-25T15:25:30.657506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293669+0000","last_change":"2024-04-25T15:25:29.307860+0000","last_active":"2024-04-25T15:25:30.293669+0000","last_peered":"2024-04-25T15:25:30.293669+0000","last_clean":"2024-04-25T15:25:30.293669+0000","last_became_active":"2024-04-25T15:25:29.307525+0000","last_became_peered":"2024-04-25T15:25:29.307525+0000","last_unstale":"2024-04-25T15:25:30.293669+0000","last_undegraded":"2024-04-25T15:25:30.293669+0000","last_fullsized":"2024-04-25T15:25:30.293669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:38.563351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.659149+0000","last_change":"2024-04-25T15:25:28.298243+0000","last_active":"2024-04-25T15:25:30.659149+0000","last_peered":"2024-04-25T15:25:30.659149+0000","last_clean":"2024-04-25T15:25:30.659149+0000","last_became_active":"2024-04-25T15:25:28.297837+0000","last_became_peered":"2024-04-25T15:25:28.297837+0000","last_unstale":"2024-04-25T15:25:30.659149+0000","last_undegraded":"2024-04-25T15:25:30.659149+0000","last_fullsized":"2024-04-25T15:25:30.659149+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299949+0000","last_change":"2024-04-25T15:25:29.308455+0000","last_active":"2024-04-25T15:25:30.299949+0000","last_peered":"2024-04-25T15:25:30.299949+0000","last_clean":"2024-04-25T15:25:30.299949+0000","last_became_active":"2024-04-25T15:25:29.307968+0000","last_became_peered":"2024-04-25T15:25:29.307968+0000","last_unstale":"2024-04-25T15:25:30.299949+0000","last_undegraded":"2024-04-25T15:25:30.299949+0000","last_fullsized":"2024-04-25T15:25:30.299949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:53.177957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293919+0000","last_change":"2024-04-25T15:25:28.298284+0000","last_active":"2024-04-25T15:25:30.293919+0000","last_peered":"2024-04-25T15:25:30.293919+0000","last_clean":"2024-04-25T15:25:30.293919+0000","last_became_active":"2024-04-25T15:25:28.297792+0000","last_became_peered":"2024-04-25T15:25:28.297792+0000","last_unstale":"2024-04-25T15:25:30.293919+0000","last_undegraded":"2024-04-25T15:25:30.293919+0000","last_fullsized":"2024-04-25T15:25:30.293919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658750+0000","last_change":"2024-04-25T15:25:29.313891+0000","last_active":"2024-04-25T15:25:30.658750+0000","last_peered":"2024-04-25T15:25:30.658750+0000","last_clean":"2024-04-25T15:25:30.658750+0000","last_became_active":"2024-04-25T15:25:29.313488+0000","last_became_peered":"2024-04-25T15:25:29.313488+0000","last_unstale":"2024-04-25T15:25:30.658750+0000","last_undegraded":"2024-04-25T15:25:30.658750+0000","last_fullsized":"2024-04-25T15:25:30.658750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658847+0000","last_change":"2024-04-25T15:25:28.301563+0000","last_active":"2024-04-25T15:25:30.658847+0000","last_peered":"2024-04-25T15:25:30.658847+0000","last_clean":"2024-04-25T15:25:30.658847+0000","last_became_active":"2024-04-25T15:25:28.301186+0000","last_became_peered":"2024-04-25T15:25:28.301186+0000","last_unstale":"2024-04-25T15:25:30.658847+0000","last_undegraded":"2024-04-25T15:25:30.658847+0000","last_fullsized":"2024-04-25T15:25:30.658847+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:25.937421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293726+0000","last_change":"2024-04-25T15:25:29.308391+0000","last_active":"2024-04-25T15:25:30.293726+0000","last_peered":"2024-04-25T15:25:30.293726+0000","last_clean":"2024-04-25T15:25:30.293726+0000","last_became_active":"2024-04-25T15:25:29.308010+0000","last_became_peered":"2024-04-25T15:25:29.308010+0000","last_unstale":"2024-04-25T15:25:30.293726+0000","last_undegraded":"2024-04-25T15:25:30.293726+0000","last_fullsized":"2024-04-25T15:25:30.293726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.312496+0000","last_change":"2024-04-25T15:25:28.304872+0000","last_active":"2024-04-25T15:25:30.312496+0000","last_peered":"2024-04-25T15:25:30.312496+0000","last_clean":"2024-04-25T15:25:30.312496+0000","last_became_active":"2024-04-25T15:25:28.304443+0000","last_became_peered":"2024-04-25T15:25:28.304443+0000","last_unstale":"2024-04-25T15:25:30.312496+0000","last_undegraded":"2024-04-25T15:25:30.312496+0000","last_fullsized":"2024-04-25T15:25:30.312496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:00.506266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658447+0000","last_change":"2024-04-25T15:25:29.326344+0000","last_active":"2024-04-25T15:25:30.658447+0000","last_peered":"2024-04-25T15:25:30.658447+0000","last_clean":"2024-04-25T15:25:30.658447+0000","last_became_active":"2024-04-25T15:25:29.326016+0000","last_became_peered":"2024-04-25T15:25:29.326016+0000","last_unstale":"2024-04-25T15:25:30.658447+0000","last_undegraded":"2024-04-25T15:25:30.658447+0000","last_fullsized":"2024-04-25T15:25:30.658447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.726792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313461+0000","last_change":"2024-04-25T15:25:28.307284+0000","last_active":"2024-04-25T15:25:30.313461+0000","last_peered":"2024-04-25T15:25:30.313461+0000","last_clean":"2024-04-25T15:25:30.313461+0000","last_became_active":"2024-04-25T15:25:28.307013+0000","last_became_peered":"2024-04-25T15:25:28.307013+0000","last_unstale":"2024-04-25T15:25:30.313461+0000","last_undegraded":"2024-04-25T15:25:30.313461+0000","last_fullsized":"2024-04-25T15:25:30.313461+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297033+0000","last_change":"2024-04-25T15:25:29.301384+0000","last_active":"2024-04-25T15:25:30.297033+0000","last_peered":"2024-04-25T15:25:30.297033+0000","last_clean":"2024-04-25T15:25:30.297033+0000","last_became_active":"2024-04-25T15:25:29.301021+0000","last_became_peered":"2024-04-25T15:25:29.301021+0000","last_unstale":"2024-04-25T15:25:30.297033+0000","last_undegraded":"2024-04-25T15:25:30.297033+0000","last_fullsized":"2024-04-25T15:25:30.297033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657611+0000","last_change":"2024-04-25T15:25:28.304086+0000","last_active":"2024-04-25T15:25:30.657611+0000","last_peered":"2024-04-25T15:25:30.657611+0000","last_clean":"2024-04-25T15:25:30.657611+0000","last_became_active":"2024-04-25T15:25:28.303771+0000","last_became_peered":"2024-04-25T15:25:28.303771+0000","last_unstale":"2024-04-25T15:25:30.657611+0000","last_undegraded":"2024-04-25T15:25:30.657611+0000","last_fullsized":"2024-04-25T15:25:30.657611+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:52.170979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.299860+0000","last_change":"2024-04-25T15:25:29.310784+0000","last_active":"2024-04-25T15:25:30.299860+0000","last_peered":"2024-04-25T15:25:30.299860+0000","last_clean":"2024-04-25T15:25:30.299860+0000","last_became_active":"2024-04-25T15:25:29.309711+0000","last_became_peered":"2024-04-25T15:25:29.309711+0000","last_unstale":"2024-04-25T15:25:30.299860+0000","last_undegraded":"2024-04-25T15:25:30.299860+0000","last_fullsized":"2024-04-25T15:25:30.299860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:16.888909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293756+0000","last_change":"2024-04-25T15:25:28.300848+0000","last_active":"2024-04-25T15:25:30.293756+0000","last_peered":"2024-04-25T15:25:30.293756+0000","last_clean":"2024-04-25T15:25:30.293756+0000","last_became_active":"2024-04-25T15:25:28.300218+0000","last_became_peered":"2024-04-25T15:25:28.300218+0000","last_unstale":"2024-04-25T15:25:30.293756+0000","last_undegraded":"2024-04-25T15:25:30.293756+0000","last_fullsized":"2024-04-25T15:25:30.293756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658212+0000","last_change":"2024-04-25T15:25:29.313291+0000","last_active":"2024-04-25T15:25:30.658212+0000","last_peered":"2024-04-25T15:25:30.658212+0000","last_clean":"2024-04-25T15:25:30.658212+0000","last_became_active":"2024-04-25T15:25:29.312960+0000","last_became_peered":"2024-04-25T15:25:29.312960+0000","last_unstale":"2024-04-25T15:25:30.658212+0000","last_undegraded":"2024-04-25T15:25:30.658212+0000","last_fullsized":"2024-04-25T15:25:30.658212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:35.455230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.293893+0000","last_change":"2024-04-25T15:25:28.301806+0000","last_active":"2024-04-25T15:25:30.293893+0000","last_peered":"2024-04-25T15:25:30.293893+0000","last_clean":"2024-04-25T15:25:30.293893+0000","last_became_active":"2024-04-25T15:25:28.301447+0000","last_became_peered":"2024-04-25T15:25:28.301447+0000","last_unstale":"2024-04-25T15:25:30.293893+0000","last_undegraded":"2024-04-25T15:25:30.293893+0000","last_fullsized":"2024-04-25T15:25:30.293893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.657861+0000","last_change":"2024-04-25T15:25:29.310336+0000","last_active":"2024-04-25T15:25:30.657861+0000","last_peered":"2024-04-25T15:25:30.657861+0000","last_clean":"2024-04-25T15:25:30.657861+0000","last_became_active":"2024-04-25T15:25:29.309914+0000","last_became_peered":"2024-04-25T15:25:29.309914+0000","last_unstale":"2024-04-25T15:25:30.657861+0000","last_undegraded":"2024-04-25T15:25:30.657861+0000","last_fullsized":"2024-04-25T15:25:30.657861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:45.338186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658401+0000","last_change":"2024-04-25T15:25:28.300815+0000","last_active":"2024-04-25T15:25:30.658401+0000","last_peered":"2024-04-25T15:25:30.658401+0000","last_clean":"2024-04-25T15:25:30.658401+0000","last_became_active":"2024-04-25T15:25:28.300487+0000","last_became_peered":"2024-04-25T15:25:28.300487+0000","last_unstale":"2024-04-25T15:25:30.658401+0000","last_undegraded":"2024-04-25T15:25:30.658401+0000","last_fullsized":"2024-04-25T15:25:30.658401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:15.881931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.313011+0000","last_change":"2024-04-25T15:25:29.310500+0000","last_active":"2024-04-25T15:25:30.313011+0000","last_peered":"2024-04-25T15:25:30.313011+0000","last_clean":"2024-04-25T15:25:30.313011+0000","last_became_active":"2024-04-25T15:25:29.310111+0000","last_became_peered":"2024-04-25T15:25:29.310111+0000","last_unstale":"2024-04-25T15:25:30.313011+0000","last_undegraded":"2024-04-25T15:25:30.313011+0000","last_fullsized":"2024-04-25T15:25:30.313011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:34:10.321070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658618+0000","last_change":"2024-04-25T15:25:29.314651+0000","last_active":"2024-04-25T15:25:30.658618+0000","last_peered":"2024-04-25T15:25:30.658618+0000","last_clean":"2024-04-25T15:25:30.658618+0000","last_became_active":"2024-04-25T15:25:29.314239+0000","last_became_peered":"2024-04-25T15:25:29.314239+0000","last_unstale":"2024-04-25T15:25:30.658618+0000","last_undegraded":"2024-04-25T15:25:30.658618+0000","last_fullsized":"2024-04-25T15:25:30.658618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:33.876839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658320+0000","last_change":"2024-04-25T15:25:28.304792+0000","last_active":"2024-04-25T15:25:30.658320+0000","last_peered":"2024-04-25T15:25:30.658320+0000","last_clean":"2024-04-25T15:25:30.658320+0000","last_became_active":"2024-04-25T15:25:28.304489+0000","last_became_peered":"2024-04-25T15:25:28.304489+0000","last_unstale":"2024-04-25T15:25:30.658320+0000","last_undegraded":"2024-04-25T15:25:30.658320+0000","last_fullsized":"2024-04-25T15:25:30.658320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:03:52.283061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658967+0000","last_change":"2024-04-25T15:25:29.318608+0000","last_active":"2024-04-25T15:25:30.658967+0000","last_peered":"2024-04-25T15:25:30.658967+0000","last_clean":"2024-04-25T15:25:30.658967+0000","last_became_active":"2024-04-25T15:25:29.317330+0000","last_became_peered":"2024-04-25T15:25:29.317330+0000","last_unstale":"2024-04-25T15:25:30.658967+0000","last_undegraded":"2024-04-25T15:25:30.658967+0000","last_fullsized":"2024-04-25T15:25:30.658967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:09.650557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658591+0000","last_change":"2024-04-25T15:25:28.305376+0000","last_active":"2024-04-25T15:25:30.658591+0000","last_peered":"2024-04-25T15:25:30.658591+0000","last_clean":"2024-04-25T15:25:30.658591+0000","last_became_active":"2024-04-25T15:25:28.305083+0000","last_became_peered":"2024-04-25T15:25:28.305083+0000","last_unstale":"2024-04-25T15:25:30.658591+0000","last_undegraded":"2024-04-25T15:25:30.658591+0000","last_fullsized":"2024-04-25T15:25:30.658591+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:38:41.711687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297061+0000","last_change":"2024-04-25T15:25:29.319184+0000","last_active":"2024-04-25T15:25:30.297061+0000","last_peered":"2024-04-25T15:25:30.297061+0000","last_clean":"2024-04-25T15:25:30.297061+0000","last_became_active":"2024-04-25T15:25:29.317757+0000","last_became_peered":"2024-04-25T15:25:29.317757+0000","last_unstale":"2024-04-25T15:25:30.297061+0000","last_undegraded":"2024-04-25T15:25:30.297061+0000","last_fullsized":"2024-04-25T15:25:30.297061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:37.023625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658361+0000","last_change":"2024-04-25T15:25:28.298019+0000","last_active":"2024-04-25T15:25:30.658361+0000","last_peered":"2024-04-25T15:25:30.658361+0000","last_clean":"2024-04-25T15:25:30.658361+0000","last_became_active":"2024-04-25T15:25:28.297570+0000","last_became_peered":"2024-04-25T15:25:28.297570+0000","last_unstale":"2024-04-25T15:25:30.658361+0000","last_undegraded":"2024-04-25T15:25:30.658361+0000","last_fullsized":"2024-04-25T15:25:30.658361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:52.810491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.658351+0000","last_change":"2024-04-25T15:25:29.316068+0000","last_active":"2024-04-25T15:25:30.658351+0000","last_peered":"2024-04-25T15:25:30.658351+0000","last_clean":"2024-04-25T15:25:30.658351+0000","last_became_active":"2024-04-25T15:25:29.315705+0000","last_became_peered":"2024-04-25T15:25:29.315705+0000","last_unstale":"2024-04-25T15:25:30.658351+0000","last_undegraded":"2024-04-25T15:25:30.658351+0000","last_fullsized":"2024-04-25T15:25:30.658351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:28.275524+0000","last_clean_scrub_stamp":"2024-04-25T15:25:28.275524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:02.118812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:25:30.297170+0000","last_change":"2024-04-25T15:25:28.300551+0000","last_active":"2024-04-25T15:25:30.297170+0000","last_peered":"2024-04-25T15:25:30.297170+0000","last_clean":"2024-04-25T15:25:30.297170+0000","last_became_active":"2024-04-25T15:25:28.299984+0000","last_became_peered":"2024-04-25T15:25:28.299984+0000","last_unstale":"2024-04-25T15:25:30.297170+0000","last_undegraded":"2024-04-25T15:25:30.297170+0000","last_fullsized":"2024-04-25T15:25:30.297170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:27.268546+0000","last_clean_scrub_stamp":"2024-04-25T15:25:27.268546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:08:02.904549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:25:38.509 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T15:25:38.509 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T15:25:38.510 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T15:25:38.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:25:38.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.655+0000 7efc4e8e7640 1 Processor -- start 2024-04-25T15:25:38.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.655+0000 7efc4e8e7640 1 -- start start 2024-04-25T15:25:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc4e8e7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc480b8290 0x7efc48160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc4e8e7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc481615f0 con 0x7efc4815df80 2024-04-25T15:25:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc4e8e7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc48164020 con 0x7efc48164b00 2024-04-25T15:25:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc4e8e7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efc48164670 con 0x7efc480b8290 2024-04-25T15:25:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc4ce84640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efc4815df80 0x7efc48160dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53728/0 (socket says 172.21.15.67:53728) 2024-04-25T15:25:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc4ce84640 1 -- 172.21.15.67:0/2162238978 learned_addr learned my addr 172.21.15.67:0/2162238978 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc477fe640 1 --2- 172.21.15.67:0/2162238978 >> v2:172.21.15.67:3300/0 conn(0x7efc480b8290 0x7efc48160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc46ffd640 1 -- 172.21.15.67:0/2162238978 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1197967823 0 0) 0x7efc481615f0 con 0x7efc4815df80 2024-04-25T15:25:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc46ffd640 1 -- 172.21.15.67:0/2162238978 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc30003520 con 0x7efc4815df80 2024-04-25T15:25:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc477fe640 1 -- 172.21.15.67:0/2162238978 >> v1:172.21.15.176:6789/0 conn(0x7efc48164b00 legacy=0x7efc480b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc477fe640 1 -- 172.21.15.67:0/2162238978 >> v1:172.21.15.67:6789/0 conn(0x7efc4815df80 legacy=0x7efc48160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.657+0000 7efc477fe640 1 -- 172.21.15.67:0/2162238978 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc48164300 con 0x7efc480b8290 2024-04-25T15:25:38.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.658+0000 7efc477fe640 1 --2- 172.21.15.67:0/2162238978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc480b8290 0x7efc48160890 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc481647f0 tx=0x7efc3802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bbc6311eb41e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:38.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.658+0000 7efc46ffd640 1 -- 172.21.15.67:0/2162238978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc38037070 con 0x7efc480b8290 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.658+0000 7efc46ffd640 1 -- 172.21.15.67:0/2162238978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc38032440 con 0x7efc480b8290 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.658+0000 7efc46ffd640 1 -- 172.21.15.67:0/2162238978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc38032810 con 0x7efc480b8290 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.659+0000 7efc4e8e7640 1 -- 172.21.15.67:0/2162238978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc480b8290 msgr2=0x7efc48160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.659+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/2162238978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc480b8290 0x7efc48160890 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7efc481647f0 tx=0x7efc3802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.659+0000 7efc4e8e7640 1 -- 172.21.15.67:0/2162238978 shutdown_connections 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.659+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/2162238978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc480b8290 0x7efc48160890 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.659+0000 7efc4e8e7640 1 -- 172.21.15.67:0/2162238978 >> 172.21.15.67:0/2162238978 conn(0x7efc480d2bc0 msgr2=0x7efc480d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.659+0000 7efc4e8e7640 1 -- 172.21.15.67:0/2162238978 shutdown_connections 2024-04-25T15:25:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.659+0000 7efc4e8e7640 1 -- 172.21.15.67:0/2162238978 wait complete. 2024-04-25T15:25:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.661+0000 7efc4e8e7640 1 Processor -- start 2024-04-25T15:25:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.661+0000 7efc4e8e7640 1 -- start start 2024-04-25T15:25:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc4e8e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc4815df80 0x7efc48154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc4e8e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc48164b00 0x7efc481517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc4e8e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc48157110 0x7efc48157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc4e8e7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efc480b4c90 con 0x7efc4815df80 2024-04-25T15:25:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc4e8e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efc480b4b10 con 0x7efc48157110 2024-04-25T15:25:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc4e8e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efc480b4e10 con 0x7efc48164b00 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc4815df80 0x7efc48154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc4815df80 0x7efc48154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38076/0 (socket says 172.21.15.67:38076) 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 -- 172.21.15.67:0/789271970 learned_addr learned my addr 172.21.15.67:0/789271970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc4ce84640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc48157110 0x7efc48157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc477fe640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc48164b00 0x7efc481517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 -- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc48164b00 msgr2=0x7efc481517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc48164b00 0x7efc481517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 -- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc48157110 msgr2=0x7efc48157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc48157110 0x7efc48157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.662+0000 7efc47fff640 1 -- 172.21.15.67:0/789271970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc481537b0 con 0x7efc4815df80 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.663+0000 7efc47fff640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc4815df80 0x7efc48154850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00b420 tx=0x7efc3c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:38.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.663+0000 7efc44ff9640 1 -- 172.21.15.67:0/789271970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c011830 con 0x7efc4815df80 2024-04-25T15:25:38.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.663+0000 7efc44ff9640 1 -- 172.21.15.67:0/789271970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc3c016070 con 0x7efc4815df80 2024-04-25T15:25:38.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.663+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc48153a90 con 0x7efc4815df80 2024-04-25T15:25:38.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.663+0000 7efc44ff9640 1 -- 172.21.15.67:0/789271970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c011c10 con 0x7efc4815df80 2024-04-25T15:25:38.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.663+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc48154050 con 0x7efc4815df80 2024-04-25T15:25:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.664+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc48050740 con 0x7efc4815df80 2024-04-25T15:25:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.664+0000 7efc44ff9640 1 -- 172.21.15.67:0/789271970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efc3c01b460 con 0x7efc4815df80 2024-04-25T15:25:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.665+0000 7efc44ff9640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc340744e0 0x7efc340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.665+0000 7efc44ff9640 1 -- 172.21.15.67:0/789271970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efc3c094cb0 con 0x7efc4815df80 2024-04-25T15:25:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.665+0000 7efc477fe640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc340744e0 0x7efc340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.666+0000 7efc477fe640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc340744e0 0x7efc340769f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc38034000 tx=0x7efc3802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:38.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.670+0000 7efc44ff9640 1 -- 172.21.15.67:0/789271970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc3c020020 con 0x7efc4815df80 2024-04-25T15:25:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.895+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7efc480d4170 con 0x7efc4815df80 2024-04-25T15:25:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.895+0000 7efc44ff9640 1 -- 172.21.15.67:0/789271970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7efc3c05f4d0 con 0x7efc4815df80 2024-04-25T15:25:38.896 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:38.897 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.897+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc340744e0 msgr2=0x7efc340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.897+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc340744e0 0x7efc340769f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc38034000 tx=0x7efc3802e000 comp rx=0 tx=0).stop 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc4815df80 msgr2=0x7efc48154850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc4815df80 0x7efc48154850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00b420 tx=0x7efc3c00b8e0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 shutdown_connections 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc340744e0 0x7efc340769f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc48157110 0x7efc48157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc48164b00 0x7efc481517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 --2- 172.21.15.67:0/789271970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc4815df80 0x7efc48154850 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 >> 172.21.15.67:0/789271970 conn(0x7efc480d2bc0 msgr2=0x7efc48160720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 shutdown_connections 2024-04-25T15:25:38.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:38.898+0000 7efc4e8e7640 1 -- 172.21.15.67:0/789271970 wait complete. 2024-04-25T15:25:38.909 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T15:25:38.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:39.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.019+0000 7f3a50e29640 1 Processor -- start 2024-04-25T15:25:39.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.020+0000 7f3a50e29640 1 -- start start 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a50e29640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3a4c164b00 0x7f3a4c0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a50e29640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a4c0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a50e29640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a4c1615f0 con 0x7f3a4c15df80 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a50e29640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3a4c1613b0 con 0x7f3a4c164b00 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a50e29640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4c164670 con 0x7f3a4c160fa0 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a49d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a4c15df80 0x7f3a4c160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53730/0 (socket says 172.21.15.67:53730) 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a49d9b640 1 -- 172.21.15.67:0/2161784190 learned_addr learned my addr 172.21.15.67:0/2161784190 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a4ad9d640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a4c0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/2161784190 >> v2:172.21.15.176:3300/0 conn(0x7f3a4c164b00 0x7f3a4c0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 969572487 0 0) 0x7f3a4c1615f0 con 0x7f3a4c15df80 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.021+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a38003520 con 0x7f3a4c15df80 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3623425522 0 0) 0x7f3a38003520 con 0x7f3a4c15df80 2024-04-25T15:25:39.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 msgr2=0x7f3a4c0b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a4c0b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 >> v2:172.21.15.176:3300/0 conn(0x7f3a4c164b00 msgr2=0x7f3a4c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 --2- 172.21.15.67:0/2161784190 >> v2:172.21.15.176:3300/0 conn(0x7f3a4c164b00 0x7f3a4c0b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c164020 con 0x7f3a4c15df80 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f3a34001670 con 0x7f3a4c15df80 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4ad9d640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a4c0b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/2161784190 >> v2:172.21.15.176:3300/0 conn(0x7f3a4c164b00 0x7f3a4c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 >> v1:172.21.15.67:6789/0 conn(0x7f3a4c15df80 legacy=0x7f3a4c160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380039b0 0x7f3a38003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38004800 0x7f3a38004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a38005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1613b0 con 0x7f3a380039b0 2024-04-25T15:25:39.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c164670 con 0x7f3a38004800 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3800b9c0 con 0x7f3a4c160fa0 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380039b0 0x7f3a38003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a49d9b640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38004800 0x7f3a38004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4ad9d640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a38005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4a59c640 1 -- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 msgr2=0x7f3a38005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a38005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4a59c640 1 -- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38004800 msgr2=0x7f3a38004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38004800 0x7f3a38004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4a59c640 1 -- 172.21.15.67:0/2161784190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a38003520 con 0x7f3a380039b0 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a49d9b640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38004800 0x7f3a38004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.022+0000 7f3a4ad9d640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c160fa0 0x7f3a38005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.023+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380039b0 0x7f3a38003e20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009c90 tx=0x7f3a38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8873cad1b2442c91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.023+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44036070 con 0x7f3a380039b0 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.023+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a44031400 con 0x7f3a380039b0 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.023+0000 7f3a4959a640 1 -- 172.21.15.67:0/2161784190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44009860 con 0x7f3a380039b0 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.024+0000 7f3a50e29640 1 -- 172.21.15.67:0/2161784190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380039b0 msgr2=0x7f3a38003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.024+0000 7f3a50e29640 1 --2- 172.21.15.67:0/2161784190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380039b0 0x7f3a38003e20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009c90 tx=0x7f3a38005830 comp rx=0 tx=0).stop 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.024+0000 7f3a4ad9d640 1 -- 172.21.15.67:0/2161784190 reap_dead start 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.024+0000 7f3a50e29640 1 -- 172.21.15.67:0/2161784190 shutdown_connections 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.024+0000 7f3a50e29640 1 -- 172.21.15.67:0/2161784190 >> 172.21.15.67:0/2161784190 conn(0x7f3a4c0d2bc0 msgr2=0x7f3a4c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.024+0000 7f3a50e29640 1 -- 172.21.15.67:0/2161784190 shutdown_connections 2024-04-25T15:25:39.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.024+0000 7f3a50e29640 1 -- 172.21.15.67:0/2161784190 wait complete. 2024-04-25T15:25:39.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.026+0000 7f3a50e29640 1 Processor -- start 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.026+0000 7f3a50e29640 1 -- start start 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a50e29640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a380039b0 0x7f3a4c0c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a50e29640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4c1618d0 0x7f3a4c161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a50e29640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c14b8a0 0x7f3a4c14bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a50e29640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c0b8240 con 0x7f3a4c1618d0 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a50e29640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c0b80c0 con 0x7f3a380039b0 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a50e29640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4c0b83c0 con 0x7f3a4c14b8a0 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4c1618d0 0x7f3a4c161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4c1618d0 0x7f3a4c161ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38092/0 (socket says 172.21.15.67:38092) 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 -- 172.21.15.67:0/3962162546 learned_addr learned my addr 172.21.15.67:0/3962162546 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a4ad9d640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c14b8a0 0x7f3a4c14bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a380039b0 0x7f3a4c0c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 -- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c14b8a0 msgr2=0x7f3a4c14bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c14b8a0 0x7f3a4c14bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 -- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a380039b0 msgr2=0x7f3a4c0c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a380039b0 0x7f3a4c0c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.027+0000 7f3a49d9b640 1 -- 172.21.15.67:0/3962162546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c162220 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.028+0000 7f3a49d9b640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4c1618d0 0x7f3a4c161ce0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400e7f0 tx=0x7f3a3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.028+0000 7f3a42ffd640 1 -- 172.21.15.67:0/3962162546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3400c870 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.028+0000 7f3a42ffd640 1 -- 172.21.15.67:0/3962162546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a34018070 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.028+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c14c590 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.028+0000 7f3a42ffd640 1 -- 172.21.15.67:0/3962162546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3400cc10 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.028+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c0ccb40 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.029+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a4c050740 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.029+0000 7f3a42ffd640 1 -- 172.21.15.67:0/3962162546 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3a3401d460 con 0x7f3a4c1618d0 2024-04-25T15:25:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.030+0000 7f3a42ffd640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a2c0744e0 0x7f3a2c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.030+0000 7f3a42ffd640 1 -- 172.21.15.67:0/3962162546 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3a34095020 con 0x7f3a4c1618d0 2024-04-25T15:25:39.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.030+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a2c0744e0 0x7f3a2c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.031+0000 7f3a4a59c640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a2c0744e0 0x7f3a2c0769f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3a44033000 tx=0x7f3a44008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.035+0000 7f3a42ffd640 1 -- 172.21.15.67:0/3962162546 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a4c162220 con 0x7f3a4c1618d0 2024-04-25T15:25:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.219+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a4c0d4170 con 0x7f3a4c1618d0 2024-04-25T15:25:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.220+0000 7f3a42ffd640 1 -- 172.21.15.67:0/3962162546 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f3a3405f160 con 0x7f3a4c1618d0 2024-04-25T15:25:39.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:39.221 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a2c0744e0 msgr2=0x7f3a2c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a2c0744e0 0x7f3a2c0769f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3a44033000 tx=0x7f3a44008030 comp rx=0 tx=0).stop 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4c1618d0 msgr2=0x7f3a4c161ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4c1618d0 0x7f3a4c161ce0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400e7f0 tx=0x7f3a3400ecb0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 shutdown_connections 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a2c0744e0 0x7f3a2c0769f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a4c14b8a0 0x7f3a4c14bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a4c1618d0 0x7f3a4c161ce0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 --2- 172.21.15.67:0/3962162546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a380039b0 0x7f3a4c0c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 >> 172.21.15.67:0/3962162546 conn(0x7f3a4c0d2bc0 msgr2=0x7f3a4c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 shutdown_connections 2024-04-25T15:25:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.222+0000 7f3a50e29640 1 -- 172.21.15.67:0/3962162546 wait complete. 2024-04-25T15:25:39.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:39.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.350+0000 7f441ce05640 1 Processor -- start 2024-04-25T15:25:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.350+0000 7f441ce05640 1 -- start start 2024-04-25T15:25:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f441ce05640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 0x7f441816b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f441ce05640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44180cea10 con 0x7f44180b7f00 2024-04-25T15:25:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f441ce05640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44180cecf0 con 0x7f44180b40b0 2024-04-25T15:25:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f441ce05640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44180b1b10 con 0x7f4418163b10 2024-04-25T15:25:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f4415d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f44180b7f00 0x7f44180cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53732/0 (socket says 172.21.15.67:53732) 2024-04-25T15:25:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f4415d9b640 1 -- 172.21.15.67:0/3902735858 learned_addr learned my addr 172.21.15.67:0/3902735858 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f4416d9d640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 0x7f441816b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.351+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458860406 0 0) 0x7f44180cea10 con 0x7f44180b7f00 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4400003520 con 0x7f44180b7f00 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1995365888 0 0) 0x7f44180cecf0 con 0x7f44180b40b0 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44180cea10 con 0x7f44180b40b0 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3691469337 0 0) 0x7f4400003520 con 0x7f44180b7f00 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 msgr2=0x7f441816b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 0x7f441816b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 >> v1:172.21.15.176:6789/0 conn(0x7f44180b40b0 legacy=0x7f44180b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44180d1f10 con 0x7f44180b7f00 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f440c001670 con 0x7f44180b7f00 2024-04-25T15:25:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4416d9d640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 0x7f441816b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 >> v1:172.21.15.67:6789/0 conn(0x7f44180b7f00 legacy=0x7f44180cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 0x7f4400003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44000040d0 0x7f4400004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4400005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f440000baf0 con 0x7f4400004d40 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f440000bca0 con 0x7f44000040d0 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f440000bea0 con 0x7f4418163b10 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4416d9d640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4400005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f441659c640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 0x7f4400003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4415d9b640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44000040d0 0x7f4400004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4416d9d640 1 -- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 msgr2=0x7f4400003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4416d9d640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418163b10 0x7f4400003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4416d9d640 1 -- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44000040d0 msgr2=0x7f4400004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4416d9d640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44000040d0 0x7f4400004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.352+0000 7f4416d9d640 1 -- 172.21.15.67:0/3902735858 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4400003520 con 0x7f4400004d40 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.353+0000 7f4416d9d640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4400005150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f441806eea0 tx=0x7f4400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423e2158f10a09d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.353+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408033020 con 0x7f4400004d40 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.353+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4408003b90 con 0x7f4400004d40 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.353+0000 7f441559a640 1 -- 172.21.15.67:0/3902735858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408009dc0 con 0x7f4400004d40 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.354+0000 7f441ce05640 1 -- 172.21.15.67:0/3902735858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 msgr2=0x7f4400005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.354+0000 7f441ce05640 1 --2- 172.21.15.67:0/3902735858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4400005150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f441806eea0 tx=0x7f4400005830 comp rx=0 tx=0).stop 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.354+0000 7f4416d9d640 1 -- 172.21.15.67:0/3902735858 reap_dead start 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.354+0000 7f441ce05640 1 -- 172.21.15.67:0/3902735858 shutdown_connections 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.354+0000 7f441ce05640 1 -- 172.21.15.67:0/3902735858 >> 172.21.15.67:0/3902735858 conn(0x7f441809b7d0 msgr2=0x7f441809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.354+0000 7f441ce05640 1 -- 172.21.15.67:0/3902735858 shutdown_connections 2024-04-25T15:25:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.354+0000 7f441ce05640 1 -- 172.21.15.67:0/3902735858 wait complete. 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.356+0000 7f441ce05640 1 Processor -- start 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.356+0000 7f441ce05640 1 -- start start 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441ce05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4418158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441ce05640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418160a30 0x7f441806d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441ce05640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4418160e40 0x7f441806dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441ce05640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f44180d3720 con 0x7f4400004d40 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441ce05640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f44180ce7b0 con 0x7f4418160e40 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441ce05640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44180d38a0 con 0x7f4418160a30 2024-04-25T15:25:39.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4418158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4418158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38102/0 (socket says 172.21.15.67:38102) 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.357+0000 7f441659c640 1 -- 172.21.15.67:0/4112407142 learned_addr learned my addr 172.21.15.67:0/4112407142 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f4415d9b640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418160a30 0x7f441806d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f4416d9d640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4418160e40 0x7f441806dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441659c640 1 -- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418160a30 msgr2=0x7f441806d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441659c640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418160a30 0x7f441806d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441659c640 1 -- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4418160e40 msgr2=0x7f441806dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441659c640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4418160e40 0x7f441806dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441659c640 1 -- 172.21.15.67:0/4112407142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441815fcb0 con 0x7f4400004d40 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441659c640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4418158460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f441000f760 tx=0x7f441000fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f4406ffd640 1 -- 172.21.15.67:0/4112407142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441000fe70 con 0x7f4400004d40 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f4406ffd640 1 -- 172.21.15.67:0/4112407142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4410018070 con 0x7f4400004d40 2024-04-25T15:25:39.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44181615a0 con 0x7f4400004d40 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f4406ffd640 1 -- 172.21.15.67:0/4112407142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441000d730 con 0x7f4400004d40 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.358+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4418161b60 con 0x7f4400004d40 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.359+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4418050740 con 0x7f4400004d40 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.360+0000 7f4406ffd640 1 -- 172.21.15.67:0/4112407142 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f441000d8d0 con 0x7f4400004d40 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.360+0000 7f4406ffd640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43f80744b0 0x7f43f80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.360+0000 7f4406ffd640 1 -- 172.21.15.67:0/4112407142 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4410095220 con 0x7f4400004d40 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.360+0000 7f4415d9b640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43f80744b0 0x7f43f80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.361+0000 7f4415d9b640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43f80744b0 0x7f43f80769c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f44181f94d0 tx=0x7f440c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.365+0000 7f4406ffd640 1 -- 172.21.15.67:0/4112407142 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f441005f9c0 con 0x7f4400004d40 2024-04-25T15:25:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.549+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f44180b2040 con 0x7f4400004d40 2024-04-25T15:25:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.549+0000 7f4406ffd640 1 -- 172.21.15.67:0/4112407142 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f441005f2c0 con 0x7f4400004d40 2024-04-25T15:25:39.550 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43f80744b0 msgr2=0x7f43f80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43f80744b0 0x7f43f80769c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f44181f94d0 tx=0x7f440c009040 comp rx=0 tx=0).stop 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 msgr2=0x7f4418158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4418158460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f441000f760 tx=0x7f441000fc20 comp rx=0 tx=0).stop 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 shutdown_connections 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43f80744b0 0x7f43f80769c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4418160e40 0x7f441806dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4418160a30 0x7f441806d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 --2- 172.21.15.67:0/4112407142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4400004d40 0x7f4418158460 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 >> 172.21.15.67:0/4112407142 conn(0x7f441809b7d0 msgr2=0x7f44180b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 shutdown_connections 2024-04-25T15:25:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.551+0000 7f441ce05640 1 -- 172.21.15.67:0/4112407142 wait complete. 2024-04-25T15:25:39.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:39.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.678+0000 7f4d6976c640 1 Processor -- start 2024-04-25T15:25:39.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.678+0000 7f4d6976c640 1 -- start start 2024-04-25T15:25:39.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d6976c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d6415e080 0x7f4d641646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d6976c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d641609b0 con 0x7f4d6406ccd0 2024-04-25T15:25:39.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d6976c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d64164cd0 con 0x7f4d64161710 2024-04-25T15:25:39.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d6976c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4d64160c30 con 0x7f4d6415e080 2024-04-25T15:25:39.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d637fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d6415e080 0x7f4d641646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d627fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d6406ccd0 0x7f4d64163fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53742/0 (socket says 172.21.15.67:53742) 2024-04-25T15:25:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d627fc640 1 -- 172.21.15.67:0/3607553771 learned_addr learned my addr 172.21.15.67:0/3607553771 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d637fe640 1 -- 172.21.15.67:0/3607553771 >> v1:172.21.15.176:6789/0 conn(0x7f4d64161710 legacy=0x7f4d6406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d637fe640 1 -- 172.21.15.67:0/3607553771 >> v1:172.21.15.67:6789/0 conn(0x7f4d6406ccd0 legacy=0x7f4d64163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.679+0000 7f4d637fe640 1 -- 172.21.15.67:0/3607553771 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d64160f10 con 0x7f4d6415e080 2024-04-25T15:25:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.680+0000 7f4d637fe640 1 --2- 172.21.15.67:0/3607553771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6415e080 0x7f4d641646b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0097a0 tx=0x7f4d5c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a85002af988ccc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.680+0000 7f4d61ffb640 1 -- 172.21.15.67:0/3607553771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c037070 con 0x7f4d6415e080 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.680+0000 7f4d61ffb640 1 -- 172.21.15.67:0/3607553771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d5c02ae50 con 0x7f4d6415e080 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.680+0000 7f4d61ffb640 1 -- 172.21.15.67:0/3607553771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c032660 con 0x7f4d6415e080 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.681+0000 7f4d6976c640 1 -- 172.21.15.67:0/3607553771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6415e080 msgr2=0x7f4d641646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.681+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3607553771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6415e080 0x7f4d641646b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0097a0 tx=0x7f4d5c02a140 comp rx=0 tx=0).stop 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.681+0000 7f4d6976c640 1 -- 172.21.15.67:0/3607553771 shutdown_connections 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.681+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3607553771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d6415e080 0x7f4d641646b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.681+0000 7f4d6976c640 1 -- 172.21.15.67:0/3607553771 >> 172.21.15.67:0/3607553771 conn(0x7f4d640d5d90 msgr2=0x7f4d640d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.681+0000 7f4d6976c640 1 -- 172.21.15.67:0/3607553771 shutdown_connections 2024-04-25T15:25:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.681+0000 7f4d6976c640 1 -- 172.21.15.67:0/3607553771 wait complete. 2024-04-25T15:25:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.683+0000 7f4d6976c640 1 Processor -- start 2024-04-25T15:25:39.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.683+0000 7f4d6976c640 1 -- start start 2024-04-25T15:25:39.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d6976c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d6415e080 0x7f4d6415ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d6976c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d64161710 0x7f4d64155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d6976c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d64156220 0x7f4d64157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d6976c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6416b280 con 0x7f4d64156220 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d6976c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6416b100 con 0x7f4d64161710 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d6976c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6416b400 con 0x7f4d6415e080 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d64156220 0x7f4d64157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d64156220 0x7f4d64157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38128/0 (socket says 172.21.15.67:38128) 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.684+0000 7f4d637fe640 1 -- 172.21.15.67:0/3065574881 learned_addr learned my addr 172.21.15.67:0/3065574881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d627fc640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d64161710 0x7f4d64155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d637fe640 1 -- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d6415e080 msgr2=0x7f4d6415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d637fe640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d6415e080 0x7f4d6415ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d637fe640 1 -- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d64161710 msgr2=0x7f4d64155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d637fe640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d64161710 0x7f4d64155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d637fe640 1 -- 172.21.15.67:0/3065574881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6415d790 con 0x7f4d64156220 2024-04-25T15:25:39.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d627fc640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d64161710 0x7f4d64155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:39.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d637fe640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d64156220 0x7f4d64157200 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0098c0 tx=0x7f4d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d5b7fe640 1 -- 172.21.15.67:0/3065574881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c042070 con 0x7f4d64156220 2024-04-25T15:25:39.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d5b7fe640 1 -- 172.21.15.67:0/3065574881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d5c032e10 con 0x7f4d64156220 2024-04-25T15:25:39.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6415b560 con 0x7f4d64156220 2024-04-25T15:25:39.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d5b7fe640 1 -- 172.21.15.67:0/3065574881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c037040 con 0x7f4d64156220 2024-04-25T15:25:39.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.685+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6415bb20 con 0x7f4d64156220 2024-04-25T15:25:39.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.686+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d64050740 con 0x7f4d64156220 2024-04-25T15:25:39.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.687+0000 7f4d5b7fe640 1 -- 172.21.15.67:0/3065574881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4d5c045070 con 0x7f4d64156220 2024-04-25T15:25:39.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.688+0000 7f4d5b7fe640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d440744b0 0x7f4d440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:39.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.688+0000 7f4d5b7fe640 1 -- 172.21.15.67:0/3065574881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4d5c0b5a10 con 0x7f4d64156220 2024-04-25T15:25:39.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.688+0000 7f4d62ffd640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d440744b0 0x7f4d440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:39.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.688+0000 7f4d62ffd640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d440744b0 0x7f4d440769c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d6408e250 tx=0x7f4d54009230 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:39.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.692+0000 7f4d5b7fe640 1 -- 172.21.15.67:0/3065574881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d5c080220 con 0x7f4d64156220 2024-04-25T15:25:39.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.876+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d641627d0 con 0x7f4d64156220 2024-04-25T15:25:39.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.876+0000 7f4d5b7fe640 1 -- 172.21.15.67:0/3065574881 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f4d5c088210 con 0x7f4d64156220 2024-04-25T15:25:39.877 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:25:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d440744b0 msgr2=0x7f4d440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d440744b0 0x7f4d440769c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d6408e250 tx=0x7f4d54009230 comp rx=0 tx=0).stop 2024-04-25T15:25:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d64156220 msgr2=0x7f4d64157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d64156220 0x7f4d64157200 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0098c0 tx=0x7f4d5c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 shutdown_connections 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d440744b0 0x7f4d440769c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d64156220 0x7f4d64157200 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d64161710 0x7f4d64155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 --2- 172.21.15.67:0/3065574881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d6415e080 0x7f4d6415ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 >> 172.21.15.67:0/3065574881 conn(0x7f4d640d5d90 msgr2=0x7f4d64161b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 shutdown_connections 2024-04-25T15:25:39.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:39.878+0000 7f4d6976c640 1 -- 172.21.15.67:0/3065574881 wait complete. 2024-04-25T15:25:39.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:25:40.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.003+0000 7f616d500640 1 Processor -- start 2024-04-25T15:25:40.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.003+0000 7f616d500640 1 -- start start 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f616d500640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f616815e080 0x7f61681646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f616d500640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61681609b0 con 0x7f616806ccd0 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f616d500640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6168164cd0 con 0x7f6168161710 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f616d500640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6168160c30 con 0x7f616815e080 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f61667fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f616806ccd0 0x7f6168163fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53756/0 (socket says 172.21.15.67:53756) 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f61667fc640 1 -- 172.21.15.67:0/2772895272 learned_addr learned my addr 172.21.15.67:0/2772895272 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f61677fe640 1 --2- 172.21.15.67:0/2772895272 >> v2:172.21.15.176:3300/0 conn(0x7f616815e080 0x7f61681646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180385793 0 0) 0x7f6150001110 con 0x7f616806ccd0 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6154003520 con 0x7f616806ccd0 2024-04-25T15:25:40.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 902056808 0 0) 0x7f6168164cd0 con 0x7f6168161710 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6150001110 con 0x7f6168161710 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 735539759 0 0) 0x7f6154003520 con 0x7f616806ccd0 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 >> v2:172.21.15.176:3300/0 conn(0x7f616815e080 msgr2=0x7f61681646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 --2- 172.21.15.67:0/2772895272 >> v2:172.21.15.176:3300/0 conn(0x7f616815e080 0x7f61681646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 >> v1:172.21.15.176:6789/0 conn(0x7f6168161710 legacy=0x7f616806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6168160f10 con 0x7f616806ccd0 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f6150001840 con 0x7f616806ccd0 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f61677fe640 1 --2- 172.21.15.67:0/2772895272 >> v2:172.21.15.176:3300/0 conn(0x7f616815e080 0x7f61681646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 >> v1:172.21.15.67:6789/0 conn(0x7f616806ccd0 legacy=0x7f6168163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6154003b40 0x7f6154003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154004800 0x7f6154004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6154005260 0x7f615400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6168160c30 con 0x7f6154003b40 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f615400c1c0 con 0x7f6154005260 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.005+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f615400c3c0 con 0x7f6154004800 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f6166ffd640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6154003b40 0x7f6154003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f61667fc640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154004800 0x7f6154004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f61677fe640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6154005260 0x7f615400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f6166ffd640 1 -- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154004800 msgr2=0x7f6154004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f6166ffd640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154004800 0x7f6154004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f6166ffd640 1 -- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6154005260 msgr2=0x7f615400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f6166ffd640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6154005260 0x7f615400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f61677fe640 1 -- 172.21.15.67:0/2772895272 reap_dead start 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.006+0000 7f6166ffd640 1 -- 172.21.15.67:0/2772895272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6154003520 con 0x7f6154003b40 2024-04-25T15:25:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f6166ffd640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6154003b40 0x7f6154003f90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f61600018a0 tx=0x7f6154005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c93fac3dc473ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61600014f0 con 0x7f6154003b40 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61600052d0 con 0x7f6154003b40 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f6165ffb640 1 -- 172.21.15.67:0/2772895272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616002ac00 con 0x7f6154003b40 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f616d500640 1 -- 172.21.15.67:0/2772895272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6154003b40 msgr2=0x7f6154003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f616d500640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6154003b40 0x7f6154003f90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f61600018a0 tx=0x7f6154005830 comp rx=0 tx=0).stop 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f616d500640 1 -- 172.21.15.67:0/2772895272 shutdown_connections 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f616d500640 1 --2- 172.21.15.67:0/2772895272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6154003b40 0x7f6154003f90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f616d500640 1 -- 172.21.15.67:0/2772895272 >> 172.21.15.67:0/2772895272 conn(0x7f61680d5d90 msgr2=0x7f61680d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f616d500640 1 -- 172.21.15.67:0/2772895272 shutdown_connections 2024-04-25T15:25:40.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.007+0000 7f616d500640 1 -- 172.21.15.67:0/2772895272 wait complete. 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.009+0000 7f616d500640 1 Processor -- start 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.009+0000 7f616d500640 1 -- start start 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f616d500640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154003b40 0x7f61680b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f616d500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680b6dd0 0x7f61680b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f616d500640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61680cde20 0x7f61680ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f616d500640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f616806c740 con 0x7f61680b6dd0 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f616d500640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f616806c5c0 con 0x7f61680cde20 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f616d500640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f616806c8c0 con 0x7f6154003b40 2024-04-25T15:25:40.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680b6dd0 0x7f61680b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680b6dd0 0x7f61680b71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38144/0 (socket says 172.21.15.67:38144) 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 -- 172.21.15.67:0/3938849626 learned_addr learned my addr 172.21.15.67:0/3938849626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f6166ffd640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154003b40 0x7f61680b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61677fe640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61680cde20 0x7f61680ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 -- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154003b40 msgr2=0x7f61680b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154003b40 0x7f61680b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 -- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61680cde20 msgr2=0x7f61680ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61680cde20 0x7f61680ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 -- 172.21.15.67:0/3938849626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6168162ba0 con 0x7f61680b6dd0 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f61667fc640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680b6dd0 0x7f61680b71e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6150007310 tx=0x7f61500077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.011+0000 7f615f7fe640 1 -- 172.21.15.67:0/3938849626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615001a070 con 0x7f61680b6dd0 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.012+0000 7f615f7fe640 1 -- 172.21.15.67:0/3938849626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f615000f9c0 con 0x7f61680b6dd0 2024-04-25T15:25:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.012+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61680b57f0 con 0x7f61680b6dd0 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.012+0000 7f615f7fe640 1 -- 172.21.15.67:0/3938849626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615000fd90 con 0x7f61680b6dd0 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.012+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61680b5db0 con 0x7f61680b6dd0 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.013+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61680cf710 con 0x7f61680b6dd0 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.013+0000 7f615f7fe640 1 -- 172.21.15.67:0/3938849626 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f615001e3f0 con 0x7f61680b6dd0 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.014+0000 7f615f7fe640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61480744e0 0x7f61480769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.014+0000 7f615f7fe640 1 -- 172.21.15.67:0/3938849626 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6150094ae0 con 0x7f61680b6dd0 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.014+0000 7f6166ffd640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61480744e0 0x7f61480769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.014+0000 7f6166ffd640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61480744e0 0x7f61480769f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6160032000 tx=0x7f6160009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.018+0000 7f615f7fe640 1 -- 172.21.15.67:0/3938849626 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61681609b0 con 0x7f61680b6dd0 2024-04-25T15:25:40.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.201+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6168050740 con 0x7f61680b6dd0 2024-04-25T15:25:40.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.202+0000 7f615f7fe640 1 -- 172.21.15.67:0/3938849626 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f615005f270 con 0x7f61680b6dd0 2024-04-25T15:25:40.203 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:25:40.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61480744e0 msgr2=0x7f61480769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61480744e0 0x7f61480769f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6160032000 tx=0x7f6160009030 comp rx=0 tx=0).stop 2024-04-25T15:25:40.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680b6dd0 msgr2=0x7f61680b71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680b6dd0 0x7f61680b71e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6150007310 tx=0x7f61500077d0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 shutdown_connections 2024-04-25T15:25:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61480744e0 0x7f61480769f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61680cde20 0x7f61680ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61680b6dd0 0x7f61680b71e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 --2- 172.21.15.67:0/3938849626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6154003b40 0x7f61680b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 >> 172.21.15.67:0/3938849626 conn(0x7f61680d5d90 msgr2=0x7f616815d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 shutdown_connections 2024-04-25T15:25:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.204+0000 7f616d500640 1 -- 172.21.15.67:0/3938849626 wait complete. 2024-04-25T15:25:40.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:25:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.326+0000 7f6add97d640 1 Processor -- start 2024-04-25T15:25:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.326+0000 7f6add97d640 1 -- start start 2024-04-25T15:25:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6add97d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 0x7f6ad80af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6add97d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6ad80af860 0x7f6ad8161da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6add97d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ad815e6c0 con 0x7f6ad806f840 2024-04-25T15:25:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6add97d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6ad806d020 con 0x7f6ad80af860 2024-04-25T15:25:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6add97d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad8071370 con 0x7f6ad815e2b0 2024-04-25T15:25:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad6ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ad806f840 0x7f6ad8070a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53772/0 (socket says 172.21.15.67:53772) 2024-04-25T15:25:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad6ffd640 1 -- 172.21.15.67:0/3224435199 learned_addr learned my addr 172.21.15.67:0/3224435199 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 0x7f6ad80af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 7591255 0 0) 0x7f6ad0001110 con 0x7f6ad806f840 2024-04-25T15:25:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad77fe640 1 --2- 172.21.15.67:0/3224435199 >> v2:172.21.15.176:3300/0 conn(0x7f6ad80af860 0x7f6ad8161da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac0003520 con 0x7f6ad806f840 2024-04-25T15:25:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2105718106 0 0) 0x7f6ac0003520 con 0x7f6ad806f840 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 msgr2=0x7f6ad80af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.328+0000 7f6ad5ffb640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 0x7f6ad80af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 >> v2:172.21.15.176:3300/0 conn(0x7f6ad80af860 msgr2=0x7f6ad8161da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 --2- 172.21.15.67:0/3224435199 >> v2:172.21.15.176:3300/0 conn(0x7f6ad80af860 0x7f6ad8161da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad806c6f0 con 0x7f6ad806f840 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f6ad0001840 con 0x7f6ad806f840 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 >> v1:172.21.15.67:6789/0 conn(0x7f6ad806f840 legacy=0x7f6ad8070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00039b0 0x7f6ac0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 0x7f6ac0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad806d020 con 0x7f6ac0004800 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad8071370 con 0x7f6ac00039b0 2024-04-25T15:25:40.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac000ba70 con 0x7f6ad815e2b0 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad6ffd640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00039b0 0x7f6ac0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad77fe640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 0x7f6ac0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad67fc640 1 -- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 msgr2=0x7f6ac0005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 0x7f6ac0005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad67fc640 1 -- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00039b0 msgr2=0x7f6ac0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00039b0 0x7f6ac0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad67fc640 1 -- 172.21.15.67:0/3224435199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac0003520 con 0x7f6ac0004800 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.329+0000 7f6ad77fe640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad815e2b0 0x7f6ac0005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.330+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009aa0 tx=0x7f6ac0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eee9736e85a3276 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.330+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc036070 con 0x7f6ac0004800 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.330+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6acc031400 con 0x7f6ac0004800 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.330+0000 7f6ad5ffb640 1 -- 172.21.15.67:0/3224435199 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc009670 con 0x7f6ac0004800 2024-04-25T15:25:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.331+0000 7f6add97d640 1 -- 172.21.15.67:0/3224435199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac0004800 msgr2=0x7f6ac0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.331+0000 7f6add97d640 1 --2- 172.21.15.67:0/3224435199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009aa0 tx=0x7f6ac0005830 comp rx=0 tx=0).stop 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.331+0000 7f6ad77fe640 1 -- 172.21.15.67:0/3224435199 reap_dead start 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.331+0000 7f6add97d640 1 -- 172.21.15.67:0/3224435199 shutdown_connections 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.331+0000 7f6add97d640 1 -- 172.21.15.67:0/3224435199 >> 172.21.15.67:0/3224435199 conn(0x7f6ad80ce760 msgr2=0x7f6ad80ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.331+0000 7f6add97d640 1 -- 172.21.15.67:0/3224435199 shutdown_connections 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.331+0000 7f6add97d640 1 -- 172.21.15.67:0/3224435199 wait complete. 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.333+0000 7f6add97d640 1 Processor -- start 2024-04-25T15:25:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.333+0000 7f6add97d640 1 -- start start 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6add97d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac0004800 0x7f6ad814a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6add97d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad81e7ca0 0x7f6ad81ea3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6add97d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad80b2260 0x7f6ad80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6add97d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad80af320 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6add97d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad806ccb0 con 0x7f6ac0004800 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6add97d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad80af4a0 con 0x7f6ad80b2260 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad81e7ca0 0x7f6ad81ea3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad81e7ca0 0x7f6ad81ea3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38160/0 (socket says 172.21.15.67:38160) 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 -- 172.21.15.67:0/3038645968 learned_addr learned my addr 172.21.15.67:0/3038645968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad6ffd640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac0004800 0x7f6ad814a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad77fe640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad80b2260 0x7f6ad80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 -- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad80b2260 msgr2=0x7f6ad80b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad80b2260 0x7f6ad80b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 -- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac0004800 msgr2=0x7f6ad814a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac0004800 0x7f6ad814a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad67fc640 1 -- 172.21.15.67:0/3038645968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad80b2bb0 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.334+0000 7f6ad6ffd640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac0004800 0x7f6ad814a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.335+0000 7f6ad67fc640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad81e7ca0 0x7f6ad81ea3e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009bc0 tx=0x7f6acc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.335+0000 7f6abffff640 1 -- 172.21.15.67:0/3038645968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc036040 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.335+0000 7f6abffff640 1 -- 172.21.15.67:0/3038645968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6acc03d680 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.335+0000 7f6abffff640 1 -- 172.21.15.67:0/3038645968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc009670 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.335+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad81f7d10 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.335+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad81f82d0 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.336+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ad8050740 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.337+0000 7f6abffff640 1 -- 172.21.15.67:0/3038645968 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6acc039040 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.337+0000 7f6abffff640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6ac40744b0 0x7f6ac40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.337+0000 7f6abffff640 1 -- 172.21.15.67:0/3038645968 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6acc0b5a90 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.337+0000 7f6ad6ffd640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6ac40744b0 0x7f6ac40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.338+0000 7f6ad6ffd640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6ac40744b0 0x7f6ac40769c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0001050 tx=0x7f6ad0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.342+0000 7f6abffff640 1 -- 172.21.15.67:0/3038645968 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ad80b2bb0 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.565+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ad8071020 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.565+0000 7f6abffff640 1 -- 172.21.15.67:0/3038645968 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7f6acc041020 con 0x7f6ad81e7ca0 2024-04-25T15:25:40.566 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:40.566 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:25:40.567 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6ac40744b0 msgr2=0x7f6ac40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6ac40744b0 0x7f6ac40769c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0001050 tx=0x7f6ad0009040 comp rx=0 tx=0).stop 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad81e7ca0 msgr2=0x7f6ad81ea3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad81e7ca0 0x7f6ad81ea3e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6acc009bc0 tx=0x7f6acc004730 comp rx=0 tx=0).stop 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 shutdown_connections 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6ac40744b0 0x7f6ac40769c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ad80b2260 0x7f6ad80b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ad81e7ca0 0x7f6ad81ea3e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.567+0000 7f6add97d640 1 --2- 172.21.15.67:0/3038645968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac0004800 0x7f6ad814a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.568+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 >> 172.21.15.67:0/3038645968 conn(0x7f6ad80ce760 msgr2=0x7f6ad80ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.568+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 shutdown_connections 2024-04-25T15:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.568+0000 7f6add97d640 1 -- 172.21.15.67:0/3038645968 wait complete. 2024-04-25T15:25:40.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.694+0000 7f1577fce640 1 Processor -- start 2024-04-25T15:25:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.694+0000 7f1577fce640 1 -- start start 2024-04-25T15:25:40.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.695+0000 7f1577fce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 0x7f15700d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.695+0000 7f1577fce640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f15700d2d70 0x7f15700d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.695+0000 7f1577fce640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15700b3c10 con 0x7f157006dcf0 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.695+0000 7f1577fce640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f15700b9520 con 0x7f15700d2d70 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.695+0000 7f1577fce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15700b4820 con 0x7f157006d8e0 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1575569640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f157006dcf0 0x7f15700d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53784/0 (socket says 172.21.15.67:53784) 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1575569640 1 -- 172.21.15.67:0/3522721561 learned_addr learned my addr 172.21.15.67:0/3522721561 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1575d6a640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 0x7f15700d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f157656b640 1 --2- 172.21.15.67:0/3522721561 >> v2:172.21.15.176:3300/0 conn(0x7f15700d2d70 0x7f15700d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854957352 0 0) 0x7f15700b3c10 con 0x7f157006dcf0 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f155c003520 con 0x7f157006dcf0 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4247934325 0 0) 0x7f155c003520 con 0x7f157006dcf0 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 msgr2=0x7f15700d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 0x7f15700d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 >> v2:172.21.15.176:3300/0 conn(0x7f15700d2d70 msgr2=0x7f15700d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 --2- 172.21.15.67:0/3522721561 >> v2:172.21.15.176:3300/0 conn(0x7f15700d2d70 0x7f15700d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15700b3ef0 con 0x7f157006dcf0 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f1560001670 con 0x7f157006dcf0 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 >> v1:172.21.15.67:6789/0 conn(0x7f157006dcf0 legacy=0x7f15700d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f155c0039b0 0x7f155c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c004800 0x7f155c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 0x7f155c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15700b9520 con 0x7f155c004800 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15700b4820 con 0x7f155c0039b0 2024-04-25T15:25:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f155c00ba70 con 0x7f157006d8e0 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1575569640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c004800 0x7f155c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f1575d6a640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f155c0039b0 0x7f155c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.696+0000 7f157656b640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 0x7f155c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1575569640 1 -- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 msgr2=0x7f155c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1575569640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f157006d8e0 0x7f155c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1575569640 1 -- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f155c0039b0 msgr2=0x7f155c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1575569640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f155c0039b0 0x7f155c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1575569640 1 -- 172.21.15.67:0/3522721561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155c003520 con 0x7f155c004800 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1575569640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c004800 0x7f155c004c10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f157006f790 tx=0x7f155c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe10c14403c0bb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560007d30 con 0x7f155c004800 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1560005af0 con 0x7f155c004800 2024-04-25T15:25:40.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.697+0000 7f1574d68640 1 -- 172.21.15.67:0/3522721561 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560007070 con 0x7f155c004800 2024-04-25T15:25:40.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.698+0000 7f1577fce640 1 -- 172.21.15.67:0/3522721561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c004800 msgr2=0x7f155c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.698+0000 7f1577fce640 1 --2- 172.21.15.67:0/3522721561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c004800 0x7f155c004c10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f157006f790 tx=0x7f155c005830 comp rx=0 tx=0).stop 2024-04-25T15:25:40.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.698+0000 7f157656b640 1 -- 172.21.15.67:0/3522721561 reap_dead start 2024-04-25T15:25:40.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.698+0000 7f1577fce640 1 -- 172.21.15.67:0/3522721561 shutdown_connections 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.698+0000 7f1577fce640 1 -- 172.21.15.67:0/3522721561 >> 172.21.15.67:0/3522721561 conn(0x7f157008d110 msgr2=0x7f157008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.698+0000 7f1577fce640 1 -- 172.21.15.67:0/3522721561 shutdown_connections 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.698+0000 7f1577fce640 1 -- 172.21.15.67:0/3522721561 wait complete. 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.700+0000 7f1577fce640 1 Processor -- start 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.700+0000 7f1577fce640 1 -- start start 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.701+0000 7f1577fce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155c004800 0x7f15700a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.701+0000 7f1577fce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1570150c70 0x7f15701e8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.701+0000 7f1577fce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1570150020 0x7f1570150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.701+0000 7f1577fce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15700b38e0 con 0x7f1570150c70 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.701+0000 7f1577fce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15700b44b0 con 0x7f1570150020 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.701+0000 7f1577fce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15700d3a20 con 0x7f155c004800 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.701+0000 7f1575569640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1570150c70 0x7f15701e8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1570150c70 0x7f15701e8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38178/0 (socket says 172.21.15.67:38178) 2024-04-25T15:25:40.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 -- 172.21.15.67:0/785581190 learned_addr learned my addr 172.21.15.67:0/785581190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575d6a640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155c004800 0x7f15700a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f157656b640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1570150020 0x7f1570150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 -- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155c004800 msgr2=0x7f15700a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155c004800 0x7f15700a0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 -- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1570150020 msgr2=0x7f1570150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1570150020 0x7f1570150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 -- 172.21.15.67:0/785581190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15700a0640 con 0x7f1570150c70 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575d6a640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155c004800 0x7f15700a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1575569640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1570150c70 0x7f15701e8db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1560001970 tx=0x7f15700b40c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f15667fc640 1 -- 172.21.15.67:0/785581190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560001630 con 0x7f1570150c70 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f15667fc640 1 -- 172.21.15.67:0/785581190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1560042070 con 0x7f1570150c70 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1570150970 con 0x7f1570150c70 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f15667fc640 1 -- 172.21.15.67:0/785581190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560007270 con 0x7f1570150c70 2024-04-25T15:25:40.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.702+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15700a0080 con 0x7f1570150c70 2024-04-25T15:25:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.703+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1570050740 con 0x7f1570150c70 2024-04-25T15:25:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.704+0000 7f15667fc640 1 -- 172.21.15.67:0/785581190 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1560007410 con 0x7f1570150c70 2024-04-25T15:25:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.704+0000 7f15667fc640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744e0 0x7f15540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.704+0000 7f15667fc640 1 -- 172.21.15.67:0/785581190 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f15600b5420 con 0x7f1570150c70 2024-04-25T15:25:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.704+0000 7f1575d6a640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744e0 0x7f15540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:40.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.705+0000 7f1575d6a640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744e0 0x7f15540769f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1568009160 tx=0x7f1568008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:40.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.709+0000 7f15667fc640 1 -- 172.21.15.67:0/785581190 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1570150970 con 0x7f1570150c70 2024-04-25T15:25:40.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.896+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15700d1fc0 con 0x7f1570150c70 2024-04-25T15:25:40.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.898+0000 7f15667fc640 1 -- 172.21.15.67:0/785581190 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f156007fbb0 con 0x7f1570150c70 2024-04-25T15:25:40.899 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:40.899 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.899+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744e0 msgr2=0x7f15540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.899+0000 7f1577fce640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744e0 0x7f15540769f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1568009160 tx=0x7f1568008040 comp rx=0 tx=0).stop 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.899+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1570150c70 msgr2=0x7f15701e8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.899+0000 7f1577fce640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1570150c70 0x7f15701e8db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1560001970 tx=0x7f15700b40c0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 shutdown_connections 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744e0 0x7f15540769f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1570150020 0x7f1570150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1570150c70 0x7f15701e8db0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 --2- 172.21.15.67:0/785581190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155c004800 0x7f15700a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:40.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 >> 172.21.15.67:0/785581190 conn(0x7f157008d110 msgr2=0x7f157006e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:40.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 shutdown_connections 2024-04-25T15:25:40.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:40.900+0000 7f1577fce640 1 -- 172.21.15.67:0/785581190 wait complete. 2024-04-25T15:25:40.912 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:40.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:41.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.027+0000 7fcb11092640 1 Processor -- start 2024-04-25T15:25:41.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.027+0000 7fcb11092640 1 -- start start 2024-04-25T15:25:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.028+0000 7fcb11092640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c06ccd0 0x7fcb0c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.028+0000 7fcb11092640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcb0c0d4a50 0x7fcb0c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.028+0000 7fcb11092640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb0c0d1850 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.028+0000 7fcb11092640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcb0c160de0 con 0x7fcb0c0d4a50 2024-04-25T15:25:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.028+0000 7fcb11092640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c1610c0 con 0x7fcb0c06ccd0 2024-04-25T15:25:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c06ccd0 0x7fcb0c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0b59e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcb0c0d4a50 0x7fcb0c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0ad9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fcb0c0d0e00 0x7fcb0c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56154/0 (socket says 172.21.15.67:56154) 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0ad9d640 1 -- 172.21.15.67:0/171375373 learned_addr learned my addr 172.21.15.67:0/171375373 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0b59e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcb0c0d4a50 0x7fcb0c0d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39728/0 (socket says 172.21.15.67:39728) 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c06ccd0 0x7fcb0c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52406/0 (socket says 172.21.15.67:52406) 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb09d9b640 1 -- 172.21.15.67:0/171375373 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544289700 0 0) 0x7fcb0c0d1850 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb09d9b640 1 -- 172.21.15.67:0/171375373 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaf4003520 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0a59c640 1 -- 172.21.15.67:0/171375373 >> v2:172.21.15.176:3300/0 conn(0x7fcb0c0d4a50 msgr2=0x7fcb0c0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/171375373 >> v2:172.21.15.176:3300/0 conn(0x7fcb0c0d4a50 0x7fcb0c0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0a59c640 1 -- 172.21.15.67:0/171375373 >> v1:172.21.15.176:6789/0 conn(0x7fcb0c0d0e00 legacy=0x7fcb0c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0a59c640 1 -- 172.21.15.67:0/171375373 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c1613a0 con 0x7fcb0c06ccd0 2024-04-25T15:25:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.029+0000 7fcb0b59e640 1 --2- 172.21.15.67:0/171375373 >> v2:172.21.15.176:3300/0 conn(0x7fcb0c0d4a50 0x7fcb0c0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:41.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.030+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/171375373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c06ccd0 0x7fcb0c0d1210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcb000097a0 tx=0x7fcb0002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d950cb75298fa22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.030+0000 7fcb09d9b640 1 -- 172.21.15.67:0/171375373 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00037070 con 0x7fcb0c06ccd0 2024-04-25T15:25:41.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.030+0000 7fcb09d9b640 1 -- 172.21.15.67:0/171375373 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb00032440 con 0x7fcb0c06ccd0 2024-04-25T15:25:41.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.030+0000 7fcb09d9b640 1 -- 172.21.15.67:0/171375373 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000327f0 con 0x7fcb0c06ccd0 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 -- 172.21.15.67:0/171375373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c06ccd0 msgr2=0x7fcb0c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 --2- 172.21.15.67:0/171375373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c06ccd0 0x7fcb0c0d1210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcb000097a0 tx=0x7fcb0002a180 comp rx=0 tx=0).stop 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 -- 172.21.15.67:0/171375373 shutdown_connections 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 --2- 172.21.15.67:0/171375373 >> v2:172.21.15.176:3300/0 conn(0x7fcb0c0d4a50 0x7fcb0c0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 --2- 172.21.15.67:0/171375373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c06ccd0 0x7fcb0c0d1210 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 -- 172.21.15.67:0/171375373 >> 172.21.15.67:0/171375373 conn(0x7fcb0c0d5d90 msgr2=0x7fcb0c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 -- 172.21.15.67:0/171375373 shutdown_connections 2024-04-25T15:25:41.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.031+0000 7fcb11092640 1 -- 172.21.15.67:0/171375373 wait complete. 2024-04-25T15:25:41.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.033+0000 7fcb11092640 1 Processor -- start 2024-04-25T15:25:41.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.033+0000 7fcb11092640 1 -- start start 2024-04-25T15:25:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb11092640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c0d0e00 0x7fcb0c150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb11092640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c0d4a50 0x7fcb0c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb11092640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c150010 0x7fcb0c0c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb11092640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c164e00 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb11092640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c0b5320 con 0x7fcb0c0d4a50 2024-04-25T15:25:41.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb11092640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c164f80 con 0x7fcb0c150010 2024-04-25T15:25:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb0ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c0d0e00 0x7fcb0c150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb0ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c0d0e00 0x7fcb0c150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38180/0 (socket says 172.21.15.67:38180) 2024-04-25T15:25:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb0ad9d640 1 -- 172.21.15.67:0/2906585807 learned_addr learned my addr 172.21.15.67:0/2906585807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c0d4a50 0x7fcb0c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.034+0000 7fcb0b59e640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c150010 0x7fcb0c0c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0ad9d640 1 -- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c150010 msgr2=0x7fcb0c0c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0ad9d640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c150010 0x7fcb0c0c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0ad9d640 1 -- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c0d4a50 msgr2=0x7fcb0c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0ad9d640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c0d4a50 0x7fcb0c0cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0ad9d640 1 -- 172.21.15.67:0/2906585807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0ce7a0 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0b59e640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c150010 0x7fcb0c0c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c0d4a50 0x7fcb0c0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb0ad9d640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c0d0e00 0x7fcb0c150c20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c3c0 tx=0x7fcb0400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcaf37fe640 1 -- 172.21.15.67:0/2906585807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04017070 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcaf37fe640 1 -- 172.21.15.67:0/2906585807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb04005ce0 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcaf37fe640 1 -- 172.21.15.67:0/2906585807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb040072f0 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c0cea80 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.035+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c0cf040 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.036+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0c050740 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.037+0000 7fcaf37fe640 1 -- 172.21.15.67:0/2906585807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb0401a070 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.037+0000 7fcaf37fe640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744e0 0x7fcaf80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.037+0000 7fcaf37fe640 1 -- 172.21.15.67:0/2906585807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcb04094600 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.038+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744e0 0x7fcaf80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.038+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744e0 0x7fcaf80769f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb00034000 tx=0x7fcb0002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.042+0000 7fcaf37fe640 1 -- 172.21.15.67:0/2906585807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb0405ed90 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.226+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcb0c0d4320 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.226+0000 7fcaf37fe640 1 -- 172.21.15.67:0/2906585807 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fcb0405e690 con 0x7fcb0c0d0e00 2024-04-25T15:25:41.227 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:41.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744e0 msgr2=0x7fcaf80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744e0 0x7fcaf80769f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcb00034000 tx=0x7fcb0002e000 comp rx=0 tx=0).stop 2024-04-25T15:25:41.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c0d0e00 msgr2=0x7fcb0c150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c0d0e00 0x7fcb0c150c20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c3c0 tx=0x7fcb0400c880 comp rx=0 tx=0).stop 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 shutdown_connections 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744e0 0x7fcaf80769f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c150010 0x7fcb0c0c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c0d4a50 0x7fcb0c0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 --2- 172.21.15.67:0/2906585807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c0d0e00 0x7fcb0c150c20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 >> 172.21.15.67:0/2906585807 conn(0x7fcb0c0d5d90 msgr2=0x7fcb0c164480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 shutdown_connections 2024-04-25T15:25:41.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.228+0000 7fcb11092640 1 -- 172.21.15.67:0/2906585807 wait complete. 2024-04-25T15:25:41.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.354+0000 7f74d44f9640 1 Processor -- start 2024-04-25T15:25:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.354+0000 7f74d44f9640 1 -- start start 2024-04-25T15:25:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d44f9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f74cc0b7e70 0x7f74cc0cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d44f9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc0b4280 con 0x7f74cc06c9a0 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d44f9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74cc0b3700 con 0x7f74cc06e890 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d44f9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f74cc0d3820 con 0x7f74cc0b7e70 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d2a96640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74cc06c9a0 0x7f74cc06cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53788/0 (socket says 172.21.15.67:53788) 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d2a96640 1 -- 172.21.15.67:0/1374643879 learned_addr learned my addr 172.21.15.67:0/1374643879 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d1a94640 1 --2- 172.21.15.67:0/1374643879 >> v2:172.21.15.176:3300/0 conn(0x7f74cc0b7e70 0x7f74cc0cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d1293640 1 -- 172.21.15.67:0/1374643879 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1304094602 0 0) 0x7f74cc0b4280 con 0x7f74cc06c9a0 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d1293640 1 -- 172.21.15.67:0/1374643879 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74b8003520 con 0x7f74cc06c9a0 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.356+0000 7f74d1293640 1 -- 172.21.15.67:0/1374643879 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 646612977 0 0) 0x7f74b8003520 con 0x7f74cc06c9a0 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.357+0000 7f74d1a94640 1 -- 172.21.15.67:0/1374643879 >> v1:172.21.15.176:6789/0 conn(0x7f74cc06e890 legacy=0x7f74cc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.357+0000 7f74d1a94640 1 -- 172.21.15.67:0/1374643879 >> v1:172.21.15.67:6789/0 conn(0x7f74cc06c9a0 legacy=0x7f74cc06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.357+0000 7f74d1a94640 1 -- 172.21.15.67:0/1374643879 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc0b39e0 con 0x7f74cc0b7e70 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.357+0000 7f74d1a94640 1 --2- 172.21.15.67:0/1374643879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0b7e70 0x7f74cc0cf410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0097a0 tx=0x7f74bc02a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5296809ce6a5be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d1293640 1 -- 172.21.15.67:0/1374643879 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc037070 con 0x7f74cc0b7e70 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d1293640 1 -- 172.21.15.67:0/1374643879 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74bc02ab60 con 0x7f74cc0b7e70 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d1293640 1 -- 172.21.15.67:0/1374643879 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c80019f0 con 0x7f74cc0b7e70 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d44f9640 1 -- 172.21.15.67:0/1374643879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0b7e70 msgr2=0x7f74cc0cf410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1374643879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0b7e70 0x7f74cc0cf410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0097a0 tx=0x7f74bc02a0d0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d44f9640 1 -- 172.21.15.67:0/1374643879 shutdown_connections 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1374643879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0b7e70 0x7f74cc0cf410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d44f9640 1 -- 172.21.15.67:0/1374643879 >> 172.21.15.67:0/1374643879 conn(0x7f74cc09b7d0 msgr2=0x7f74cc09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d44f9640 1 -- 172.21.15.67:0/1374643879 shutdown_connections 2024-04-25T15:25:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.358+0000 7f74d44f9640 1 -- 172.21.15.67:0/1374643879 wait complete. 2024-04-25T15:25:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.360+0000 7f74d44f9640 1 Processor -- start 2024-04-25T15:25:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.360+0000 7f74d44f9640 1 -- start start 2024-04-25T15:25:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.361+0000 7f74d44f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74cc06c9a0 0x7f74cc0cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.361+0000 7f74d44f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc06e890 0x7f74cc0cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.361+0000 7f74d44f9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0c6f00 0x7f74cc15fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.361+0000 7f74d44f9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc0d0300 con 0x7f74cc06e890 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.361+0000 7f74d44f9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc0b3fa0 con 0x7f74cc0c6f00 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.361+0000 7f74d44f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f74cc06d4c0 con 0x7f74cc06c9a0 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc06e890 0x7f74cc0cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc06e890 0x7f74cc0cc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38182/0 (socket says 172.21.15.67:38182) 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 -- 172.21.15.67:0/1254161867 learned_addr learned my addr 172.21.15.67:0/1254161867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 -- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74cc06c9a0 msgr2=0x7f74cc0cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74cc06c9a0 0x7f74cc0cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d2a96640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0c6f00 0x7f74cc15fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 -- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0c6f00 msgr2=0x7f74cc15fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0c6f00 0x7f74cc15fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 -- 172.21.15.67:0/1254161867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc161610 con 0x7f74cc06e890 2024-04-25T15:25:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d2a96640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0c6f00 0x7f74cc15fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d1a94640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc06e890 0x7f74cc0cc1c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0098c0 tx=0x7f74bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74c2ffd640 1 -- 172.21.15.67:0/1254161867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc042020 con 0x7f74cc06e890 2024-04-25T15:25:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74c2ffd640 1 -- 172.21.15.67:0/1254161867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74bc003a40 con 0x7f74cc06e890 2024-04-25T15:25:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74c2ffd640 1 -- 172.21.15.67:0/1254161867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74bc037040 con 0x7f74cc06e890 2024-04-25T15:25:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74cc0c6780 con 0x7f74cc06e890 2024-04-25T15:25:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.362+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74cc15c8d0 con 0x7f74cc06e890 2024-04-25T15:25:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.363+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74cc0c6ca0 con 0x7f74cc06e890 2024-04-25T15:25:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.364+0000 7f74c2ffd640 1 -- 172.21.15.67:0/1254161867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f74bc045020 con 0x7f74cc06e890 2024-04-25T15:25:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.364+0000 7f74c2ffd640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f74b00744b0 0x7f74b00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.365+0000 7f74c2ffd640 1 -- 172.21.15.67:0/1254161867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f74bc0b5820 con 0x7f74cc06e890 2024-04-25T15:25:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.365+0000 7f74d2295640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f74b00744b0 0x7f74b00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.365+0000 7f74d2295640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f74b00744b0 0x7f74b00769c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f74c4001610 tx=0x7f74c4007950 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.369+0000 7f74c2ffd640 1 -- 172.21.15.67:0/1254161867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74cc0c6ca0 con 0x7f74cc06e890 2024-04-25T15:25:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.553+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74cc1613a0 con 0x7f74cc06e890 2024-04-25T15:25:41.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.554+0000 7f74c2ffd640 1 -- 172.21.15.67:0/1254161867 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f74bc04c030 con 0x7f74cc06e890 2024-04-25T15:25:41.555 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f74b00744b0 msgr2=0x7f74b00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f74b00744b0 0x7f74b00769c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f74c4001610 tx=0x7f74c4007950 comp rx=0 tx=0).stop 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc06e890 msgr2=0x7f74cc0cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc06e890 0x7f74cc0cc1c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0098c0 tx=0x7f74bc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 shutdown_connections 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f74b00744b0 0x7f74b00769c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74cc0c6f00 0x7f74cc15fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74cc06e890 0x7f74cc0cc1c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 --2- 172.21.15.67:0/1254161867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74cc06c9a0 0x7f74cc0cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 >> 172.21.15.67:0/1254161867 conn(0x7f74cc09b7d0 msgr2=0x7f74cc06d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:41.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 shutdown_connections 2024-04-25T15:25:41.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.556+0000 7f74d44f9640 1 -- 172.21.15.67:0/1254161867 wait complete. 2024-04-25T15:25:41.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:25:41.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.681+0000 7f8a4bd84640 1 Processor -- start 2024-04-25T15:25:41.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.681+0000 7f8a4bd84640 1 -- start start 2024-04-25T15:25:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a4bd84640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8a440d0e00 0x7f8a440d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a4bd84640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a440d1850 con 0x7f8a440d4a50 2024-04-25T15:25:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a4bd84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a44160de0 con 0x7f8a4406c5c0 2024-04-25T15:25:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a4bd84640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8a441610c0 con 0x7f8a440d0e00 2024-04-25T15:25:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a4a321640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a440d4a50 0x7f8a440d4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53802/0 (socket says 172.21.15.67:53802) 2024-04-25T15:25:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a4a321640 1 -- 172.21.15.67:0/439387904 learned_addr learned my addr 172.21.15.67:0/439387904 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:41.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a49b20640 1 --2- 172.21.15.67:0/439387904 >> v2:172.21.15.176:3300/0 conn(0x7f8a440d0e00 0x7f8a440d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a48b1e640 1 -- 172.21.15.67:0/439387904 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 927297021 0 0) 0x7f8a440d1850 con 0x7f8a440d4a50 2024-04-25T15:25:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a48b1e640 1 -- 172.21.15.67:0/439387904 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a30003520 con 0x7f8a440d4a50 2024-04-25T15:25:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a49b20640 1 -- 172.21.15.67:0/439387904 >> v1:172.21.15.176:6789/0 conn(0x7f8a4406c5c0 legacy=0x7f8a4406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a49b20640 1 -- 172.21.15.67:0/439387904 >> v1:172.21.15.67:6789/0 conn(0x7f8a440d4a50 legacy=0x7f8a440d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.683+0000 7f8a49b20640 1 -- 172.21.15.67:0/439387904 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a441613a0 con 0x7f8a440d0e00 2024-04-25T15:25:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.684+0000 7f8a49b20640 1 --2- 172.21.15.67:0/439387904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 0x7f8a440d1210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a400097a0 tx=0x7f8a4002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a246d67228390699 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.684+0000 7f8a48b1e640 1 -- 172.21.15.67:0/439387904 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40037070 con 0x7f8a440d0e00 2024-04-25T15:25:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.684+0000 7f8a48b1e640 1 -- 172.21.15.67:0/439387904 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a40032440 con 0x7f8a440d0e00 2024-04-25T15:25:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.684+0000 7f8a48b1e640 1 -- 172.21.15.67:0/439387904 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400327f0 con 0x7f8a440d0e00 2024-04-25T15:25:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.685+0000 7f8a4bd84640 1 -- 172.21.15.67:0/439387904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 msgr2=0x7f8a440d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.685+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/439387904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 0x7f8a440d1210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8a400097a0 tx=0x7f8a4002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.685+0000 7f8a4bd84640 1 -- 172.21.15.67:0/439387904 shutdown_connections 2024-04-25T15:25:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.685+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/439387904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 0x7f8a440d1210 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.685+0000 7f8a4bd84640 1 -- 172.21.15.67:0/439387904 >> 172.21.15.67:0/439387904 conn(0x7f8a440d5d90 msgr2=0x7f8a440d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.685+0000 7f8a4bd84640 1 -- 172.21.15.67:0/439387904 shutdown_connections 2024-04-25T15:25:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.685+0000 7f8a4bd84640 1 -- 172.21.15.67:0/439387904 wait complete. 2024-04-25T15:25:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.687+0000 7f8a4bd84640 1 Processor -- start 2024-04-25T15:25:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.687+0000 7f8a4bd84640 1 -- start start 2024-04-25T15:25:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4bd84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a4406c5c0 0x7f8a440cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4bd84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 0x7f8a440b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4bd84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a440d4a50 0x7f8a440cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4bd84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44164e00 con 0x7f8a4406c5c0 2024-04-25T15:25:41.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4bd84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a440b5320 con 0x7f8a440d0e00 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4bd84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a44164f80 con 0x7f8a440d4a50 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a49b20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a4406c5c0 0x7f8a440cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a49b20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a4406c5c0 0x7f8a440cbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38186/0 (socket says 172.21.15.67:38186) 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a49b20640 1 -- 172.21.15.67:0/2704346961 learned_addr learned my addr 172.21.15.67:0/2704346961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4931f640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 0x7f8a440b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.688+0000 7f8a4a321640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a440d4a50 0x7f8a440cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a49b20640 1 -- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a440d4a50 msgr2=0x7f8a440cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a49b20640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a440d4a50 0x7f8a440cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a49b20640 1 -- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 msgr2=0x7f8a440b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a49b20640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 0x7f8a440b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a49b20640 1 -- 172.21.15.67:0/2704346961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a440ca880 con 0x7f8a4406c5c0 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a49b20640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a4406c5c0 0x7f8a440cbd90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8a40035040 tx=0x7f8a400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a3a7fc640 1 -- 172.21.15.67:0/2704346961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40037050 con 0x7f8a4406c5c0 2024-04-25T15:25:41.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a3a7fc640 1 -- 172.21.15.67:0/2704346961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a40043070 con 0x7f8a4406c5c0 2024-04-25T15:25:41.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a440cacf0 con 0x7f8a4406c5c0 2024-04-25T15:25:41.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a3a7fc640 1 -- 172.21.15.67:0/2704346961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40006e10 con 0x7f8a4406c5c0 2024-04-25T15:25:41.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.689+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a440cb2b0 con 0x7f8a4406c5c0 2024-04-25T15:25:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.690+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a44050740 con 0x7f8a4406c5c0 2024-04-25T15:25:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.691+0000 7f8a3a7fc640 1 -- 172.21.15.67:0/2704346961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8a40032810 con 0x7f8a4406c5c0 2024-04-25T15:25:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.691+0000 7f8a3a7fc640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a280744b0 0x7f8a280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.691+0000 7f8a3a7fc640 1 -- 172.21.15.67:0/2704346961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8a400b5bd0 con 0x7f8a4406c5c0 2024-04-25T15:25:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.691+0000 7f8a4931f640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a280744b0 0x7f8a280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.692+0000 7f8a4931f640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a280744b0 0x7f8a280769c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a440b5810 tx=0x7f8a34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.696+0000 7f8a3a7fc640 1 -- 172.21.15.67:0/2704346961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a4007fd10 con 0x7f8a4406c5c0 2024-04-25T15:25:41.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.881+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a440d4320 con 0x7f8a4406c5c0 2024-04-25T15:25:41.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.881+0000 7f8a3a7fc640 1 -- 172.21.15.67:0/2704346961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f8a40084010 con 0x7f8a4406c5c0 2024-04-25T15:25:41.882 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:25:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a280744b0 msgr2=0x7f8a280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a280744b0 0x7f8a280769c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a440b5810 tx=0x7f8a34009040 comp rx=0 tx=0).stop 2024-04-25T15:25:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a4406c5c0 msgr2=0x7f8a440cbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a4406c5c0 0x7f8a440cbd90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8a40035040 tx=0x7f8a400046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 shutdown_connections 2024-04-25T15:25:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a280744b0 0x7f8a280769c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a440d4a50 0x7f8a440cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a440d0e00 0x7f8a440b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 --2- 172.21.15.67:0/2704346961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a4406c5c0 0x7f8a440cbd90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 >> 172.21.15.67:0/2704346961 conn(0x7f8a440d5d90 msgr2=0x7f8a44161e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 shutdown_connections 2024-04-25T15:25:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:41.883+0000 7f8a4bd84640 1 -- 172.21.15.67:0/2704346961 wait complete. 2024-04-25T15:25:41.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:25:42.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.006+0000 7f8adc051640 1 Processor -- start 2024-04-25T15:25:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.006+0000 7f8adc051640 1 -- start start 2024-04-25T15:25:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8adc051640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ad40d0e00 0x7f8ad40d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8adc051640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8ad40d4a50 0x7f8ad40d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8adc051640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad40d1850 con 0x7f8ad406c5c0 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8adc051640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad4160de0 con 0x7f8ad40d0e00 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8adc051640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad41610c0 con 0x7f8ad40d4a50 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8ad9ded640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ad40d0e00 0x7f8ad40d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8ad9ded640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ad40d0e00 0x7f8ad40d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38192/0 (socket says 172.21.15.67:38192) 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8ad9ded640 1 -- 172.21.15.67:0/1374667825 learned_addr learned my addr 172.21.15.67:0/1374667825 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.007+0000 7f8ada5ee640 1 --2- 172.21.15.67:0/1374667825 >> v2:172.21.15.176:3300/0 conn(0x7f8ad40d4a50 0x7f8ad40d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.008+0000 7f8ad9ded640 1 -- 172.21.15.67:0/1374667825 >> v2:172.21.15.176:3300/0 conn(0x7f8ad40d4a50 msgr2=0x7f8ad40d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.008+0000 7f8ad9ded640 1 --2- 172.21.15.67:0/1374667825 >> v2:172.21.15.176:3300/0 conn(0x7f8ad40d4a50 0x7f8ad40d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.008+0000 7f8ad9ded640 1 -- 172.21.15.67:0/1374667825 >> v1:172.21.15.176:6789/0 conn(0x7f8ad406c5c0 legacy=0x7f8ad406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.008+0000 7f8ad9ded640 1 -- 172.21.15.67:0/1374667825 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad41613a0 con 0x7f8ad40d0e00 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.008+0000 7f8ad9ded640 1 --2- 172.21.15.67:0/1374667825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad40d1210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40d1750 tx=0x7f8ad002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a293ba9c9a7fa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.008+0000 7f8ad8deb640 1 -- 172.21.15.67:0/1374667825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0037070 con 0x7f8ad40d0e00 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8ad8deb640 1 -- 172.21.15.67:0/1374667825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad0032440 con 0x7f8ad40d0e00 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8ad8deb640 1 -- 172.21.15.67:0/1374667825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0032810 con 0x7f8ad40d0e00 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 -- 172.21.15.67:0/1374667825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 msgr2=0x7f8ad40d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 --2- 172.21.15.67:0/1374667825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad40d1210 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40d1750 tx=0x7f8ad002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 -- 172.21.15.67:0/1374667825 shutdown_connections 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 --2- 172.21.15.67:0/1374667825 >> v2:172.21.15.176:3300/0 conn(0x7f8ad40d4a50 0x7f8ad40d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 --2- 172.21.15.67:0/1374667825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad40d1210 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 -- 172.21.15.67:0/1374667825 >> 172.21.15.67:0/1374667825 conn(0x7f8ad40d5d90 msgr2=0x7f8ad40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 -- 172.21.15.67:0/1374667825 shutdown_connections 2024-04-25T15:25:42.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.009+0000 7f8adc051640 1 -- 172.21.15.67:0/1374667825 wait complete. 2024-04-25T15:25:42.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.011+0000 7f8adc051640 1 Processor -- start 2024-04-25T15:25:42.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.011+0000 7f8adc051640 1 -- start start 2024-04-25T15:25:42.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8adc051640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ad406c5c0 0x7f8ad415c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8adc051640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad41f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8adc051640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ad40d4a50 0x7f8ad41f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8adc051640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad4164e00 con 0x7f8ad40d0e00 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8adc051640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40b5320 con 0x7f8ad406c5c0 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8adc051640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad4164f80 con 0x7f8ad40d4a50 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8ad95ec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad41f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8ad95ec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad41f7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38198/0 (socket says 172.21.15.67:38198) 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.012+0000 7f8ad95ec640 1 -- 172.21.15.67:0/42568828 learned_addr learned my addr 172.21.15.67:0/42568828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ada5ee640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ad40d4a50 0x7f8ad41f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ad95ec640 1 -- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ad40d4a50 msgr2=0x7f8ad41f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ad95ec640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ad40d4a50 0x7f8ad41f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ad9ded640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ad406c5c0 0x7f8ad415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ad95ec640 1 -- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ad406c5c0 msgr2=0x7f8ad415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ad95ec640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ad406c5c0 0x7f8ad415c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ad95ec640 1 -- 172.21.15.67:0/42568828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad4157a50 con 0x7f8ad40d0e00 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8ad95ec640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad41f7210 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c3c0 tx=0x7f8ac400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8aca7fc640 1 -- 172.21.15.67:0/42568828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40052d0 con 0x7f8ad40d0e00 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8aca7fc640 1 -- 172.21.15.67:0/42568828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac4005490 con 0x7f8ad40d0e00 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad40cb3d0 con 0x7f8ad40d0e00 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8aca7fc640 1 -- 172.21.15.67:0/42568828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4007260 con 0x7f8ad40d0e00 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.013+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad415be70 con 0x7f8ad40d0e00 2024-04-25T15:25:42.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.014+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad4050740 con 0x7f8ad40d0e00 2024-04-25T15:25:42.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.015+0000 7f8aca7fc640 1 -- 172.21.15.67:0/42568828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8ac4007400 con 0x7f8ad40d0e00 2024-04-25T15:25:42.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.015+0000 7f8aca7fc640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8ab80744b0 0x7f8ab80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.015+0000 7f8aca7fc640 1 -- 172.21.15.67:0/42568828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8ac40946f0 con 0x7f8ad40d0e00 2024-04-25T15:25:42.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.016+0000 7f8ad9ded640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8ab80744b0 0x7f8ab80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.016+0000 7f8ad9ded640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8ab80744b0 0x7f8ab80769c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0034000 tx=0x7f8ad002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.020+0000 7f8aca7fc640 1 -- 172.21.15.67:0/42568828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ac405ee90 con 0x7f8ad40d0e00 2024-04-25T15:25:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.243+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ad406cc40 con 0x7f8ad40d0e00 2024-04-25T15:25:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.244+0000 7f8aca7fc640 1 -- 172.21.15.67:0/42568828 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7f8ac405e790 con 0x7f8ad40d0e00 2024-04-25T15:25:42.245 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:42.245 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:25:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8ab80744b0 msgr2=0x7f8ab80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8ab80744b0 0x7f8ab80769c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0034000 tx=0x7f8ad002e000 comp rx=0 tx=0).stop 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 msgr2=0x7f8ad41f7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad41f7210 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c3c0 tx=0x7f8ac400c880 comp rx=0 tx=0).stop 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 shutdown_connections 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8ab80744b0 0x7f8ab80769c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ad40d4a50 0x7f8ad41f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ad40d0e00 0x7f8ad41f7210 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 --2- 172.21.15.67:0/42568828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ad406c5c0 0x7f8ad415c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.245+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 >> 172.21.15.67:0/42568828 conn(0x7f8ad40d5d90 msgr2=0x7f8ad41602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.246+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 shutdown_connections 2024-04-25T15:25:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.246+0000 7f8adc051640 1 -- 172.21.15.67:0/42568828 wait complete. 2024-04-25T15:25:42.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.369+0000 7fc05f39e640 1 Processor -- start 2024-04-25T15:25:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.369+0000 7fc05f39e640 1 -- start start 2024-04-25T15:25:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.370+0000 7fc05f39e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc05806ccd0 0x7fc0580d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.370+0000 7fc05f39e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0580d4a30 0x7fc0580d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.370+0000 7fc05f39e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0580d1830 con 0x7fc0580d0de0 2024-04-25T15:25:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.370+0000 7fc05f39e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc058160dc0 con 0x7fc0580d4a30 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05f39e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc0581610a0 con 0x7fc05806ccd0 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05d93b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0580d4a30 0x7fc0580d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05d93b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0580d4a30 0x7fc0580d4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38204/0 (socket says 172.21.15.67:38204) 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05d93b640 1 -- 172.21.15.67:0/2877517707 learned_addr learned my addr 172.21.15.67:0/2877517707 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05c939640 1 --2- 172.21.15.67:0/2877517707 >> v2:172.21.15.176:3300/0 conn(0x7fc05806ccd0 0x7fc0580d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05d93b640 1 -- 172.21.15.67:0/2877517707 >> v2:172.21.15.176:3300/0 conn(0x7fc05806ccd0 msgr2=0x7fc0580d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05d93b640 1 --2- 172.21.15.67:0/2877517707 >> v2:172.21.15.176:3300/0 conn(0x7fc05806ccd0 0x7fc0580d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05d93b640 1 -- 172.21.15.67:0/2877517707 >> v1:172.21.15.176:6789/0 conn(0x7fc0580d0de0 legacy=0x7fc05806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05d93b640 1 -- 172.21.15.67:0/2877517707 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc058161380 con 0x7fc0580d4a30 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.371+0000 7fc05c939640 1 --2- 172.21.15.67:0/2877517707 >> v2:172.21.15.176:3300/0 conn(0x7fc05806ccd0 0x7fc0580d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.372+0000 7fc05d93b640 1 --2- 172.21.15.67:0/2877517707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580d4e40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc05808f150 tx=0x7fc05002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7132132bd84acb13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.372+0000 7fc04ffff640 1 -- 172.21.15.67:0/2877517707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050036070 con 0x7fc0580d4a30 2024-04-25T15:25:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.372+0000 7fc04ffff640 1 -- 172.21.15.67:0/2877517707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc050031600 con 0x7fc0580d4a30 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.372+0000 7fc04ffff640 1 -- 172.21.15.67:0/2877517707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0500319d0 con 0x7fc0580d4a30 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 -- 172.21.15.67:0/2877517707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 msgr2=0x7fc0580d4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 --2- 172.21.15.67:0/2877517707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580d4e40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc05808f150 tx=0x7fc05002a660 comp rx=0 tx=0).stop 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 -- 172.21.15.67:0/2877517707 shutdown_connections 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 --2- 172.21.15.67:0/2877517707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580d4e40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 --2- 172.21.15.67:0/2877517707 >> v2:172.21.15.176:3300/0 conn(0x7fc05806ccd0 0x7fc0580d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 -- 172.21.15.67:0/2877517707 >> 172.21.15.67:0/2877517707 conn(0x7fc0580d5d70 msgr2=0x7fc0580d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 -- 172.21.15.67:0/2877517707 shutdown_connections 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.373+0000 7fc05f39e640 1 -- 172.21.15.67:0/2877517707 wait complete. 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.374+0000 7fc05f39e640 1 Processor -- start 2024-04-25T15:25:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.375+0000 7fc05f39e640 1 -- start start 2024-04-25T15:25:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05f39e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0580d0de0 0x7fc0581f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05f39e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05f39e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0580b7d90 0x7fc0580cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05f39e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc058164de0 con 0x7fc0580d4a30 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05f39e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc0580b5300 con 0x7fc0580b7d90 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05f39e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc058164f60 con 0x7fc0580d0de0 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38210/0 (socket says 172.21.15.67:38210) 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 -- 172.21.15.67:0/112494549 learned_addr learned my addr 172.21.15.67:0/112494549 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05d93b640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0580b7d90 0x7fc0580cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05d13a640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0580d0de0 0x7fc0581f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 -- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0580d0de0 msgr2=0x7fc0581f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0580d0de0 0x7fc0581f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 -- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0580b7d90 msgr2=0x7fc0580cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0580b7d90 0x7fc0580cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 -- 172.21.15.67:0/112494549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0580b82a0 con 0x7fc0580d4a30 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.376+0000 7fc05c939640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580cd930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc04800d7d0 tx=0x7fc04800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.377+0000 7fc04dffb640 1 -- 172.21.15.67:0/112494549 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048004030 con 0x7fc0580d4a30 2024-04-25T15:25:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.377+0000 7fc04dffb640 1 -- 172.21.15.67:0/112494549 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0480041d0 con 0x7fc0580d4a30 2024-04-25T15:25:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.377+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0580b8540 con 0x7fc0580d4a30 2024-04-25T15:25:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.377+0000 7fc04dffb640 1 -- 172.21.15.67:0/112494549 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0480096a0 con 0x7fc0580d4a30 2024-04-25T15:25:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.377+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0580cd130 con 0x7fc0580d4a30 2024-04-25T15:25:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.378+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc058050740 con 0x7fc0580d4a30 2024-04-25T15:25:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.379+0000 7fc04dffb640 1 -- 172.21.15.67:0/112494549 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc048009840 con 0x7fc0580d4a30 2024-04-25T15:25:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.379+0000 7fc04dffb640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc03c0744e0 0x7fc03c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.379+0000 7fc04dffb640 1 -- 172.21.15.67:0/112494549 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc0480952d0 con 0x7fc0580d4a30 2024-04-25T15:25:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.380+0000 7fc05d13a640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc03c0744e0 0x7fc03c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.380+0000 7fc05d13a640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc03c0744e0 0x7fc03c0769f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc054006fd0 tx=0x7fc054009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.384+0000 7fc04dffb640 1 -- 172.21.15.67:0/112494549 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc04805f4a0 con 0x7fc0580d4a30 2024-04-25T15:25:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.569+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0580d4300 con 0x7fc0580d4a30 2024-04-25T15:25:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.570+0000 7fc04dffb640 1 -- 172.21.15.67:0/112494549 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7fc0480637a0 con 0x7fc0580d4a30 2024-04-25T15:25:42.571 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:42.571 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc03c0744e0 msgr2=0x7fc03c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc03c0744e0 0x7fc03c0769f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc054006fd0 tx=0x7fc054009060 comp rx=0 tx=0).stop 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 msgr2=0x7fc0580cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580cd930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc04800d7d0 tx=0x7fc04800dc90 comp rx=0 tx=0).stop 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 shutdown_connections 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc03c0744e0 0x7fc03c0769f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0580b7d90 0x7fc0580cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0580d4a30 0x7fc0580cd930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 --2- 172.21.15.67:0/112494549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0580d0de0 0x7fc0581f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 >> 172.21.15.67:0/112494549 conn(0x7fc0580d5d70 msgr2=0x7fc0580d50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 shutdown_connections 2024-04-25T15:25:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.572+0000 7fc05f39e640 1 -- 172.21.15.67:0/112494549 wait complete. 2024-04-25T15:25:42.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:25:42.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.695+0000 7fe460234640 1 Processor -- start 2024-04-25T15:25:42.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.696+0000 7fe460234640 1 -- start start 2024-04-25T15:25:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe460234640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe45806ccd0 0x7fe4580d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe460234640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4580d4a30 0x7fe4580d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe460234640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4580d1830 con 0x7fe4580d0de0 2024-04-25T15:25:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe460234640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe458160dc0 con 0x7fe4580d4a30 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe460234640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe4581610a0 con 0x7fe45806ccd0 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe45d7cf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe45806ccd0 0x7fe4580d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe45e7d1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4580d4a30 0x7fe4580d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe45d7cf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe45806ccd0 0x7fe4580d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52472/0 (socket says 172.21.15.67:52472) 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe45e7d1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4580d4a30 0x7fe4580d4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39806/0 (socket says 172.21.15.67:39806) 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe45dfd0640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe4580d0de0 0x7fe45806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56208/0 (socket says 172.21.15.67:56208) 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.697+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1754728689 learned_addr learned my addr 172.21.15.67:0/1754728689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.698+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1754728689 >> v2:172.21.15.176:3300/0 conn(0x7fe4580d4a30 msgr2=0x7fe4580d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.698+0000 7fe45d7cf640 1 --2- 172.21.15.67:0/1754728689 >> v2:172.21.15.176:3300/0 conn(0x7fe4580d4a30 0x7fe4580d4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.698+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1754728689 >> v1:172.21.15.176:6789/0 conn(0x7fe4580d0de0 legacy=0x7fe45806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.698+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1754728689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe458161380 con 0x7fe45806ccd0 2024-04-25T15:25:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.698+0000 7fe45d7cf640 1 --2- 172.21.15.67:0/1754728689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe45806ccd0 0x7fe4580d11f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009870 tx=0x7fe44c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd3b6d7b5c7fdf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe45cfce640 1 -- 172.21.15.67:0/1754728689 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c037070 con 0x7fe45806ccd0 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe45cfce640 1 -- 172.21.15.67:0/1754728689 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44c032440 con 0x7fe45806ccd0 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe45cfce640 1 -- 172.21.15.67:0/1754728689 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c032810 con 0x7fe45806ccd0 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 -- 172.21.15.67:0/1754728689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe45806ccd0 msgr2=0x7fe4580d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 --2- 172.21.15.67:0/1754728689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe45806ccd0 0x7fe4580d11f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009870 tx=0x7fe44c02a270 comp rx=0 tx=0).stop 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 -- 172.21.15.67:0/1754728689 shutdown_connections 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 --2- 172.21.15.67:0/1754728689 >> v2:172.21.15.176:3300/0 conn(0x7fe4580d4a30 0x7fe4580d4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 --2- 172.21.15.67:0/1754728689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe45806ccd0 0x7fe4580d11f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 -- 172.21.15.67:0/1754728689 >> 172.21.15.67:0/1754728689 conn(0x7fe4580d5d70 msgr2=0x7fe4580d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 -- 172.21.15.67:0/1754728689 shutdown_connections 2024-04-25T15:25:42.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.699+0000 7fe460234640 1 -- 172.21.15.67:0/1754728689 wait complete. 2024-04-25T15:25:42.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.701+0000 7fe460234640 1 Processor -- start 2024-04-25T15:25:42.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.701+0000 7fe460234640 1 -- start start 2024-04-25T15:25:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe460234640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4580d0de0 0x7fe4581f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe460234640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4580d4a30 0x7fe4580ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe460234640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580caa40 0x7fe4580b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe460234640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe458164de0 con 0x7fe4580d4a30 2024-04-25T15:25:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe460234640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580b5300 con 0x7fe4580caa40 2024-04-25T15:25:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe460234640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe458164f60 con 0x7fe4580d0de0 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe45d7cf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4580d4a30 0x7fe4580ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe45d7cf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4580d4a30 0x7fe4580ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38212/0 (socket says 172.21.15.67:38212) 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.702+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1800104006 learned_addr learned my addr 172.21.15.67:0/1800104006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45e7d1640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580caa40 0x7fe4580b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45dfd0640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4580d0de0 0x7fe4581f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4580d0de0 msgr2=0x7fe4581f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45d7cf640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4580d0de0 0x7fe4581f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580caa40 msgr2=0x7fe4580b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45d7cf640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580caa40 0x7fe4580b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45d7cf640 1 -- 172.21.15.67:0/1800104006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4580cb280 con 0x7fe4580d4a30 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45dfd0640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4580d0de0 0x7fe4581f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe45d7cf640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4580d4a30 0x7fe4580ca500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe44c02a2b0 tx=0x7fe44c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe4567fc640 1 -- 172.21.15.67:0/1800104006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c037040 con 0x7fe4580d4a30 2024-04-25T15:25:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe4567fc640 1 -- 172.21.15.67:0/1800104006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44c042070 con 0x7fe4580d4a30 2024-04-25T15:25:42.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4580cb560 con 0x7fe4580d4a30 2024-04-25T15:25:42.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe4567fc640 1 -- 172.21.15.67:0/1800104006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c030880 con 0x7fe4580d4a30 2024-04-25T15:25:42.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.703+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4580cd1c0 con 0x7fe4580d4a30 2024-04-25T15:25:42.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.704+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe458050740 con 0x7fe4580d4a30 2024-04-25T15:25:42.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.705+0000 7fe4567fc640 1 -- 172.21.15.67:0/1800104006 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe44c030a20 con 0x7fe4580d4a30 2024-04-25T15:25:42.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.705+0000 7fe4567fc640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4440744e0 0x7fe4440769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:42.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.705+0000 7fe4567fc640 1 -- 172.21.15.67:0/1800104006 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe44c0b5a90 con 0x7fe4580d4a30 2024-04-25T15:25:42.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.706+0000 7fe45dfd0640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4440744e0 0x7fe4440769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:42.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.706+0000 7fe45dfd0640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4440744e0 0x7fe4440769f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe450007fd0 tx=0x7fe450007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.710+0000 7fe4567fc640 1 -- 172.21.15.67:0/1800104006 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe44c080220 con 0x7fe4580d4a30 2024-04-25T15:25:42.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.935+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4580d4300 con 0x7fe4580d4a30 2024-04-25T15:25:42.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.935+0000 7fe4567fc640 1 -- 172.21.15.67:0/1800104006 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7fe44c07fb20 con 0x7fe4580d4a30 2024-04-25T15:25:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:42.937 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:25:42.937 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:25:42.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4440744e0 msgr2=0x7fe4440769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4440744e0 0x7fe4440769f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe450007fd0 tx=0x7fe450007d60 comp rx=0 tx=0).stop 2024-04-25T15:25:42.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4580d4a30 msgr2=0x7fe4580ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4580d4a30 0x7fe4580ca500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe44c02a2b0 tx=0x7fe44c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 shutdown_connections 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4440744e0 0x7fe4440769f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580caa40 0x7fe4580b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4580d4a30 0x7fe4580ca500 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 --2- 172.21.15.67:0/1800104006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4580d0de0 0x7fe4581f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 >> 172.21.15.67:0/1800104006 conn(0x7fe4580d5d70 msgr2=0x7fe4581641b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 shutdown_connections 2024-04-25T15:25:42.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:42.937+0000 7fe460234640 1 -- 172.21.15.67:0/1800104006 wait complete. 2024-04-25T15:25:43.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:25:28.675855+0000', 'modified': '2024-04-25T15:25:31.686830+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4287': {'gid': 4287, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.176:6836/4089268988', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6834', 'nonce': 4089268988}, {'type': 'v1', 'addr': '172.21.15.176:6836', 'nonce': 4089268988}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4287, 'qdb_cluster': [4287]} 2024-04-25T15:25:43.352 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:25:43.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T15:25:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.466+0000 7f075be92640 1 Processor -- start 2024-04-25T15:25:43.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.466+0000 7f075be92640 1 -- start start 2024-04-25T15:25:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.467+0000 7f075be92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 0x7f07540d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.467+0000 7f075be92640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07540b3c10 con 0x7f07540d4130 2024-04-25T15:25:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.467+0000 7f075be92640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07540b3ef0 con 0x7f075406d8e0 2024-04-25T15:25:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.467+0000 7f075be92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07540b9520 con 0x7f075406dcf0 2024-04-25T15:25:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.467+0000 7f075942d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07540d4130 0x7f07540d1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53804/0 (socket says 172.21.15.67:53804) 2024-04-25T15:25:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.467+0000 7f075942d640 1 -- 172.21.15.67:0/3919592573 learned_addr learned my addr 172.21.15.67:0/3919592573 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:43.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f075a42f640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 0x7f07540d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874687261 0 0) 0x7f07540b3c10 con 0x7f07540d4130 2024-04-25T15:25:43.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0740003520 con 0x7f07540d4130 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3868900444 0 0) 0x7f0740003520 con 0x7f07540d4130 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 msgr2=0x7f07540d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 0x7f07540d2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 >> v1:172.21.15.176:6789/0 conn(0x7f075406d8e0 legacy=0x7f07540d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07540b41d0 con 0x7f07540d4130 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f0744001670 con 0x7f07540d4130 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 >> v1:172.21.15.67:6789/0 conn(0x7f07540d4130 legacy=0x7f07540d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 0x7f07400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0740003f60 0x7f0740004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0740004d40 0x7f0740005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07540b9520 con 0x7f0740004d40 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f074000bc00 con 0x7f0740003f60 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.468+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f074000be00 con 0x7f075406dcf0 2024-04-25T15:25:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075a42f640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0740004d40 0x7f0740005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f0759c2e640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 0x7f07400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075942d640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0740003f60 0x7f0740004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075a42f640 1 -- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 msgr2=0x7f07400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075a42f640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075406dcf0 0x7f07400039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075a42f640 1 -- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0740003f60 msgr2=0x7f0740004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075a42f640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0740003f60 0x7f0740004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075a42f640 1 -- 172.21.15.67:0/3919592573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0740003520 con 0x7f0740004d40 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.469+0000 7f075942d640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0740003f60 0x7f0740004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f075a42f640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0740004d40 0x7f0740005150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f074c009fc0 tx=0x7f0740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f690172d019acb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c034020 con 0x7f0740004d40 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c0305f0 con 0x7f0740004d40 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f0758c2c640 1 -- 172.21.15.67:0/3919592573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c009c10 con 0x7f0740004d40 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f075be92640 1 -- 172.21.15.67:0/3919592573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0740004d40 msgr2=0x7f0740005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f075be92640 1 --2- 172.21.15.67:0/3919592573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0740004d40 0x7f0740005150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f074c009fc0 tx=0x7f0740005830 comp rx=0 tx=0).stop 2024-04-25T15:25:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f075a42f640 1 -- 172.21.15.67:0/3919592573 reap_dead start 2024-04-25T15:25:43.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f075be92640 1 -- 172.21.15.67:0/3919592573 shutdown_connections 2024-04-25T15:25:43.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.470+0000 7f075be92640 1 -- 172.21.15.67:0/3919592573 >> 172.21.15.67:0/3919592573 conn(0x7f075408d110 msgr2=0x7f075408d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:43.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.471+0000 7f075be92640 1 -- 172.21.15.67:0/3919592573 shutdown_connections 2024-04-25T15:25:43.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.471+0000 7f075be92640 1 -- 172.21.15.67:0/3919592573 wait complete. 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.472+0000 7f075be92640 1 Processor -- start 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.472+0000 7f075be92640 1 -- start start 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f075409d570 0x7f07541ea460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075415f590 0x7f0754151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07540a0ae0 0x7f07540a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07540b38e0 con 0x7f075409d570 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f07540b4820 con 0x7f07540a0ae0 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07540d2470 con 0x7f075415f590 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f075409d570 0x7f07541ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f075409d570 0x7f07541ea460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38230/0 (socket says 172.21.15.67:38230) 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 -- 172.21.15.67:0/363129949 learned_addr learned my addr 172.21.15.67:0/363129949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075942d640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075415f590 0x7f0754151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075a42f640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07540a0ae0 0x7f07540a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 -- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075415f590 msgr2=0x7f0754151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075415f590 0x7f0754151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 -- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07540a0ae0 msgr2=0x7f07540a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07540a0ae0 0x7f07540a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 -- 172.21.15.67:0/363129949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07540b8190 con 0x7f075409d570 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f0759c2e640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f075409d570 0x7f07541ea460 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f075000f7f0 tx=0x7f075000fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f074a7fc640 1 -- 172.21.15.67:0/363129949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07500050d0 con 0x7f075409d570 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f074a7fc640 1 -- 172.21.15.67:0/363129949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0750018070 con 0x7f075409d570 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075409db50 con 0x7f075409d570 2024-04-25T15:25:43.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f074a7fc640 1 -- 172.21.15.67:0/363129949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075000d730 con 0x7f075409d570 2024-04-25T15:25:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.474+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f075409e110 con 0x7f075409d570 2024-04-25T15:25:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.476+0000 7f074a7fc640 1 -- 172.21.15.67:0/363129949 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f075000d8d0 con 0x7f075409d570 2024-04-25T15:25:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.476+0000 7f074a7fc640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07380744b0 0x7f07380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.477+0000 7f074a7fc640 1 -- 172.21.15.67:0/363129949 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0750094ee0 con 0x7f075409d570 2024-04-25T15:25:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.477+0000 7f075942d640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07380744b0 0x7f07380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.477+0000 7f075942d640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07380744b0 0x7f07380769c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f07540a0870 tx=0x7f0744007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.478+0000 7f0762e1a740 1 Processor -- start 2024-04-25T15:25:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.478+0000 7f0762e1a740 1 -- start start 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a61270 0x55fc27e2d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e29fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a40590 0x55fc27e03260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f0762e1a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fc27d67740 con 0x55fc27a40590 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55fc27d033a0 con 0x55fc27a61270 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55fc27d63700 con 0x55fc27a383b0 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f075a42f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a40590 0x55fc27e03260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f075a42f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a40590 0x55fc27e03260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38244/0 (socket says 172.21.15.67:38244) 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f075a42f640 1 -- 172.21.15.67:0/1092964360 learned_addr learned my addr 172.21.15.67:0/1092964360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f0759c2e640 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a61270 0x55fc27e2d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.479+0000 7f075942d640 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e29fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f075a42f640 1 -- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 msgr2=0x55fc27e29fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f075a42f640 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e29fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f075a42f640 1 -- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a61270 msgr2=0x55fc27e2d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f075a42f640 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a61270 0x55fc27e2d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f075a42f640 1 -- 172.21.15.67:0/1092964360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc27d1f750 con 0x55fc27a40590 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f075a42f640 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a40590 0x55fc27e03260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f074c02af30 tx=0x7f074c02d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28db5fd01171cf55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f07377fe640 1 -- 172.21.15.67:0/1092964360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c034040 con 0x55fc27a40590 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f07377fe640 1 -- 172.21.15.67:0/1092964360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c003990 con 0x55fc27a40590 2024-04-25T15:25:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.480+0000 7f07377fe640 1 -- 172.21.15.67:0/1092964360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c003d60 con 0x55fc27a40590 2024-04-25T15:25:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 -- 172.21.15.67:0/1092964360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a40590 msgr2=0x55fc27e03260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a40590 0x55fc27e03260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f074c02af30 tx=0x7f074c02d000 comp rx=0 tx=0).stop 2024-04-25T15:25:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 -- 172.21.15.67:0/1092964360 shutdown_connections 2024-04-25T15:25:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a40590 0x55fc27e03260 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e29fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1092964360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a61270 0x55fc27e2d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 -- 172.21.15.67:0/1092964360 >> 172.21.15.67:0/1092964360 conn(0x55fc27a553c0 msgr2=0x55fc27e029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:43.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 -- 172.21.15.67:0/1092964360 shutdown_connections 2024-04-25T15:25:43.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.481+0000 7f0762e1a740 1 -- 172.21.15.67:0/1092964360 wait complete. 2024-04-25T15:25:43.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.482+0000 7f0762e1a740 1 -- start start 2024-04-25T15:25:43.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.482+0000 7f0762e1a740 1 Processor -- start 2024-04-25T15:25:43.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e5ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a40590 0x55fc27e30250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a61270 0x55fc27e30790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f0762e1a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fc27ceefe0 con 0x55fc27a61270 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55fc27e277d0 con 0x55fc27a40590 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55fc27e47320 con 0x55fc27a383b0 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f0762e1a740 10 client.0 _ll_register_callbacks cb 0x55fc27d05150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f075a42f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a61270 0x55fc27e30790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f075a42f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a61270 0x55fc27e30790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38254/0 (socket says 172.21.15.67:38254) 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.483+0000 7f075a42f640 1 -- 172.21.15.67:0/1304514025 learned_addr learned my addr 172.21.15.67:0/1304514025 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0759c2e640 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e5ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f075942d640 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a40590 0x55fc27e30250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f075a42f640 1 -- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 msgr2=0x55fc27e5ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f075a42f640 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e5ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f075a42f640 1 -- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a40590 msgr2=0x55fc27e30250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f075a42f640 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a40590 0x55fc27e30250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f075a42f640 1 -- 172.21.15.67:0/1304514025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc27a792d0 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f075a42f640 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a61270 0x55fc27e30790 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f074c007090 tx=0x7f074c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed3025ae333a093 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0734ff9640 1 -- 172.21.15.67:0/1304514025 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c051060 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0734ff9640 1 -- 172.21.15.67:0/1304514025 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f074c044d80 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f074c003990 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0734ff9640 1 -- 172.21.15.67:0/1304514025 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f074c057050 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0762e1a740 10 client.4650 fetch_fsmap learned FSMap version 7 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55fc27b436b0 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0734ff9640 1 -- 172.21.15.67:0/1304514025 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c003990 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0734ff9640 1 -- 172.21.15.67:0/1304514025 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7f074c05a020 con 0x55fc27a61270 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0762e1a740 10 client.4650 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.484+0000 7f0762e1a740 10 client.4650 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.485+0000 7f0762e1a740 20 client.4650 populate_metadata read hostname 'smithi067' 2024-04-25T15:25:43.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.485+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc27a1fe40 0x55fc27e48070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.485+0000 7f0762e1a740 4 client.4650 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.485+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 --> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc27d75350 con 0x55fc27a1fe40 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.485+0000 7f0759c2e640 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc27a1fe40 0x55fc27e48070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.485+0000 7f0759c2e640 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc27a1fe40 0x55fc27e48070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25fb10977410fc8d server_cookie=984b1395f0bbd31e in_seq=0 out_seq=0 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.485+0000 7f0734ff9640 10 client.4650 ms_handle_connect on v2:172.21.15.176:6834/4089268988 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0734ff9640 1 -- 172.21.15.67:0/1304514025 <== mds.0 v2:172.21.15.176:6834/4089268988 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fc27d75350 con 0x55fc27a1fe40 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0734ff9640 10 client.4650 handle_command_reply: tid=0 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 client.4650 shutdown 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a61270 msgr2=0x55fc27e30790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a61270 0x55fc27e30790 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f074c007090 tx=0x7f074c004ab0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 shutdown_connections 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc27a1fe40 0x55fc27e48070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a61270 0x55fc27e30790 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a40590 0x55fc27e30250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0734ff9640 0 client.4650 ms_handle_reset on v2:172.21.15.176:6834/4089268988 2024-04-25T15:25:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 --2- 172.21.15.67:0/1304514025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a383b0 0x55fc27e5ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 >> 172.21.15.67:0/1304514025 conn(0x55fc27a553c0 msgr2=0x55fc27e01d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 shutdown_connections 2024-04-25T15:25:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.486+0000 7f0762e1a740 1 -- 172.21.15.67:0/1304514025 wait complete. 2024-04-25T15:25:43.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.487+0000 7f0762e1a740 20 client.4650 trim_cache size 0 max 16384 2024-04-25T15:25:43.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.506+0000 7f0762e1a740 1 Processor -- start 2024-04-25T15:25:43.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.506+0000 7f0762e1a740 1 -- start start 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a1fe40 0x55fc27e03fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a383b0 0x55fc27e48a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e26af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0762e1a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0754202490 con 0x55fc27a383b0 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f07541e83c0 con 0x55fc27a1fe40 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0754201f40 con 0x55fc27a40590 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a383b0 0x55fc27e48a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a383b0 0x55fc27e48a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38268/0 (socket says 172.21.15.67:38268) 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 -- 172.21.15.67:0/3986601714 learned_addr learned my addr 172.21.15.67:0/3986601714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0759c2e640 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a1fe40 0x55fc27e03fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075a42f640 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e26af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 -- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 msgr2=0x55fc27e26af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e26af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 -- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a1fe40 msgr2=0x55fc27e03fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a1fe40 0x55fc27e03fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075942d640 1 -- 172.21.15.67:0/3986601714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc27e007c0 con 0x55fc27a383b0 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f0759c2e640 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a1fe40 0x55fc27e03fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.507+0000 7f075a42f640 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e26af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f075942d640 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a383b0 0x55fc27e48a90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0744001d20 tx=0x7f0744014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed02b1aee3ba810 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0736ffd640 1 -- 172.21.15.67:0/3986601714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074401b070 con 0x55fc27a383b0 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0736ffd640 1 -- 172.21.15.67:0/3986601714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0744029070 con 0x55fc27a383b0 2024-04-25T15:25:43.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0736ffd640 1 -- 172.21.15.67:0/3986601714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07440147e0 con 0x55fc27a383b0 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 -- 172.21.15.67:0/3986601714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a383b0 msgr2=0x55fc27e48a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a383b0 0x55fc27e48a90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f0744001d20 tx=0x7f0744014000 comp rx=0 tx=0).stop 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 -- 172.21.15.67:0/3986601714 shutdown_connections 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e26af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a383b0 0x55fc27e48a90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 --2- 172.21.15.67:0/3986601714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a1fe40 0x55fc27e03fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 -- 172.21.15.67:0/3986601714 >> 172.21.15.67:0/3986601714 conn(0x55fc27a553c0 msgr2=0x55fc27e4b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.508+0000 7f0762e1a740 1 -- 172.21.15.67:0/3986601714 shutdown_connections 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.509+0000 7f0762e1a740 1 -- 172.21.15.67:0/3986601714 wait complete. 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.509+0000 7f0762e1a740 1 -- start start 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.509+0000 7f0762e1a740 1 Processor -- start 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a1fe40 0x55fc27e02d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a383b0 0x55fc27e69b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0762e1a740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e6a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0762e1a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fc27e271f0 con 0x55fc27a1fe40 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55fc27e27510 con 0x55fc27a383b0 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0762e1a740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55fc27e470c0 con 0x55fc27a40590 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0762e1a740 10 client.0 _ll_register_callbacks cb 0x7f07500078d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a1fe40 0x55fc27e02d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a1fe40 0x55fc27e02d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38282/0 (socket says 172.21.15.67:38282) 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 -- 172.21.15.67:0/2274251896 learned_addr learned my addr 172.21.15.67:0/2274251896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f075942d640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a383b0 0x55fc27e69b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f075a42f640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e6a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 -- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 msgr2=0x55fc27e6a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e6a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 -- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a383b0 msgr2=0x55fc27e69b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a383b0 0x55fc27e69b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 -- 172.21.15.67:0/2274251896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc27e03890 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f075942d640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a383b0 0x55fc27e69b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f075a42f640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e6a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0759c2e640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a1fe40 0x55fc27e02d60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f075005f040 tx=0x7f075005ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122ac874a4f1e376 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.511+0000 7f0734ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0734ff9640 1 -- 172.21.15.67:0/2274251896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750067dd0 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0734ff9640 1 -- 172.21.15.67:0/2274251896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0750067940 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0744029070 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0734ff9640 1 -- 172.21.15.67:0/2274251896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f0750080050 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 10 client.4656 fetch_fsmap learned FSMap version 7 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55fc27ec5dc0 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0734ff9640 1 -- 172.21.15.67:0/2274251896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0744029070 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0734ff9640 1 -- 172.21.15.67:0/2274251896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7f075000dbc0 con 0x55fc27a1fe40 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 10 client.4656 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T15:25:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 10 client.4656 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 20 client.4656 populate_metadata read hostname 'smithi067' 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc279f5280 0x55fc27e49110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 4 client.4656 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 --> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] -- command(tid 0: {"prefix": "status"}) -- 0x55fc27e4c550 con 0x55fc279f5280 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.512+0000 7f075942d640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc279f5280 0x55fc27e49110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0734ff9640 10 client.4656 ms_handle_connect on v2:172.21.15.176:6834/4089268988 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f075942d640 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc279f5280 0x55fc27e49110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f35bdbd65a101b44 server_cookie=ce9c16c818ca523b in_seq=0 out_seq=0 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0734ff9640 1 -- 172.21.15.67:0/2274251896 <== mds.0 v2:172.21.15.176:6834/4089268988 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55fc27e4c550 con 0x55fc279f5280 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0734ff9640 10 client.4656 handle_command_reply: tid=0 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 client.4656 shutdown 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a1fe40 msgr2=0x55fc27e02d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a1fe40 0x55fc27e02d60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f075005f040 tx=0x7f075005ec80 comp rx=0 tx=0).stop 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 shutdown_connections 2024-04-25T15:25:43.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:6834/4089268988,v1:172.21.15.176:6836/4089268988] conn(0x55fc279f5280 0x55fc27e49110 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fc27a40590 0x55fc27e6a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fc27a383b0 0x55fc27e69b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0734ff9640 0 client.4656 ms_handle_reset on v2:172.21.15.176:6834/4089268988 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 --2- 172.21.15.67:0/2274251896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fc27a1fe40 0x55fc27e02d60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 >> 172.21.15.67:0/2274251896 conn(0x55fc27a553c0 msgr2=0x55fc27e30010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 shutdown_connections 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 1 -- 172.21.15.67:0/2274251896 wait complete. 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.513+0000 7f0762e1a740 20 client.4656 trim_cache size 0 max 16384 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b43d0acb-252a-4f7a-bf14-34d789b94b95", 2024-04-25T15:25:43.519 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4287, 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 14.212143737, 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 7, 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 20, 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 20, 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 20.918860635000001 2024-04-25T15:25:43.520 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07380744b0 msgr2=0x7f07380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07380744b0 0x7f07380769c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f07540a0870 tx=0x7f0744007040 comp rx=0 tx=0).stop 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f075409d570 msgr2=0x7f07541ea460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f075409d570 0x7f07541ea460 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f075000f7f0 tx=0x7f075000fcb0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 shutdown_connections 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07380744b0 0x7f07380769c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07540a0ae0 0x7f07540a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f075415f590 0x7f0754151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 --2- 172.21.15.67:0/363129949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f075409d570 0x7f07541ea460 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:43.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 >> 172.21.15.67:0/363129949 conn(0x7f075408d110 msgr2=0x7f07540b8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:43.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 shutdown_connections 2024-04-25T15:25:43.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:43.514+0000 7f075be92640 1 -- 172.21.15.67:0/363129949 wait complete. 2024-04-25T15:25:43.526 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T15:25:43.536 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T15:25:43.536 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T15:25:43.536 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T15:25:43.536 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T15:25:43.563 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T15:25:43.564 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-25T15:25:43.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.564 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:25:43.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-25T15:25:43.646 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-25T15:25:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:25:43.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.647 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:25:43.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-25T15:25:43.769 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-25T15:25:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:25:43.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.771 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:25:43.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.789 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link delete ceph-brx 2024-04-25T15:25:43.860 INFO:teuthology.orchestra.run.smithi176.stderr:Cannot find device "ceph-brx" 2024-04-25T15:25:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:25:43.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.862 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:25:43.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:43.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link delete ceph-brx 2024-04-25T15:25:43.986 INFO:teuthology.orchestra.run.smithi176.stderr:Cannot find device "ceph-brx" 2024-04-25T15:25:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:25:43.988 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-25T15:25:43.988 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:25:43.988 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:25:43.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:25:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.100+0000 7f94bc415640 1 Processor -- start 2024-04-25T15:25:44.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.100+0000 7f94bc415640 1 -- start start 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94bc415640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94b40b8290 0x7f94b4160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94bc415640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b4160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94bc415640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b41615f0 con 0x7f94b4164b00 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94bc415640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f94b4164670 con 0x7f94b40b8290 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94bc415640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f94b4164020 con 0x7f94b415df80 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94b99b0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94b40b8290 0x7f94b4160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94b99b0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94b40b8290 0x7f94b4160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38296/0 (socket says 172.21.15.67:38296) 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.101+0000 7f94b99b0640 1 -- 172.21.15.67:0/303879131 learned_addr learned my addr 172.21.15.67:0/303879131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.102+0000 7f94ba9b2640 1 --2- 172.21.15.67:0/303879131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b4160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.102+0000 7f94b99b0640 1 -- 172.21.15.67:0/303879131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 msgr2=0x7f94b4160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.102+0000 7f94b99b0640 1 --2- 172.21.15.67:0/303879131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b4160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.102+0000 7f94b99b0640 1 -- 172.21.15.67:0/303879131 >> v1:172.21.15.176:6789/0 conn(0x7f94b4164b00 legacy=0x7f94b40b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:44.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.102+0000 7f94b99b0640 1 -- 172.21.15.67:0/303879131 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b4164300 con 0x7f94b40b8290 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.102+0000 7f94b99b0640 1 --2- 172.21.15.67:0/303879131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b40b8290 0x7f94b4160890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009870 tx=0x7f94a402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e518209591a9a8d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94b91af640 1 -- 172.21.15.67:0/303879131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4037070 con 0x7f94b40b8290 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94b91af640 1 -- 172.21.15.67:0/303879131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a4032440 con 0x7f94b40b8290 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94b91af640 1 -- 172.21.15.67:0/303879131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4032810 con 0x7f94b40b8290 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 -- 172.21.15.67:0/303879131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b40b8290 msgr2=0x7f94b4160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 --2- 172.21.15.67:0/303879131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b40b8290 0x7f94b4160890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009870 tx=0x7f94a402a270 comp rx=0 tx=0).stop 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 -- 172.21.15.67:0/303879131 shutdown_connections 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 --2- 172.21.15.67:0/303879131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b4160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 --2- 172.21.15.67:0/303879131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b40b8290 0x7f94b4160890 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 -- 172.21.15.67:0/303879131 >> 172.21.15.67:0/303879131 conn(0x7f94b40d2bc0 msgr2=0x7f94b40d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 -- 172.21.15.67:0/303879131 shutdown_connections 2024-04-25T15:25:44.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.103+0000 7f94bc415640 1 -- 172.21.15.67:0/303879131 wait complete. 2024-04-25T15:25:44.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.105+0000 7f94bc415640 1 Processor -- start 2024-04-25T15:25:44.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.105+0000 7f94bc415640 1 -- start start 2024-04-25T15:25:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.106+0000 7f94bc415640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b40b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.106+0000 7f94bc415640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b4164b00 0x7f94b4152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.106+0000 7f94bc415640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b40c8dc0 0x7f94b40d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.106+0000 7f94bc415640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f94b40b4a00 con 0x7f94b4164b00 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.106+0000 7f94bc415640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f94b40b4880 con 0x7f94b40c8dc0 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.106+0000 7f94bc415640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f94b40b4b80 con 0x7f94b415df80 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b4164b00 0x7f94b4152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b4164b00 0x7f94b4152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38308/0 (socket says 172.21.15.67:38308) 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 -- 172.21.15.67:0/1856998955 learned_addr learned my addr 172.21.15.67:0/1856998955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94ba9b2640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b40c8dc0 0x7f94b40d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94ba1b1640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b40b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 -- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 msgr2=0x7f94b40b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b40b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 -- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b40c8dc0 msgr2=0x7f94b40d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:44.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b40c8dc0 0x7f94b40d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 -- 172.21.15.67:0/1856998955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b4153410 con 0x7f94b4164b00 2024-04-25T15:25:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94b99b0640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b4164b00 0x7f94b4152bc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f94a4000c00 tx=0x7f94a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94aaffd640 1 -- 172.21.15.67:0/1856998955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4037040 con 0x7f94b4164b00 2024-04-25T15:25:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94aaffd640 1 -- 172.21.15.67:0/1856998955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a4042070 con 0x7f94b4164b00 2024-04-25T15:25:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b41536f0 con 0x7f94b4164b00 2024-04-25T15:25:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94aaffd640 1 -- 172.21.15.67:0/1856998955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4030880 con 0x7f94b4164b00 2024-04-25T15:25:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.107+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b4153cb0 con 0x7f94b4164b00 2024-04-25T15:25:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.108+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b4050740 con 0x7f94b4164b00 2024-04-25T15:25:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.109+0000 7f94aaffd640 1 -- 172.21.15.67:0/1856998955 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f94a4030a20 con 0x7f94b4164b00 2024-04-25T15:25:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.109+0000 7f94aaffd640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94980744b0 0x7f94980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.110+0000 7f94aaffd640 1 -- 172.21.15.67:0/1856998955 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f94a40b5a50 con 0x7f94b4164b00 2024-04-25T15:25:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.110+0000 7f94ba1b1640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94980744b0 0x7f94980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.110+0000 7f94ba1b1640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94980744b0 0x7f94980769c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f94b0007fd0 tx=0x7f94b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.114+0000 7f94aaffd640 1 -- 172.21.15.67:0/1856998955 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94a4080300 con 0x7f94b4164b00 2024-04-25T15:25:44.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.338+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f94b41531a0 con 0x7f94b4164b00 2024-04-25T15:25:44.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.339+0000 7f94aaffd640 1 -- 172.21.15.67:0/1856998955 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f94a407fc00 con 0x7f94b4164b00 2024-04-25T15:25:44.339 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.340+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94980744b0 msgr2=0x7f94980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.340+0000 7f94bc415640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94980744b0 0x7f94980769c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f94b0007fd0 tx=0x7f94b0009040 comp rx=0 tx=0).stop 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b4164b00 msgr2=0x7f94b4152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b4164b00 0x7f94b4152bc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f94a4000c00 tx=0x7f94a40046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 shutdown_connections 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f94980744b0 0x7f94980769c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b40c8dc0 0x7f94b40d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94b4164b00 0x7f94b4152bc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 --2- 172.21.15.67:0/1856998955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94b415df80 0x7f94b40b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:44.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 >> 172.21.15.67:0/1856998955 conn(0x7f94b40d2bc0 msgr2=0x7f94b40d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:44.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 shutdown_connections 2024-04-25T15:25:44.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44.341+0000 7f94bc415640 1 -- 172.21.15.67:0/1856998955 wait complete. 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:25:44.353 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:25:44.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:44.353 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:25:44.368 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:25:44.369 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:44.370 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:25:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:25:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42755sec preferred_lft 42755sec 2024-04-25T15:25:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:25:44.370 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:44.370 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T15:25:44.371 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:44.371 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:44.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-04-25T15:25:44.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-04-25T15:25:44.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-04-25T15:25:44.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T15:25:44.371 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:44.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:44.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:44.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:44.548 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:25:44.573 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:25:44.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:44.575 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:25:44.637 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:25:44.637 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:25:44.638 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:25:44.638 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:25:44.638 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:25:44.638 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:44.639 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:44.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:25:44.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:25:44.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:25:44.639 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:44.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:44.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:44.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:44.906 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:25:44.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:44.922 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-25T15:25:44.977 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:44.977 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:44.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:25:44.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T15:25:44.978 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:45.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:45.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:45.098 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T15:25:45.098 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:45.099 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:45.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T15:25:45.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T15:25:45.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T15:25:45.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T15:25:45.099 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T15:25:45.099 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:45.231 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:45.231 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:45.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:25:45.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T15:25:45.231 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:45.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:45.311 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:45.311 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:25:45.327 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:25:45.327 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:45.327 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:25:45.379 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T15:25:45.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:45.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:45.482 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:45.483 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:45.596 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:45.596 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:45.596 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:25:45.596 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:25:45.596 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:25:45.597 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T15:25:45.597 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:25:45.597 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:25:45.597 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:25:45.597 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:25:45.598 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:25:45.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:45.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:25:45.628 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:45.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:45.730 INFO:teuthology.orchestra.run.smithi067.stdout:client4538 2024-04-25T15:25:45.731 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:45.731 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:25:45.747 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:45.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:45.849 INFO:teuthology.orchestra.run.smithi067.stdout:client4538 2024-04-25T15:25:45.849 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:45.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:45.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:45.912 INFO:teuthology.orchestra.run.smithi067.stdout:client4538 2024-04-25T15:25:45.912 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/b43d0acb-252a-4f7a-bf14-34d789b94b95.client4538/status') 2024-04-25T15:25:45.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:45.975 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4538 (3)172.21.15.67:0/1871215720 2024-04-25T15:25:45.976 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-04-25T15:25:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-04-25T15:25:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-04-25T15:25:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:70 bytes copied, 2.4517e-05 s, 2.9 MB/s 2024-04-25T15:25:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:45.984 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-04-25T15:25:45.984 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:25:45.984 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T15:25:45.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:25:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.094+0000 7f0819b0d640 1 Processor -- start 2024-04-25T15:25:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.094+0000 7f0819b0d640 1 -- start start 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.095+0000 7f0819b0d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f081406ccd0 0x7f08140d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.095+0000 7f0819b0d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f08140d4a50 0x7f08140d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.095+0000 7f0819b0d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08140d1850 con 0x7f08140d0e00 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.095+0000 7f0819b0d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0814160de0 con 0x7f08140d4a50 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.095+0000 7f0819b0d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f08141610c0 con 0x7f081406ccd0 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.095+0000 7f0812ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f081406ccd0 0x7f08140d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0813fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f08140d4a50 0x7f08140d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0812ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f081406ccd0 0x7f08140d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52572/0 (socket says 172.21.15.67:52572) 2024-04-25T15:25:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0812ffd640 1 -- 172.21.15.67:0/4082837263 learned_addr learned my addr 172.21.15.67:0/4082837263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f08137fe640 1 --1- 172.21.15.67:0/4082837263 >> v1:172.21.15.176:6789/0 conn(0x7f08140d0e00 0x7f081406c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56224/0 (socket says 172.21.15.67:56224) 2024-04-25T15:25:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0812ffd640 1 -- 172.21.15.67:0/4082837263 >> v2:172.21.15.176:3300/0 conn(0x7f08140d4a50 msgr2=0x7f08140d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0812ffd640 1 --2- 172.21.15.67:0/4082837263 >> v2:172.21.15.176:3300/0 conn(0x7f08140d4a50 0x7f08140d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0812ffd640 1 -- 172.21.15.67:0/4082837263 >> v1:172.21.15.176:6789/0 conn(0x7f08140d0e00 legacy=0x7f081406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0812ffd640 1 -- 172.21.15.67:0/4082837263 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08141613a0 con 0x7f081406ccd0 2024-04-25T15:25:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.096+0000 7f0813fff640 1 --2- 172.21.15.67:0/4082837263 >> v2:172.21.15.176:3300/0 conn(0x7f08140d4a50 0x7f08140d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.097+0000 7f0812ffd640 1 --2- 172.21.15.67:0/4082837263 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f081406ccd0 0x7f08140d1210 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0808009870 tx=0x7f080802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae58db605df48b6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.097+0000 7f08127fc640 1 -- 172.21.15.67:0/4082837263 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808037070 con 0x7f081406ccd0 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.097+0000 7f08127fc640 1 -- 172.21.15.67:0/4082837263 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0808032440 con 0x7f081406ccd0 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.097+0000 7f08127fc640 1 -- 172.21.15.67:0/4082837263 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808032810 con 0x7f081406ccd0 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 -- 172.21.15.67:0/4082837263 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f081406ccd0 msgr2=0x7f08140d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 --2- 172.21.15.67:0/4082837263 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f081406ccd0 0x7f08140d1210 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0808009870 tx=0x7f080802a270 comp rx=0 tx=0).stop 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 -- 172.21.15.67:0/4082837263 shutdown_connections 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 --2- 172.21.15.67:0/4082837263 >> v2:172.21.15.176:3300/0 conn(0x7f08140d4a50 0x7f08140d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 --2- 172.21.15.67:0/4082837263 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f081406ccd0 0x7f08140d1210 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 -- 172.21.15.67:0/4082837263 >> 172.21.15.67:0/4082837263 conn(0x7f08140d5d90 msgr2=0x7f08140d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 -- 172.21.15.67:0/4082837263 shutdown_connections 2024-04-25T15:25:46.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.098+0000 7f0819b0d640 1 -- 172.21.15.67:0/4082837263 wait complete. 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.099+0000 7f0819b0d640 1 Processor -- start 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.100+0000 7f0819b0d640 1 -- start start 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0819b0d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08140d0e00 0x7f081414e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0819b0d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08140d4a50 0x7f08140b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0819b0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08140b5c30 0x7f08140cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0819b0d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0814164e00 con 0x7f08140b5c30 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0819b0d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f08140b5320 con 0x7f08140d4a50 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0819b0d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0814164f80 con 0x7f08140d0e00 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08140b5c30 0x7f08140cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08140b5c30 0x7f08140cd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38318/0 (socket says 172.21.15.67:38318) 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 -- 172.21.15.67:0/2721600667 learned_addr learned my addr 172.21.15.67:0/2721600667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:46.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f08137fe640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08140d0e00 0x7f081414e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0812ffd640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08140d4a50 0x7f08140b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 -- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08140d0e00 msgr2=0x7f081414e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08140d0e00 0x7f081414e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 -- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08140d4a50 msgr2=0x7f08140b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08140d4a50 0x7f08140b56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0813fff640 1 -- 172.21.15.67:0/2721600667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08140b6470 con 0x7f08140b5c30 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.101+0000 7f0812ffd640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08140d4a50 0x7f08140b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.102+0000 7f0813fff640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08140b5c30 0x7f08140cd8e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f080c00e850 tx=0x7f080c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.102+0000 7f07fbfff640 1 -- 172.21.15.67:0/2721600667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c017070 con 0x7f08140b5c30 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.102+0000 7f07fbfff640 1 -- 172.21.15.67:0/2721600667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f080c00c940 con 0x7f08140b5c30 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.102+0000 7f07fbfff640 1 -- 172.21.15.67:0/2721600667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c00ccf0 con 0x7f08140b5c30 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.102+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08140b6750 con 0x7f08140b5c30 2024-04-25T15:25:46.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.102+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f081414e6d0 con 0x7f08140b5c30 2024-04-25T15:25:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.103+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08141eb660 con 0x7f08140b5c30 2024-04-25T15:25:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.104+0000 7f07fbfff640 1 -- 172.21.15.67:0/2721600667 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f080c01a070 con 0x7f08140b5c30 2024-04-25T15:25:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.104+0000 7f07fbfff640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08000744b0 0x7f08000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.104+0000 7f07fbfff640 1 -- 172.21.15.67:0/2721600667 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f080c095450 con 0x7f08140b5c30 2024-04-25T15:25:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.104+0000 7f08137fe640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08000744b0 0x7f08000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.105+0000 7f08137fe640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08000744b0 0x7f08000769c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0804007fd0 tx=0x7f0804001630 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.109+0000 7f07fbfff640 1 -- 172.21.15.67:0/2721600667 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f080c05f510 con 0x7f08140b5c30 2024-04-25T15:25:46.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.334+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f08140d4320 con 0x7f08140b5c30 2024-04-25T15:25:46.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.334+0000 7f07fbfff640 1 -- 172.21.15.67:0/2721600667 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f080c063810 con 0x7f08140b5c30 2024-04-25T15:25:46.336 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:25:46.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08000744b0 msgr2=0x7f08000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08000744b0 0x7f08000769c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0804007fd0 tx=0x7f0804001630 comp rx=0 tx=0).stop 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08140b5c30 msgr2=0x7f08140cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08140b5c30 0x7f08140cd8e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f080c00e850 tx=0x7f080c00ed10 comp rx=0 tx=0).stop 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 shutdown_connections 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08000744b0 0x7f08000769c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08140b5c30 0x7f08140cd8e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08140d4a50 0x7f08140b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 --2- 172.21.15.67:0/2721600667 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08140d0e00 0x7f081414e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 >> 172.21.15.67:0/2721600667 conn(0x7f08140d5d90 msgr2=0x7f08141631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.336+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 shutdown_connections 2024-04-25T15:25:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46.337+0000 7f0819b0d640 1 -- 172.21.15.67:0/2721600667 wait complete. 2024-04-25T15:25:46.349 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:25:46.349 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:25:46.349 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:25:46.350 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T15:25:46.350 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T15:25:46.350 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:25:46.350 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:25:46.350 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:25:46.350 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:25:46.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:46.350 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:46.366 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42753sec preferred_lft 42753sec 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:46.367 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7482:42ff:fe30:5ffc/64 scope link 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::885:bff:fe04:f6cb/64 scope link tentative 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:46.368 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:25:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:25:46.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:46.431 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:25:46.488 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:25:46.488 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:25:46.489 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:25:46.489 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:25:46.489 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:25:46.489 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:46.489 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:46.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:25:46.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:25:46.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:25:46.489 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:46.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:46.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:46.616 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:25:46.631 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:25:46.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:46.631 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-25T15:25:46.686 INFO:teuthology.orchestra.run.smithi067.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-25T15:25:46.686 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:46.686 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:46.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:25:46.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-25T15:25:46.687 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:46.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:46.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:46.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:46.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T15:25:46.827 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d2:a2:7d:71:70:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d0a2:7dff:fe71:70ee/64 scope link 2024-04-25T15:25:46.828 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:46.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T15:25:46.893 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:46.893 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:46.893 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether d2:a2:7d:71:70:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::d0a2:7dff:fe71:70ee/64 scope link 2024-04-25T15:25:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:46.895 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-25T15:25:46.896 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:47.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:47.047 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:47.047 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:25:47.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.1 up 2024-04-25T15:25:47.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T15:25:47.047 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:25:47.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:47.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:47.123 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T15:25:47.139 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T15:25:47.139 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:47.139 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T15:25:47.194 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-04-25T15:25:47.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:47.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:47.288 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:47.288 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:47.300 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:47.300 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:47.300 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:25:47.300 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 2@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:25:47.300 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:25:47.301 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-25T15:25:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:25:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:25:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:25:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:25:47.302 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:25:47.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:47.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-25T15:25:47.327 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:47.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:47.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:47.425 INFO:teuthology.orchestra.run.smithi067.stdout:client4547 2024-04-25T15:25:47.425 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:47.425 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:25:47.441 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:47.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:47.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:47.538 INFO:teuthology.orchestra.run.smithi067.stdout:client4547 2024-04-25T15:25:47.538 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:47.595 INFO:teuthology.orchestra.run.smithi067.stdout:client4547 2024-04-25T15:25:47.595 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/b43d0acb-252a-4f7a-bf14-34d789b94b95.client4547/status') 2024-04-25T15:25:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:47.653 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4547 (3)172.21.15.67:0/3554924325 2024-04-25T15:25:47.654 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-04-25T15:25:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-04-25T15:25:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-04-25T15:25:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:70 bytes copied, 2.5703e-05 s, 2.7 MB/s 2024-04-25T15:25:47.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:47.661 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-04-25T15:25:47.661 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:25:47.661 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:25:47.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:25:47.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.771+0000 7f038a6ee640 1 Processor -- start 2024-04-25T15:25:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.771+0000 7f038a6ee640 1 -- start start 2024-04-25T15:25:47.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f038a6ee640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03840d0e00 0x7f03840d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:47.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f038a6ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f038406c5c0 0x7f038406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:47.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f038a6ee640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f03840b7810 0x7f03840b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f038a6ee640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f038406cf10 con 0x7f03840d0e00 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f038a6ee640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f03840d1850 con 0x7f03840b7810 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f038a6ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0384160de0 con 0x7f038406c5c0 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03840d0e00 0x7f03840d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03840d0e00 0x7f03840d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59082/0 (socket says 172.21.15.67:59082) 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 -- 172.21.15.67:0/1351229751 learned_addr learned my addr 172.21.15.67:0/1351229751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0388c8b640 1 --2- 172.21.15.67:0/1351229751 >> v2:172.21.15.176:3300/0 conn(0x7f03840b7810 0x7f03840b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f03837fe640 1 --2- 172.21.15.67:0/1351229751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f038406c5c0 0x7f038406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 -- 172.21.15.67:0/1351229751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f038406c5c0 msgr2=0x7f038406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 --2- 172.21.15.67:0/1351229751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f038406c5c0 0x7f038406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 -- 172.21.15.67:0/1351229751 >> v2:172.21.15.176:3300/0 conn(0x7f03840b7810 msgr2=0x7f03840b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 --2- 172.21.15.67:0/1351229751 >> v2:172.21.15.176:3300/0 conn(0x7f03840b7810 0x7f03840b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.772+0000 7f0383fff640 1 -- 172.21.15.67:0/1351229751 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03841610c0 con 0x7f03840d0e00 2024-04-25T15:25:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.773+0000 7f0383fff640 1 --2- 172.21.15.67:0/1351229751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03840d0e00 0x7f03840d1210 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f038406d1e0 tx=0x7f037802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c033d92f97236bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.773+0000 7f0382ffd640 1 -- 172.21.15.67:0/1351229751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378037070 con 0x7f03840d0e00 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.773+0000 7f0382ffd640 1 -- 172.21.15.67:0/1351229751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0378032440 con 0x7f03840d0e00 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.773+0000 7f0382ffd640 1 -- 172.21.15.67:0/1351229751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378032810 con 0x7f03840d0e00 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 -- 172.21.15.67:0/1351229751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03840d0e00 msgr2=0x7f03840d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1351229751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03840d0e00 0x7f03840d1210 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f038406d1e0 tx=0x7f037802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 -- 172.21.15.67:0/1351229751 shutdown_connections 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1351229751 >> v2:172.21.15.176:3300/0 conn(0x7f03840b7810 0x7f03840b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1351229751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f038406c5c0 0x7f038406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1351229751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03840d0e00 0x7f03840d1210 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 -- 172.21.15.67:0/1351229751 >> 172.21.15.67:0/1351229751 conn(0x7f03840d5d90 msgr2=0x7f03840d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 -- 172.21.15.67:0/1351229751 shutdown_connections 2024-04-25T15:25:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.774+0000 7f038a6ee640 1 -- 172.21.15.67:0/1351229751 wait complete. 2024-04-25T15:25:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.776+0000 7f038a6ee640 1 Processor -- start 2024-04-25T15:25:47.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.776+0000 7f038a6ee640 1 -- start start 2024-04-25T15:25:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f038a6ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f038406c5c0 0x7f038415de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f038a6ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03840b7810 0x7f03841572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f038a6ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03840d0e00 0x7f038415be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f038a6ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03840b5320 con 0x7f038406c5c0 2024-04-25T15:25:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f038a6ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0384160ad0 con 0x7f03840d0e00 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f038a6ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0384164e00 con 0x7f03840b7810 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f0383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f038406c5c0 0x7f038415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f0383fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f038406c5c0 0x7f038415de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59094/0 (socket says 172.21.15.67:59094) 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.777+0000 7f0383fff640 1 -- 172.21.15.67:0/1161789368 learned_addr learned my addr 172.21.15.67:0/1161789368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f03837fe640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03840b7810 0x7f03841572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0388c8b640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03840d0e00 0x7f038415be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0383fff640 1 -- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03840b7810 msgr2=0x7f03841572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0383fff640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03840b7810 0x7f03841572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0383fff640 1 -- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03840d0e00 msgr2=0x7f038415be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0383fff640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03840d0e00 0x7f038415be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0383fff640 1 -- 172.21.15.67:0/1161789368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03840c98d0 con 0x7f038406c5c0 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0383fff640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f038406c5c0 0x7f038415de40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f037802a220 tx=0x7f03780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0380ff9640 1 -- 172.21.15.67:0/1161789368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378041020 con 0x7f038406c5c0 2024-04-25T15:25:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0380ff9640 1 -- 172.21.15.67:0/1161789368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03780306c0 con 0x7f038406c5c0 2024-04-25T15:25:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03840c9bb0 con 0x7f038406c5c0 2024-04-25T15:25:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f0380ff9640 1 -- 172.21.15.67:0/1161789368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378037040 con 0x7f038406c5c0 2024-04-25T15:25:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.778+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03840ca170 con 0x7f038406c5c0 2024-04-25T15:25:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.779+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0384050740 con 0x7f038406c5c0 2024-04-25T15:25:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.780+0000 7f0380ff9640 1 -- 172.21.15.67:0/1161789368 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0378044020 con 0x7f038406c5c0 2024-04-25T15:25:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.780+0000 7f0380ff9640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03700744e0 0x7f03700769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.781+0000 7f0380ff9640 1 -- 172.21.15.67:0/1161789368 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f03780b5890 con 0x7f038406c5c0 2024-04-25T15:25:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.781+0000 7f03837fe640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03700744e0 0x7f03700769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:47.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.781+0000 7f03837fe640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03700744e0 0x7f03700769f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f03841645b0 tx=0x7f0374008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:47.785+0000 7f0380ff9640 1 -- 172.21.15.67:0/1161789368 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f037804b030 con 0x7f038406c5c0 2024-04-25T15:25:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.009+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f03840b5ee0 con 0x7f038406c5c0 2024-04-25T15:25:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.010+0000 7f0380ff9640 1 -- 172.21.15.67:0/1161789368 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f037807fa60 con 0x7f038406c5c0 2024-04-25T15:25:48.012 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:25:48.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03700744e0 msgr2=0x7f03700769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:48.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03700744e0 0x7f03700769f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f03841645b0 tx=0x7f0374008040 comp rx=0 tx=0).stop 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f038406c5c0 msgr2=0x7f038415de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f038406c5c0 0x7f038415de40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f037802a220 tx=0x7f03780046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 shutdown_connections 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03700744e0 0x7f03700769f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03840d0e00 0x7f038415be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03840b7810 0x7f03841572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 --2- 172.21.15.67:0/1161789368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f038406c5c0 0x7f038415de40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.011+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 >> 172.21.15.67:0/1161789368 conn(0x7f03840d5d90 msgr2=0x7f0384160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.012+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 shutdown_connections 2024-04-25T15:25:48.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:48.012+0000 7f038a6ee640 1 -- 172.21.15.67:0/1161789368 wait complete. 2024-04-25T15:25:48.024 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:25:48.024 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:25:48.024 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:25:48.024 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:25:48.024 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:25:48.025 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:25:48.025 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:25:48.025 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:25:48.025 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:25:48.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:48.025 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:25:48.043 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:48.043 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:48.044 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:25:48.045 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:48.045 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:25:48.045 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42740sec preferred_lft 42740sec 2024-04-25T15:25:48.045 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:25:48.045 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:48.045 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T15:25:48.045 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:48.045 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:48.045 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link add name ceph-brx type bridge 2024-04-25T15:25:48.045 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip addr flush dev ceph-brx 2024-04-25T15:25:48.045 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set ceph-brx up 2024-04-25T15:25:48.046 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T15:25:48.046 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:48.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:48.227 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:48.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:48.233 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:25:48.257 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:25:48.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:48.258 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:25:48.322 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:25:48.322 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:25:48.322 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:25:48.323 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:25:48.323 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:25:48.323 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:48.323 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:48.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:25:48.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:25:48.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:25:48.323 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:48.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:48.569 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:48.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:48.576 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:25:48.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:48.590 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list-id 2024-04-25T15:25:48.646 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:48.646 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:48.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:25:48.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-25T15:25:48.646 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:48.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:48.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:48.761 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-25T15:25:48.761 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:48.761 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:48.761 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-25T15:25:48.761 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T15:25:48.761 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-25T15:25:48.761 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-25T15:25:48.762 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-25T15:25:48.762 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:48.810 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:48.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:48.882 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:48.883 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:48.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:25:48.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T15:25:48.883 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:48.931 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:48.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:48.958 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:48.958 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:25:48.972 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:25:48.973 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:48.973 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:25:49.027 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T15:25:49.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:49.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:49.132 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:49.132 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:49.249 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:49.249 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:49.249 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:25:49.249 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:25:49.249 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:25:49.250 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:25:49.250 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:25:49.250 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:25:49.250 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:25:49.250 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:25:49.252 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:25:49.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:49.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:25:49.275 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:49.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:49.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:49.370 INFO:teuthology.orchestra.run.smithi176.stdout:client4654 2024-04-25T15:25:49.370 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:49.371 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:25:49.386 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:49.473 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:49.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:49.480 INFO:teuthology.orchestra.run.smithi176.stdout:client4654 2024-04-25T15:25:49.480 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:49.528 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:49.529 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:49.535 INFO:teuthology.orchestra.run.smithi176.stdout:client4654 2024-04-25T15:25:49.536 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/b43d0acb-252a-4f7a-bf14-34d789b94b95.client4654/status') 2024-04-25T15:25:49.583 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:49.592 INFO:teuthology.orchestra.run.smithi176.stdout:instance: client.4654 (3)192.168.0.1:0/2761355947 2024-04-25T15:25:49.592 INFO:teuthology.orchestra.run.smithi176.stdout:blocklisted: false 2024-04-25T15:25:49.593 INFO:teuthology.orchestra.run.smithi176.stderr:0+1 records in 2024-04-25T15:25:49.593 INFO:teuthology.orchestra.run.smithi176.stderr:0+1 records out 2024-04-25T15:25:49.593 INFO:teuthology.orchestra.run.smithi176.stderr:69 bytes copied, 2.3376e-05 s, 3.0 MB/s 2024-04-25T15:25:49.593 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:49.599 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-04-25T15:25:49.600 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:25:49.600 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T15:25:49.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:25:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.724+0000 7f4bad0f2640 1 Processor -- start 2024-04-25T15:25:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.724+0000 7f4bad0f2640 1 -- start start 2024-04-25T15:25:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.725+0000 7f4bad0f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba80b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.725+0000 7f4bad0f2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba80cf4b0 con 0x7f4ba80b82d0 2024-04-25T15:25:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.725+0000 7f4bad0f2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba80cf790 con 0x7f4ba80d2200 2024-04-25T15:25:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.725+0000 7f4bad0f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba80d00c0 con 0x7f4ba8161760 2024-04-25T15:25:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.725+0000 7f4ba659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ba80b82d0 0x7f4ba80d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59048/0 (socket says 172.21.15.67:59048) 2024-04-25T15:25:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.725+0000 7f4ba659c640 1 -- 172.21.15.67:0/1633666456 learned_addr learned my addr 172.21.15.67:0/1633666456 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:49.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.725+0000 7f4ba6d9d640 1 --2- 172.21.15.67:0/1633666456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba80b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.726+0000 7f4ba5d9b640 1 -- 172.21.15.67:0/1633666456 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839169074 0 0) 0x7f4ba80cf4b0 con 0x7f4ba80b82d0 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.726+0000 7f4ba5d9b640 1 -- 172.21.15.67:0/1633666456 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b90003520 con 0x7f4ba80b82d0 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.726+0000 7f4ba5d9b640 1 -- 172.21.15.67:0/1633666456 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499993441 0 0) 0x7f4ba80cf790 con 0x7f4ba80d2200 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.726+0000 7f4ba5d9b640 1 -- 172.21.15.67:0/1633666456 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ba80cf4b0 con 0x7f4ba80d2200 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.726+0000 7f4ba6d9d640 1 -- 172.21.15.67:0/1633666456 >> v1:172.21.15.176:6789/0 conn(0x7f4ba80d2200 legacy=0x7f4ba80b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.726+0000 7f4ba6d9d640 1 -- 172.21.15.67:0/1633666456 >> v1:172.21.15.67:6789/0 conn(0x7f4ba80b82d0 legacy=0x7f4ba80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.726+0000 7f4ba6d9d640 1 -- 172.21.15.67:0/1633666456 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba80cfa70 con 0x7f4ba8161760 2024-04-25T15:25:49.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.727+0000 7f4ba6d9d640 1 --2- 172.21.15.67:0/1633666456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba80b7d90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b980097d0 tx=0x7f4b9802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dcabe9cfc2622a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.727+0000 7f4ba5d9b640 1 -- 172.21.15.67:0/1633666456 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98037070 con 0x7f4ba8161760 2024-04-25T15:25:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.727+0000 7f4ba5d9b640 1 -- 172.21.15.67:0/1633666456 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b98032440 con 0x7f4ba8161760 2024-04-25T15:25:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.727+0000 7f4ba5d9b640 1 -- 172.21.15.67:0/1633666456 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98032810 con 0x7f4ba8161760 2024-04-25T15:25:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.727+0000 7f4bad0f2640 1 -- 172.21.15.67:0/1633666456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 msgr2=0x7f4ba80b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.728+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/1633666456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba80b7d90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b980097d0 tx=0x7f4b9802a210 comp rx=0 tx=0).stop 2024-04-25T15:25:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.728+0000 7f4bad0f2640 1 -- 172.21.15.67:0/1633666456 shutdown_connections 2024-04-25T15:25:49.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.728+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/1633666456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba80b7d90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.728+0000 7f4bad0f2640 1 -- 172.21.15.67:0/1633666456 >> 172.21.15.67:0/1633666456 conn(0x7f4ba809b7d0 msgr2=0x7f4ba809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.728+0000 7f4bad0f2640 1 -- 172.21.15.67:0/1633666456 shutdown_connections 2024-04-25T15:25:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.728+0000 7f4bad0f2640 1 -- 172.21.15.67:0/1633666456 wait complete. 2024-04-25T15:25:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.730+0000 7f4bad0f2640 1 Processor -- start 2024-04-25T15:25:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.730+0000 7f4bad0f2640 1 -- start start 2024-04-25T15:25:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4bad0f2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba80b82d0 0x7f4ba8156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4bad0f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba80d2200 0x7f4ba815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4bad0f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4bad0f2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba80b7940 con 0x7f4ba80b82d0 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4bad0f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba8161b70 con 0x7f4ba80d2200 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4bad0f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba80b2af0 con 0x7f4ba8161760 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4ba6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba80b82d0 0x7f4ba8156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4ba6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba80b82d0 0x7f4ba8156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59106/0 (socket says 172.21.15.67:59106) 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4ba6d9d640 1 -- 172.21.15.67:0/668868449 learned_addr learned my addr 172.21.15.67:0/668868449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4ba659c640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba80d2200 0x7f4ba815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.731+0000 7f4ba759e640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4ba6d9d640 1 -- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 msgr2=0x7f4ba815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4ba6d9d640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba815bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4ba6d9d640 1 -- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba80d2200 msgr2=0x7f4ba815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4ba6d9d640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba80d2200 0x7f4ba815d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4ba6d9d640 1 -- 172.21.15.67:0/668868449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba8157250 con 0x7f4ba80b82d0 2024-04-25T15:25:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4ba659c640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba80d2200 0x7f4ba815d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4ba6d9d640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba80b82d0 0x7f4ba8156d10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4b980098f0 tx=0x7f4b980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4b8f7fe640 1 -- 172.21.15.67:0/668868449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98041020 con 0x7f4ba80b82d0 2024-04-25T15:25:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4b8f7fe640 1 -- 172.21.15.67:0/668868449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b9803d6e0 con 0x7f4ba80b82d0 2024-04-25T15:25:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba81589a0 con 0x7f4ba80b82d0 2024-04-25T15:25:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4b8f7fe640 1 -- 172.21.15.67:0/668868449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b98037040 con 0x7f4ba80b82d0 2024-04-25T15:25:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.732+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba8160c20 con 0x7f4ba80b82d0 2024-04-25T15:25:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.733+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba8050740 con 0x7f4ba80b82d0 2024-04-25T15:25:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.734+0000 7f4b8f7fe640 1 -- 172.21.15.67:0/668868449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4b98044020 con 0x7f4ba80b82d0 2024-04-25T15:25:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.734+0000 7f4b8f7fe640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b940744e0 0x7f4b940769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.735+0000 7f4b8f7fe640 1 -- 172.21.15.67:0/668868449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4b980b5750 con 0x7f4ba80b82d0 2024-04-25T15:25:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.735+0000 7f4ba659c640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b940744e0 0x7f4b940769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:49.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.735+0000 7f4ba659c640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b940744e0 0x7f4b940769f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80cfa40 tx=0x7f4b9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.739+0000 7f4b8f7fe640 1 -- 172.21.15.67:0/668868449 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b9804b030 con 0x7f4ba80b82d0 2024-04-25T15:25:49.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.963+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4ba8160e60 con 0x7f4ba80b82d0 2024-04-25T15:25:49.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.963+0000 7f4b8f7fe640 1 -- 172.21.15.67:0/668868449 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f4b9807f930 con 0x7f4ba80b82d0 2024-04-25T15:25:49.965 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:25:49.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b940744e0 msgr2=0x7f4b940769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:49.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b940744e0 0x7f4b940769f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80cfa40 tx=0x7f4b9c009040 comp rx=0 tx=0).stop 2024-04-25T15:25:49.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba80b82d0 msgr2=0x7f4ba8156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba80b82d0 0x7f4ba8156d10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4b980098f0 tx=0x7f4b980046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 shutdown_connections 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b940744e0 0x7f4b940769f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba8161760 0x7f4ba815bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba80d2200 0x7f4ba815d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 --2- 172.21.15.67:0/668868449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba80b82d0 0x7f4ba8156d10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 >> 172.21.15.67:0/668868449 conn(0x7f4ba809b7d0 msgr2=0x7f4ba806c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 shutdown_connections 2024-04-25T15:25:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:49.965+0000 7f4bad0f2640 1 -- 172.21.15.67:0/668868449 wait complete. 2024-04-25T15:25:49.978 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:25:49.978 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:25:49.978 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:25:49.978 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T15:25:49.979 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T15:25:49.979 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:25:49.979 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:25:49.979 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:25:49.979 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:25:49.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:49.979 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:25:49.995 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:49.995 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:49.995 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:49.995 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:49.995 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:49.995 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:49.995 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42739sec preferred_lft 42739sec 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:49.996 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link tentative 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:49.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:49.997 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:25:50.058 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:25:50.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:50.060 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:25:50.116 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:25:50.116 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:25:50.116 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:25:50.116 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:25:50.116 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:25:50.116 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:50.117 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:50.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:25:50.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:25:50.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:25:50.117 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:50.203 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:50.233 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:50.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:50.239 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:25:50.253 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:25:50.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:50.254 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list-id 2024-04-25T15:25:50.308 INFO:teuthology.orchestra.run.smithi176.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-25T15:25:50.308 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:50.309 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:50.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:25:50.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-25T15:25:50.309 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:50.397 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:50.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:50.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:50.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T15:25:50.447 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 1a:be:95:26:6b:90 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::18be:95ff:fe26:6b90/64 scope link 2024-04-25T15:25:50.448 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:50.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T15:25:50.512 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:25:50.512 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 1a:be:95:26:6b:90 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::18be:95ff:fe26:6b90/64 scope link 2024-04-25T15:25:50.513 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:25:50.514 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-25T15:25:50.514 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:50.514 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:50.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-25T15:25:50.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T15:25:50.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-25T15:25:50.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-25T15:25:50.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-25T15:25:50.515 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:50.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:50.658 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:50.664 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:25:50.664 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:25:50.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.1 up 2024-04-25T15:25:50.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T15:25:50.664 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:25:50.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:50.732 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:50.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:50.739 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T15:25:50.753 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T15:25:50.753 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:25:50.754 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T15:25:50.808 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-04-25T15:25:50.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:50.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:50.899 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:50.899 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:50.908 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:25:50.908 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:25:50.908 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:25:50.908 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 3@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:25:50.908 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:25:50.908 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-25T15:25:50.909 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:25:50.909 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:25:50.909 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:25:50.909 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:25:50.910 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:25:50.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:25:50.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-25T15:25:50.934 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:51.021 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:51.023 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:51.029 INFO:teuthology.orchestra.run.smithi176.stdout:client4680 2024-04-25T15:25:51.029 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:25:51.029 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:25:51.044 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:51.132 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:51.134 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:51.140 INFO:teuthology.orchestra.run.smithi176.stdout:client4680 2024-04-25T15:25:51.140 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:25:51.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:25:51.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:51.195 INFO:teuthology.orchestra.run.smithi176.stdout:client4680 2024-04-25T15:25:51.195 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/b43d0acb-252a-4f7a-bf14-34d789b94b95.client4680/status') 2024-04-25T15:25:51.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:25:51.252 INFO:teuthology.orchestra.run.smithi176.stdout:instance: client.4680 (3)172.21.15.176:0/1047005840 2024-04-25T15:25:51.252 INFO:teuthology.orchestra.run.smithi176.stdout:blocklisted: false 2024-04-25T15:25:51.252 INFO:teuthology.orchestra.run.smithi176.stderr:0+1 records in 2024-04-25T15:25:51.252 INFO:teuthology.orchestra.run.smithi176.stderr:0+1 records out 2024-04-25T15:25:51.253 INFO:teuthology.orchestra.run.smithi176.stderr:71 bytes copied, 2.4311e-05 s, 2.9 MB/s 2024-04-25T15:25:51.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:25:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:25:51.259 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-25T15:25:51.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:51.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.401+0000 7f195cba8640 1 Processor -- start 2024-04-25T15:25:51.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.401+0000 7f195cba8640 1 -- start start 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.402+0000 7f195cba8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19580d0de0 0x7f19580d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.402+0000 7f195cba8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f195806c5c0 0x7f195806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.402+0000 7f195cba8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f195806cf10 con 0x7f19580b77f0 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.402+0000 7f195cba8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f19580d5a40 con 0x7f195806c5c0 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.402+0000 7f195cba8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f19580d1830 con 0x7f19580d0de0 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f195659c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19580d0de0 0x7f19580d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f1956d9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f19580b77f0 0x7f19580b7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:59190/0 (socket says 172.21.15.67:59190) 2024-04-25T15:25:51.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f1956d9d640 1 -- 172.21.15.67:0/4061249139 learned_addr learned my addr 172.21.15.67:0/4061249139 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f194ffff640 1 --2- 172.21.15.67:0/4061249139 >> v2:172.21.15.176:3300/0 conn(0x7f195806c5c0 0x7f195806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f1955d9b640 1 -- 172.21.15.67:0/4061249139 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214697372 0 0) 0x7f1950001490 con 0x7f19580b77f0 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f1955d9b640 1 -- 172.21.15.67:0/4061249139 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1944003520 con 0x7f19580b77f0 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f195659c640 1 -- 172.21.15.67:0/4061249139 >> v2:172.21.15.176:3300/0 conn(0x7f195806c5c0 msgr2=0x7f195806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f195659c640 1 --2- 172.21.15.67:0/4061249139 >> v2:172.21.15.176:3300/0 conn(0x7f195806c5c0 0x7f195806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f195659c640 1 -- 172.21.15.67:0/4061249139 >> v1:172.21.15.176:6789/0 conn(0x7f19580b77f0 legacy=0x7f19580b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f195659c640 1 -- 172.21.15.67:0/4061249139 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1958160dc0 con 0x7f19580d0de0 2024-04-25T15:25:51.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.403+0000 7f194ffff640 1 --2- 172.21.15.67:0/4061249139 >> v2:172.21.15.176:3300/0 conn(0x7f195806c5c0 0x7f195806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:51.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.404+0000 7f195659c640 1 --2- 172.21.15.67:0/4061249139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19580d0de0 0x7f19580d11f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f19480097a0 tx=0x7f194802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=189b114ff6af4c04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:51.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.404+0000 7f1955d9b640 1 -- 172.21.15.67:0/4061249139 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948037070 con 0x7f19580d0de0 2024-04-25T15:25:51.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.404+0000 7f1955d9b640 1 -- 172.21.15.67:0/4061249139 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1948032440 con 0x7f19580d0de0 2024-04-25T15:25:51.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.404+0000 7f1955d9b640 1 -- 172.21.15.67:0/4061249139 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948032810 con 0x7f19580d0de0 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 -- 172.21.15.67:0/4061249139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19580d0de0 msgr2=0x7f19580d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 --2- 172.21.15.67:0/4061249139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19580d0de0 0x7f19580d11f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f19480097a0 tx=0x7f194802a180 comp rx=0 tx=0).stop 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 -- 172.21.15.67:0/4061249139 shutdown_connections 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 --2- 172.21.15.67:0/4061249139 >> v2:172.21.15.176:3300/0 conn(0x7f195806c5c0 0x7f195806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 --2- 172.21.15.67:0/4061249139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19580d0de0 0x7f19580d11f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 -- 172.21.15.67:0/4061249139 >> 172.21.15.67:0/4061249139 conn(0x7f19580d5d70 msgr2=0x7f19580d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 -- 172.21.15.67:0/4061249139 shutdown_connections 2024-04-25T15:25:51.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.405+0000 7f195cba8640 1 -- 172.21.15.67:0/4061249139 wait complete. 2024-04-25T15:25:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.407+0000 7f195cba8640 1 Processor -- start 2024-04-25T15:25:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.407+0000 7f195cba8640 1 -- start start 2024-04-25T15:25:51.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f195cba8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f195806c5c0 0x7f19580cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f195cba8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19580b77f0 0x7f195815f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f195cba8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19580d0de0 0x7f19580ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f195cba8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1958160ab0 con 0x7f19580b77f0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f195cba8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1958161380 con 0x7f19580d0de0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f195cba8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f19580b5300 con 0x7f195806c5c0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f194ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19580b77f0 0x7f195815f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f194ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19580b77f0 0x7f195815f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59108/0 (socket says 172.21.15.67:59108) 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f194ffff640 1 -- 172.21.15.67:0/2503912241 learned_addr learned my addr 172.21.15.67:0/2503912241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f195659c640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f195806c5c0 0x7f19580cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.408+0000 7f1956d9d640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19580d0de0 0x7f19580ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194ffff640 1 -- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f195806c5c0 msgr2=0x7f19580cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194ffff640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f195806c5c0 0x7f19580cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194ffff640 1 -- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19580d0de0 msgr2=0x7f19580ac680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194ffff640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19580d0de0 0x7f19580ac680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194ffff640 1 -- 172.21.15.67:0/2503912241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19580cff10 con 0x7f19580b77f0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194ffff640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19580b77f0 0x7f195815f6f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f194000e990 tx=0x7f194000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:51.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194effd640 1 -- 172.21.15.67:0/2503912241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940017070 con 0x7f19580b77f0 2024-04-25T15:25:51.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194effd640 1 -- 172.21.15.67:0/2503912241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f194000c980 con 0x7f19580b77f0 2024-04-25T15:25:51.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195815fc30 con 0x7f19580b77f0 2024-04-25T15:25:51.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f194effd640 1 -- 172.21.15.67:0/2503912241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194000cd50 con 0x7f19580b77f0 2024-04-25T15:25:51.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.409+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19580ccf50 con 0x7f19580b77f0 2024-04-25T15:25:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.410+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1958050740 con 0x7f19580b77f0 2024-04-25T15:25:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.411+0000 7f194effd640 1 -- 172.21.15.67:0/2503912241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f194001a070 con 0x7f19580b77f0 2024-04-25T15:25:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.411+0000 7f194effd640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19380744e0 0x7f19380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.412+0000 7f194effd640 1 -- 172.21.15.67:0/2503912241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1940095020 con 0x7f19580b77f0 2024-04-25T15:25:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.412+0000 7f195659c640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19380744e0 0x7f19380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.412+0000 7f195659c640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19380744e0 0x7f19380769f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1948034000 tx=0x7f194802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:51.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.416+0000 7f194effd640 1 -- 172.21.15.67:0/2503912241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f194005f160 con 0x7f19580b77f0 2024-04-25T15:25:51.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.603+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19580d4300 con 0x7f19580b77f0 2024-04-25T15:25:51.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.604+0000 7f194effd640 1 -- 172.21.15.67:0/2503912241 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f1940063460 con 0x7f19580b77f0 2024-04-25T15:25:51.607 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:51.607 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19380744e0 msgr2=0x7f19380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19380744e0 0x7f19380769f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1948034000 tx=0x7f194802e000 comp rx=0 tx=0).stop 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19580b77f0 msgr2=0x7f195815f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19580b77f0 0x7f195815f6f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f194000e990 tx=0x7f194000ee50 comp rx=0 tx=0).stop 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 shutdown_connections 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19380744e0 0x7f19380769f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19580d0de0 0x7f19580ac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19580b77f0 0x7f195815f6f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 --2- 172.21.15.67:0/2503912241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f195806c5c0 0x7f19580cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 >> 172.21.15.67:0/2503912241 conn(0x7f19580d5d70 msgr2=0x7f19580b7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 shutdown_connections 2024-04-25T15:25:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.607+0000 7f195cba8640 1 -- 172.21.15.67:0/2503912241 wait complete. 2024-04-25T15:25:51.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:51.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:51.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.733+0000 7fab6f8cd640 1 Processor -- start 2024-04-25T15:25:51.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.733+0000 7fab6f8cd640 1 -- start start 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.734+0000 7fab6f8cd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab680b8290 0x7fab68160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.734+0000 7fab6f8cd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab681615f0 con 0x7fab68164b00 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.734+0000 7fab6f8cd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab68164020 con 0x7fab6815df80 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.734+0000 7fab6f8cd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fab68164670 con 0x7fab680b8290 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.735+0000 7fab6d669640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fab68164b00 0x7fab680b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59058/0 (socket says 172.21.15.67:59058) 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.735+0000 7fab6d669640 1 -- 172.21.15.67:0/3387093028 learned_addr learned my addr 172.21.15.67:0/3387093028 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.735+0000 7fab6ce68640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab680b8290 0x7fab68160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.735+0000 7fab6ce68640 1 -- 172.21.15.67:0/3387093028 >> v1:172.21.15.176:6789/0 conn(0x7fab6815df80 legacy=0x7fab68160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.735+0000 7fab6ce68640 1 -- 172.21.15.67:0/3387093028 >> v1:172.21.15.67:6789/0 conn(0x7fab68164b00 legacy=0x7fab680b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.735+0000 7fab5ffff640 1 -- 172.21.15.67:0/3387093028 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361274640 0 0) 0x7fab681615f0 con 0x7fab68164b00 2024-04-25T15:25:51.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.735+0000 7fab6ce68640 1 -- 172.21.15.67:0/3387093028 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab68164300 con 0x7fab680b8290 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.736+0000 7fab6ce68640 1 --2- 172.21.15.67:0/3387093028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab680b8290 0x7fab68160890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fab681647f0 tx=0x7fab5802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9cd06423f819f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.736+0000 7fab5ffff640 1 -- 172.21.15.67:0/3387093028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab58037070 con 0x7fab680b8290 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.736+0000 7fab5ffff640 1 -- 172.21.15.67:0/3387093028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab58032440 con 0x7fab680b8290 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.736+0000 7fab5ffff640 1 -- 172.21.15.67:0/3387093028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab58032810 con 0x7fab680b8290 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.736+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3387093028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab680b8290 msgr2=0x7fab68160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.736+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3387093028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab680b8290 0x7fab68160890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fab681647f0 tx=0x7fab5802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.737+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3387093028 shutdown_connections 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.737+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3387093028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab680b8290 0x7fab68160890 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.737+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3387093028 >> 172.21.15.67:0/3387093028 conn(0x7fab680d2bc0 msgr2=0x7fab680d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.737+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3387093028 shutdown_connections 2024-04-25T15:25:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.737+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3387093028 wait complete. 2024-04-25T15:25:51.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.738+0000 7fab6f8cd640 1 Processor -- start 2024-04-25T15:25:51.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.738+0000 7fab6f8cd640 1 -- start start 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6f8cd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab6815df80 0x7fab68163120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6f8cd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab68164b00 0x7fab68157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6f8cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6814b310 0x7fab681580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6f8cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab680b4c90 con 0x7fab6814b310 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6f8cd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fab680b4b10 con 0x7fab6815df80 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6f8cd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fab680b4e10 con 0x7fab68164b00 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6814b310 0x7fab681580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6814b310 0x7fab681580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59140/0 (socket says 172.21.15.67:59140) 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 -- 172.21.15.67:0/3086478279 learned_addr learned my addr 172.21.15.67:0/3086478279 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6ce68640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab68164b00 0x7fab68157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6d669640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab6815df80 0x7fab68163120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 -- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab68164b00 msgr2=0x7fab68157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab68164b00 0x7fab68157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 -- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab6815df80 msgr2=0x7fab68163120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab6815df80 0x7fab68163120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 -- 172.21.15.67:0/3086478279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab68156190 con 0x7fab6814b310 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6de6a640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6814b310 0x7fab681580d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab6000ba10 tx=0x7fab6000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab5dffb640 1 -- 172.21.15.67:0/3086478279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60019070 con 0x7fab6814b310 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab5dffb640 1 -- 172.21.15.67:0/3086478279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab6000cb40 con 0x7fab6814b310 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab5dffb640 1 -- 172.21.15.67:0/3086478279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60014440 con 0x7fab6814b310 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.740+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab68156470 con 0x7fab6814b310 2024-04-25T15:25:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.741+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab68156a30 con 0x7fab6814b310 2024-04-25T15:25:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.742+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab68050740 con 0x7fab6814b310 2024-04-25T15:25:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.742+0000 7fab5dffb640 1 -- 172.21.15.67:0/3086478279 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fab6001c070 con 0x7fab6814b310 2024-04-25T15:25:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.743+0000 7fab5dffb640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab4c0744b0 0x7fab4c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.743+0000 7fab5dffb640 1 -- 172.21.15.67:0/3086478279 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fab60094d40 con 0x7fab6814b310 2024-04-25T15:25:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.743+0000 7fab6d669640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab4c0744b0 0x7fab4c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.744+0000 7fab6d669640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab4c0744b0 0x7fab4c0769c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab64007fd0 tx=0x7fab64007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.748+0000 7fab5dffb640 1 -- 172.21.15.67:0/3086478279 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab6005ee80 con 0x7fab6814b310 2024-04-25T15:25:51.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.935+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fab680d03b0 con 0x7fab6814b310 2024-04-25T15:25:51.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.936+0000 7fab5dffb640 1 -- 172.21.15.67:0/3086478279 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fab60063180 con 0x7fab6814b310 2024-04-25T15:25:51.938 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab4c0744b0 msgr2=0x7fab4c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab4c0744b0 0x7fab4c0769c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fab64007fd0 tx=0x7fab64007de0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6814b310 msgr2=0x7fab681580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6814b310 0x7fab681580d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab6000ba10 tx=0x7fab6000bed0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 shutdown_connections 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab4c0744b0 0x7fab4c0769c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab6814b310 0x7fab681580d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab68164b00 0x7fab68157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 --2- 172.21.15.67:0/3086478279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab6815df80 0x7fab68163120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:51.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 >> 172.21.15.67:0/3086478279 conn(0x7fab680d2bc0 msgr2=0x7fab6815e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:51.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 shutdown_connections 2024-04-25T15:25:51.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:51.938+0000 7fab6f8cd640 1 -- 172.21.15.67:0/3086478279 wait complete. 2024-04-25T15:25:51.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:52.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.064+0000 7f4089c34640 1 Processor -- start 2024-04-25T15:25:52.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.064+0000 7f4089c34640 1 -- start start 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f4089c34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4084164b00 0x7f40840b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f4089c34640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f408415df80 0x7f40840b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f4089c34640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40841615f0 con 0x7f4084160890 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f4089c34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f40841613b0 con 0x7f4084164b00 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f4089c34640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4084164670 con 0x7f408415df80 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f40837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4084164b00 0x7f40840b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f40837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4084164b00 0x7f40840b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59144/0 (socket says 172.21.15.67:59144) 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f40837fe640 1 -- 172.21.15.67:0/3822546542 learned_addr learned my addr 172.21.15.67:0/3822546542 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f4082ffd640 1 --2- 172.21.15.67:0/3822546542 >> v2:172.21.15.176:3300/0 conn(0x7f408415df80 0x7f40840b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f40837fe640 1 -- 172.21.15.67:0/3822546542 >> v2:172.21.15.176:3300/0 conn(0x7f408415df80 msgr2=0x7f40840b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:52.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f40837fe640 1 --2- 172.21.15.67:0/3822546542 >> v2:172.21.15.176:3300/0 conn(0x7f408415df80 0x7f40840b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f40837fe640 1 -- 172.21.15.67:0/3822546542 >> v1:172.21.15.176:6789/0 conn(0x7f4084160890 legacy=0x7f4084160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f40837fe640 1 -- 172.21.15.67:0/3822546542 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4084164020 con 0x7f4084164b00 2024-04-25T15:25:52.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.066+0000 7f4082ffd640 1 --2- 172.21.15.67:0/3822546542 >> v2:172.21.15.176:3300/0 conn(0x7f408415df80 0x7f40840b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.067+0000 7f40837fe640 1 --2- 172.21.15.67:0/3822546542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40840b7b80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f40840d2540 tx=0x7f407402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3754cc366330048 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.067+0000 7f40827fc640 1 -- 172.21.15.67:0/3822546542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074037070 con 0x7f4084164b00 2024-04-25T15:25:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.067+0000 7f40827fc640 1 -- 172.21.15.67:0/3822546542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4074032440 con 0x7f4084164b00 2024-04-25T15:25:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.067+0000 7f40827fc640 1 -- 172.21.15.67:0/3822546542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074032810 con 0x7f4084164b00 2024-04-25T15:25:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 -- 172.21.15.67:0/3822546542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 msgr2=0x7f40840b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 --2- 172.21.15.67:0/3822546542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40840b7b80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f40840d2540 tx=0x7f407402a230 comp rx=0 tx=0).stop 2024-04-25T15:25:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 -- 172.21.15.67:0/3822546542 shutdown_connections 2024-04-25T15:25:52.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 --2- 172.21.15.67:0/3822546542 >> v2:172.21.15.176:3300/0 conn(0x7f408415df80 0x7f40840b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 --2- 172.21.15.67:0/3822546542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40840b7b80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 -- 172.21.15.67:0/3822546542 >> 172.21.15.67:0/3822546542 conn(0x7f40840d2bc0 msgr2=0x7f40840d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:52.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 -- 172.21.15.67:0/3822546542 shutdown_connections 2024-04-25T15:25:52.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.068+0000 7f4089c34640 1 -- 172.21.15.67:0/3822546542 wait complete. 2024-04-25T15:25:52.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.069+0000 7f4089c34640 1 Processor -- start 2024-04-25T15:25:52.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.070+0000 7f4089c34640 1 -- start start 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4089c34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408415df80 0x7f40840c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4089c34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4084160890 0x7f4084153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4089c34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40841588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4089c34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40840b4a00 con 0x7f4084164b00 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4089c34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f40840b4880 con 0x7f408415df80 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4089c34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f40840b4b80 con 0x7f4084160890 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40841588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40841588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59150/0 (socket says 172.21.15.67:59150) 2024-04-25T15:25:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 -- 172.21.15.67:0/824064559 learned_addr learned my addr 172.21.15.67:0/824064559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f40837fe640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408415df80 0x7f40840c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4082ffd640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4084160890 0x7f4084153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 -- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4084160890 msgr2=0x7f4084153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4084160890 0x7f4084153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 -- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408415df80 msgr2=0x7f40840c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408415df80 0x7f40840c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f4083fff640 1 -- 172.21.15.67:0/824064559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40840b5010 con 0x7f4084164b00 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.071+0000 7f40837fe640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408415df80 0x7f40840c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.072+0000 7f4083fff640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40841588a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f407c00ba10 tx=0x7f407c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.072+0000 7f406bfff640 1 -- 172.21.15.67:0/824064559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c019070 con 0x7f4084164b00 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.072+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40840b52f0 con 0x7f4084164b00 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.072+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40840b58b0 con 0x7f4084164b00 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.072+0000 7f406bfff640 1 -- 172.21.15.67:0/824064559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407c00cb40 con 0x7f4084164b00 2024-04-25T15:25:52.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.072+0000 7f406bfff640 1 -- 172.21.15.67:0/824064559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c023060 con 0x7f4084164b00 2024-04-25T15:25:52.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.073+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4064005290 con 0x7f4084164b00 2024-04-25T15:25:52.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.073+0000 7f406bfff640 1 -- 172.21.15.67:0/824064559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f407c01c070 con 0x7f4084164b00 2024-04-25T15:25:52.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.074+0000 7f406bfff640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40700744b0 0x7f40700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.074+0000 7f406bfff640 1 -- 172.21.15.67:0/824064559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f407c094d70 con 0x7f4084164b00 2024-04-25T15:25:52.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.074+0000 7f40837fe640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40700744b0 0x7f40700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.075+0000 7f40837fe640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40700744b0 0x7f40700769c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4074034000 tx=0x7f407402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.079+0000 7f406bfff640 1 -- 172.21.15.67:0/824064559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f407c05ef40 con 0x7f4084164b00 2024-04-25T15:25:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.264+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4064005850 con 0x7f4084164b00 2024-04-25T15:25:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.264+0000 7f406bfff640 1 -- 172.21.15.67:0/824064559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f407c063240 con 0x7f4084164b00 2024-04-25T15:25:52.267 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40700744b0 msgr2=0x7f40700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40700744b0 0x7f40700769c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4074034000 tx=0x7f407402e000 comp rx=0 tx=0).stop 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 msgr2=0x7f40841588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40841588a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f407c00ba10 tx=0x7f407c00bed0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 shutdown_connections 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40700744b0 0x7f40700769c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4084164b00 0x7f40841588a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4084160890 0x7f4084153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 --2- 172.21.15.67:0/824064559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408415df80 0x7f40840c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 >> 172.21.15.67:0/824064559 conn(0x7f40840d2bc0 msgr2=0x7f40840d0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:52.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 shutdown_connections 2024-04-25T15:25:52.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.267+0000 7f4089c34640 1 -- 172.21.15.67:0/824064559 wait complete. 2024-04-25T15:25:52.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:25:52.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.394+0000 7fbd6d8e7640 1 Processor -- start 2024-04-25T15:25:52.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.394+0000 7fbd6d8e7640 1 -- start start 2024-04-25T15:25:52.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd6d8e7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd680b2e10 0x7fbd680ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd6d8e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680d23f0 0x7fbd680d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd6d8e7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbd680b5520 0x7fbd680b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd6d8e7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbd680b9620 con 0x7fbd680b2e10 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd6d8e7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbd680ceca0 con 0x7fbd680b5520 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd6d8e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbd680ceed0 con 0x7fbd680d23f0 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd680b2e10 0x7fbd680ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd680b2e10 0x7fbd680ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59154/0 (socket says 172.21.15.67:59154) 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 -- 172.21.15.67:0/2556056824 learned_addr learned my addr 172.21.15.67:0/2556056824 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd677fe640 1 --2- 172.21.15.67:0/2556056824 >> v2:172.21.15.176:3300/0 conn(0x7fbd680b5520 0x7fbd680b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 -- 172.21.15.67:0/2556056824 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680d23f0 msgr2=0x7fbd680d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd667fc640 1 --2- 172.21.15.67:0/2556056824 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680d23f0 0x7fbd680d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 --2- 172.21.15.67:0/2556056824 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680d23f0 0x7fbd680d2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 -- 172.21.15.67:0/2556056824 >> v2:172.21.15.176:3300/0 conn(0x7fbd680b5520 msgr2=0x7fbd680b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 --2- 172.21.15.67:0/2556056824 >> v2:172.21.15.176:3300/0 conn(0x7fbd680b5520 0x7fbd680b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 -- 172.21.15.67:0/2556056824 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd680cf1b0 con 0x7fbd680b2e10 2024-04-25T15:25:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd677fe640 1 --2- 172.21.15.67:0/2556056824 >> v2:172.21.15.176:3300/0 conn(0x7fbd680b5520 0x7fbd680b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:52.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd66ffd640 1 --2- 172.21.15.67:0/2556056824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680b2e10 0x7fbd680ce760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd68069550 tx=0x7fbd6002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5696fa0e499b131c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd65ffb640 1 -- 172.21.15.67:0/2556056824 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60036070 con 0x7fbd680b2e10 2024-04-25T15:25:52.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd65ffb640 1 -- 172.21.15.67:0/2556056824 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd60031600 con 0x7fbd680b2e10 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.396+0000 7fbd65ffb640 1 -- 172.21.15.67:0/2556056824 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd600319d0 con 0x7fbd680b2e10 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/2556056824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680b2e10 msgr2=0x7fbd680ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/2556056824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680b2e10 0x7fbd680ce760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd68069550 tx=0x7fbd6002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/2556056824 shutdown_connections 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/2556056824 >> v2:172.21.15.176:3300/0 conn(0x7fbd680b5520 0x7fbd680b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/2556056824 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680d23f0 0x7fbd680d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/2556056824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680b2e10 0x7fbd680ce760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/2556056824 >> 172.21.15.67:0/2556056824 conn(0x7fbd680d3aa0 msgr2=0x7fbd680d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/2556056824 shutdown_connections 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.397+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/2556056824 wait complete. 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.399+0000 7fbd6d8e7640 1 Processor -- start 2024-04-25T15:25:52.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.399+0000 7fbd6d8e7640 1 -- start start 2024-04-25T15:25:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd6d8e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd680b2e10 0x7fbd6809d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd6d8e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680b5520 0x7fbd681f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd6d8e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680d23f0 0x7fbd681f8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd6d8e7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd680cfa50 con 0x7fbd680d23f0 2024-04-25T15:25:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd6d8e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbd680cf770 con 0x7fbd680b2e10 2024-04-25T15:25:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd6d8e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbd680cff70 con 0x7fbd680b5520 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680d23f0 0x7fbd681f8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680d23f0 0x7fbd681f8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59168/0 (socket says 172.21.15.67:59168) 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.400+0000 7fbd677fe640 1 -- 172.21.15.67:0/1151314725 learned_addr learned my addr 172.21.15.67:0/1151314725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd66ffd640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd680b2e10 0x7fbd6809d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd667fc640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680b5520 0x7fbd681f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd677fe640 1 -- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680b5520 msgr2=0x7fbd681f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd677fe640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680b5520 0x7fbd681f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd677fe640 1 -- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd680b2e10 msgr2=0x7fbd6809d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd677fe640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd680b2e10 0x7fbd6809d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd677fe640 1 -- 172.21.15.67:0/1151314725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd681607e0 con 0x7fbd680d23f0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd677fe640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680d23f0 0x7fbd681f8220 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800afd0 tx=0x7fbd5800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd4ffff640 1 -- 172.21.15.67:0/1151314725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58015020 con 0x7fbd680d23f0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd4ffff640 1 -- 172.21.15.67:0/1151314725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd58004440 con 0x7fbd680d23f0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd68160ac0 con 0x7fbd680d23f0 2024-04-25T15:25:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd4ffff640 1 -- 172.21.15.67:0/1151314725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58003850 con 0x7fbd680d23f0 2024-04-25T15:25:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.401+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd68161080 con 0x7fbd680d23f0 2024-04-25T15:25:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.402+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd680b3bf0 con 0x7fbd680d23f0 2024-04-25T15:25:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.403+0000 7fbd4ffff640 1 -- 172.21.15.67:0/1151314725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbd58018020 con 0x7fbd680d23f0 2024-04-25T15:25:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.404+0000 7fbd4ffff640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbd540744e0 0x7fbd540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.404+0000 7fbd4ffff640 1 -- 172.21.15.67:0/1151314725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbd58095070 con 0x7fbd680d23f0 2024-04-25T15:25:52.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.404+0000 7fbd66ffd640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbd540744e0 0x7fbd540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.405+0000 7fbd66ffd640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbd540744e0 0x7fbd540769f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbd60033000 tx=0x7fbd60009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.408+0000 7fbd4ffff640 1 -- 172.21.15.67:0/1151314725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd5801f020 con 0x7fbd680d23f0 2024-04-25T15:25:52.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.595+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbd680d4700 con 0x7fbd680d23f0 2024-04-25T15:25:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.595+0000 7fbd4ffff640 1 -- 172.21.15.67:0/1151314725 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fbd5805f1c0 con 0x7fbd680d23f0 2024-04-25T15:25:52.597 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:25:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbd540744e0 msgr2=0x7fbd540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbd540744e0 0x7fbd540769f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbd60033000 tx=0x7fbd60009000 comp rx=0 tx=0).stop 2024-04-25T15:25:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680d23f0 msgr2=0x7fbd681f8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680d23f0 0x7fbd681f8220 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800afd0 tx=0x7fbd5800ca20 comp rx=0 tx=0).stop 2024-04-25T15:25:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 shutdown_connections 2024-04-25T15:25:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbd540744e0 0x7fbd540769f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd680d23f0 0x7fbd681f8220 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbd680b5520 0x7fbd681f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 --2- 172.21.15.67:0/1151314725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd680b2e10 0x7fbd6809d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 >> 172.21.15.67:0/1151314725 conn(0x7fbd680d3aa0 msgr2=0x7fbd6815e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 shutdown_connections 2024-04-25T15:25:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.597+0000 7fbd6d8e7640 1 -- 172.21.15.67:0/1151314725 wait complete. 2024-04-25T15:25:52.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.721+0000 7f2701ed8640 1 Processor -- start 2024-04-25T15:25:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.721+0000 7f2701ed8640 1 -- start start 2024-04-25T15:25:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f2701ed8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d1210 0x7f26fc0d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f2701ed8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26fc0d1850 con 0x7f26fc0d0e00 2024-04-25T15:25:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f2701ed8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26fc160de0 con 0x7f26fc06ccd0 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f2701ed8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc0d1620 con 0x7f26fc0d1210 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f26fc0d0e00 0x7f26fc06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59070/0 (socket says 172.21.15.67:59070) 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fb7fe640 1 -- 172.21.15.67:0/3476294165 learned_addr learned my addr 172.21.15.67:0/3476294165 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fbfff640 1 --2- 172.21.15.67:0/3476294165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d1210 0x7f26fc0d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fa7fc640 1 -- 172.21.15.67:0/3476294165 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676237088 0 0) 0x7f26fc0d1850 con 0x7f26fc0d0e00 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fa7fc640 1 -- 172.21.15.67:0/3476294165 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e4003520 con 0x7f26fc0d0e00 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fa7fc640 1 -- 172.21.15.67:0/3476294165 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143070905 0 0) 0x7f26fc160de0 con 0x7f26fc06ccd0 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fbfff640 1 -- 172.21.15.67:0/3476294165 >> v1:172.21.15.176:6789/0 conn(0x7f26fc06ccd0 legacy=0x7f26fc0d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fbfff640 1 -- 172.21.15.67:0/3476294165 >> v1:172.21.15.67:6789/0 conn(0x7f26fc0d0e00 legacy=0x7f26fc06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fbfff640 1 -- 172.21.15.67:0/3476294165 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc1610c0 con 0x7f26fc0d1210 2024-04-25T15:25:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.722+0000 7f26fbfff640 1 --2- 172.21.15.67:0/3476294165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d1210 0x7f26fc0d5160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26f40097d0 tx=0x7f26f402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c4a22069cd8d04a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f26fa7fc640 1 -- 172.21.15.67:0/3476294165 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4037070 con 0x7f26fc0d1210 2024-04-25T15:25:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f26fa7fc640 1 -- 172.21.15.67:0/3476294165 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f4032440 con 0x7f26fc0d1210 2024-04-25T15:25:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f26fa7fc640 1 -- 172.21.15.67:0/3476294165 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4032810 con 0x7f26fc0d1210 2024-04-25T15:25:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f2701ed8640 1 -- 172.21.15.67:0/3476294165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d1210 msgr2=0x7f26fc0d5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f2701ed8640 1 --2- 172.21.15.67:0/3476294165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d1210 0x7f26fc0d5160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26f40097d0 tx=0x7f26f402a210 comp rx=0 tx=0).stop 2024-04-25T15:25:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f2701ed8640 1 -- 172.21.15.67:0/3476294165 shutdown_connections 2024-04-25T15:25:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f2701ed8640 1 --2- 172.21.15.67:0/3476294165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d1210 0x7f26fc0d5160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f2701ed8640 1 -- 172.21.15.67:0/3476294165 >> 172.21.15.67:0/3476294165 conn(0x7f26fc0d5d90 msgr2=0x7f26fc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f2701ed8640 1 -- 172.21.15.67:0/3476294165 shutdown_connections 2024-04-25T15:25:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.723+0000 7f2701ed8640 1 -- 172.21.15.67:0/3476294165 wait complete. 2024-04-25T15:25:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.725+0000 7f2701ed8640 1 Processor -- start 2024-04-25T15:25:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.725+0000 7f2701ed8640 1 -- start start 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f2701ed8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d0e00 0x7f26fc0cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f2701ed8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0d1210 0x7f26fc14def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f2701ed8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26fc14e630 0x7f26fc0b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f2701ed8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc0b5320 con 0x7f26fc14e630 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f2701ed8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc160ad0 con 0x7f26fc0d1210 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f2701ed8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc164e00 con 0x7f26fc0d0e00 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26fc14e630 0x7f26fc0b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26fc14e630 0x7f26fc0b80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59170/0 (socket says 172.21.15.67:59170) 2024-04-25T15:25:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 -- 172.21.15.67:0/548594397 learned_addr learned my addr 172.21.15.67:0/548594397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fb7fe640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d0e00 0x7f26fc0cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26faffd640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0d1210 0x7f26fc14def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 -- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d0e00 msgr2=0x7f26fc0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d0e00 0x7f26fc0cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 -- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0d1210 msgr2=0x7f26fc14def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0d1210 0x7f26fc14def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 -- 172.21.15.67:0/548594397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc0b69e0 con 0x7f26fc14e630 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.727+0000 7f26fbfff640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26fc14e630 0x7f26fc0b80d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009720 tx=0x7f26f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.728+0000 7f26e3fff640 1 -- 172.21.15.67:0/548594397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4037040 con 0x7f26fc14e630 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.728+0000 7f26e3fff640 1 -- 172.21.15.67:0/548594397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f4004230 con 0x7f26fc14e630 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.728+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26fc0b6cc0 con 0x7f26fc14e630 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.728+0000 7f26e3fff640 1 -- 172.21.15.67:0/548594397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4003610 con 0x7f26fc14e630 2024-04-25T15:25:52.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.728+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26fc0b6fa0 con 0x7f26fc14e630 2024-04-25T15:25:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.729+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26fc050740 con 0x7f26fc14e630 2024-04-25T15:25:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.729+0000 7f26e3fff640 1 -- 172.21.15.67:0/548594397 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f26f40037b0 con 0x7f26fc14e630 2024-04-25T15:25:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.730+0000 7f26e3fff640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f26e80744b0 0x7f26e80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.730+0000 7f26e3fff640 1 -- 172.21.15.67:0/548594397 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f26f40b5e80 con 0x7f26fc14e630 2024-04-25T15:25:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.730+0000 7f26fb7fe640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f26e80744b0 0x7f26e80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:52.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.731+0000 7f26fb7fe640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f26e80744b0 0x7f26e80769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f26ec005c00 tx=0x7f26ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.735+0000 7f26e3fff640 1 -- 172.21.15.67:0/548594397 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26f407ffd0 con 0x7f26fc14e630 2024-04-25T15:25:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.921+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26fc0cc5e0 con 0x7f26fc14e630 2024-04-25T15:25:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.923+0000 7f26e3fff640 1 -- 172.21.15.67:0/548594397 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f26f40842d0 con 0x7f26fc14e630 2024-04-25T15:25:52.924 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:52.924 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f26e80744b0 msgr2=0x7f26e80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f26e80744b0 0x7f26e80769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f26ec005c00 tx=0x7f26ec009040 comp rx=0 tx=0).stop 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26fc14e630 msgr2=0x7f26fc0b80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26fc14e630 0x7f26fc0b80d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009720 tx=0x7f26f40046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 shutdown_connections 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f26e80744b0 0x7f26e80769c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26fc14e630 0x7f26fc0b80d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0d1210 0x7f26fc14def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 --2- 172.21.15.67:0/548594397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26fc0d0e00 0x7f26fc0cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.924+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 >> 172.21.15.67:0/548594397 conn(0x7f26fc0d5d90 msgr2=0x7f26fc15fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.925+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 shutdown_connections 2024-04-25T15:25:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:52.925+0000 7f2701ed8640 1 -- 172.21.15.67:0/548594397 wait complete. 2024-04-25T15:25:52.937 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:52.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:53.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.051+0000 7f6743882640 1 Processor -- start 2024-04-25T15:25:53.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.051+0000 7f6743882640 1 -- start start 2024-04-25T15:25:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6743882640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f673c0d1210 0x7f673c0d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6743882640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f673c0d1850 con 0x7f673c0d0e00 2024-04-25T15:25:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6743882640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f673c160de0 con 0x7f673c06ccd0 2024-04-25T15:25:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6743882640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f673c0d1620 con 0x7f673c0d1210 2024-04-25T15:25:53.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f674161e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f673c0d0e00 0x7f673c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59086/0 (socket says 172.21.15.67:59086) 2024-04-25T15:25:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f674161e640 1 -- 172.21.15.67:0/2275429070 learned_addr learned my addr 172.21.15.67:0/2275429070 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6741e1f640 1 --2- 172.21.15.67:0/2275429070 >> v2:172.21.15.67:3300/0 conn(0x7f673c0d1210 0x7f673c0d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6741e1f640 1 -- 172.21.15.67:0/2275429070 >> v1:172.21.15.176:6789/0 conn(0x7f673c06ccd0 legacy=0x7f673c0d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6741e1f640 1 -- 172.21.15.67:0/2275429070 >> v1:172.21.15.67:6789/0 conn(0x7f673c0d0e00 legacy=0x7f673c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f673bfff640 1 -- 172.21.15.67:0/2275429070 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126023427 0 0) 0x7f673c0d1850 con 0x7f673c0d0e00 2024-04-25T15:25:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.053+0000 7f6741e1f640 1 -- 172.21.15.67:0/2275429070 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673c1610c0 con 0x7f673c0d1210 2024-04-25T15:25:53.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.054+0000 7f6741e1f640 1 --2- 172.21.15.67:0/2275429070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c0d1210 0x7f673c0d5160 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f672c0097a0 tx=0x7f672c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2894543ca1fbfbe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.054+0000 7f673bfff640 1 -- 172.21.15.67:0/2275429070 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c037070 con 0x7f673c0d1210 2024-04-25T15:25:53.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.054+0000 7f673bfff640 1 -- 172.21.15.67:0/2275429070 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672c032440 con 0x7f673c0d1210 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.054+0000 7f673bfff640 1 -- 172.21.15.67:0/2275429070 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c032810 con 0x7f673c0d1210 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.055+0000 7f6743882640 1 -- 172.21.15.67:0/2275429070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c0d1210 msgr2=0x7f673c0d5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.055+0000 7f6743882640 1 --2- 172.21.15.67:0/2275429070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c0d1210 0x7f673c0d5160 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f672c0097a0 tx=0x7f672c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.055+0000 7f6743882640 1 -- 172.21.15.67:0/2275429070 shutdown_connections 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.055+0000 7f6743882640 1 --2- 172.21.15.67:0/2275429070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c0d1210 0x7f673c0d5160 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.055+0000 7f6743882640 1 -- 172.21.15.67:0/2275429070 >> 172.21.15.67:0/2275429070 conn(0x7f673c0d5d90 msgr2=0x7f673c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.055+0000 7f6743882640 1 -- 172.21.15.67:0/2275429070 shutdown_connections 2024-04-25T15:25:53.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.055+0000 7f6743882640 1 -- 172.21.15.67:0/2275429070 wait complete. 2024-04-25T15:25:53.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.057+0000 7f6743882640 1 Processor -- start 2024-04-25T15:25:53.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.057+0000 7f6743882640 1 -- start start 2024-04-25T15:25:53.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6743882640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f673c0d0e00 0x7f673c0cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6743882640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f673c0d1210 0x7f673c155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6743882640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c15a5a0 0x7f673c15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6743882640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f673c0b5320 con 0x7f673c15a5a0 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6743882640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f673c160ad0 con 0x7f673c0d1210 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6743882640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f673c164e00 con 0x7f673c0d0e00 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6741e1f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c15a5a0 0x7f673c15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6741e1f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c15a5a0 0x7f673c15ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59184/0 (socket says 172.21.15.67:59184) 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.058+0000 7f6741e1f640 1 -- 172.21.15.67:0/505120333 learned_addr learned my addr 172.21.15.67:0/505120333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f674161e640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f673c0d0e00 0x7f673c0cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6740e1d640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f673c0d1210 0x7f673c155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6741e1f640 1 -- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f673c0d0e00 msgr2=0x7f673c0cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6741e1f640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f673c0d0e00 0x7f673c0cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6741e1f640 1 -- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f673c0d1210 msgr2=0x7f673c155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6741e1f640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f673c0d1210 0x7f673c155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6741e1f640 1 -- 172.21.15.67:0/505120333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673c0cb4e0 con 0x7f673c15a5a0 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f674161e640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f673c0d0e00 0x7f673c0cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6740e1d640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f673c0d1210 0x7f673c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6741e1f640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c15a5a0 0x7f673c15ac20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f672c035040 tx=0x7f672c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6739ffb640 1 -- 172.21.15.67:0/505120333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c037050 con 0x7f673c15a5a0 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6739ffb640 1 -- 172.21.15.67:0/505120333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672c038040 con 0x7f673c15a5a0 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673c0cb7c0 con 0x7f673c15a5a0 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6739ffb640 1 -- 172.21.15.67:0/505120333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c004490 con 0x7f673c15a5a0 2024-04-25T15:25:53.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.059+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f673c0cbd80 con 0x7f673c15a5a0 2024-04-25T15:25:53.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.060+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f673c050740 con 0x7f673c15a5a0 2024-04-25T15:25:53.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.061+0000 7f6739ffb640 1 -- 172.21.15.67:0/505120333 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f672c030980 con 0x7f673c15a5a0 2024-04-25T15:25:53.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.061+0000 7f6739ffb640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67280744b0 0x7f67280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.061+0000 7f6739ffb640 1 -- 172.21.15.67:0/505120333 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f672c0b5db0 con 0x7f673c15a5a0 2024-04-25T15:25:53.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.061+0000 7f674161e640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67280744b0 0x7f67280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.062+0000 7f674161e640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67280744b0 0x7f67280769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67340012e0 tx=0x7f6734009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.066+0000 7f6739ffb640 1 -- 172.21.15.67:0/505120333 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f672c0805d0 con 0x7f673c15a5a0 2024-04-25T15:25:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.250+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f673c0d4320 con 0x7f673c15a5a0 2024-04-25T15:25:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.250+0000 7f6739ffb640 1 -- 172.21.15.67:0/505120333 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f672c032e20 con 0x7f673c15a5a0 2024-04-25T15:25:53.251 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67280744b0 msgr2=0x7f67280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67280744b0 0x7f67280769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67340012e0 tx=0x7f6734009040 comp rx=0 tx=0).stop 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c15a5a0 msgr2=0x7f673c15ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c15a5a0 0x7f673c15ac20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f672c035040 tx=0x7f672c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 shutdown_connections 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67280744b0 0x7f67280769c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f673c15a5a0 0x7f673c15ac20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f673c0d1210 0x7f673c155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 --2- 172.21.15.67:0/505120333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f673c0d0e00 0x7f673c0cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 >> 172.21.15.67:0/505120333 conn(0x7f673c0d5d90 msgr2=0x7f673c0bb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 shutdown_connections 2024-04-25T15:25:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.253+0000 7f6743882640 1 -- 172.21.15.67:0/505120333 wait complete. 2024-04-25T15:25:53.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:53.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.381+0000 7f850153f640 1 Processor -- start 2024-04-25T15:25:53.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.381+0000 7f850153f640 1 -- start start 2024-04-25T15:25:53.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.382+0000 7f850153f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc161730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f850153f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84fc160cb0 0x7f84fc0b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f850153f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84fc0b2470 con 0x7f84fc0b59c0 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f850153f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f84fc06c5c0 con 0x7f84fc160cb0 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f850153f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f84fc06c8a0 con 0x7f84fc161320 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84fb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84fc160cb0 0x7f84fc0b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84fb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84fc160cb0 0x7f84fc0b5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59190/0 (socket says 172.21.15.67:59190) 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84fb7fe640 1 -- 172.21.15.67:0/4168703084 learned_addr learned my addr 172.21.15.67:0/4168703084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84faffd640 1 --2- 172.21.15.67:0/4168703084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc161730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84fb7fe640 1 -- 172.21.15.67:0/4168703084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 msgr2=0x7f84fc161730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84fb7fe640 1 --2- 172.21.15.67:0/4168703084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc161730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84fb7fe640 1 -- 172.21.15.67:0/4168703084 >> v1:172.21.15.176:6789/0 conn(0x7f84fc0b59c0 legacy=0x7f84fc1605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.383+0000 7f84fb7fe640 1 -- 172.21.15.67:0/4168703084 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84fc0b2710 con 0x7f84fc160cb0 2024-04-25T15:25:53.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.384+0000 7f84fb7fe640 1 --2- 172.21.15.67:0/4168703084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc160cb0 0x7f84fc0b5dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009e50 tx=0x7f84f402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ee00ebd5478004 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.384+0000 7f84fa7fc640 1 -- 172.21.15.67:0/4168703084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f4036070 con 0x7f84fc160cb0 2024-04-25T15:25:53.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.384+0000 7f84fa7fc640 1 -- 172.21.15.67:0/4168703084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f4031600 con 0x7f84fc160cb0 2024-04-25T15:25:53.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.384+0000 7f84fa7fc640 1 -- 172.21.15.67:0/4168703084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40319d0 con 0x7f84fc160cb0 2024-04-25T15:25:53.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 -- 172.21.15.67:0/4168703084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc160cb0 msgr2=0x7f84fc0b5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 --2- 172.21.15.67:0/4168703084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc160cb0 0x7f84fc0b5dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f84f4009e50 tx=0x7f84f402a680 comp rx=0 tx=0).stop 2024-04-25T15:25:53.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 -- 172.21.15.67:0/4168703084 shutdown_connections 2024-04-25T15:25:53.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 --2- 172.21.15.67:0/4168703084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc160cb0 0x7f84fc0b5dd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 --2- 172.21.15.67:0/4168703084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc161730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 -- 172.21.15.67:0/4168703084 >> 172.21.15.67:0/4168703084 conn(0x7f84fc09b7d0 msgr2=0x7f84fc0b52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:53.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 -- 172.21.15.67:0/4168703084 shutdown_connections 2024-04-25T15:25:53.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.385+0000 7f850153f640 1 -- 172.21.15.67:0/4168703084 wait complete. 2024-04-25T15:25:53.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.387+0000 7f850153f640 1 Processor -- start 2024-04-25T15:25:53.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.387+0000 7f850153f640 1 -- start start 2024-04-25T15:25:53.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f850153f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc0b59c0 0x7f84fc1541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f850153f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc0cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f850153f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84fc0cf580 0x7f84fc0c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f850153f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc0d2360 con 0x7f84fc0b59c0 2024-04-25T15:25:53.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f850153f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc161e90 con 0x7f84fc0cf580 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f850153f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f84fc0d24e0 con 0x7f84fc161320 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc0b59c0 0x7f84fc1541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc0b59c0 0x7f84fc1541c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59198/0 (socket says 172.21.15.67:59198) 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 -- 172.21.15.67:0/3933252686 learned_addr learned my addr 172.21.15.67:0/3933252686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84fb7fe640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84fc0cf580 0x7f84fc0c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84f3fff640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc0cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 -- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 msgr2=0x7f84fc0cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc0cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 -- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84fc0cf580 msgr2=0x7f84fc0c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84fc0cf580 0x7f84fc0c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84faffd640 1 -- 172.21.15.67:0/3933252686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84fc0cd6f0 con 0x7f84fc0b59c0 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.388+0000 7f84f3fff640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc0cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.389+0000 7f84faffd640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc0b59c0 0x7f84fc1541c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00d830 tx=0x7f84ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.389+0000 7f84f37fe640 1 -- 172.21.15.67:0/3933252686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec005990 con 0x7f84fc0b59c0 2024-04-25T15:25:53.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.389+0000 7f84f37fe640 1 -- 172.21.15.67:0/3933252686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84ec005b50 con 0x7f84fc0b59c0 2024-04-25T15:25:53.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.389+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84fc0cd9d0 con 0x7f84fc0b59c0 2024-04-25T15:25:53.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.389+0000 7f84f37fe640 1 -- 172.21.15.67:0/3933252686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0096a0 con 0x7f84fc0b59c0 2024-04-25T15:25:53.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.389+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84fc0cdf90 con 0x7f84fc0b59c0 2024-04-25T15:25:53.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.390+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84fc050740 con 0x7f84fc0b59c0 2024-04-25T15:25:53.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.390+0000 7f84f37fe640 1 -- 172.21.15.67:0/3933252686 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f84ec009a50 con 0x7f84fc0b59c0 2024-04-25T15:25:53.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.391+0000 7f84f37fe640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84dc0744e0 0x7f84dc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.391+0000 7f84f37fe640 1 -- 172.21.15.67:0/3933252686 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f84ec095640 con 0x7f84fc0b59c0 2024-04-25T15:25:53.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.391+0000 7f84f3fff640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84dc0744e0 0x7f84dc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.392+0000 7f84f3fff640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84dc0744e0 0x7f84dc0769f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84fc155190 tx=0x7f84e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.396+0000 7f84f37fe640 1 -- 172.21.15.67:0/3933252686 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f84ec05fdd0 con 0x7f84fc0b59c0 2024-04-25T15:25:53.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.581+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f84fc0764f0 con 0x7f84fc0b59c0 2024-04-25T15:25:53.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.582+0000 7f84f37fe640 1 -- 172.21.15.67:0/3933252686 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f84ec05f6d0 con 0x7f84fc0b59c0 2024-04-25T15:25:53.583 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84dc0744e0 msgr2=0x7f84dc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84dc0744e0 0x7f84dc0769f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84fc155190 tx=0x7f84e4009040 comp rx=0 tx=0).stop 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc0b59c0 msgr2=0x7f84fc1541c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc0b59c0 0x7f84fc1541c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00d830 tx=0x7f84ec00dcf0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 shutdown_connections 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84dc0744e0 0x7f84dc0769f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84fc0cf580 0x7f84fc0c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84fc161320 0x7f84fc0cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 --2- 172.21.15.67:0/3933252686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84fc0b59c0 0x7f84fc1541c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 >> 172.21.15.67:0/3933252686 conn(0x7f84fc09b7d0 msgr2=0x7f84fc15fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 shutdown_connections 2024-04-25T15:25:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.584+0000 7f850153f640 1 -- 172.21.15.67:0/3933252686 wait complete. 2024-04-25T15:25:53.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:25:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.710+0000 7f8a7677a640 1 Processor -- start 2024-04-25T15:25:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.710+0000 7f8a7677a640 1 -- start start 2024-04-25T15:25:53.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a7677a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8a7006ccd0 0x7f8a7015e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a7677a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a70163fa0 0x7f8a701643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a7677a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a701609b0 con 0x7f8a70161710 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a7677a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8a70164cd0 con 0x7f8a70163fa0 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a7677a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8a70160c30 con 0x7f8a7006ccd0 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a74d17640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a70163fa0 0x7f8a701643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a74d17640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a70163fa0 0x7f8a701643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59200/0 (socket says 172.21.15.67:59200) 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a6ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a70161710 0x7f8a7006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59100/0 (socket says 172.21.15.67:59100) 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.711+0000 7f8a74d17640 1 -- 172.21.15.67:0/1749127348 learned_addr learned my addr 172.21.15.67:0/1749127348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a6f7fe640 1 --2- 172.21.15.67:0/1749127348 >> v2:172.21.15.176:3300/0 conn(0x7f8a7006ccd0 0x7f8a7015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a74d17640 1 -- 172.21.15.67:0/1749127348 >> v2:172.21.15.176:3300/0 conn(0x7f8a7006ccd0 msgr2=0x7f8a7015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a74d17640 1 --2- 172.21.15.67:0/1749127348 >> v2:172.21.15.176:3300/0 conn(0x7f8a7006ccd0 0x7f8a7015e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a74d17640 1 -- 172.21.15.67:0/1749127348 >> v1:172.21.15.67:6789/0 conn(0x7f8a70161710 legacy=0x7f8a7006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a74d17640 1 -- 172.21.15.67:0/1749127348 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a70160f10 con 0x7f8a70163fa0 2024-04-25T15:25:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a6effd640 1 -- 172.21.15.67:0/1749127348 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252923303 0 0) 0x7f8a60001110 con 0x7f8a70161710 2024-04-25T15:25:53.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a6f7fe640 1 --2- 172.21.15.67:0/1749127348 >> v2:172.21.15.176:3300/0 conn(0x7f8a7006ccd0 0x7f8a7015e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.712+0000 7f8a74d17640 1 --2- 172.21.15.67:0/1749127348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a701643b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8a700d2770 tx=0x7f8a6802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1039925e28f91d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a6effd640 1 -- 172.21.15.67:0/1749127348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68036070 con 0x7f8a70163fa0 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a6effd640 1 -- 172.21.15.67:0/1749127348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68031600 con 0x7f8a70163fa0 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a6effd640 1 -- 172.21.15.67:0/1749127348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a680319d0 con 0x7f8a70163fa0 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 -- 172.21.15.67:0/1749127348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 msgr2=0x7f8a701643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 --2- 172.21.15.67:0/1749127348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a701643b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8a700d2770 tx=0x7f8a6802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 -- 172.21.15.67:0/1749127348 shutdown_connections 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 --2- 172.21.15.67:0/1749127348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a701643b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 --2- 172.21.15.67:0/1749127348 >> v2:172.21.15.176:3300/0 conn(0x7f8a7006ccd0 0x7f8a7015e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 -- 172.21.15.67:0/1749127348 >> 172.21.15.67:0/1749127348 conn(0x7f8a700d5d90 msgr2=0x7f8a700d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:53.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 -- 172.21.15.67:0/1749127348 shutdown_connections 2024-04-25T15:25:53.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.713+0000 7f8a7677a640 1 -- 172.21.15.67:0/1749127348 wait complete. 2024-04-25T15:25:53.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.715+0000 7f8a7677a640 1 Processor -- start 2024-04-25T15:25:53.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.715+0000 7f8a7677a640 1 -- start start 2024-04-25T15:25:53.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a7677a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a70161710 0x7f8a700c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a7677a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a7015a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a7677a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a701574d0 0x7f8a7015a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a7677a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70164a70 con 0x7f8a70163fa0 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a7677a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a701648f0 con 0x7f8a70161710 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a7677a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a700d49f0 con 0x7f8a701574d0 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a7015a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a7015a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59206/0 (socket says 172.21.15.67:59206) 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 -- 172.21.15.67:0/3891352212 learned_addr learned my addr 172.21.15.67:0/3891352212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6ffff640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a70161710 0x7f8a700c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a74d17640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a701574d0 0x7f8a7015a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 -- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a701574d0 msgr2=0x7f8a7015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a701574d0 0x7f8a7015a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 -- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a70161710 msgr2=0x7f8a700c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a70161710 0x7f8a700c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 -- 172.21.15.67:0/3891352212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a700c9c10 con 0x7f8a70163fa0 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6f7fe640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a7015a0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400c820 tx=0x7f8a6400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6cff9640 1 -- 172.21.15.67:0/3891352212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64015070 con 0x7f8a70163fa0 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6cff9640 1 -- 172.21.15.67:0/3891352212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a64007a30 con 0x7f8a70163fa0 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a700cb880 con 0x7f8a70163fa0 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.717+0000 7f8a6cff9640 1 -- 172.21.15.67:0/3891352212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64007de0 con 0x7f8a70163fa0 2024-04-25T15:25:53.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.718+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a700cbe40 con 0x7f8a70163fa0 2024-04-25T15:25:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.718+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a700d10a0 con 0x7f8a70163fa0 2024-04-25T15:25:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.719+0000 7f8a6cff9640 1 -- 172.21.15.67:0/3891352212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8a64018070 con 0x7f8a70163fa0 2024-04-25T15:25:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.720+0000 7f8a6cff9640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a5c0744b0 0x7f8a5c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.720+0000 7f8a6cff9640 1 -- 172.21.15.67:0/3891352212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8a640950a0 con 0x7f8a70163fa0 2024-04-25T15:25:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.720+0000 7f8a6ffff640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a5c0744b0 0x7f8a5c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.720+0000 7f8a6ffff640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a5c0744b0 0x7f8a5c0769c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8a60007fd0 tx=0x7f8a60007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.724+0000 7f8a6cff9640 1 -- 172.21.15.67:0/3891352212 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a700d10a0 con 0x7f8a70163fa0 2024-04-25T15:25:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.908+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a700c9eb0 con 0x7f8a70163fa0 2024-04-25T15:25:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.909+0000 7f8a6cff9640 1 -- 172.21.15.67:0/3891352212 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f8a6405f1e0 con 0x7f8a70163fa0 2024-04-25T15:25:53.909 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:25:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a5c0744b0 msgr2=0x7f8a5c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a5c0744b0 0x7f8a5c0769c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8a60007fd0 tx=0x7f8a60007de0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 msgr2=0x7f8a7015a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a7015a0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400c820 tx=0x7f8a6400cce0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 shutdown_connections 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a5c0744b0 0x7f8a5c0769c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a701574d0 0x7f8a7015a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a70163fa0 0x7f8a7015a0f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 --2- 172.21.15.67:0/3891352212 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a70161710 0x7f8a700c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.910+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 >> 172.21.15.67:0/3891352212 conn(0x7f8a700d5d90 msgr2=0x7f8a700d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.911+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 shutdown_connections 2024-04-25T15:25:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:53.911+0000 7f8a7677a640 1 -- 172.21.15.67:0/3891352212 wait complete. 2024-04-25T15:25:53.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.033+0000 7fb9ecbd7640 1 Processor -- start 2024-04-25T15:25:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.033+0000 7fb9ecbd7640 1 -- start start 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9ecbd7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9e806d8e0 0x7fb9e80d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9ecbd7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806dcf0 0x7fb9e80d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9ecbd7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9e80b3c10 con 0x7fb9e80d04a0 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9ecbd7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb9e80b9520 con 0x7fb9e806d8e0 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9ecbd7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e80b4820 con 0x7fb9e806dcf0 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9e659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9e806d8e0 0x7fb9e80d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9e659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9e806d8e0 0x7fb9e80d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59212/0 (socket says 172.21.15.67:59212) 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9e659c640 1 -- 172.21.15.67:0/521699080 learned_addr learned my addr 172.21.15.67:0/521699080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9dffff640 1 --2- 172.21.15.67:0/521699080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806dcf0 0x7fb9e80d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9e659c640 1 -- 172.21.15.67:0/521699080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806dcf0 msgr2=0x7fb9e80d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9e659c640 1 --2- 172.21.15.67:0/521699080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806dcf0 0x7fb9e80d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9e659c640 1 -- 172.21.15.67:0/521699080 >> v1:172.21.15.176:6789/0 conn(0x7fb9e80d04a0 legacy=0x7fb9e80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.035+0000 7fb9e659c640 1 -- 172.21.15.67:0/521699080 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e80b3ef0 con 0x7fb9e806d8e0 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.036+0000 7fb9e659c640 1 --2- 172.21.15.67:0/521699080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806d8e0 0x7fb9e80d3a20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009870 tx=0x7fb9d802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327f5d9cc5b91d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.036+0000 7fb9e5d9b640 1 -- 172.21.15.67:0/521699080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8037070 con 0x7fb9e806d8e0 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.036+0000 7fb9e5d9b640 1 -- 172.21.15.67:0/521699080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d8032440 con 0x7fb9e806d8e0 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.036+0000 7fb9e5d9b640 1 -- 172.21.15.67:0/521699080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8032810 con 0x7fb9e806d8e0 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/521699080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806d8e0 msgr2=0x7fb9e80d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/521699080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806d8e0 0x7fb9e80d3a20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009870 tx=0x7fb9d802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/521699080 shutdown_connections 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/521699080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806dcf0 0x7fb9e80d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/521699080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806d8e0 0x7fb9e80d3a20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/521699080 >> 172.21.15.67:0/521699080 conn(0x7fb9e808d110 msgr2=0x7fb9e808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:54.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/521699080 shutdown_connections 2024-04-25T15:25:54.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.037+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/521699080 wait complete. 2024-04-25T15:25:54.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.039+0000 7fb9ecbd7640 1 Processor -- start 2024-04-25T15:25:54.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.039+0000 7fb9ecbd7640 1 -- start start 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9ecbd7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806d8e0 0x7fb9e8150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9ecbd7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806dcf0 0x7fb9e81eab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9ecbd7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9e80d04a0 0x7fb9e809f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9ecbd7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e80b38e0 con 0x7fb9e806dcf0 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9ecbd7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e80b44b0 con 0x7fb9e80d04a0 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9ecbd7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e80d0fc0 con 0x7fb9e806d8e0 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806dcf0 0x7fb9e81eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806dcf0 0x7fb9e81eab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59218/0 (socket says 172.21.15.67:59218) 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9dffff640 1 -- 172.21.15.67:0/4077256448 learned_addr learned my addr 172.21.15.67:0/4077256448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9e6d9d640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9e80d04a0 0x7fb9e809f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.040+0000 7fb9e659c640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806d8e0 0x7fb9e8150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9dffff640 1 -- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806d8e0 msgr2=0x7fb9e8150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9dffff640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806d8e0 0x7fb9e8150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9dffff640 1 -- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9e80d04a0 msgr2=0x7fb9e809f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9dffff640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9e80d04a0 0x7fb9e809f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9dffff640 1 -- 172.21.15.67:0/4077256448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e81eb600 con 0x7fb9e806dcf0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9dffff640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806dcf0 0x7fb9e81eab00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000d5f0 tx=0x7fb9d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9deffd640 1 -- 172.21.15.67:0/4077256448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00125f0 con 0x7fb9e806dcf0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9deffd640 1 -- 172.21.15.67:0/4077256448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d0019070 con 0x7fb9e806dcf0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e81eb8e0 con 0x7fb9e806dcf0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9deffd640 1 -- 172.21.15.67:0/4077256448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00129d0 con 0x7fb9e806dcf0 2024-04-25T15:25:54.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.041+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e81ebea0 con 0x7fb9e806dcf0 2024-04-25T15:25:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.042+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e81eb370 con 0x7fb9e806dcf0 2024-04-25T15:25:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.043+0000 7fb9deffd640 1 -- 172.21.15.67:0/4077256448 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb9d0012b70 con 0x7fb9e806dcf0 2024-04-25T15:25:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.043+0000 7fb9deffd640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9c80744b0 0x7fb9c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.043+0000 7fb9deffd640 1 -- 172.21.15.67:0/4077256448 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb9d0095530 con 0x7fb9e806dcf0 2024-04-25T15:25:54.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.043+0000 7fb9e659c640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9c80744b0 0x7fb9c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.044+0000 7fb9e659c640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9c80744b0 0x7fb9c80769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8034000 tx=0x7fb9d802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.048+0000 7fb9deffd640 1 -- 172.21.15.67:0/4077256448 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9d005f670 con 0x7fb9e806dcf0 2024-04-25T15:25:54.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.236+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9e8050740 con 0x7fb9e806dcf0 2024-04-25T15:25:54.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.237+0000 7fb9deffd640 1 -- 172.21.15.67:0/4077256448 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7fb9d0063970 con 0x7fb9e806dcf0 2024-04-25T15:25:54.238 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:54.238 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9c80744b0 msgr2=0x7fb9c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9c80744b0 0x7fb9c80769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8034000 tx=0x7fb9d802e000 comp rx=0 tx=0).stop 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806dcf0 msgr2=0x7fb9e81eab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806dcf0 0x7fb9e81eab00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000d5f0 tx=0x7fb9d000dab0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 shutdown_connections 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9c80744b0 0x7fb9c80769c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9e80d04a0 0x7fb9e809f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9e806dcf0 0x7fb9e81eab00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 --2- 172.21.15.67:0/4077256448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9e806d8e0 0x7fb9e8150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 >> 172.21.15.67:0/4077256448 conn(0x7fb9e808d110 msgr2=0x7fb9e80d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 shutdown_connections 2024-04-25T15:25:54.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.239+0000 7fb9ecbd7640 1 -- 172.21.15.67:0/4077256448 wait complete. 2024-04-25T15:25:54.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:54.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.367+0000 7ffa651c0640 1 Processor -- start 2024-04-25T15:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.367+0000 7ffa651c0640 1 -- start start 2024-04-25T15:25:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.368+0000 7ffa651c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa60164640 0x7ffa600b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.368+0000 7ffa651c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa600d3170 con 0x7ffa60164ad0 2024-04-25T15:25:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.368+0000 7ffa651c0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa600d3450 con 0x7ffa600b0a70 2024-04-25T15:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.368+0000 7ffa651c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ffa600d2f40 con 0x7ffa60164640 2024-04-25T15:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.368+0000 7ffa5ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffa60164ad0 0x7ffa60163f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59102/0 (socket says 172.21.15.67:59102) 2024-04-25T15:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.368+0000 7ffa57fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa60164640 0x7ffa600b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.368+0000 7ffa5ed9d640 1 -- 172.21.15.67:0/2958398855 learned_addr learned my addr 172.21.15.67:0/2958398855 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.369+0000 7ffa57fff640 1 -- 172.21.15.67:0/2958398855 >> v1:172.21.15.176:6789/0 conn(0x7ffa600b0a70 legacy=0x7ffa600b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.369+0000 7ffa57fff640 1 -- 172.21.15.67:0/2958398855 >> v1:172.21.15.67:6789/0 conn(0x7ffa60164ad0 legacy=0x7ffa60163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.369+0000 7ffa57fff640 1 -- 172.21.15.67:0/2958398855 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6006d6d0 con 0x7ffa60164640 2024-04-25T15:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.369+0000 7ffa57fff640 1 --2- 172.21.15.67:0/2958398855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164640 0x7ffa600b2ca0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffa480097a0 tx=0x7ffa4802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a9f87167474194 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa5e59c640 1 -- 172.21.15.67:0/2958398855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48037070 con 0x7ffa60164640 2024-04-25T15:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa5e59c640 1 -- 172.21.15.67:0/2958398855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa48032440 con 0x7ffa60164640 2024-04-25T15:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa5e59c640 1 -- 172.21.15.67:0/2958398855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48032810 con 0x7ffa60164640 2024-04-25T15:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa651c0640 1 -- 172.21.15.67:0/2958398855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164640 msgr2=0x7ffa600b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa651c0640 1 --2- 172.21.15.67:0/2958398855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164640 0x7ffa600b2ca0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ffa480097a0 tx=0x7ffa4802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa651c0640 1 -- 172.21.15.67:0/2958398855 shutdown_connections 2024-04-25T15:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa651c0640 1 --2- 172.21.15.67:0/2958398855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164640 0x7ffa600b2ca0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa651c0640 1 -- 172.21.15.67:0/2958398855 >> 172.21.15.67:0/2958398855 conn(0x7ffa600d3aa0 msgr2=0x7ffa600d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa651c0640 1 -- 172.21.15.67:0/2958398855 shutdown_connections 2024-04-25T15:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.370+0000 7ffa651c0640 1 -- 172.21.15.67:0/2958398855 wait complete. 2024-04-25T15:25:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.372+0000 7ffa651c0640 1 Processor -- start 2024-04-25T15:25:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.372+0000 7ffa651c0640 1 -- start start 2024-04-25T15:25:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa651c0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa600b0a70 0x7ffa60157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa651c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164ad0 0x7ffa600caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa651c0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa600b2350 0x7ffa600b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa651c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffa600b31e0 con 0x7ffa60164ad0 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa651c0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa60164ee0 con 0x7ffa600b0a70 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa651c0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa600b9430 con 0x7ffa600b2350 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164ad0 0x7ffa600caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164ad0 0x7ffa600caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59232/0 (socket says 172.21.15.67:59232) 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 -- 172.21.15.67:0/3606940506 learned_addr learned my addr 172.21.15.67:0/3606940506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa5ed9d640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa600b0a70 0x7ffa60157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa5f59e640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa600b2350 0x7ffa600b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 -- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa600b2350 msgr2=0x7ffa600b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa600b2350 0x7ffa600b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 -- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa600b0a70 msgr2=0x7ffa60157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa600b0a70 0x7ffa60157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 -- 172.21.15.67:0/3606940506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa600b2ca0 con 0x7ffa60164ad0 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa5ed9d640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa600b0a70 0x7ffa60157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa57fff640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164ad0 0x7ffa600caae0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffa48000c00 tx=0x7ffa480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa577fe640 1 -- 172.21.15.67:0/3606940506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48041070 con 0x7ffa60164ad0 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa577fe640 1 -- 172.21.15.67:0/3606940506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4803c890 con 0x7ffa60164ad0 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa60158530 con 0x7ffa60164ad0 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.374+0000 7ffa577fe640 1 -- 172.21.15.67:0/3606940506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48037040 con 0x7ffa60164ad0 2024-04-25T15:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.375+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa600cb860 con 0x7ffa60164ad0 2024-04-25T15:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.376+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa600d4770 con 0x7ffa60164ad0 2024-04-25T15:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.376+0000 7ffa577fe640 1 -- 172.21.15.67:0/3606940506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffa48044070 con 0x7ffa60164ad0 2024-04-25T15:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.377+0000 7ffa577fe640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.377+0000 7ffa577fe640 1 -- 172.21.15.67:0/3606940506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ffa480b5e10 con 0x7ffa60164ad0 2024-04-25T15:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.377+0000 7ffa5ed9d640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.378+0000 7ffa5ed9d640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffa50000f60 tx=0x7ffa50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.381+0000 7ffa577fe640 1 -- 172.21.15.67:0/3606940506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa600b2ca0 con 0x7ffa60164ad0 2024-04-25T15:25:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.564+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffa6015fb90 con 0x7ffa60164ad0 2024-04-25T15:25:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.564+0000 7ffa577fe640 1 -- 172.21.15.67:0/3606940506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7ffa480806c0 con 0x7ffa60164ad0 2024-04-25T15:25:54.566 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 msgr2=0x7ffa400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffa50000f60 tx=0x7ffa50009040 comp rx=0 tx=0).stop 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164ad0 msgr2=0x7ffa600caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164ad0 0x7ffa600caae0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffa48000c00 tx=0x7ffa480046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 shutdown_connections 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa600b2350 0x7ffa600b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.566+0000 7ffa651c0640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa60164ad0 0x7ffa600caae0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.567+0000 7ffa651c0640 1 --2- 172.21.15.67:0/3606940506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa600b0a70 0x7ffa60157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.567+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 >> 172.21.15.67:0/3606940506 conn(0x7ffa600d3aa0 msgr2=0x7ffa6015eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.567+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 shutdown_connections 2024-04-25T15:25:54.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.567+0000 7ffa651c0640 1 -- 172.21.15.67:0/3606940506 wait complete. 2024-04-25T15:25:54.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:54.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.693+0000 7f893c500640 1 Processor -- start 2024-04-25T15:25:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.694+0000 7f893c500640 1 -- start start 2024-04-25T15:25:54.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f893c500640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89340b8290 0x7f8934160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f893c500640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89341615f0 con 0x7f8934164b00 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f893c500640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8934164020 con 0x7f893415df80 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f893c500640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8934164670 con 0x7f89340b8290 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f8939a9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89340b8290 0x7f8934160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f893a29c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8934164b00 0x7f89340b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59104/0 (socket says 172.21.15.67:59104) 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f893a29c640 1 -- 172.21.15.67:0/442448536 learned_addr learned my addr 172.21.15.67:0/442448536 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.695+0000 7f8939a9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89340b8290 0x7f8934160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59248/0 (socket says 172.21.15.67:59248) 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f8939a9b640 1 -- 172.21.15.67:0/442448536 >> v1:172.21.15.176:6789/0 conn(0x7f893415df80 legacy=0x7f8934160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f8939a9b640 1 -- 172.21.15.67:0/442448536 >> v1:172.21.15.67:6789/0 conn(0x7f8934164b00 legacy=0x7f89340b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f893929a640 1 -- 172.21.15.67:0/442448536 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805158652 0 0) 0x7f89341615f0 con 0x7f8934164b00 2024-04-25T15:25:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f8939a9b640 1 -- 172.21.15.67:0/442448536 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8934164300 con 0x7f89340b8290 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f8939a9b640 1 --2- 172.21.15.67:0/442448536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b8290 0x7f8934160890 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f89341647f0 tx=0x7f892802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76b4e15cdfd523a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f893929a640 1 -- 172.21.15.67:0/442448536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928037070 con 0x7f89340b8290 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f893929a640 1 -- 172.21.15.67:0/442448536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8928032440 con 0x7f89340b8290 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.696+0000 7f893929a640 1 -- 172.21.15.67:0/442448536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928032810 con 0x7f89340b8290 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.697+0000 7f893c500640 1 -- 172.21.15.67:0/442448536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b8290 msgr2=0x7f8934160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.697+0000 7f893c500640 1 --2- 172.21.15.67:0/442448536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b8290 0x7f8934160890 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f89341647f0 tx=0x7f892802a210 comp rx=0 tx=0).stop 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.697+0000 7f893c500640 1 -- 172.21.15.67:0/442448536 shutdown_connections 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.697+0000 7f893c500640 1 --2- 172.21.15.67:0/442448536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b8290 0x7f8934160890 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.697+0000 7f893c500640 1 -- 172.21.15.67:0/442448536 >> 172.21.15.67:0/442448536 conn(0x7f89340d2bc0 msgr2=0x7f89340d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.697+0000 7f893c500640 1 -- 172.21.15.67:0/442448536 shutdown_connections 2024-04-25T15:25:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.697+0000 7f893c500640 1 -- 172.21.15.67:0/442448536 wait complete. 2024-04-25T15:25:54.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.699+0000 7f893c500640 1 Processor -- start 2024-04-25T15:25:54.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.699+0000 7f893c500640 1 -- start start 2024-04-25T15:25:54.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893c500640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f893415df80 0x7f8934157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893c500640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8934164b00 0x7f89340b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893c500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b72c0 0x7f8934162640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893c500640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89340b4c90 con 0x7f89340b72c0 2024-04-25T15:25:54.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893c500640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89340b4b10 con 0x7f893415df80 2024-04-25T15:25:54.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893c500640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89340b4e10 con 0x7f8934164b00 2024-04-25T15:25:54.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893aa9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b72c0 0x7f8934162640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.700+0000 7f893aa9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b72c0 0x7f8934162640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59264/0 (socket says 172.21.15.67:59264) 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893aa9d640 1 -- 172.21.15.67:0/1763028557 learned_addr learned my addr 172.21.15.67:0/1763028557 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893a29c640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f893415df80 0x7f8934157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f8939a9b640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8934164b00 0x7f89340b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893aa9d640 1 -- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8934164b00 msgr2=0x7f89340b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893aa9d640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8934164b00 0x7f89340b6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893aa9d640 1 -- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f893415df80 msgr2=0x7f8934157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893aa9d640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f893415df80 0x7f8934157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893aa9d640 1 -- 172.21.15.67:0/1763028557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89340b55e0 con 0x7f89340b72c0 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893a29c640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f893415df80 0x7f8934157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f8939a9b640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8934164b00 0x7f89340b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893aa9d640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b72c0 0x7f8934162640 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f892400c680 tx=0x7f892400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f8932ffd640 1 -- 172.21.15.67:0/1763028557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924003b70 con 0x7f89340b72c0 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f8932ffd640 1 -- 172.21.15.67:0/1763028557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8924003d10 con 0x7f89340b72c0 2024-04-25T15:25:54.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89340b58c0 con 0x7f89340b72c0 2024-04-25T15:25:54.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f8932ffd640 1 -- 172.21.15.67:0/1763028557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8924005310 con 0x7f89340b72c0 2024-04-25T15:25:54.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.701+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89340b5e80 con 0x7f89340b72c0 2024-04-25T15:25:54.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.702+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89340b5350 con 0x7f89340b72c0 2024-04-25T15:25:54.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.703+0000 7f8932ffd640 1 -- 172.21.15.67:0/1763028557 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f89240054b0 con 0x7f89340b72c0 2024-04-25T15:25:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.703+0000 7f8932ffd640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89200744b0 0x7f89200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.704+0000 7f8932ffd640 1 -- 172.21.15.67:0/1763028557 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8924095070 con 0x7f89340b72c0 2024-04-25T15:25:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.704+0000 7f893a29c640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89200744b0 0x7f89200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.704+0000 7f893a29c640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89200744b0 0x7f89200769c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f892c007fd0 tx=0x7f892c007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:54.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.708+0000 7f8932ffd640 1 -- 172.21.15.67:0/1763028557 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f892405f800 con 0x7f89340b72c0 2024-04-25T15:25:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.896+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f893415e950 con 0x7f89340b72c0 2024-04-25T15:25:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.896+0000 7f8932ffd640 1 -- 172.21.15.67:0/1763028557 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f893415e950 con 0x7f89340b72c0 2024-04-25T15:25:54.897 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:25:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89200744b0 msgr2=0x7f89200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89200744b0 0x7f89200769c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f892c007fd0 tx=0x7f892c007dc0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b72c0 msgr2=0x7f8934162640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b72c0 0x7f8934162640 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f892400c680 tx=0x7f892400cb40 comp rx=0 tx=0).stop 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 shutdown_connections 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89200744b0 0x7f89200769c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89340b72c0 0x7f8934162640 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8934164b00 0x7f89340b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 --2- 172.21.15.67:0/1763028557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f893415df80 0x7f8934157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 >> 172.21.15.67:0/1763028557 conn(0x7f89340d2bc0 msgr2=0x7f893415f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 shutdown_connections 2024-04-25T15:25:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:54.898+0000 7f893c500640 1 -- 172.21.15.67:0/1763028557 wait complete. 2024-04-25T15:25:54.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:25:55.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.024+0000 7fe37752e640 1 Processor -- start 2024-04-25T15:25:55.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.024+0000 7fe37752e640 1 -- start start 2024-04-25T15:25:55.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.025+0000 7fe37752e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe370164b00 0x7fe3700b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.025+0000 7fe37752e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37015df80 0x7fe3700b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.025+0000 7fe37752e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe370160890 0x7fe370160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.025+0000 7fe37752e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe370164670 con 0x7fe370164b00 2024-04-25T15:25:55.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.025+0000 7fe37752e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe3701615f0 con 0x7fe370160890 2024-04-25T15:25:55.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.025+0000 7fe37752e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe370164020 con 0x7fe37015df80 2024-04-25T15:25:55.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.025+0000 7fe3752ca640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe370164b00 0x7fe3700b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe3752ca640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe370164b00 0x7fe3700b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59270/0 (socket says 172.21.15.67:59270) 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe3752ca640 1 -- 172.21.15.67:0/939590144 learned_addr learned my addr 172.21.15.67:0/939590144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe375acb640 1 --2- 172.21.15.67:0/939590144 >> v2:172.21.15.176:3300/0 conn(0x7fe370160890 0x7fe370160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe374ac9640 1 --2- 172.21.15.67:0/939590144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37015df80 0x7fe3700b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe3752ca640 1 -- 172.21.15.67:0/939590144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37015df80 msgr2=0x7fe3700b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe3752ca640 1 --2- 172.21.15.67:0/939590144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37015df80 0x7fe3700b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe3752ca640 1 -- 172.21.15.67:0/939590144 >> v2:172.21.15.176:3300/0 conn(0x7fe370160890 msgr2=0x7fe370160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe3752ca640 1 --2- 172.21.15.67:0/939590144 >> v2:172.21.15.176:3300/0 conn(0x7fe370160890 0x7fe370160ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe3752ca640 1 -- 172.21.15.67:0/939590144 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe370164300 con 0x7fe370164b00 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.026+0000 7fe374ac9640 1 --2- 172.21.15.67:0/939590144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37015df80 0x7fe3700b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:55.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe3752ca640 1 --2- 172.21.15.67:0/939590144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe370164b00 0x7fe3700b7b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe37006f110 tx=0x7fe36c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f536f1bfffa3659 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe367fff640 1 -- 172.21.15.67:0/939590144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c036070 con 0x7fe370164b00 2024-04-25T15:25:55.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe367fff640 1 -- 172.21.15.67:0/939590144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c031600 con 0x7fe370164b00 2024-04-25T15:25:55.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe367fff640 1 -- 172.21.15.67:0/939590144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c0319d0 con 0x7fe370164b00 2024-04-25T15:25:55.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe37752e640 1 -- 172.21.15.67:0/939590144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe370164b00 msgr2=0x7fe3700b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe37752e640 1 --2- 172.21.15.67:0/939590144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe370164b00 0x7fe3700b7b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe37006f110 tx=0x7fe36c02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe37752e640 1 -- 172.21.15.67:0/939590144 shutdown_connections 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe37752e640 1 --2- 172.21.15.67:0/939590144 >> v2:172.21.15.176:3300/0 conn(0x7fe370160890 0x7fe370160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe37752e640 1 --2- 172.21.15.67:0/939590144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37015df80 0x7fe3700b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe37752e640 1 --2- 172.21.15.67:0/939590144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe370164b00 0x7fe3700b7b80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.027+0000 7fe37752e640 1 -- 172.21.15.67:0/939590144 >> 172.21.15.67:0/939590144 conn(0x7fe3700d2bc0 msgr2=0x7fe3700d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.028+0000 7fe37752e640 1 -- 172.21.15.67:0/939590144 shutdown_connections 2024-04-25T15:25:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.028+0000 7fe37752e640 1 -- 172.21.15.67:0/939590144 wait complete. 2024-04-25T15:25:55.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.029+0000 7fe37752e640 1 Processor -- start 2024-04-25T15:25:55.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.029+0000 7fe37752e640 1 -- start start 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe37752e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37015df80 0x7fe3701593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe37752e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe370160890 0x7fe3701539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe37752e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe370164b00 0x7fe370157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe37752e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700b4a00 con 0x7fe37015df80 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe37752e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700b4880 con 0x7fe370164b00 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe37752e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3700b4b80 con 0x7fe370160890 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37015df80 0x7fe3701593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37015df80 0x7fe3701593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59272/0 (socket says 172.21.15.67:59272) 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 -- 172.21.15.67:0/4101528759 learned_addr learned my addr 172.21.15.67:0/4101528759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe374ac9640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe370160890 0x7fe3701539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe375acb640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe370164b00 0x7fe370157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 -- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe370160890 msgr2=0x7fe3701539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe370160890 0x7fe3701539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 -- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe370164b00 msgr2=0x7fe370157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe370164b00 0x7fe370157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 -- 172.21.15.67:0/4101528759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe370154730 con 0x7fe37015df80 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe375acb640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe370164b00 0x7fe370157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe374ac9640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe370160890 0x7fe3701539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe3752ca640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37015df80 0x7fe3701593a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe36c034040 tx=0x7fe36c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.031+0000 7fe365ffb640 1 -- 172.21.15.67:0/4101528759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c03f020 con 0x7fe37015df80 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.032+0000 7fe365ffb640 1 -- 172.21.15.67:0/4101528759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c037040 con 0x7fe37015df80 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.032+0000 7fe365ffb640 1 -- 172.21.15.67:0/4101528759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c036070 con 0x7fe37015df80 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.032+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe370154a10 con 0x7fe37015df80 2024-04-25T15:25:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.032+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe370154fd0 con 0x7fe37015df80 2024-04-25T15:25:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.033+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe370050740 con 0x7fe37015df80 2024-04-25T15:25:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.033+0000 7fe365ffb640 1 -- 172.21.15.67:0/4101528759 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe36c042020 con 0x7fe37015df80 2024-04-25T15:25:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.034+0000 7fe365ffb640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3540744b0 0x7fe3540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.034+0000 7fe365ffb640 1 -- 172.21.15.67:0/4101528759 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe36c0b5810 con 0x7fe37015df80 2024-04-25T15:25:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.034+0000 7fe374ac9640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3540744b0 0x7fe3540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.035+0000 7fe374ac9640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3540744b0 0x7fe3540769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe3701528b0 tx=0x7fe360008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.038+0000 7fe365ffb640 1 -- 172.21.15.67:0/4101528759 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe36c049030 con 0x7fe37015df80 2024-04-25T15:25:55.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.225+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe3700d4170 con 0x7fe37015df80 2024-04-25T15:25:55.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.225+0000 7fe365ffb640 1 -- 172.21.15.67:0/4101528759 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fe36c07ffa0 con 0x7fe37015df80 2024-04-25T15:25:55.226 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:25:55.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3540744b0 msgr2=0x7fe3540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3540744b0 0x7fe3540769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe3701528b0 tx=0x7fe360008040 comp rx=0 tx=0).stop 2024-04-25T15:25:55.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37015df80 msgr2=0x7fe3701593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37015df80 0x7fe3701593a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe36c034040 tx=0x7fe36c004a40 comp rx=0 tx=0).stop 2024-04-25T15:25:55.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 shutdown_connections 2024-04-25T15:25:55.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3540744b0 0x7fe3540769c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe370164b00 0x7fe370157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe370160890 0x7fe3701539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 --2- 172.21.15.67:0/4101528759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe37015df80 0x7fe3701593a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 >> 172.21.15.67:0/4101528759 conn(0x7fe3700d2bc0 msgr2=0x7fe3700d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:55.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 shutdown_connections 2024-04-25T15:25:55.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.227+0000 7fe37752e640 1 -- 172.21.15.67:0/4101528759 wait complete. 2024-04-25T15:25:55.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:55.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.350+0000 7f51a8084640 1 Processor -- start 2024-04-25T15:25:55.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.350+0000 7f51a8084640 1 -- start start 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a8084640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51a006d8e0 0x7f51a00d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a8084640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f51a006dcf0 0x7f51a00d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a8084640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a00d04a0 0x7f51a00d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a8084640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f51a00b9520 con 0x7f51a006d8e0 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a8084640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f51a00b4820 con 0x7f51a006dcf0 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a8084640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f51a00b3c10 con 0x7f51a00d04a0 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51a006d8e0 0x7f51a00d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51a006d8e0 0x7f51a00d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59278/0 (socket says 172.21.15.67:59278) 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 -- 172.21.15.67:0/2936576723 learned_addr learned my addr 172.21.15.67:0/2936576723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a561f640 1 --2- 172.21.15.67:0/2936576723 >> v2:172.21.15.176:3300/0 conn(0x7f51a006dcf0 0x7f51a00d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a6621640 1 --2- 172.21.15.67:0/2936576723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a00d04a0 0x7f51a00d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 -- 172.21.15.67:0/2936576723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a00d04a0 msgr2=0x7f51a00d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 --2- 172.21.15.67:0/2936576723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a00d04a0 0x7f51a00d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 -- 172.21.15.67:0/2936576723 >> v2:172.21.15.176:3300/0 conn(0x7f51a006dcf0 msgr2=0x7f51a00d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 --2- 172.21.15.67:0/2936576723 >> v2:172.21.15.176:3300/0 conn(0x7f51a006dcf0 0x7f51a00d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a5e20640 1 -- 172.21.15.67:0/2936576723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a00b3ef0 con 0x7f51a006d8e0 2024-04-25T15:25:55.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.352+0000 7f51a6621640 1 --2- 172.21.15.67:0/2936576723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a00d04a0 0x7f51a00d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.353+0000 7f51a5e20640 1 --2- 172.21.15.67:0/2936576723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a00d3a20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51a006f790 tx=0x7f519802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee64b480719a6df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.353+0000 7f51a4e1e640 1 -- 172.21.15.67:0/2936576723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198037070 con 0x7f51a006d8e0 2024-04-25T15:25:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.353+0000 7f51a4e1e640 1 -- 172.21.15.67:0/2936576723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5198032440 con 0x7f51a006d8e0 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.353+0000 7f51a4e1e640 1 -- 172.21.15.67:0/2936576723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198032810 con 0x7f51a006d8e0 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 -- 172.21.15.67:0/2936576723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 msgr2=0x7f51a00d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 --2- 172.21.15.67:0/2936576723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a00d3a20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51a006f790 tx=0x7f519802a140 comp rx=0 tx=0).stop 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 -- 172.21.15.67:0/2936576723 shutdown_connections 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 --2- 172.21.15.67:0/2936576723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a00d04a0 0x7f51a00d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 --2- 172.21.15.67:0/2936576723 >> v2:172.21.15.176:3300/0 conn(0x7f51a006dcf0 0x7f51a00d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 --2- 172.21.15.67:0/2936576723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a00d3a20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 -- 172.21.15.67:0/2936576723 >> 172.21.15.67:0/2936576723 conn(0x7f51a008d110 msgr2=0x7f51a008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 -- 172.21.15.67:0/2936576723 shutdown_connections 2024-04-25T15:25:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.354+0000 7f51a8084640 1 -- 172.21.15.67:0/2936576723 wait complete. 2024-04-25T15:25:55.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.356+0000 7f51a8084640 1 Processor -- start 2024-04-25T15:25:55.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.356+0000 7f51a8084640 1 -- start start 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a8084640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a015fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a8084640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a006dcf0 0x7f51a01e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a8084640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51a00d04a0 0x7f51a01e9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a8084640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f51a00b38e0 con 0x7f51a006d8e0 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a8084640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f51a00b44b0 con 0x7f51a00d04a0 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a8084640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f51a00d0df0 con 0x7f51a006dcf0 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a015fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a015fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59282/0 (socket says 172.21.15.67:59282) 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 -- 172.21.15.67:0/3577780505 learned_addr learned my addr 172.21.15.67:0/3577780505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a561f640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a006dcf0 0x7f51a01e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a6621640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51a00d04a0 0x7f51a01e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 -- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a006dcf0 msgr2=0x7f51a01e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a006dcf0 0x7f51a01e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 -- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51a00d04a0 msgr2=0x7f51a01e9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51a00d04a0 0x7f51a01e9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a5e20640 1 -- 172.21.15.67:0/3577780505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a009f220 con 0x7f51a006d8e0 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a6621640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51a00d04a0 0x7f51a01e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.357+0000 7f51a561f640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a006dcf0 0x7f51a01e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.358+0000 7f51a5e20640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a015fc90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f519802a180 tx=0x7f51980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.358+0000 7f519e7fc640 1 -- 172.21.15.67:0/3577780505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198041030 con 0x7f51a006d8e0 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.358+0000 7f519e7fc640 1 -- 172.21.15.67:0/3577780505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5198030440 con 0x7f51a006d8e0 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.358+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a009f500 con 0x7f51a006d8e0 2024-04-25T15:25:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.358+0000 7f519e7fc640 1 -- 172.21.15.67:0/3577780505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5198037040 con 0x7f51a006d8e0 2024-04-25T15:25:55.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.358+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a009fac0 con 0x7f51a006d8e0 2024-04-25T15:25:55.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.359+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a0050740 con 0x7f51a006d8e0 2024-04-25T15:25:55.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.360+0000 7f519e7fc640 1 -- 172.21.15.67:0/3577780505 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5198044070 con 0x7f51a006d8e0 2024-04-25T15:25:55.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.360+0000 7f519e7fc640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f518c0744e0 0x7f518c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.360+0000 7f519e7fc640 1 -- 172.21.15.67:0/3577780505 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f51980b5970 con 0x7f51a006d8e0 2024-04-25T15:25:55.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.360+0000 7f51a561f640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f518c0744e0 0x7f518c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.361+0000 7f51a561f640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f518c0744e0 0x7f518c0769f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f51a00d0150 tx=0x7f5194008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.365+0000 7f519e7fc640 1 -- 172.21.15.67:0/3577780505 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5198080100 con 0x7f51a006d8e0 2024-04-25T15:25:55.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.551+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51a01613f0 con 0x7f51a006d8e0 2024-04-25T15:25:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.552+0000 7f519e7fc640 1 -- 172.21.15.67:0/3577780505 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f519807fa00 con 0x7f51a006d8e0 2024-04-25T15:25:55.554 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:55.554 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f518c0744e0 msgr2=0x7f518c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f518c0744e0 0x7f518c0769f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f51a00d0150 tx=0x7f5194008040 comp rx=0 tx=0).stop 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 msgr2=0x7f51a015fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a015fc90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f519802a180 tx=0x7f51980046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 shutdown_connections 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f518c0744e0 0x7f518c0769f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51a00d04a0 0x7f51a01e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51a006dcf0 0x7f51a01e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 --2- 172.21.15.67:0/3577780505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51a006d8e0 0x7f51a015fc90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 >> 172.21.15.67:0/3577780505 conn(0x7f51a008d110 msgr2=0x7f51a00d1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 shutdown_connections 2024-04-25T15:25:55.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.555+0000 7f51a8084640 1 -- 172.21.15.67:0/3577780505 wait complete. 2024-04-25T15:25:55.567 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:55.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.682+0000 7fb2d3c30640 1 Processor -- start 2024-04-25T15:25:55.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.682+0000 7fb2d3c30640 1 -- start start 2024-04-25T15:25:55.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d3c30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc06e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d3c30640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2cc06e590 0x7fb2cc0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d3c30640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2cc161970 con 0x7fb2cc0d1270 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d3c30640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb2cc1614e0 con 0x7fb2cc06e590 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d3c30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc0d1a40 con 0x7fb2cc15e0c0 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d11cb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2cc06e590 0x7fb2cc0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d11cb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2cc06e590 0x7fb2cc0d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59284/0 (socket says 172.21.15.67:59284) 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d21cd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb2cc0d1270 0x7fb2cc0b7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59120/0 (socket says 172.21.15.67:59120) 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d11cb640 1 -- 172.21.15.67:0/166020800 learned_addr learned my addr 172.21.15.67:0/166020800 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.683+0000 7fb2d19cc640 1 --2- 172.21.15.67:0/166020800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.684+0000 7fb2d11cb640 1 -- 172.21.15.67:0/166020800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 msgr2=0x7fb2cc06e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.684+0000 7fb2d11cb640 1 --2- 172.21.15.67:0/166020800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc06e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.684+0000 7fb2d11cb640 1 -- 172.21.15.67:0/166020800 >> v1:172.21.15.67:6789/0 conn(0x7fb2cc0d1270 legacy=0x7fb2cc0b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.684+0000 7fb2d11cb640 1 -- 172.21.15.67:0/166020800 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc06e9a0 con 0x7fb2cc06e590 2024-04-25T15:25:55.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.684+0000 7fb2d09ca640 1 -- 172.21.15.67:0/166020800 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3558396771 0 0) 0x7fb2cc161970 con 0x7fb2cc0d1270 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.684+0000 7fb2d11cb640 1 --2- 172.21.15.67:0/166020800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0d0d30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009870 tx=0x7fb2c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2225b3e52a2c39a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.684+0000 7fb2d09ca640 1 -- 172.21.15.67:0/166020800 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0037070 con 0x7fb2cc06e590 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d09ca640 1 -- 172.21.15.67:0/166020800 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c0032440 con 0x7fb2cc06e590 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d09ca640 1 -- 172.21.15.67:0/166020800 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0032810 con 0x7fb2cc06e590 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 -- 172.21.15.67:0/166020800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 msgr2=0x7fb2cc0d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/166020800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0d0d30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009870 tx=0x7fb2c002a270 comp rx=0 tx=0).stop 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 -- 172.21.15.67:0/166020800 shutdown_connections 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/166020800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0d0d30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/166020800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc06e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 -- 172.21.15.67:0/166020800 >> 172.21.15.67:0/166020800 conn(0x7fb2cc0d2800 msgr2=0x7fb2cc0d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 -- 172.21.15.67:0/166020800 shutdown_connections 2024-04-25T15:25:55.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.685+0000 7fb2d3c30640 1 -- 172.21.15.67:0/166020800 wait complete. 2024-04-25T15:25:55.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.687+0000 7fb2d3c30640 1 Processor -- start 2024-04-25T15:25:55.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.687+0000 7fb2d3c30640 1 -- start start 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d3c30640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d3c30640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0d1270 0x7fb2cc0b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d3c30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc0cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d3c30640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc161170 con 0x7fb2cc06e590 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d3c30640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc160e90 con 0x7fb2cc0d1270 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d3c30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc0d1680 con 0x7fb2cc15e0c0 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d19cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d19cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0cf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59290/0 (socket says 172.21.15.67:59290) 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.688+0000 7fb2d19cc640 1 -- 172.21.15.67:0/3358377806 learned_addr learned my addr 172.21.15.67:0/3358377806 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d21cd640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc0cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d11cb640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0d1270 0x7fb2cc0b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d19cc640 1 -- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 msgr2=0x7fb2cc0cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d19cc640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc0cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d19cc640 1 -- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0d1270 msgr2=0x7fb2cc0b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d19cc640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0d1270 0x7fb2cc0b7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d19cc640 1 -- 172.21.15.67:0/3358377806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc0b8540 con 0x7fb2cc06e590 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d21cd640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc0cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d11cb640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0d1270 0x7fb2cc0b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d19cc640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0cf6f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d620 tx=0x7fb2c400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2ca7fc640 1 -- 172.21.15.67:0/3358377806 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400b620 con 0x7fb2cc06e590 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2ca7fc640 1 -- 172.21.15.67:0/3358377806 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c4019070 con 0x7fb2cc06e590 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2cc1f9f10 con 0x7fb2cc06e590 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2ca7fc640 1 -- 172.21.15.67:0/3358377806 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400ba00 con 0x7fb2cc06e590 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.689+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2cc1fa4d0 con 0x7fb2cc06e590 2024-04-25T15:25:55.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.690+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2cc050740 con 0x7fb2cc06e590 2024-04-25T15:25:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.691+0000 7fb2ca7fc640 1 -- 172.21.15.67:0/3358377806 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb2c400bba0 con 0x7fb2cc06e590 2024-04-25T15:25:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.691+0000 7fb2ca7fc640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2b80744b0 0x7fb2b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.691+0000 7fb2ca7fc640 1 -- 172.21.15.67:0/3358377806 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb2c40955d0 con 0x7fb2cc06e590 2024-04-25T15:25:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.692+0000 7fb2d11cb640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2b80744b0 0x7fb2b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.692+0000 7fb2d11cb640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2b80744b0 0x7fb2b80769c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0034000 tx=0x7fb2c002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:55.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.696+0000 7fb2ca7fc640 1 -- 172.21.15.67:0/3358377806 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb2c405fe80 con 0x7fb2cc06e590 2024-04-25T15:25:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.881+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb2cc0d43b0 con 0x7fb2cc06e590 2024-04-25T15:25:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.882+0000 7fb2ca7fc640 1 -- 172.21.15.67:0/3358377806 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fb2c4066e20 con 0x7fb2cc06e590 2024-04-25T15:25:55.883 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2b80744b0 msgr2=0x7fb2b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2b80744b0 0x7fb2b80769c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0034000 tx=0x7fb2c002e000 comp rx=0 tx=0).stop 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 msgr2=0x7fb2cc0cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0cf6f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d620 tx=0x7fb2c400dae0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 shutdown_connections 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2b80744b0 0x7fb2b80769c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc15e0c0 0x7fb2cc0cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0d1270 0x7fb2cc0b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 --2- 172.21.15.67:0/3358377806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc06e590 0x7fb2cc0cf6f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 >> 172.21.15.67:0/3358377806 conn(0x7fb2cc0d2800 msgr2=0x7fb2cc0d5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 shutdown_connections 2024-04-25T15:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:55.884+0000 7fb2d3c30640 1 -- 172.21.15.67:0/3358377806 wait complete. 2024-04-25T15:25:55.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:56.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.011+0000 7f70d09ce640 1 Processor -- start 2024-04-25T15:25:56.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.011+0000 7f70d09ce640 1 -- start start 2024-04-25T15:25:56.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.012+0000 7f70d09ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc0d0e00 0x7f70cc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.012+0000 7f70d09ce640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f70cc06c5c0 0x7f70cc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.012+0000 7f70d09ce640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70cc06cf10 con 0x7f70cc0b7810 2024-04-25T15:25:56.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.012+0000 7f70d09ce640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f70cc0d5a60 con 0x7f70cc06c5c0 2024-04-25T15:25:56.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.012+0000 7f70d09ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70cc0d1850 con 0x7f70cc0d0e00 2024-04-25T15:25:56.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70ca59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc0d0e00 0x7f70cc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70c9d9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f70cc06c5c0 0x7f70cc06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70ca59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc0d0e00 0x7f70cc0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:44886/0 (socket says 172.21.15.67:44886) 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70ca59c640 1 -- 172.21.15.67:0/455707684 learned_addr learned my addr 172.21.15.67:0/455707684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70ca59c640 1 -- 172.21.15.67:0/455707684 >> v2:172.21.15.176:3300/0 conn(0x7f70cc06c5c0 msgr2=0x7f70cc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70ca59c640 1 --2- 172.21.15.67:0/455707684 >> v2:172.21.15.176:3300/0 conn(0x7f70cc06c5c0 0x7f70cc06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70ca59c640 1 -- 172.21.15.67:0/455707684 >> v1:172.21.15.176:6789/0 conn(0x7f70cc0b7810 legacy=0x7f70cc0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70ca59c640 1 -- 172.21.15.67:0/455707684 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70cc160de0 con 0x7f70cc0d0e00 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.013+0000 7f70c9d9b640 1 --2- 172.21.15.67:0/455707684 >> v2:172.21.15.176:3300/0 conn(0x7f70cc06c5c0 0x7f70cc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.014+0000 7f70ca59c640 1 --2- 172.21.15.67:0/455707684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc0d0e00 0x7f70cc0d1210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009e50 tx=0x7f70bc02a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3632afc09a970a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.014+0000 7f70c959a640 1 -- 172.21.15.67:0/455707684 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc036070 con 0x7f70cc0d0e00 2024-04-25T15:25:56.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.014+0000 7f70c959a640 1 -- 172.21.15.67:0/455707684 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70bc031600 con 0x7f70cc0d0e00 2024-04-25T15:25:56.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.014+0000 7f70c959a640 1 -- 172.21.15.67:0/455707684 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0319d0 con 0x7f70cc0d0e00 2024-04-25T15:25:56.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 -- 172.21.15.67:0/455707684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc0d0e00 msgr2=0x7f70cc0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 --2- 172.21.15.67:0/455707684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc0d0e00 0x7f70cc0d1210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009e50 tx=0x7f70bc02a730 comp rx=0 tx=0).stop 2024-04-25T15:25:56.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 -- 172.21.15.67:0/455707684 shutdown_connections 2024-04-25T15:25:56.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 --2- 172.21.15.67:0/455707684 >> v2:172.21.15.176:3300/0 conn(0x7f70cc06c5c0 0x7f70cc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 --2- 172.21.15.67:0/455707684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc0d0e00 0x7f70cc0d1210 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 -- 172.21.15.67:0/455707684 >> 172.21.15.67:0/455707684 conn(0x7f70cc0d5d90 msgr2=0x7f70cc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:56.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 -- 172.21.15.67:0/455707684 shutdown_connections 2024-04-25T15:25:56.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.015+0000 7f70d09ce640 1 -- 172.21.15.67:0/455707684 wait complete. 2024-04-25T15:25:56.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.017+0000 7f70d09ce640 1 Processor -- start 2024-04-25T15:25:56.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.017+0000 7f70d09ce640 1 -- start start 2024-04-25T15:25:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.018+0000 7f70d09ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc06c5c0 0x7f70cc15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.018+0000 7f70d09ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70cc0b7810 0x7f70cc1573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.018+0000 7f70d09ce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70cc0d0e00 0x7f70cc158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.018+0000 7f70d09ce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc160ad0 con 0x7f70cc0b7810 2024-04-25T15:25:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.018+0000 7f70d09ce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc1613a0 con 0x7f70cc0d0e00 2024-04-25T15:25:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.018+0000 7f70d09ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70cc0b5320 con 0x7f70cc06c5c0 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70cc0b7810 0x7f70cc1573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70cc0b7810 0x7f70cc1573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59292/0 (socket says 172.21.15.67:59292) 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 -- 172.21.15.67:0/1754704337 learned_addr learned my addr 172.21.15.67:0/1754704337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70cad9d640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70cc0d0e00 0x7f70cc158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70ca59c640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc06c5c0 0x7f70cc15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 -- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc06c5c0 msgr2=0x7f70cc15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc06c5c0 0x7f70cc15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 -- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70cc0d0e00 msgr2=0x7f70cc158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70cc0d0e00 0x7f70cc158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 -- 172.21.15.67:0/1754704337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70cc15af20 con 0x7f70cc0b7810 2024-04-25T15:25:56.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c9d9b640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70cc0b7810 0x7f70cc1573d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e990 tx=0x7f70b400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c2ffd640 1 -- 172.21.15.67:0/1754704337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4017070 con 0x7f70cc0b7810 2024-04-25T15:25:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70cc15ac40 con 0x7f70cc0b7810 2024-04-25T15:25:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c2ffd640 1 -- 172.21.15.67:0/1754704337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b400c9e0 con 0x7f70cc0b7810 2024-04-25T15:25:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.019+0000 7f70c2ffd640 1 -- 172.21.15.67:0/1754704337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b400cdb0 con 0x7f70cc0b7810 2024-04-25T15:25:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.020+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70cc157f90 con 0x7f70cc0b7810 2024-04-25T15:25:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.021+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70cc15a9b0 con 0x7f70cc0b7810 2024-04-25T15:25:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.022+0000 7f70c2ffd640 1 -- 172.21.15.67:0/1754704337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f70b401a070 con 0x7f70cc0b7810 2024-04-25T15:25:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.022+0000 7f70c2ffd640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70ac0744b0 0x7f70ac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.022+0000 7f70c2ffd640 1 -- 172.21.15.67:0/1754704337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f70b4095120 con 0x7f70cc0b7810 2024-04-25T15:25:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.023+0000 7f70ca59c640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70ac0744b0 0x7f70ac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.023+0000 7f70ca59c640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70ac0744b0 0x7f70ac0769c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f70bc033000 tx=0x7f70bc009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.027+0000 7f70c2ffd640 1 -- 172.21.15.67:0/1754704337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70b405f1e0 con 0x7f70cc0b7810 2024-04-25T15:25:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.213+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70cc050740 con 0x7f70cc0b7810 2024-04-25T15:25:56.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.213+0000 7f70c2ffd640 1 -- 172.21.15.67:0/1754704337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f70b40634e0 con 0x7f70cc0b7810 2024-04-25T15:25:56.214 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:25:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.215+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70ac0744b0 msgr2=0x7f70ac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.215+0000 7f70d09ce640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70ac0744b0 0x7f70ac0769c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f70bc033000 tx=0x7f70bc009000 comp rx=0 tx=0).stop 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.215+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70cc0b7810 msgr2=0x7f70cc1573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.215+0000 7f70d09ce640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70cc0b7810 0x7f70cc1573d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e990 tx=0x7f70b400ee50 comp rx=0 tx=0).stop 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 shutdown_connections 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70ac0744b0 0x7f70ac0769c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70cc0d0e00 0x7f70cc158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70cc0b7810 0x7f70cc1573d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 --2- 172.21.15.67:0/1754704337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70cc06c5c0 0x7f70cc15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 >> 172.21.15.67:0/1754704337 conn(0x7f70cc0d5d90 msgr2=0x7f70cc0b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 shutdown_connections 2024-04-25T15:25:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.216+0000 7f70d09ce640 1 -- 172.21.15.67:0/1754704337 wait complete. 2024-04-25T15:25:56.228 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:25:56.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.341+0000 7f5d6e815640 1 Processor -- start 2024-04-25T15:25:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.341+0000 7f5d6e815640 1 -- start start 2024-04-25T15:25:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d6e815640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 0x7f5d680d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d6e815640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5d6806c5c0 0x7f5d6806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d6e815640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d6806cf10 con 0x7f5d680b7810 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d6e815640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5d680d5a60 con 0x7f5d6806c5c0 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d6e815640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d680d1850 con 0x7f5d680d0e00 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d6cdb2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5d680b7810 0x7f5d680b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59124/0 (socket says 172.21.15.67:59124) 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d6cdb2640 1 -- 172.21.15.67:0/4170046542 learned_addr learned my addr 172.21.15.67:0/4170046542 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d677fe640 1 --2- 172.21.15.67:0/4170046542 >> v2:172.21.15.176:3300/0 conn(0x7f5d6806c5c0 0x7f5d6806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d67fff640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 0x7f5d680d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 647286991 0 0) 0x7f5d60001490 con 0x7f5d680b7810 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d50003520 con 0x7f5d680b7810 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1280281153 0 0) 0x7f5d50003520 con 0x7f5d680b7810 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 msgr2=0x7f5d680d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 0x7f5d680d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 >> v2:172.21.15.176:3300/0 conn(0x7f5d6806c5c0 msgr2=0x7f5d6806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 --2- 172.21.15.67:0/4170046542 >> v2:172.21.15.176:3300/0 conn(0x7f5d6806c5c0 0x7f5d6806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d68160de0 con 0x7f5d680b7810 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f5d60001bc0 con 0x7f5d680b7810 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 >> v1:172.21.15.67:6789/0 conn(0x7f5d680b7810 legacy=0x7f5d680b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d500039b0 0x7f5d50003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 0x7f5d50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d50004d40 0x7f5d50005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d680d5a60 con 0x7f5d500039b0 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5d680d1850 con 0x7f5d50004d40 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.343+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d5000b9c0 con 0x7f5d680d0e00 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d67fff640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d500039b0 0x7f5d50003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d677fe640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 0x7f5d50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d6cdb2640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d50004d40 0x7f5d50005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d67fff640 1 -- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 msgr2=0x7f5d50004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d67fff640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d680d0e00 0x7f5d50004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d67fff640 1 -- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d50004d40 msgr2=0x7f5d50005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d67fff640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d50004d40 0x7f5d50005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.344+0000 7f5d67fff640 1 -- 172.21.15.67:0/4170046542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d50003520 con 0x7f5d500039b0 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d67fff640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d500039b0 0x7f5d50003e20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009ac0 tx=0x7f5d50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88aec486f153789 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c036070 con 0x7f5d500039b0 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5c031400 con 0x7f5d500039b0 2024-04-25T15:25:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d66ffd640 1 -- 172.21.15.67:0/4170046542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c009690 con 0x7f5d500039b0 2024-04-25T15:25:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d6e815640 1 -- 172.21.15.67:0/4170046542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d500039b0 msgr2=0x7f5d50003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d6e815640 1 --2- 172.21.15.67:0/4170046542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d500039b0 0x7f5d50003e20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009ac0 tx=0x7f5d50005830 comp rx=0 tx=0).stop 2024-04-25T15:25:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d6cdb2640 1 -- 172.21.15.67:0/4170046542 reap_dead start 2024-04-25T15:25:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d6e815640 1 -- 172.21.15.67:0/4170046542 shutdown_connections 2024-04-25T15:25:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d6e815640 1 -- 172.21.15.67:0/4170046542 >> 172.21.15.67:0/4170046542 conn(0x7f5d680d5d90 msgr2=0x7f5d680d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.345+0000 7f5d6e815640 1 -- 172.21.15.67:0/4170046542 shutdown_connections 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.346+0000 7f5d6e815640 1 -- 172.21.15.67:0/4170046542 wait complete. 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.347+0000 7f5d6e815640 1 Processor -- start 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.347+0000 7f5d6e815640 1 -- start start 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6e815640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d500039b0 0x7f5d68156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6e815640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d680ca6b0 0x7f5d68162900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6e815640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d68154dd0 0x7f5d68163e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6e815640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d68160ad0 con 0x7f5d68154dd0 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6e815640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5d681613a0 con 0x7f5d680ca6b0 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6e815640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d680b5320 con 0x7f5d500039b0 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d68154dd0 0x7f5d68163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d68154dd0 0x7f5d68163e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59304/0 (socket says 172.21.15.67:59304) 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 -- 172.21.15.67:0/1716636997 learned_addr learned my addr 172.21.15.67:0/1716636997 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d67fff640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d500039b0 0x7f5d68156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d677fe640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d680ca6b0 0x7f5d68162900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 -- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d500039b0 msgr2=0x7f5d68156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d500039b0 0x7f5d68156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 -- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d680ca6b0 msgr2=0x7f5d68162900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d680ca6b0 0x7f5d68162900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 -- 172.21.15.67:0/1716636997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d68163120 con 0x7f5d68154dd0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d6cdb2640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d68154dd0 0x7f5d68163e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d600013d0 tx=0x7f5d6000aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.349+0000 7f5d64ff9640 1 -- 172.21.15.67:0/1716636997 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d600078c0 con 0x7f5d68154dd0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.350+0000 7f5d64ff9640 1 -- 172.21.15.67:0/1716636997 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d60018070 con 0x7f5d68154dd0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.350+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d68163400 con 0x7f5d68154dd0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.350+0000 7f5d64ff9640 1 -- 172.21.15.67:0/1716636997 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60013630 con 0x7f5d68154dd0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.350+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d68158010 con 0x7f5d68154dd0 2024-04-25T15:25:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.350+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d68050740 con 0x7f5d68154dd0 2024-04-25T15:25:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.351+0000 7f5d64ff9640 1 -- 172.21.15.67:0/1716636997 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5d600137d0 con 0x7f5d68154dd0 2024-04-25T15:25:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.352+0000 7f5d64ff9640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d540744e0 0x7f5d540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.352+0000 7f5d64ff9640 1 -- 172.21.15.67:0/1716636997 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5d60095780 con 0x7f5d68154dd0 2024-04-25T15:25:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.352+0000 7f5d67fff640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d540744e0 0x7f5d540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.352+0000 7f5d67fff640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d540744e0 0x7f5d540769f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c033000 tx=0x7f5d5c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.356+0000 7f5d64ff9640 1 -- 172.21.15.67:0/1716636997 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d6806cf10 con 0x7f5d68154dd0 2024-04-25T15:25:56.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.542+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5d68162eb0 con 0x7f5d68154dd0 2024-04-25T15:25:56.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.543+0000 7f5d64ff9640 1 -- 172.21.15.67:0/1716636997 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f5d6005ff10 con 0x7f5d68154dd0 2024-04-25T15:25:56.544 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:25:56.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d540744e0 msgr2=0x7f5d540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d540744e0 0x7f5d540769f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c033000 tx=0x7f5d5c008030 comp rx=0 tx=0).stop 2024-04-25T15:25:56.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d68154dd0 msgr2=0x7f5d68163e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d68154dd0 0x7f5d68163e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5d600013d0 tx=0x7f5d6000aec0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 shutdown_connections 2024-04-25T15:25:56.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d540744e0 0x7f5d540769f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d68154dd0 0x7f5d68163e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d680ca6b0 0x7f5d68162900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 --2- 172.21.15.67:0/1716636997 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d500039b0 0x7f5d68156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 >> 172.21.15.67:0/1716636997 conn(0x7f5d680d5d90 msgr2=0x7f5d680bab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:56.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 shutdown_connections 2024-04-25T15:25:56.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.545+0000 7f5d6e815640 1 -- 172.21.15.67:0/1716636997 wait complete. 2024-04-25T15:25:56.583 INFO:tasks.cephfs_test_runner:Starting test: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) test: 2024-04-25T15:25:56.583 INFO:tasks.cephfs_test_runner:test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2024-04-25T15:25:56.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:56.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.694+0000 7fae6d249640 1 Processor -- start 2024-04-25T15:25:56.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.694+0000 7fae6d249640 1 -- start start 2024-04-25T15:25:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae6d249640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fae68164b00 0x7fae680b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae6d249640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae681615f0 con 0x7fae68160fa0 2024-04-25T15:25:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae6d249640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae68164020 con 0x7fae6815df80 2024-04-25T15:25:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae6d249640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fae681613b0 con 0x7fae68164b00 2024-04-25T15:25:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae6759e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fae68160fa0 0x7fae680b4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59140/0 (socket says 172.21.15.67:59140) 2024-04-25T15:25:56.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae6759e640 1 -- 172.21.15.67:0/1080438729 learned_addr learned my addr 172.21.15.67:0/1080438729 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:56.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae66d9d640 1 --2- 172.21.15.67:0/1080438729 >> v2:172.21.15.176:3300/0 conn(0x7fae68164b00 0x7fae680b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae65d9b640 1 -- 172.21.15.67:0/1080438729 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701349182 0 0) 0x7fae681615f0 con 0x7fae68160fa0 2024-04-25T15:25:56.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae65d9b640 1 -- 172.21.15.67:0/1080438729 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae54003520 con 0x7fae68160fa0 2024-04-25T15:25:56.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae66d9d640 1 -- 172.21.15.67:0/1080438729 >> v1:172.21.15.176:6789/0 conn(0x7fae6815df80 legacy=0x7fae68160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae66d9d640 1 -- 172.21.15.67:0/1080438729 >> v1:172.21.15.67:6789/0 conn(0x7fae68160fa0 legacy=0x7fae680b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.696+0000 7fae66d9d640 1 -- 172.21.15.67:0/1080438729 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae68164300 con 0x7fae68164b00 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.697+0000 7fae66d9d640 1 --2- 172.21.15.67:0/1080438729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 0x7fae680b7b80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fae600097a0 tx=0x7fae6002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c7f69618fc31238 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.697+0000 7fae65d9b640 1 -- 172.21.15.67:0/1080438729 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae60037070 con 0x7fae68164b00 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.697+0000 7fae65d9b640 1 -- 172.21.15.67:0/1080438729 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae60032440 con 0x7fae68164b00 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.697+0000 7fae65d9b640 1 -- 172.21.15.67:0/1080438729 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae60032810 con 0x7fae68164b00 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.698+0000 7fae6d249640 1 -- 172.21.15.67:0/1080438729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 msgr2=0x7fae680b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.698+0000 7fae6d249640 1 --2- 172.21.15.67:0/1080438729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 0x7fae680b7b80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fae600097a0 tx=0x7fae6002a230 comp rx=0 tx=0).stop 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.698+0000 7fae6d249640 1 -- 172.21.15.67:0/1080438729 shutdown_connections 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.698+0000 7fae6d249640 1 --2- 172.21.15.67:0/1080438729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 0x7fae680b7b80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.698+0000 7fae6d249640 1 -- 172.21.15.67:0/1080438729 >> 172.21.15.67:0/1080438729 conn(0x7fae680d2bc0 msgr2=0x7fae680d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.698+0000 7fae6d249640 1 -- 172.21.15.67:0/1080438729 shutdown_connections 2024-04-25T15:25:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.698+0000 7fae6d249640 1 -- 172.21.15.67:0/1080438729 wait complete. 2024-04-25T15:25:56.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.700+0000 7fae6d249640 1 Processor -- start 2024-04-25T15:25:56.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.700+0000 7fae6d249640 1 -- start start 2024-04-25T15:25:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.701+0000 7fae6d249640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6815df80 0x7fae6814ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.701+0000 7fae6d249640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae68160fa0 0x7fae6814c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.701+0000 7fae6d249640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 0x7fae68149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.701+0000 7fae6d249640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae680b84d0 con 0x7fae6815df80 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.701+0000 7fae6d249640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae680b8350 con 0x7fae68164b00 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.701+0000 7fae6d249640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae68164670 con 0x7fae68160fa0 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6815df80 0x7fae6814ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6815df80 0x7fae6814ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59308/0 (socket says 172.21.15.67:59308) 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 -- 172.21.15.67:0/3540770912 learned_addr learned my addr 172.21.15.67:0/3540770912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae6659c640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae68160fa0 0x7fae6814c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae6759e640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 0x7fae68149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 -- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae68160fa0 msgr2=0x7fae6814c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae68160fa0 0x7fae6814c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 -- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 msgr2=0x7fae68149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 0x7fae68149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 -- 172.21.15.67:0/3540770912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6814d2f0 con 0x7fae6815df80 2024-04-25T15:25:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae66d9d640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6815df80 0x7fae6814ae70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae600098c0 tx=0x7fae600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae5f7fe640 1 -- 172.21.15.67:0/3540770912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae60037040 con 0x7fae6815df80 2024-04-25T15:25:56.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae5f7fe640 1 -- 172.21.15.67:0/3540770912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae60030500 con 0x7fae6815df80 2024-04-25T15:25:56.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6814ba50 con 0x7fae6815df80 2024-04-25T15:25:56.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae5f7fe640 1 -- 172.21.15.67:0/3540770912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae600308d0 con 0x7fae6815df80 2024-04-25T15:25:56.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.702+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6814c010 con 0x7fae6815df80 2024-04-25T15:25:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.703+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae6814c250 con 0x7fae6815df80 2024-04-25T15:25:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.704+0000 7fae5f7fe640 1 -- 172.21.15.67:0/3540770912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fae60044020 con 0x7fae6815df80 2024-04-25T15:25:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.704+0000 7fae5f7fe640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae480744b0 0x7fae480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.705+0000 7fae5f7fe640 1 -- 172.21.15.67:0/3540770912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fae600b53d0 con 0x7fae6815df80 2024-04-25T15:25:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.705+0000 7fae6659c640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae480744b0 0x7fae480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.705+0000 7fae6659c640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae480744b0 0x7fae480769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae680b69e0 tx=0x7fae50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:56.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.709+0000 7fae5f7fe640 1 -- 172.21.15.67:0/3540770912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae6814c250 con 0x7fae6815df80 2024-04-25T15:25:56.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.898+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae68050740 con 0x7fae6815df80 2024-04-25T15:25:56.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.899+0000 7fae5f7fe640 1 -- 172.21.15.67:0/3540770912 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7fae6004b020 con 0x7fae6815df80 2024-04-25T15:25:56.900 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:56.900 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.901+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae480744b0 msgr2=0x7fae480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.901+0000 7fae6d249640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae480744b0 0x7fae480769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fae680b69e0 tx=0x7fae50009040 comp rx=0 tx=0).stop 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.901+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6815df80 msgr2=0x7fae6814ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.901+0000 7fae6d249640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6815df80 0x7fae6814ae70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae600098c0 tx=0x7fae600046c0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 shutdown_connections 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae480744b0 0x7fae480769c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae68164b00 0x7fae68149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae68160fa0 0x7fae6814c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 --2- 172.21.15.67:0/3540770912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6815df80 0x7fae6814ae70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 >> 172.21.15.67:0/3540770912 conn(0x7fae680d2bc0 msgr2=0x7fae6806e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 shutdown_connections 2024-04-25T15:25:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:56.902+0000 7fae6d249640 1 -- 172.21.15.67:0/3540770912 wait complete. 2024-04-25T15:25:56.914 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:56.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.029+0000 7fb70851f640 1 Processor -- start 2024-04-25T15:25:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.029+0000 7fb70851f640 1 -- start start 2024-04-25T15:25:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb70851f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb70006ccd0 0x7fb7000d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb70851f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7000d1850 con 0x7fb7000d0e00 2024-04-25T15:25:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb70851f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb700160de0 con 0x7fb7000d4a50 2024-04-25T15:25:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb70851f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb7001610c0 con 0x7fb70006ccd0 2024-04-25T15:25:57.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7062bb640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7000d0e00 0x7fb70006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59156/0 (socket says 172.21.15.67:59156) 2024-04-25T15:25:57.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7062bb640 1 -- 172.21.15.67:0/1476710605 learned_addr learned my addr 172.21.15.67:0/1476710605 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb705aba640 1 --2- 172.21.15.67:0/1476710605 >> v2:172.21.15.176:3300/0 conn(0x7fb70006ccd0 0x7fb7000d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321265485 0 0) 0x7fb7000d1850 con 0x7fb7000d0e00 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6e8003520 con 0x7fb7000d0e00 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 966939495 0 0) 0x7fb6e8003520 con 0x7fb7000d0e00 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 >> v2:172.21.15.176:3300/0 conn(0x7fb70006ccd0 msgr2=0x7fb7000d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 --2- 172.21.15.67:0/1476710605 >> v2:172.21.15.176:3300/0 conn(0x7fb70006ccd0 0x7fb7000d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 >> v1:172.21.15.176:6789/0 conn(0x7fb7000d4a50 legacy=0x7fb7000d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7001613a0 con 0x7fb7000d0e00 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fb6f8001670 con 0x7fb7000d0e00 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 >> v1:172.21.15.67:6789/0 conn(0x7fb7000d0e00 legacy=0x7fb70006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e80039b0 0x7fb6e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6e8004800 0x7fb6e8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6e80051c0 0x7fb6e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7001610c0 con 0x7fb6e8004800 2024-04-25T15:25:57.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.031+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e800c160 con 0x7fb6e80051c0 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e800c360 con 0x7fb6e80039b0 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb705aba640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6e8004800 0x7fb6e8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb7062bb640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e80039b0 0x7fb6e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb706abc640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6e80051c0 0x7fb6e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb705aba640 1 -- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e80039b0 msgr2=0x7fb6e8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb705aba640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e80039b0 0x7fb6e8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb705aba640 1 -- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6e80051c0 msgr2=0x7fb6e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb705aba640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6e80051c0 0x7fb6e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb706abc640 1 -- 172.21.15.67:0/1476710605 reap_dead start 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.032+0000 7fb705aba640 1 -- 172.21.15.67:0/1476710605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e8003520 con 0x7fb6e8004800 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.033+0000 7fb705aba640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6e8004800 0x7fb6e8004c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009ac0 tx=0x7fb6e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b22116f46c67ccf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.033+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4036070 con 0x7fb6e8004800 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.033+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f4031400 con 0x7fb6e8004800 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.033+0000 7fb7052b9640 1 -- 172.21.15.67:0/1476710605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4009690 con 0x7fb6e8004800 2024-04-25T15:25:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.033+0000 7fb70851f640 1 -- 172.21.15.67:0/1476710605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6e8004800 msgr2=0x7fb6e8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.033+0000 7fb70851f640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6e8004800 0x7fb6e8004c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009ac0 tx=0x7fb6e8005830 comp rx=0 tx=0).stop 2024-04-25T15:25:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.034+0000 7fb70851f640 1 -- 172.21.15.67:0/1476710605 shutdown_connections 2024-04-25T15:25:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.034+0000 7fb70851f640 1 --2- 172.21.15.67:0/1476710605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6e8004800 0x7fb6e8004c10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.034+0000 7fb70851f640 1 -- 172.21.15.67:0/1476710605 >> 172.21.15.67:0/1476710605 conn(0x7fb7000d5d90 msgr2=0x7fb7000d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.034+0000 7fb70851f640 1 -- 172.21.15.67:0/1476710605 shutdown_connections 2024-04-25T15:25:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.034+0000 7fb70851f640 1 -- 172.21.15.67:0/1476710605 wait complete. 2024-04-25T15:25:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.035+0000 7fb70851f640 1 Processor -- start 2024-04-25T15:25:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.035+0000 7fb70851f640 1 -- start start 2024-04-25T15:25:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e8004800 0x7fb700150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7000b8050 0x7fb700150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000cd070 0x7fb7000cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb700164e00 con 0x7fb7000cd070 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7000b5320 con 0x7fb7000b8050 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb700164f80 con 0x7fb6e8004800 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000cd070 0x7fb7000cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000cd070 0x7fb7000cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59338/0 (socket says 172.21.15.67:59338) 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 -- 172.21.15.67:0/3606483076 learned_addr learned my addr 172.21.15.67:0/3606483076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb7062bb640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e8004800 0x7fb700150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb705aba640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7000b8050 0x7fb700150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 -- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e8004800 msgr2=0x7fb700150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e8004800 0x7fb700150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 -- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7000b8050 msgr2=0x7fb700150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7000b8050 0x7fb700150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 -- 172.21.15.67:0/3606483076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7000b6fb0 con 0x7fb7000cd070 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb706abc640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000cd070 0x7fb7000cd480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb7000d4050 tx=0x7fb6f000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb6feffd640 1 -- 172.21.15.67:0/3606483076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0019070 con 0x7fb7000cd070 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb6feffd640 1 -- 172.21.15.67:0/3606483076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f0006e20 con 0x7fb7000cd070 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7000b7290 con 0x7fb7000cd070 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb6feffd640 1 -- 172.21.15.67:0/3606483076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0003b90 con 0x7fb7000cd070 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.037+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7000b7850 con 0x7fb7000cd070 2024-04-25T15:25:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.039+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7000b6d10 con 0x7fb7000cd070 2024-04-25T15:25:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.039+0000 7fb6feffd640 1 -- 172.21.15.67:0/3606483076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb6f001c070 con 0x7fb7000cd070 2024-04-25T15:25:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.040+0000 7fb6feffd640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6ec0744b0 0x7fb6ec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.040+0000 7fb6feffd640 1 -- 172.21.15.67:0/3606483076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb6f0094580 con 0x7fb7000cd070 2024-04-25T15:25:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.040+0000 7fb7062bb640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6ec0744b0 0x7fb6ec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.040+0000 7fb7062bb640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6ec0744b0 0x7fb6ec0769c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8001eb0 tx=0x7fb6f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.044+0000 7fb6feffd640 1 -- 172.21.15.67:0/3606483076 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7000b6d10 con 0x7fb7000cd070 2024-04-25T15:25:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.230+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb700050740 con 0x7fb7000cd070 2024-04-25T15:25:57.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.231+0000 7fb6feffd640 1 -- 172.21.15.67:0/3606483076 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fb6f005ed10 con 0x7fb7000cd070 2024-04-25T15:25:57.232 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6ec0744b0 msgr2=0x7fb6ec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6ec0744b0 0x7fb6ec0769c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8001eb0 tx=0x7fb6f8009040 comp rx=0 tx=0).stop 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000cd070 msgr2=0x7fb7000cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000cd070 0x7fb7000cd480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb7000d4050 tx=0x7fb6f000ff70 comp rx=0 tx=0).stop 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 shutdown_connections 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6ec0744b0 0x7fb6ec0769c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7000cd070 0x7fb7000cd480 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7000b8050 0x7fb700150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 --2- 172.21.15.67:0/3606483076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6e8004800 0x7fb700150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 >> 172.21.15.67:0/3606483076 conn(0x7fb7000d5d90 msgr2=0x7fb7001600c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 shutdown_connections 2024-04-25T15:25:57.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.233+0000 7fb70851f640 1 -- 172.21.15.67:0/3606483076 wait complete. 2024-04-25T15:25:57.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:57.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.359+0000 7f834c0ee640 1 Processor -- start 2024-04-25T15:25:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.359+0000 7f834c0ee640 1 -- start start 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f834c0ee640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f834406dc10 0x7f8344160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f834c0ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8344075e80 0x7f83440ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f834c0ee640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83440cf4e0 con 0x7f83440cf0d0 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f834c0ee640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f83440b2310 con 0x7f834406dc10 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f834c0ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f83440b25f0 con 0x7f8344075e80 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f8349e8a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f834406dc10 0x7f8344160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f8349e8a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f834406dc10 0x7f8344160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59342/0 (socket says 172.21.15.67:59342) 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.360+0000 7f8349e8a640 1 -- 172.21.15.67:0/1191818536 learned_addr learned my addr 172.21.15.67:0/1191818536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.361+0000 7f8349689640 1 --2- 172.21.15.67:0/1191818536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8344075e80 0x7f83440ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.361+0000 7f8349e8a640 1 -- 172.21.15.67:0/1191818536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8344075e80 msgr2=0x7f83440ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.361+0000 7f8349e8a640 1 --2- 172.21.15.67:0/1191818536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8344075e80 0x7f83440ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.361+0000 7f8349e8a640 1 -- 172.21.15.67:0/1191818536 >> v1:172.21.15.176:6789/0 conn(0x7f83440cf0d0 legacy=0x7f834406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.361+0000 7f8349e8a640 1 -- 172.21.15.67:0/1191818536 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83440b28d0 con 0x7f834406dc10 2024-04-25T15:25:57.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.361+0000 7f8349689640 1 --2- 172.21.15.67:0/1191818536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8344075e80 0x7f83440ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.361+0000 7f8349e8a640 1 --2- 172.21.15.67:0/1191818536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406dc10 0x7f8344160750 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f833c009870 tx=0x7f833c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f261abd80b52211 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f8348e88640 1 -- 172.21.15.67:0/1191818536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c037070 con 0x7f834406dc10 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f8348e88640 1 -- 172.21.15.67:0/1191818536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833c02ae50 con 0x7f834406dc10 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f8348e88640 1 -- 172.21.15.67:0/1191818536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c032860 con 0x7f834406dc10 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f834c0ee640 1 -- 172.21.15.67:0/1191818536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406dc10 msgr2=0x7f8344160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1191818536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406dc10 0x7f8344160750 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f833c009870 tx=0x7f833c02a300 comp rx=0 tx=0).stop 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f834c0ee640 1 -- 172.21.15.67:0/1191818536 shutdown_connections 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1191818536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8344075e80 0x7f83440ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1191818536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834406dc10 0x7f8344160750 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.362+0000 7f834c0ee640 1 -- 172.21.15.67:0/1191818536 >> 172.21.15.67:0/1191818536 conn(0x7f834409b7d0 msgr2=0x7f83440b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.363+0000 7f834c0ee640 1 -- 172.21.15.67:0/1191818536 shutdown_connections 2024-04-25T15:25:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.363+0000 7f834c0ee640 1 -- 172.21.15.67:0/1191818536 wait complete. 2024-04-25T15:25:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.364+0000 7f834c0ee640 1 Processor -- start 2024-04-25T15:25:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.364+0000 7f834c0ee640 1 -- start start 2024-04-25T15:25:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834c0ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f834406dc10 0x7f83441f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834c0ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8344075e80 0x7f834414f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834c0ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83440cf0d0 0x7f834409f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834c0ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8344160c90 con 0x7f83440cf0d0 2024-04-25T15:25:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834c0ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8344076440 con 0x7f8344075e80 2024-04-25T15:25:57.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834c0ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8344160e10 con 0x7f834406dc10 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83440cf0d0 0x7f834409f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83440cf0d0 0x7f834409f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59348/0 (socket says 172.21.15.67:59348) 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 -- 172.21.15.67:0/1295051942 learned_addr learned my addr 172.21.15.67:0/1295051942 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f8349e8a640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f834406dc10 0x7f83441f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f8349689640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8344075e80 0x7f834414f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 -- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f834406dc10 msgr2=0x7f83441f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f834406dc10 0x7f83441f36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 -- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8344075e80 msgr2=0x7f834414f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8344075e80 0x7f834414f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 -- 172.21.15.67:0/1295051942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83441e5ef0 con 0x7f83440cf0d0 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f834a68b640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83440cf0d0 0x7f834409f9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f834000c680 tx=0x7f834000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.366+0000 7f833a7fc640 1 -- 172.21.15.67:0/1295051942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340003b70 con 0x7f83440cf0d0 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.367+0000 7f833a7fc640 1 -- 172.21.15.67:0/1295051942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8340003d10 con 0x7f83440cf0d0 2024-04-25T15:25:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.367+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83441e61d0 con 0x7f83440cf0d0 2024-04-25T15:25:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.367+0000 7f833a7fc640 1 -- 172.21.15.67:0/1295051942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340005310 con 0x7f83440cf0d0 2024-04-25T15:25:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.367+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83441e6790 con 0x7f83440cf0d0 2024-04-25T15:25:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.367+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8344050740 con 0x7f83440cf0d0 2024-04-25T15:25:57.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.368+0000 7f833a7fc640 1 -- 172.21.15.67:0/1295051942 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f83400054b0 con 0x7f83440cf0d0 2024-04-25T15:25:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.369+0000 7f833a7fc640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83280744e0 0x7f83280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.369+0000 7f833a7fc640 1 -- 172.21.15.67:0/1295051942 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8340094ef0 con 0x7f83440cf0d0 2024-04-25T15:25:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.369+0000 7f8349e8a640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83280744e0 0x7f83280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.369+0000 7f8349e8a640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83280744e0 0x7f83280769f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f833c034000 tx=0x7f833c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.373+0000 7f833a7fc640 1 -- 172.21.15.67:0/1295051942 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83440cf4e0 con 0x7f83440cf0d0 2024-04-25T15:25:57.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.558+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f834409ff30 con 0x7f83440cf0d0 2024-04-25T15:25:57.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.559+0000 7f833a7fc640 1 -- 172.21.15.67:0/1295051942 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f834005f030 con 0x7f83440cf0d0 2024-04-25T15:25:57.560 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83280744e0 msgr2=0x7f83280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83280744e0 0x7f83280769f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f833c034000 tx=0x7f833c02e000 comp rx=0 tx=0).stop 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83440cf0d0 msgr2=0x7f834409f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83440cf0d0 0x7f834409f9f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f834000c680 tx=0x7f834000cb40 comp rx=0 tx=0).stop 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 shutdown_connections 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83280744e0 0x7f83280769f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83440cf0d0 0x7f834409f9f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8344075e80 0x7f834414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 --2- 172.21.15.67:0/1295051942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f834406dc10 0x7f83441f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 >> 172.21.15.67:0/1295051942 conn(0x7f834409b7d0 msgr2=0x7f83440d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:57.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 shutdown_connections 2024-04-25T15:25:57.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.561+0000 7f834c0ee640 1 -- 172.21.15.67:0/1295051942 wait complete. 2024-04-25T15:25:57.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:25:57.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.687+0000 7f3e17692640 1 Processor -- start 2024-04-25T15:25:57.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.687+0000 7f3e17692640 1 -- start start 2024-04-25T15:25:57.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.688+0000 7f3e17692640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 0x7f3e1006e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.688+0000 7f3e17692640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3e1006e590 0x7f3e100d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.688+0000 7f3e17692640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e10161970 con 0x7f3e100d1270 2024-04-25T15:25:57.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.688+0000 7f3e17692640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3e101614e0 con 0x7f3e1006e590 2024-04-25T15:25:57.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.688+0000 7f3e17692640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e100d1a40 con 0x7f3e1015e0c0 2024-04-25T15:25:57.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.688+0000 7f3e15c2f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3e100d1270 0x7f3e100b7800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60104/0 (socket says 172.21.15.67:60104) 2024-04-25T15:25:57.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.688+0000 7f3e15c2f640 1 -- 172.21.15.67:0/1771783527 learned_addr learned my addr 172.21.15.67:0/1771783527 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:57.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e14c2d640 1 --2- 172.21.15.67:0/1771783527 >> v2:172.21.15.176:3300/0 conn(0x7f3e1006e590 0x7f3e100d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e1542e640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 0x7f3e1006e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271916801 0 0) 0x7f3e10161970 con 0x7f3e100d1270 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dfc003520 con 0x7f3e100d1270 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1471656995 0 0) 0x7f3dfc003520 con 0x7f3e100d1270 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 msgr2=0x7f3e1006e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 0x7f3e1006e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 >> v2:172.21.15.176:3300/0 conn(0x7f3e1006e590 msgr2=0x7f3e100d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 --2- 172.21.15.67:0/1771783527 >> v2:172.21.15.176:3300/0 conn(0x7f3e1006e590 0x7f3e100d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1006e9a0 con 0x7f3e100d1270 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f3e080019f0 con 0x7f3e100d1270 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 >> v1:172.21.15.67:6789/0 conn(0x7f3e100d1270 legacy=0x7f3e100b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dfc0039b0 0x7f3dfc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 0x7f3dfc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e101614e0 con 0x7f3dfc0039b0 2024-04-25T15:25:57.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3e100d1a40 con 0x7f3dfc004800 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3dfc00b9c0 con 0x7f3e1015e0c0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e1542e640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dfc0039b0 0x7f3dfc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e14c2d640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.689+0000 7f3e15c2f640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 0x7f3dfc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e1542e640 1 -- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 msgr2=0x7f3dfc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e1542e640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e1015e0c0 0x7f3dfc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e1542e640 1 -- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dfc004800 msgr2=0x7f3dfc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e1542e640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dfc004800 0x7f3dfc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e1542e640 1 -- 172.21.15.67:0/1771783527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc003520 con 0x7f3dfc0039b0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e1542e640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dfc0039b0 0x7f3dfc003e20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009ac0 tx=0x7f3dfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfc36c15aef9817 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c036070 con 0x7f3dfc0039b0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e0c031400 con 0x7f3dfc0039b0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.690+0000 7f3e07fff640 1 -- 172.21.15.67:0/1771783527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c009690 con 0x7f3dfc0039b0 2024-04-25T15:25:57.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.691+0000 7f3e17692640 1 -- 172.21.15.67:0/1771783527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dfc0039b0 msgr2=0x7f3dfc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.691+0000 7f3e17692640 1 --2- 172.21.15.67:0/1771783527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dfc0039b0 0x7f3dfc003e20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009ac0 tx=0x7f3dfc005830 comp rx=0 tx=0).stop 2024-04-25T15:25:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.691+0000 7f3e15c2f640 1 -- 172.21.15.67:0/1771783527 reap_dead start 2024-04-25T15:25:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.691+0000 7f3e17692640 1 -- 172.21.15.67:0/1771783527 shutdown_connections 2024-04-25T15:25:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.691+0000 7f3e17692640 1 -- 172.21.15.67:0/1771783527 >> 172.21.15.67:0/1771783527 conn(0x7f3e100d2800 msgr2=0x7f3e100d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.691+0000 7f3e17692640 1 -- 172.21.15.67:0/1771783527 shutdown_connections 2024-04-25T15:25:57.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.691+0000 7f3e17692640 1 -- 172.21.15.67:0/1771783527 wait complete. 2024-04-25T15:25:57.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.693+0000 7f3e17692640 1 Processor -- start 2024-04-25T15:25:57.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.693+0000 7f3e17692640 1 -- start start 2024-04-25T15:25:57.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.694+0000 7f3e17692640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dfc0039b0 0x7f3e1014c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.694+0000 7f3e17692640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e10149ba0 0x7f3e101eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.694+0000 7f3e17692640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e101eb770 0x7f3e100b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.694+0000 7f3e17692640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10161170 con 0x7f3e101eb770 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.694+0000 7f3e17692640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10160e90 con 0x7f3e10149ba0 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.694+0000 7f3e17692640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e100d1680 con 0x7f3dfc0039b0 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e101eb770 0x7f3e100b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e101eb770 0x7f3e100b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60040/0 (socket says 172.21.15.67:60040) 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 -- 172.21.15.67:0/4200087536 learned_addr learned my addr 172.21.15.67:0/4200087536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e14c2d640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e10149ba0 0x7f3e101eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 -- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dfc0039b0 msgr2=0x7f3e1014c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e1542e640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dfc0039b0 0x7f3e1014c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dfc0039b0 0x7f3e1014c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 -- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e10149ba0 msgr2=0x7f3e101eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e10149ba0 0x7f3e101eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 -- 172.21.15.67:0/4200087536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e101ea190 con 0x7f3e101eb770 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e14c2d640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e10149ba0 0x7f3e101eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e1542e640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dfc0039b0 0x7f3e1014c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e15c2f640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e101eb770 0x7f3e100b7bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3e100b57b0 tx=0x7f3e08007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e05ffb640 1 -- 172.21.15.67:0/4200087536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0800abb0 con 0x7f3e101eb770 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e05ffb640 1 -- 172.21.15.67:0/4200087536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e08018070 con 0x7f3e101eb770 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e05ffb640 1 -- 172.21.15.67:0/4200087536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e080133f0 con 0x7f3e101eb770 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e101ea470 con 0x7f3e101eb770 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.695+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e101eaa30 con 0x7f3e101eb770 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.696+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e10050740 con 0x7f3e101eb770 2024-04-25T15:25:57.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.697+0000 7f3e05ffb640 1 -- 172.21.15.67:0/4200087536 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3e08013590 con 0x7f3e101eb770 2024-04-25T15:25:57.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.697+0000 7f3e05ffb640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3df40744b0 0x7f3df40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:57.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.697+0000 7f3e05ffb640 1 -- 172.21.15.67:0/4200087536 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3e080958d0 con 0x7f3e101eb770 2024-04-25T15:25:57.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.698+0000 7f3e1542e640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3df40744b0 0x7f3df40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:57.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.698+0000 7f3e1542e640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3df40744b0 0x7f3df40769c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c033000 tx=0x7f3e0c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:57.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.702+0000 7f3e05ffb640 1 -- 172.21.15.67:0/4200087536 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e08060060 con 0x7f3e101eb770 2024-04-25T15:25:57.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.889+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3e10162f40 con 0x7f3e101eb770 2024-04-25T15:25:57.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.889+0000 7f3e05ffb640 1 -- 172.21.15.67:0/4200087536 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f3e0805f960 con 0x7f3e101eb770 2024-04-25T15:25:57.890 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:25:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3df40744b0 msgr2=0x7f3df40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3df40744b0 0x7f3df40769c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c033000 tx=0x7f3e0c008030 comp rx=0 tx=0).stop 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e101eb770 msgr2=0x7f3e100b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e101eb770 0x7f3e100b7bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3e100b57b0 tx=0x7f3e08007a90 comp rx=0 tx=0).stop 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 shutdown_connections 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3df40744b0 0x7f3df40769c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e101eb770 0x7f3e100b7bd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e10149ba0 0x7f3e101eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 --2- 172.21.15.67:0/4200087536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dfc0039b0 0x7f3e1014c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 >> 172.21.15.67:0/4200087536 conn(0x7f3e100d2800 msgr2=0x7f3e1015d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 shutdown_connections 2024-04-25T15:25:57.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:57.891+0000 7f3e17692640 1 -- 172.21.15.67:0/4200087536 wait complete. 2024-04-25T15:25:57.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values' 2024-04-25T15:25:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.014+0000 7fbb83f38640 1 Processor -- start 2024-04-25T15:25:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.014+0000 7fbb83f38640 1 -- start start 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb83f38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb7c0d0e00 0x7fbb7c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb83f38640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb83f38640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb7c06cf10 con 0x7fbb7c0b7810 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb83f38640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbb7c0d5a60 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb83f38640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbb7c0d1850 con 0x7fbb7c06c5c0 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb81cd4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb7c0d0e00 0x7fbb7c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb81cd4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb7c0d0e00 0x7fbb7c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60044/0 (socket says 172.21.15.67:60044) 2024-04-25T15:25:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb824d5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb7c0b7810 0x7fbb7c0b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60110/0 (socket says 172.21.15.67:60110) 2024-04-25T15:25:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb81cd4640 1 -- 172.21.15.67:0/961030307 learned_addr learned my addr 172.21.15.67:0/961030307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb814d3640 1 --2- 172.21.15.67:0/961030307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb81cd4640 1 -- 172.21.15.67:0/961030307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 msgr2=0x7fbb7c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb81cd4640 1 --2- 172.21.15.67:0/961030307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb81cd4640 1 -- 172.21.15.67:0/961030307 >> v1:172.21.15.67:6789/0 conn(0x7fbb7c0b7810 legacy=0x7fbb7c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.016+0000 7fbb81cd4640 1 -- 172.21.15.67:0/961030307 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7c160de0 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.017+0000 7fbb81cd4640 1 --2- 172.21.15.67:0/961030307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c0d1210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009870 tx=0x7fbb7402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9818ac0c972e88ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.017+0000 7fbb80cd2640 1 -- 172.21.15.67:0/961030307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74037070 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.017+0000 7fbb80cd2640 1 -- 172.21.15.67:0/961030307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb74032440 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.017+0000 7fbb80cd2640 1 -- 172.21.15.67:0/961030307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74032810 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 -- 172.21.15.67:0/961030307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 msgr2=0x7fbb7c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 --2- 172.21.15.67:0/961030307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c0d1210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009870 tx=0x7fbb7402a270 comp rx=0 tx=0).stop 2024-04-25T15:25:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 -- 172.21.15.67:0/961030307 shutdown_connections 2024-04-25T15:25:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 --2- 172.21.15.67:0/961030307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 --2- 172.21.15.67:0/961030307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c0d1210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 -- 172.21.15.67:0/961030307 >> 172.21.15.67:0/961030307 conn(0x7fbb7c0d5d90 msgr2=0x7fbb7c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 -- 172.21.15.67:0/961030307 shutdown_connections 2024-04-25T15:25:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.018+0000 7fbb83f38640 1 -- 172.21.15.67:0/961030307 wait complete. 2024-04-25T15:25:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.020+0000 7fbb83f38640 1 Processor -- start 2024-04-25T15:25:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.020+0000 7fbb83f38640 1 -- start start 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb83f38640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c0c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb83f38640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb7c0b7810 0x7fbb7c15d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb83f38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c15db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb83f38640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c160ad0 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb83f38640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1613a0 con 0x7fbb7c0b7810 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb83f38640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbb7c0b5320 con 0x7fbb7c06c5c0 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c15db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60060/0 (socket says 172.21.15.67:60060) 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 -- 172.21.15.67:0/2108614378 learned_addr learned my addr 172.21.15.67:0/2108614378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb814d3640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb7c0b7810 0x7fbb7c15d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb81cd4640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c0c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 -- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 msgr2=0x7fbb7c0c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c0c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 -- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb7c0b7810 msgr2=0x7fbb7c15d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb7c0b7810 0x7fbb7c15d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.021+0000 7fbb824d5640 1 -- 172.21.15.67:0/2108614378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7c148c50 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.022+0000 7fbb824d5640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c15db70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00c6f0 tx=0x7fbb6c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.022+0000 7fbb7a7fc640 1 -- 172.21.15.67:0/2108614378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c017070 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.022+0000 7fbb7a7fc640 1 -- 172.21.15.67:0/2108614378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c005da0 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.022+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7c0cb030 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.022+0000 7fbb7a7fc640 1 -- 172.21.15.67:0/2108614378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c00a840 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.022+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7c145e40 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.023+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb7c0cae20 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.024+0000 7fbb7a7fc640 1 -- 172.21.15.67:0/2108614378 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbb6c01a070 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.024+0000 7fbb7a7fc640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbb680744b0 0x7fbb680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.024+0000 7fbb7a7fc640 1 -- 172.21.15.67:0/2108614378 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbb6c094650 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.024+0000 7fbb81cd4640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbb680744b0 0x7fbb680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.025+0000 7fbb81cd4640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbb680744b0 0x7fbb680769c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb74034000 tx=0x7fbb7402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.029+0000 7fbb7a7fc640 1 -- 172.21.15.67:0/2108614378 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb7c06cf10 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.285+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]} v 0) -- 0x7fbb7c050740 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.399+0000 7fbb7a7fc640 1 -- 172.21.15.67:0/2108614378 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]}]=0 v48) ==== 163+0+0 (secure 0 0 0) 0x7fbb6c05e820 con 0x7fbb7c0d0e00 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.401+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbb680744b0 msgr2=0x7fbb680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.401+0000 7fbb83f38640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbb680744b0 0x7fbb680769c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbb74034000 tx=0x7fbb7402e000 comp rx=0 tx=0).stop 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.401+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 msgr2=0x7fbb7c15db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.401+0000 7fbb83f38640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c15db70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00c6f0 tx=0x7fbb6c00cbb0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 shutdown_connections 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbb680744b0 0x7fbb680769c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb7c0d0e00 0x7fbb7c15db70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb7c0b7810 0x7fbb7c15d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 --2- 172.21.15.67:0/2108614378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb7c06c5c0 0x7fbb7c0c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 >> 172.21.15.67:0/2108614378 conn(0x7fbb7c0d5d90 msgr2=0x7fbb7c1643e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 shutdown_connections 2024-04-25T15:25:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.402+0000 7fbb83f38640 1 -- 172.21.15.67:0/2108614378 wait complete. 2024-04-25T15:25:58.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:25:58.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:25:58.489 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:25:58.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.528+0000 7f1f19972640 1 Processor -- start 2024-04-25T15:25:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.528+0000 7f1f19972640 1 -- start start 2024-04-25T15:25:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f19972640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f140d3070 0x7f1f140b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f19972640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f140ce770 0x7f1f14160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f19972640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f140d2040 con 0x7f1f1406e740 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f19972640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1f140d3480 con 0x7f1f140d3070 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f19972640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140d22b0 con 0x7f1f140ce770 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f12ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f140d3070 0x7f1f140b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f12ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1f140d3070 0x7f1f140b5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60066/0 (socket says 172.21.15.67:60066) 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.529+0000 7f1f12ffd640 1 -- 172.21.15.67:0/115673444 learned_addr learned my addr 172.21.15.67:0/115673444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.530+0000 7f1f137fe640 1 --2- 172.21.15.67:0/115673444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f140ce770 0x7f1f14160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.530+0000 7f1f12ffd640 1 -- 172.21.15.67:0/115673444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f140ce770 msgr2=0x7f1f14160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.530+0000 7f1f12ffd640 1 --2- 172.21.15.67:0/115673444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f140ce770 0x7f1f14160aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.530+0000 7f1f12ffd640 1 -- 172.21.15.67:0/115673444 >> v1:172.21.15.176:6789/0 conn(0x7f1f1406e740 legacy=0x7f1f140ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.530+0000 7f1f12ffd640 1 -- 172.21.15.67:0/115673444 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f140d2590 con 0x7f1f140d3070 2024-04-25T15:25:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.530+0000 7f1f137fe640 1 --2- 172.21.15.67:0/115673444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f140ce770 0x7f1f14160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.530+0000 7f1f12ffd640 1 --2- 172.21.15.67:0/115673444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f140b5f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009870 tx=0x7f1f0402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6609087e1df69f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f11ffb640 1 -- 172.21.15.67:0/115673444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04037070 con 0x7f1f140d3070 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f11ffb640 1 -- 172.21.15.67:0/115673444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f04032440 con 0x7f1f140d3070 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f11ffb640 1 -- 172.21.15.67:0/115673444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04032810 con 0x7f1f140d3070 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 -- 172.21.15.67:0/115673444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 msgr2=0x7f1f140b5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 --2- 172.21.15.67:0/115673444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f140b5f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009870 tx=0x7f1f0402a270 comp rx=0 tx=0).stop 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 -- 172.21.15.67:0/115673444 shutdown_connections 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 --2- 172.21.15.67:0/115673444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f140ce770 0x7f1f14160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 --2- 172.21.15.67:0/115673444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f140b5f80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 -- 172.21.15.67:0/115673444 >> 172.21.15.67:0/115673444 conn(0x7f1f140d3aa0 msgr2=0x7f1f140d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 -- 172.21.15.67:0/115673444 shutdown_connections 2024-04-25T15:25:58.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.531+0000 7f1f19972640 1 -- 172.21.15.67:0/115673444 wait complete. 2024-04-25T15:25:58.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.533+0000 7f1f19972640 1 Processor -- start 2024-04-25T15:25:58.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.533+0000 7f1f19972640 1 -- start start 2024-04-25T15:25:58.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f19972640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f1406e740 0x7f1f141f9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f19972640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f140ce770 0x7f1f141fa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f19972640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f141fa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f19972640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1f140ceb80 con 0x7f1f140d3070 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f19972640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1f140d2b50 con 0x7f1f140ce770 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f19972640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140ced00 con 0x7f1f1406e740 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f141fa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f141fa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60074/0 (socket says 172.21.15.67:60074) 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 -- 172.21.15.67:0/3584287712 learned_addr learned my addr 172.21.15.67:0/3584287712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f127fc640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f140ce770 0x7f1f141fa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f12ffd640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f1406e740 0x7f1f141f9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 -- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f1406e740 msgr2=0x7f1f141f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f1406e740 0x7f1f141f9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 -- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f140ce770 msgr2=0x7f1f141fa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f140ce770 0x7f1f141fa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 -- 172.21.15.67:0/3584287712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f141f7920 con 0x7f1f140d3070 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f127fc640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f140ce770 0x7f1f141fa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f12ffd640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f1406e740 0x7f1f141f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.535+0000 7f1f137fe640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f141fa860 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00cdb0 tx=0x7f1f0c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.536+0000 7f1efbfff640 1 -- 172.21.15.67:0/3584287712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c00f680 con 0x7f1f140d3070 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.536+0000 7f1efbfff640 1 -- 172.21.15.67:0/3584287712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f0c00f840 con 0x7f1f140d3070 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.536+0000 7f1efbfff640 1 -- 172.21.15.67:0/3584287712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c00fc30 con 0x7f1f140d3070 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.536+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f14151d00 con 0x7f1f140d3070 2024-04-25T15:25:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.536+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f141522c0 con 0x7f1f140d3070 2024-04-25T15:25:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.537+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f14050740 con 0x7f1f140d3070 2024-04-25T15:25:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.537+0000 7f1efbfff640 1 -- 172.21.15.67:0/3584287712 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1f0c0213f0 con 0x7f1f140d3070 2024-04-25T15:25:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.538+0000 7f1efbfff640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1f000744b0 0x7f1f000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.538+0000 7f1efbfff640 1 -- 172.21.15.67:0/3584287712 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1f0c095ea0 con 0x7f1f140d3070 2024-04-25T15:25:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.538+0000 7f1f12ffd640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1f000744b0 0x7f1f000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.539+0000 7f1f12ffd640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1f000744b0 0x7f1f000769c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f04034000 tx=0x7f1f0402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:58.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.542+0000 7f1efbfff640 1 -- 172.21.15.67:0/3584287712 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f0c014030 con 0x7f1f140d3070 2024-04-25T15:25:58.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.729+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f1f140d4920 con 0x7f1f140d3070 2024-04-25T15:25:58.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.732+0000 7f1efbfff640 1 -- 172.21.15.67:0/3584287712 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f1f0c060750 con 0x7f1f140d3070 2024-04-25T15:25:58.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1f000744b0 msgr2=0x7f1f000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1f000744b0 0x7f1f000769c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f04034000 tx=0x7f1f0402e000 comp rx=0 tx=0).stop 2024-04-25T15:25:58.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 msgr2=0x7f1f141fa860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:58.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f141fa860 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00cdb0 tx=0x7f1f0c00a650 comp rx=0 tx=0).stop 2024-04-25T15:25:58.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 shutdown_connections 2024-04-25T15:25:58.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1f000744b0 0x7f1f000769c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f140d3070 0x7f1f141fa860 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f140ce770 0x7f1f141fa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 --2- 172.21.15.67:0/3584287712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f1406e740 0x7f1f141f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:58.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 >> 172.21.15.67:0/3584287712 conn(0x7f1f140d3aa0 msgr2=0x7f1f1406d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:58.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 shutdown_connections 2024-04-25T15:25:58.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:58.734+0000 7f1f19972640 1 -- 172.21.15.67:0/3584287712 wait complete. 2024-04-25T15:25:58.746 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:25:58.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:25:58.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:25:58.873 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:25:58.874 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:58.874 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:25:58.889 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:58.889 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:25:58.945 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:25:58.945 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:25:59.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:25:59.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.152+0000 7f88e3196640 1 Processor -- start 2024-04-25T15:25:59.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.152+0000 7f88e3196640 1 -- start start 2024-04-25T15:25:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e3196640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88dc0d0e00 0x7f88dc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e3196640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f88dc0d4a50 0x7f88dc0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e3196640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88dc0d1850 con 0x7f88dc06c5c0 2024-04-25T15:25:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e3196640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f88dc160de0 con 0x7f88dc0d0e00 2024-04-25T15:25:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e3196640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f88dc1610c0 con 0x7f88dc0d4a50 2024-04-25T15:25:59.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e0f32640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88dc0d0e00 0x7f88dc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e0f32640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88dc0d0e00 0x7f88dc0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60082/0 (socket says 172.21.15.67:60082) 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88dbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f88dc06c5c0 0x7f88dc06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60112/0 (socket says 172.21.15.67:60112) 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.153+0000 7f88e0f32640 1 -- 172.21.15.67:0/2213263951 learned_addr learned my addr 172.21.15.67:0/2213263951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.154+0000 7f88e1733640 1 --2- 172.21.15.67:0/2213263951 >> v2:172.21.15.176:3300/0 conn(0x7f88dc0d4a50 0x7f88dc0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.154+0000 7f88e0f32640 1 -- 172.21.15.67:0/2213263951 >> v2:172.21.15.176:3300/0 conn(0x7f88dc0d4a50 msgr2=0x7f88dc0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.154+0000 7f88e0f32640 1 --2- 172.21.15.67:0/2213263951 >> v2:172.21.15.176:3300/0 conn(0x7f88dc0d4a50 0x7f88dc0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.154+0000 7f88e0f32640 1 -- 172.21.15.67:0/2213263951 >> v1:172.21.15.67:6789/0 conn(0x7f88dc06c5c0 legacy=0x7f88dc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.154+0000 7f88e0f32640 1 -- 172.21.15.67:0/2213263951 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc1613a0 con 0x7f88dc0d0e00 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.154+0000 7f88e1733640 1 --2- 172.21.15.67:0/2213263951 >> v2:172.21.15.176:3300/0 conn(0x7f88dc0d4a50 0x7f88dc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.154+0000 7f88e0f32640 1 --2- 172.21.15.67:0/2213263951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d0e00 0x7f88dc0d1210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0d1750 tx=0x7f88d002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89801ba84d314f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88db7fe640 1 -- 172.21.15.67:0/2213263951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0037070 con 0x7f88dc0d0e00 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88db7fe640 1 -- 172.21.15.67:0/2213263951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d0032440 con 0x7f88dc0d0e00 2024-04-25T15:25:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88db7fe640 1 -- 172.21.15.67:0/2213263951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d0032810 con 0x7f88dc0d0e00 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 -- 172.21.15.67:0/2213263951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d0e00 msgr2=0x7f88dc0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 --2- 172.21.15.67:0/2213263951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d0e00 0x7f88dc0d1210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0d1750 tx=0x7f88d002a200 comp rx=0 tx=0).stop 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 -- 172.21.15.67:0/2213263951 shutdown_connections 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 --2- 172.21.15.67:0/2213263951 >> v2:172.21.15.176:3300/0 conn(0x7f88dc0d4a50 0x7f88dc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 --2- 172.21.15.67:0/2213263951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d0e00 0x7f88dc0d1210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 -- 172.21.15.67:0/2213263951 >> 172.21.15.67:0/2213263951 conn(0x7f88dc0d5d90 msgr2=0x7f88dc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 -- 172.21.15.67:0/2213263951 shutdown_connections 2024-04-25T15:25:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.155+0000 7f88e3196640 1 -- 172.21.15.67:0/2213263951 wait complete. 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.157+0000 7f88e3196640 1 Processor -- start 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.157+0000 7f88e3196640 1 -- start start 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e3196640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88dc06c5c0 0x7f88dc0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e3196640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88dc0d0e00 0x7f88dc14ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e3196640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d4a50 0x7f88dc14f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e3196640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc164e00 con 0x7f88dc0d4a50 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e3196640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc0b5320 con 0x7f88dc0d0e00 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e3196640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f88dc164f80 con 0x7f88dc06c5c0 2024-04-25T15:25:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d4a50 0x7f88dc14f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d4a50 0x7f88dc14f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60088/0 (socket says 172.21.15.67:60088) 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 -- 172.21.15.67:0/3143760166 learned_addr learned my addr 172.21.15.67:0/3143760166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88dbfff640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88dc0d0e00 0x7f88dc14ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e0f32640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88dc06c5c0 0x7f88dc0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 -- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88dc06c5c0 msgr2=0x7f88dc0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88dc06c5c0 0x7f88dc0cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 -- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88dc0d0e00 msgr2=0x7f88dc14ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88dc0d0e00 0x7f88dc14ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 -- 172.21.15.67:0/3143760166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc14d750 con 0x7f88dc0d4a50 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.159+0000 7f88e1733640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d4a50 0x7f88dc14f500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f88d400cd50 tx=0x7f88d400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.160+0000 7f88d97fa640 1 -- 172.21.15.67:0/3143760166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d400f680 con 0x7f88dc0d4a50 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.160+0000 7f88d97fa640 1 -- 172.21.15.67:0/3143760166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d400f840 con 0x7f88dc0d4a50 2024-04-25T15:25:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.160+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88dc14da30 con 0x7f88dc0d4a50 2024-04-25T15:25:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.160+0000 7f88d97fa640 1 -- 172.21.15.67:0/3143760166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d400fc30 con 0x7f88dc0d4a50 2024-04-25T15:25:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.160+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88dc14dff0 con 0x7f88dc0d4a50 2024-04-25T15:25:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.161+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88dc050740 con 0x7f88dc0d4a50 2024-04-25T15:25:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.161+0000 7f88d97fa640 1 -- 172.21.15.67:0/3143760166 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f88d40213f0 con 0x7f88dc0d4a50 2024-04-25T15:25:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.162+0000 7f88d97fa640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88c00744b0 0x7f88c00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.162+0000 7f88d97fa640 1 -- 172.21.15.67:0/3143760166 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f88d4095e20 con 0x7f88dc0d4a50 2024-04-25T15:25:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.162+0000 7f88e0f32640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88c00744b0 0x7f88c00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.163+0000 7f88e0f32640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88c00744b0 0x7f88c00769c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88d0034000 tx=0x7f88d002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.167+0000 7f88d97fa640 1 -- 172.21.15.67:0/3143760166 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f88d4014030 con 0x7f88dc0d4a50 2024-04-25T15:25:59.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.353+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88dc0b5790 con 0x7f88dc0d4a50 2024-04-25T15:25:59.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.354+0000 7f88d97fa640 1 -- 172.21.15.67:0/3143760166 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f88d405fff0 con 0x7f88dc0d4a50 2024-04-25T15:25:59.355 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:25:59.355 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:25:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88c00744b0 msgr2=0x7f88c00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88c00744b0 0x7f88c00769c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88d0034000 tx=0x7f88d002e000 comp rx=0 tx=0).stop 2024-04-25T15:25:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d4a50 msgr2=0x7f88dc14f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d4a50 0x7f88dc14f500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f88d400cd50 tx=0x7f88d400a650 comp rx=0 tx=0).stop 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 shutdown_connections 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88c00744b0 0x7f88c00769c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88dc0d4a50 0x7f88dc14f500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88dc0d0e00 0x7f88dc14ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 --2- 172.21.15.67:0/3143760166 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f88dc06c5c0 0x7f88dc0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 >> 172.21.15.67:0/3143760166 conn(0x7f88dc0d5d90 msgr2=0x7f88dc0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 shutdown_connections 2024-04-25T15:25:59.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.356+0000 7f88e3196640 1 -- 172.21.15.67:0/3143760166 wait complete. 2024-04-25T15:25:59.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:25:59.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:25:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.485+0000 7f15703b4640 1 Processor -- start 2024-04-25T15:25:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.485+0000 7f15703b4640 1 -- start start 2024-04-25T15:25:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f15703b4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f15680b2e60 0x7f15680d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f15703b4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1568164020 0x7f1568164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f15703b4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1568164970 con 0x7f15680d1e20 2024-04-25T15:25:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f15703b4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f15680d23f0 con 0x7f1568164020 2024-04-25T15:25:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f15703b4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f15680d26d0 con 0x7f15680b2e60 2024-04-25T15:25:59.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156e951640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1568164020 0x7f1568164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156e951640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1568164020 0x7f1568164430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60090/0 (socket says 172.21.15.67:60090) 2024-04-25T15:25:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156e951640 1 -- 172.21.15.67:0/1657210947 learned_addr learned my addr 172.21.15.67:0/1657210947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156d94f640 1 --2- 172.21.15.67:0/1657210947 >> v2:172.21.15.176:3300/0 conn(0x7f15680b2e60 0x7f15680d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156e951640 1 -- 172.21.15.67:0/1657210947 >> v2:172.21.15.176:3300/0 conn(0x7f15680b2e60 msgr2=0x7f15680d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156e951640 1 --2- 172.21.15.67:0/1657210947 >> v2:172.21.15.176:3300/0 conn(0x7f15680b2e60 0x7f15680d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156e951640 1 -- 172.21.15.67:0/1657210947 >> v1:172.21.15.176:6789/0 conn(0x7f15680d1e20 legacy=0x7f156808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.487+0000 7f156e951640 1 -- 172.21.15.67:0/1657210947 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15680d29b0 con 0x7f1568164020 2024-04-25T15:25:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.488+0000 7f156e951640 1 --2- 172.21.15.67:0/1657210947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1568164020 0x7f1568164430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1568074650 tx=0x7f155802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c807edf2e1117eea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.488+0000 7f156d14e640 1 -- 172.21.15.67:0/1657210947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558036070 con 0x7f1568164020 2024-04-25T15:25:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.488+0000 7f156d14e640 1 -- 172.21.15.67:0/1657210947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1558031600 con 0x7f1568164020 2024-04-25T15:25:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.488+0000 7f156d14e640 1 -- 172.21.15.67:0/1657210947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15580319f0 con 0x7f1568164020 2024-04-25T15:25:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 -- 172.21.15.67:0/1657210947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1568164020 msgr2=0x7f1568164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 --2- 172.21.15.67:0/1657210947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1568164020 0x7f1568164430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1568074650 tx=0x7f155802a7c0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 -- 172.21.15.67:0/1657210947 shutdown_connections 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 --2- 172.21.15.67:0/1657210947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1568164020 0x7f1568164430 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 --2- 172.21.15.67:0/1657210947 >> v2:172.21.15.176:3300/0 conn(0x7f15680b2e60 0x7f15680d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 -- 172.21.15.67:0/1657210947 >> 172.21.15.67:0/1657210947 conn(0x7f15680d0530 msgr2=0x7f15680d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 -- 172.21.15.67:0/1657210947 shutdown_connections 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.489+0000 7f15703b4640 1 -- 172.21.15.67:0/1657210947 wait complete. 2024-04-25T15:25:59.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.491+0000 7f15703b4640 1 Processor -- start 2024-04-25T15:25:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.491+0000 7f15703b4640 1 -- start start 2024-04-25T15:25:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f15703b4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15680b2e60 0x7f15680cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f15703b4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15680d1e20 0x7f15680ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f15703b4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1568164020 0x7f1568160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f15703b4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15680ce070 con 0x7f15680b2e60 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f15703b4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15680d2f70 con 0x7f1568164020 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f15703b4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15680ce1f0 con 0x7f15680d1e20 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f156e150640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15680b2e60 0x7f15680cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f156e150640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15680b2e60 0x7f15680cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60098/0 (socket says 172.21.15.67:60098) 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.492+0000 7f156e150640 1 -- 172.21.15.67:0/566578948 learned_addr learned my addr 172.21.15.67:0/566578948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156d94f640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15680d1e20 0x7f15680ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e150640 1 -- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15680d1e20 msgr2=0x7f15680ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e150640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15680d1e20 0x7f15680ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e150640 1 -- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1568164020 msgr2=0x7f1568160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e951640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1568164020 0x7f1568160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e150640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1568164020 0x7f1568160b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e150640 1 -- 172.21.15.67:0/566578948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15680b8700 con 0x7f15680b2e60 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156d94f640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15680d1e20 0x7f15680ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:59.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e951640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1568164020 0x7f1568160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:25:59.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f156e150640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15680b2e60 0x7f15680cc560 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f156000c3f0 tx=0x7f156000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f1566ffd640 1 -- 172.21.15.67:0/566578948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15600052d0 con 0x7f15680b2e60 2024-04-25T15:25:59.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f1566ffd640 1 -- 172.21.15.67:0/566578948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1560005490 con 0x7f15680b2e60 2024-04-25T15:25:59.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f1566ffd640 1 -- 172.21.15.67:0/566578948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560007260 con 0x7f15680b2e60 2024-04-25T15:25:59.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15680cb7a0 con 0x7f15680b2e60 2024-04-25T15:25:59.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.493+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15680cbd60 con 0x7f15680b2e60 2024-04-25T15:25:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.494+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1568050740 con 0x7f15680b2e60 2024-04-25T15:25:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.495+0000 7f1566ffd640 1 -- 172.21.15.67:0/566578948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1560007400 con 0x7f15680b2e60 2024-04-25T15:25:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.495+0000 7f1566ffd640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744b0 0x7f15540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.495+0000 7f1566ffd640 1 -- 172.21.15.67:0/566578948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f15600945c0 con 0x7f15680b2e60 2024-04-25T15:25:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.495+0000 7f156d94f640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744b0 0x7f15540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.496+0000 7f156d94f640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744b0 0x7f15540769c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f155c006fd0 tx=0x7f155c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.500+0000 7f1566ffd640 1 -- 172.21.15.67:0/566578948 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1568164970 con 0x7f15680b2e60 2024-04-25T15:25:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.685+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f156806ebd0 con 0x7f15680b2e60 2024-04-25T15:25:59.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.685+0000 7f1566ffd640 1 -- 172.21.15.67:0/566578948 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f156005ede0 con 0x7f15680b2e60 2024-04-25T15:25:59.686 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:25:59.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744b0 msgr2=0x7f15540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744b0 0x7f15540769c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f155c006fd0 tx=0x7f155c008040 comp rx=0 tx=0).stop 2024-04-25T15:25:59.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15680b2e60 msgr2=0x7f15680cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15680b2e60 0x7f15680cc560 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f156000c3f0 tx=0x7f156000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 shutdown_connections 2024-04-25T15:25:59.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15540744b0 0x7f15540769c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1568164020 0x7f1568160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15680d1e20 0x7f15680ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 --2- 172.21.15.67:0/566578948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15680b2e60 0x7f15680cc560 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 >> 172.21.15.67:0/566578948 conn(0x7f15680d0530 msgr2=0x7f15680cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:59.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 shutdown_connections 2024-04-25T15:25:59.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.687+0000 7f15703b4640 1 -- 172.21.15.67:0/566578948 wait complete. 2024-04-25T15:25:59.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:25:59.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.814+0000 7f334551f640 1 Processor -- start 2024-04-25T15:25:59.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.815+0000 7f334551f640 1 -- start start 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f334551f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f334015d240 0x7f334006d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f334551f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 0x7f33400b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f334551f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33400b80b0 con 0x7f334015e170 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f334551f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f334015e650 con 0x7f334015d240 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f334551f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f33400cf320 con 0x7f33400b7760 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f333effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f334015e170 0x7f33400b3d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60124/0 (socket says 172.21.15.67:60124) 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f333effd640 1 -- 172.21.15.67:0/684906816 learned_addr learned my addr 172.21.15.67:0/684906816 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:25:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f333e7fc640 1 --2- 172.21.15.67:0/684906816 >> v2:172.21.15.176:3300/0 conn(0x7f334015d240 0x7f334006d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f333f7fe640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 0x7f33400b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133024162 0 0) 0x7f3338001110 con 0x7f334015e170 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.816+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3328003520 con 0x7f334015e170 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3269622207 0 0) 0x7f3328003520 con 0x7f334015e170 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 msgr2=0x7f33400b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 0x7f33400b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 >> v2:172.21.15.176:3300/0 conn(0x7f334015d240 msgr2=0x7f334006d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 --2- 172.21.15.67:0/684906816 >> v2:172.21.15.176:3300/0 conn(0x7f334015d240 0x7f334006d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33400cf600 con 0x7f334015e170 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f3338001840 con 0x7f334015e170 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 >> v1:172.21.15.67:6789/0 conn(0x7f334015e170 legacy=0x7f33400b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33280039b0 0x7f3328003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 0x7f3328004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004d40 0x7f3328005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f334015e650 con 0x7f3328004d40 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f33400cf320 con 0x7f33280039b0 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f332800b9c0 con 0x7f33400b7760 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333e7fc640 1 --2- 172.21.15.67:0/684906816 >> v2:172.21.15.176:3300/0 conn(0x7f334015d240 0x7f334006d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333f7fe640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004d40 0x7f3328005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333effd640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33280039b0 0x7f3328003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333e7fc640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 0x7f3328004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333f7fe640 1 -- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 msgr2=0x7f3328004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333f7fe640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33400b7760 0x7f3328004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333f7fe640 1 -- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33280039b0 msgr2=0x7f3328003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333f7fe640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33280039b0 0x7f3328003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.817+0000 7f333f7fe640 1 -- 172.21.15.67:0/684906816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3328003520 con 0x7f3328004d40 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.818+0000 7f333f7fe640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004d40 0x7f3328005150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3330009f90 tx=0x7f3328005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1495f6eab0b3f7b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.818+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330009c10 con 0x7f3328004d40 2024-04-25T15:25:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.818+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3330036070 con 0x7f3328004d40 2024-04-25T15:25:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.818+0000 7f333dffb640 1 -- 172.21.15.67:0/684906816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330031850 con 0x7f3328004d40 2024-04-25T15:25:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.819+0000 7f334551f640 1 -- 172.21.15.67:0/684906816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004d40 msgr2=0x7f3328005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.819+0000 7f334551f640 1 --2- 172.21.15.67:0/684906816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3328004d40 0x7f3328005150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3330009f90 tx=0x7f3328005830 comp rx=0 tx=0).stop 2024-04-25T15:25:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.819+0000 7f333f7fe640 1 -- 172.21.15.67:0/684906816 reap_dead start 2024-04-25T15:25:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.819+0000 7f334551f640 1 -- 172.21.15.67:0/684906816 shutdown_connections 2024-04-25T15:25:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.819+0000 7f334551f640 1 -- 172.21.15.67:0/684906816 >> 172.21.15.67:0/684906816 conn(0x7f334009b7d0 msgr2=0x7f334009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.819+0000 7f334551f640 1 -- 172.21.15.67:0/684906816 shutdown_connections 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.819+0000 7f334551f640 1 -- 172.21.15.67:0/684906816 wait complete. 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.821+0000 7f334551f640 1 Processor -- start 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.821+0000 7f334551f640 1 -- start start 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f334551f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3328004d40 0x7f33401f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f334551f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3340155430 0x7f33401f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f334551f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33401f66a0 0x7f33400b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f334551f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f334006dc10 con 0x7f33401f66a0 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f334551f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3340160ce0 con 0x7f3328004d40 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f334551f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f334006dd90 con 0x7f3340155430 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f333f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33401f66a0 0x7f33400b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f333f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33401f66a0 0x7f33400b1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60110/0 (socket says 172.21.15.67:60110) 2024-04-25T15:25:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f333f7fe640 1 -- 172.21.15.67:0/2240200203 learned_addr learned my addr 172.21.15.67:0/2240200203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f333effd640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3328004d40 0x7f33401f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.822+0000 7f333e7fc640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3340155430 0x7f33401f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f333f7fe640 1 -- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3340155430 msgr2=0x7f33401f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f333f7fe640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3340155430 0x7f33401f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f333f7fe640 1 -- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3328004d40 msgr2=0x7f33401f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f333f7fe640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3328004d40 0x7f33401f7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f333f7fe640 1 -- 172.21.15.67:0/2240200203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33400b2840 con 0x7f33401f66a0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f333f7fe640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33401f66a0 0x7f33400b1850 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3330033040 tx=0x7f3330009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f33277fe640 1 -- 172.21.15.67:0/2240200203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330009c10 con 0x7f33401f66a0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f33277fe640 1 -- 172.21.15.67:0/2240200203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3330003990 con 0x7f33401f66a0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33401f7bc0 con 0x7f33401f66a0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f33277fe640 1 -- 172.21.15.67:0/2240200203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330003d80 con 0x7f33401f66a0 2024-04-25T15:25:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.823+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33401f6ba0 con 0x7f33401f66a0 2024-04-25T15:25:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.824+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3340050740 con 0x7f33401f66a0 2024-04-25T15:25:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.825+0000 7f33277fe640 1 -- 172.21.15.67:0/2240200203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3330005d10 con 0x7f33401f66a0 2024-04-25T15:25:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.825+0000 7f33277fe640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f332c0744e0 0x7f332c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:25:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.825+0000 7f33277fe640 1 -- 172.21.15.67:0/2240200203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f33300b6b00 con 0x7f33401f66a0 2024-04-25T15:25:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.826+0000 7f333effd640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f332c0744e0 0x7f332c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:25:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.826+0000 7f333effd640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f332c0744e0 0x7f332c0769f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f33380012f0 tx=0x7f3338009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:25:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:25:59.830+0000 7f33277fe640 1 -- 172.21.15.67:0/2240200203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f333004a060 con 0x7f33401f66a0 2024-04-25T15:26:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.013+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f33400d3910 con 0x7f33401f66a0 2024-04-25T15:26:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.013+0000 7f33277fe640 1 -- 172.21.15.67:0/2240200203 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f333002e050 con 0x7f33401f66a0 2024-04-25T15:26:00.014 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f332c0744e0 msgr2=0x7f332c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f332c0744e0 0x7f332c0769f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f33380012f0 tx=0x7f3338009040 comp rx=0 tx=0).stop 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33401f66a0 msgr2=0x7f33400b1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33401f66a0 0x7f33400b1850 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3330033040 tx=0x7f3330009750 comp rx=0 tx=0).stop 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 shutdown_connections 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f332c0744e0 0x7f332c0769f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33401f66a0 0x7f33400b1850 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3340155430 0x7f33401f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 --2- 172.21.15.67:0/2240200203 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3328004d40 0x7f33401f7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 >> 172.21.15.67:0/2240200203 conn(0x7f334009b7d0 msgr2=0x7f3340071360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 shutdown_connections 2024-04-25T15:26:00.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.015+0000 7f334551f640 1 -- 172.21.15.67:0/2240200203 wait complete. 2024-04-25T15:26:00.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:00.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.141+0000 7f97fda1a640 1 Processor -- start 2024-04-25T15:26:00.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.141+0000 7f97fda1a640 1 -- start start 2024-04-25T15:26:00.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.142+0000 7f97fda1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d0de0 0x7f97f80d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.142+0000 7f97fda1a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f80d1830 con 0x7f97f80d4a30 2024-04-25T15:26:00.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.142+0000 7f97fda1a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f8160dc0 con 0x7f97f806c5c0 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.142+0000 7f97fda1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97f81610a0 con 0x7f97f80d0de0 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f97f80d4a30 0x7f97f80d4e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60136/0 (socket says 172.21.15.67:60136) 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f77fe640 1 -- 172.21.15.67:0/2704429744 learned_addr learned my addr 172.21.15.67:0/2704429744 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f6ffd640 1 --2- 172.21.15.67:0/2704429744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d0de0 0x7f97f80d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f5ffb640 1 -- 172.21.15.67:0/2704429744 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 608128916 0 0) 0x7f97f80d1830 con 0x7f97f80d4a30 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f5ffb640 1 -- 172.21.15.67:0/2704429744 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97e4003520 con 0x7f97f80d4a30 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f5ffb640 1 -- 172.21.15.67:0/2704429744 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736759551 0 0) 0x7f97f8160dc0 con 0x7f97f806c5c0 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f5ffb640 1 -- 172.21.15.67:0/2704429744 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97f80d1830 con 0x7f97f806c5c0 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f6ffd640 1 -- 172.21.15.67:0/2704429744 >> v1:172.21.15.176:6789/0 conn(0x7f97f806c5c0 legacy=0x7f97f806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f6ffd640 1 -- 172.21.15.67:0/2704429744 >> v1:172.21.15.67:6789/0 conn(0x7f97f80d4a30 legacy=0x7f97f80d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.143+0000 7f97f6ffd640 1 -- 172.21.15.67:0/2704429744 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f8161380 con 0x7f97f80d0de0 2024-04-25T15:26:00.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.144+0000 7f97f6ffd640 1 --2- 172.21.15.67:0/2704429744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d0de0 0x7f97f80d11f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f97f00097d0 tx=0x7f97f002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd416a6bce71508 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:00.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.144+0000 7f97f5ffb640 1 -- 172.21.15.67:0/2704429744 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0037070 con 0x7f97f80d0de0 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.144+0000 7f97f5ffb640 1 -- 172.21.15.67:0/2704429744 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97f0032440 con 0x7f97f80d0de0 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.144+0000 7f97f5ffb640 1 -- 172.21.15.67:0/2704429744 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00327f0 con 0x7f97f80d0de0 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.145+0000 7f97fda1a640 1 -- 172.21.15.67:0/2704429744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d0de0 msgr2=0x7f97f80d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.145+0000 7f97fda1a640 1 --2- 172.21.15.67:0/2704429744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d0de0 0x7f97f80d11f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f97f00097d0 tx=0x7f97f002a210 comp rx=0 tx=0).stop 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.145+0000 7f97fda1a640 1 -- 172.21.15.67:0/2704429744 shutdown_connections 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.145+0000 7f97fda1a640 1 --2- 172.21.15.67:0/2704429744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d0de0 0x7f97f80d11f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.145+0000 7f97fda1a640 1 -- 172.21.15.67:0/2704429744 >> 172.21.15.67:0/2704429744 conn(0x7f97f80d5d70 msgr2=0x7f97f80d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.145+0000 7f97fda1a640 1 -- 172.21.15.67:0/2704429744 shutdown_connections 2024-04-25T15:26:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.145+0000 7f97fda1a640 1 -- 172.21.15.67:0/2704429744 wait complete. 2024-04-25T15:26:00.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.147+0000 7f97fda1a640 1 Processor -- start 2024-04-25T15:26:00.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.147+0000 7f97fda1a640 1 -- start start 2024-04-25T15:26:00.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97fda1a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97f806c5c0 0x7f97f815de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97fda1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97f80d0de0 0x7f97f80cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97fda1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d4a30 0x7f97f80c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97fda1a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97f8164de0 con 0x7f97f80d0de0 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97fda1a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f97f80b5300 con 0x7f97f806c5c0 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97fda1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97f8164f60 con 0x7f97f80d4a30 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97f67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97f80d0de0 0x7f97f80cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97f67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97f80d0de0 0x7f97f80cebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60120/0 (socket says 172.21.15.67:60120) 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97f67fc640 1 -- 172.21.15.67:0/3297073095 learned_addr learned my addr 172.21.15.67:0/3297073095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97f77fe640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d4a30 0x7f97f80c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.148+0000 7f97f6ffd640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97f806c5c0 0x7f97f815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97f67fc640 1 -- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d4a30 msgr2=0x7f97f80c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97f67fc640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d4a30 0x7f97f80c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97f67fc640 1 -- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97f806c5c0 msgr2=0x7f97f815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97f67fc640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97f806c5c0 0x7f97f815de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97f67fc640 1 -- 172.21.15.67:0/3297073095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f80cf120 con 0x7f97f80d0de0 2024-04-25T15:26:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97f6ffd640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97f806c5c0 0x7f97f815de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:00.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97f67fc640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97f80d0de0 0x7f97f80cebe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f97e000c430 tx=0x7f97e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:00.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97effff640 1 -- 172.21.15.67:0/3297073095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0017070 con 0x7f97f80d0de0 2024-04-25T15:26:00.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97effff640 1 -- 172.21.15.67:0/3297073095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e00052f0 con 0x7f97f80d0de0 2024-04-25T15:26:00.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97effff640 1 -- 172.21.15.67:0/3297073095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0003830 con 0x7f97f80d0de0 2024-04-25T15:26:00.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f80cfb30 con 0x7f97f80d0de0 2024-04-25T15:26:00.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.149+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f81fa470 con 0x7f97f80d0de0 2024-04-25T15:26:00.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.150+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f8050740 con 0x7f97f80d0de0 2024-04-25T15:26:00.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.151+0000 7f97effff640 1 -- 172.21.15.67:0/3297073095 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f97e001a070 con 0x7f97f80d0de0 2024-04-25T15:26:00.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.151+0000 7f97effff640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f97d80744b0 0x7f97d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.151+0000 7f97effff640 1 -- 172.21.15.67:0/3297073095 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f97e0094530 con 0x7f97f80d0de0 2024-04-25T15:26:00.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.151+0000 7f97f6ffd640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f97d80744b0 0x7f97d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.152+0000 7f97f6ffd640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f97d80744b0 0x7f97d80769c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f97f0034000 tx=0x7f97f002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:00.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.156+0000 7f97effff640 1 -- 172.21.15.67:0/3297073095 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97e005e670 con 0x7f97f80d0de0 2024-04-25T15:26:00.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.341+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f97f80d4300 con 0x7f97f80d0de0 2024-04-25T15:26:00.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.342+0000 7f97effff640 1 -- 172.21.15.67:0/3297073095 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f97e0062970 con 0x7f97f80d0de0 2024-04-25T15:26:00.343 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.343+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f97d80744b0 msgr2=0x7f97d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.343+0000 7f97fda1a640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f97d80744b0 0x7f97d80769c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f97f0034000 tx=0x7f97f002e000 comp rx=0 tx=0).stop 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.343+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97f80d0de0 msgr2=0x7f97f80cebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.343+0000 7f97fda1a640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97f80d0de0 0x7f97f80cebe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f97e000c430 tx=0x7f97e000c8f0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 shutdown_connections 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f97d80744b0 0x7f97d80769c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f80d4a30 0x7f97f80c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97f80d0de0 0x7f97f80cebe0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 --2- 172.21.15.67:0/3297073095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97f806c5c0 0x7f97f815de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 >> 172.21.15.67:0/3297073095 conn(0x7f97f80d5d70 msgr2=0x7f97f8161880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 shutdown_connections 2024-04-25T15:26:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.344+0000 7f97fda1a640 1 -- 172.21.15.67:0/3297073095 wait complete. 2024-04-25T15:26:00.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:26:00.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.468+0000 7f50e6253640 1 Processor -- start 2024-04-25T15:26:00.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.468+0000 7f50e6253640 1 -- start start 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.469+0000 7f50e6253640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f50e0164ac0 0x7f50e00ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.469+0000 7f50e6253640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50e006ed60 con 0x7f50e01646b0 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.469+0000 7f50e6253640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50e01608b0 con 0x7f50e00b8ca0 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.469+0000 7f50e6253640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f50e00d35a0 con 0x7f50e0164ac0 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.469+0000 7f50deffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50e01646b0 0x7f50e00ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60146/0 (socket says 172.21.15.67:60146) 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.469+0000 7f50deffd640 1 -- 172.21.15.67:0/603188271 learned_addr learned my addr 172.21.15.67:0/603188271 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189347946 0 0) 0x7f50e006ed60 con 0x7f50e01646b0 2024-04-25T15:26:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50dffff640 1 --2- 172.21.15.67:0/603188271 >> v2:172.21.15.176:3300/0 conn(0x7f50e0164ac0 0x7f50e00ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50c8003520 con 0x7f50e01646b0 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2543832581 0 0) 0x7f50c8003520 con 0x7f50e01646b0 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 >> v2:172.21.15.176:3300/0 conn(0x7f50e0164ac0 msgr2=0x7f50e00ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 --2- 172.21.15.67:0/603188271 >> v2:172.21.15.176:3300/0 conn(0x7f50e0164ac0 0x7f50e00ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 >> v1:172.21.15.176:6789/0 conn(0x7f50e00b8ca0 legacy=0x7f50e0163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e00d32c0 con 0x7f50e01646b0 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f50d4001670 con 0x7f50e01646b0 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 >> v1:172.21.15.67:6789/0 conn(0x7f50e01646b0 legacy=0x7f50e00ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50c80039b0 0x7f50c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50c8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50c80051c0 0x7f50c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50c800c130 con 0x7f50c8004800 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f50c800c2e0 con 0x7f50c80039b0 2024-04-25T15:26:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f50c800c4e0 con 0x7f50c80051c0 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50deffd640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50c8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50df7fe640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50c80039b0 0x7f50c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50dffff640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50c80051c0 0x7f50c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50deffd640 1 -- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50c80051c0 msgr2=0x7f50c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50deffd640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50c80051c0 0x7f50c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50deffd640 1 -- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50c80039b0 msgr2=0x7f50c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50deffd640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50c80039b0 0x7f50c8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50deffd640 1 -- 172.21.15.67:0/603188271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c8003520 con 0x7f50c8004800 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.470+0000 7f50dffff640 1 -- 172.21.15.67:0/603188271 reap_dead start 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.471+0000 7f50deffd640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50c8004c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50e0074840 tx=0x7f50c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed41e1be4b938212 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.471+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4007d30 con 0x7f50c8004800 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.471+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d4005af0 con 0x7f50c8004800 2024-04-25T15:26:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.471+0000 7f50de7fc640 1 -- 172.21.15.67:0/603188271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4007070 con 0x7f50c8004800 2024-04-25T15:26:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.472+0000 7f50e6253640 1 -- 172.21.15.67:0/603188271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 msgr2=0x7f50c8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.472+0000 7f50e6253640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50c8004c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50e0074840 tx=0x7f50c8005830 comp rx=0 tx=0).stop 2024-04-25T15:26:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.472+0000 7f50e6253640 1 -- 172.21.15.67:0/603188271 shutdown_connections 2024-04-25T15:26:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.472+0000 7f50e6253640 1 --2- 172.21.15.67:0/603188271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50c8004c10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.472+0000 7f50e6253640 1 -- 172.21.15.67:0/603188271 >> 172.21.15.67:0/603188271 conn(0x7f50e00d3aa0 msgr2=0x7f50e00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.472+0000 7f50e6253640 1 -- 172.21.15.67:0/603188271 shutdown_connections 2024-04-25T15:26:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.472+0000 7f50e6253640 1 -- 172.21.15.67:0/603188271 wait complete. 2024-04-25T15:26:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.474+0000 7f50e6253640 1 Processor -- start 2024-04-25T15:26:00.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.474+0000 7f50e6253640 1 -- start start 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50e6253640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50e01520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50e6253640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e01f9b40 0x7f50e0160a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50e6253640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50e0151860 0x7f50e00c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50e6253640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50e006e8d0 con 0x7f50c8004800 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50e6253640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f50e006e560 con 0x7f50e01f9b40 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50e6253640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f50e00cfd60 con 0x7f50e0151860 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50e01520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50e01520a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60134/0 (socket says 172.21.15.67:60134) 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50df7fe640 1 -- 172.21.15.67:0/2212661562 learned_addr learned my addr 172.21.15.67:0/2212661562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.475+0000 7f50deffd640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e01f9b40 0x7f50e0160a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50dffff640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50e0151860 0x7f50e00c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50df7fe640 1 -- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50e0151860 msgr2=0x7f50e00c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50df7fe640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50e0151860 0x7f50e00c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50df7fe640 1 -- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e01f9b40 msgr2=0x7f50e0160a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50df7fe640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e01f9b40 0x7f50e0160a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50df7fe640 1 -- 172.21.15.67:0/2212661562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e01fa320 con 0x7f50c8004800 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50df7fe640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50e01520a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f50d000f620 tx=0x7f50d000fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50c7fff640 1 -- 172.21.15.67:0/2212661562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00052e0 con 0x7f50c8004800 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50c7fff640 1 -- 172.21.15.67:0/2212661562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d0018070 con 0x7f50c8004800 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50c7fff640 1 -- 172.21.15.67:0/2212661562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0003810 con 0x7f50c8004800 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e01fa600 con 0x7f50c8004800 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.476+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e00c7750 con 0x7f50c8004800 2024-04-25T15:26:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.477+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e00c7540 con 0x7f50c8004800 2024-04-25T15:26:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.478+0000 7f50c7fff640 1 -- 172.21.15.67:0/2212661562 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f50d00039b0 con 0x7f50c8004800 2024-04-25T15:26:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.478+0000 7f50c7fff640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f50cc0744b0 0x7f50cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.478+0000 7f50c7fff640 1 -- 172.21.15.67:0/2212661562 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f50d00950c0 con 0x7f50c8004800 2024-04-25T15:26:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.479+0000 7f50deffd640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f50cc0744b0 0x7f50cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.479+0000 7f50deffd640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f50cc0744b0 0x7f50cc0769c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f50d4032000 tx=0x7f50d402d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:00.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.483+0000 7f50c7fff640 1 -- 172.21.15.67:0/2212661562 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f50d005f850 con 0x7f50c8004800 2024-04-25T15:26:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.685+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f50e0050740 con 0x7f50c8004800 2024-04-25T15:26:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.685+0000 7f50c7fff640 1 -- 172.21.15.67:0/2212661562 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f50d005f150 con 0x7f50c8004800 2024-04-25T15:26:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:26:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f50cc0744b0 msgr2=0x7f50cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f50cc0744b0 0x7f50cc0769c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f50d4032000 tx=0x7f50d402d000 comp rx=0 tx=0).stop 2024-04-25T15:26:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 msgr2=0x7f50e01520a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50e01520a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f50d000f620 tx=0x7f50d000fae0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 shutdown_connections 2024-04-25T15:26:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f50cc0744b0 0x7f50cc0769c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f50e0151860 0x7f50e00c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50e01f9b40 0x7f50e0160a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 --2- 172.21.15.67:0/2212661562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50c8004800 0x7f50e01520a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 >> 172.21.15.67:0/2212661562 conn(0x7f50e00d3aa0 msgr2=0x7f50e00cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 shutdown_connections 2024-04-25T15:26:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:00.688+0000 7f50e6253640 1 -- 172.21.15.67:0/2212661562 wait complete. 2024-04-25T15:26:00.700 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T15:26:00.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:00.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-25T15:26:00.725 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:26:00.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:00.725 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:26:00.785 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:00.785 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:00.807 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:00.807 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:00.865 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:00.866 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:00.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:01.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.034+0000 7ff4b4c6e640 1 Processor -- start 2024-04-25T15:26:01.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.034+0000 7ff4b4c6e640 1 -- start start 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4b4c6e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b00d0e00 0x7ff4b00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4b4c6e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4b006c5c0 0x7ff4b006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4b4c6e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b006cf10 con 0x7ff4b00b7810 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4b4c6e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b00d5a60 con 0x7ff4b006c5c0 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4b4c6e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b00d1850 con 0x7ff4b00d0e00 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4a5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4b006c5c0 0x7ff4b006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4a5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4b006c5c0 0x7ff4b006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60144/0 (socket says 172.21.15.67:60144) 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4aed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff4b00b7810 0x7ff4b00b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60162/0 (socket says 172.21.15.67:60162) 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.035+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/1681074468 learned_addr learned my addr 172.21.15.67:0/1681074468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.036+0000 7ff4ae59c640 1 --2- 172.21.15.67:0/1681074468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b00d0e00 0x7ff4b00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.036+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/1681074468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b00d0e00 msgr2=0x7ff4b00d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.036+0000 7ff4a5d9b640 1 --2- 172.21.15.67:0/1681074468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b00d0e00 0x7ff4b00d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.036+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/1681074468 >> v1:172.21.15.67:6789/0 conn(0x7ff4b00b7810 legacy=0x7ff4b00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.036+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/1681074468 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b0160de0 con 0x7ff4b006c5c0 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.036+0000 7ff4a5d9b640 1 --2- 172.21.15.67:0/1681074468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b006c5c0 0x7ff4b006c9d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff498009870 tx=0x7ff49802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb2244908107ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4add9b640 1 -- 172.21.15.67:0/1681074468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498037070 con 0x7ff4b006c5c0 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4add9b640 1 -- 172.21.15.67:0/1681074468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff498032440 con 0x7ff4b006c5c0 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4add9b640 1 -- 172.21.15.67:0/1681074468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498032810 con 0x7ff4b006c5c0 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/1681074468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b006c5c0 msgr2=0x7ff4b006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/1681074468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b006c5c0 0x7ff4b006c9d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff498009870 tx=0x7ff49802a270 comp rx=0 tx=0).stop 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/1681074468 shutdown_connections 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/1681074468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b006c5c0 0x7ff4b006c9d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/1681074468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b00d0e00 0x7ff4b00d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/1681074468 >> 172.21.15.67:0/1681074468 conn(0x7ff4b00d5d90 msgr2=0x7ff4b00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:01.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/1681074468 shutdown_connections 2024-04-25T15:26:01.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.037+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/1681074468 wait complete. 2024-04-25T15:26:01.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.039+0000 7ff4b4c6e640 1 Processor -- start 2024-04-25T15:26:01.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.039+0000 7ff4b4c6e640 1 -- start start 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4b4c6e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b006c5c0 0x7ff4b0159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4b4c6e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00b7810 0x7ff4b00cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4b4c6e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b00d0e00 0x7ff4b00cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4b4c6e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b0160ad0 con 0x7ff4b00b7810 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4b4c6e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b01613a0 con 0x7ff4b00d0e00 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4b4c6e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b00b5320 con 0x7ff4b006c5c0 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00b7810 0x7ff4b00cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00b7810 0x7ff4b00cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60150/0 (socket says 172.21.15.67:60150) 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/3731782880 learned_addr learned my addr 172.21.15.67:0/3731782880 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b006c5c0 msgr2=0x7ff4b0159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b006c5c0 0x7ff4b0159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b00d0e00 msgr2=0x7ff4b00cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4aed9d640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b00d0e00 0x7ff4b00cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b00d0e00 0x7ff4b00cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 -- 172.21.15.67:0/3731782880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b0162140 con 0x7ff4b00b7810 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a5d9b640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00b7810 0x7ff4b00cf430 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff498000c00 tx=0x7ff4980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.041+0000 7ff4a77fe640 1 -- 172.21.15.67:0/3731782880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498041020 con 0x7ff4b00b7810 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.042+0000 7ff4a77fe640 1 -- 172.21.15.67:0/3731782880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4980306c0 con 0x7ff4b00b7810 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.042+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b00b41d0 con 0x7ff4b00b7810 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.042+0000 7ff4a77fe640 1 -- 172.21.15.67:0/3731782880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498037040 con 0x7ff4b00b7810 2024-04-25T15:26:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.042+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b0162420 con 0x7ff4b00b7810 2024-04-25T15:26:01.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.043+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b0050740 con 0x7ff4b00b7810 2024-04-25T15:26:01.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.043+0000 7ff4a77fe640 1 -- 172.21.15.67:0/3731782880 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff498044020 con 0x7ff4b00b7810 2024-04-25T15:26:01.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.044+0000 7ff4a77fe640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4900744b0 0x7ff4900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.044+0000 7ff4a77fe640 1 -- 172.21.15.67:0/3731782880 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff4980b5a20 con 0x7ff4b00b7810 2024-04-25T15:26:01.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.044+0000 7ff4ae59c640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4900744b0 0x7ff4900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.045+0000 7ff4ae59c640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4900744b0 0x7ff4900769c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0006fd0 tx=0x7ff4a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.048+0000 7ff4a77fe640 1 -- 172.21.15.67:0/3731782880 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff49804b030 con 0x7ff4b00b7810 2024-04-25T15:26:01.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.234+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4b00d4320 con 0x7ff4b00b7810 2024-04-25T15:26:01.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.235+0000 7ff4a77fe640 1 -- 172.21.15.67:0/3731782880 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7ff498080240 con 0x7ff4b00b7810 2024-04-25T15:26:01.236 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:01.237 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4900744b0 msgr2=0x7ff4900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4900744b0 0x7ff4900769c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0006fd0 tx=0x7ff4a0008040 comp rx=0 tx=0).stop 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00b7810 msgr2=0x7ff4b00cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00b7810 0x7ff4b00cf430 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff498000c00 tx=0x7ff4980046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 shutdown_connections 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4900744b0 0x7ff4900769c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b00d0e00 0x7ff4b00cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b00b7810 0x7ff4b00cf430 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 --2- 172.21.15.67:0/3731782880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4b006c5c0 0x7ff4b0159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.237+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 >> 172.21.15.67:0/3731782880 conn(0x7ff4b00d5d90 msgr2=0x7ff4b00b6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.238+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 shutdown_connections 2024-04-25T15:26:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.238+0000 7ff4b4c6e640 1 -- 172.21.15.67:0/3731782880 wait complete. 2024-04-25T15:26:01.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:01.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:01.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.366+0000 7f9332ee0640 1 Processor -- start 2024-04-25T15:26:01.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.366+0000 7f9332ee0640 1 -- start start 2024-04-25T15:26:01.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.367+0000 7f9332ee0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c15e080 0x7f932c1646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.367+0000 7f9332ee0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932c1609b0 con 0x7f932c161710 2024-04-25T15:26:01.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.367+0000 7f9332ee0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932c164cd0 con 0x7f932c06ccd0 2024-04-25T15:26:01.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.367+0000 7f9332ee0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f932c160c30 con 0x7f932c15e080 2024-04-25T15:26:01.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.367+0000 7f9330c7c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f932c161710 0x7f932c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60168/0 (socket says 172.21.15.67:60168) 2024-04-25T15:26:01.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.367+0000 7f9330c7c640 1 -- 172.21.15.67:0/976095738 learned_addr learned my addr 172.21.15.67:0/976095738 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:01.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.367+0000 7f933147d640 1 --2- 172.21.15.67:0/976095738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c15e080 0x7f932c1646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744814731 0 0) 0x7f9324001490 con 0x7f932c161710 2024-04-25T15:26:01.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9318003520 con 0x7f932c161710 2024-04-25T15:26:01.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568573143 0 0) 0x7f932c164cd0 con 0x7f932c06ccd0 2024-04-25T15:26:01.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9324001490 con 0x7f932c06ccd0 2024-04-25T15:26:01.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1937534247 0 0) 0x7f9318003520 con 0x7f932c161710 2024-04-25T15:26:01.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f933147d640 1 -- 172.21.15.67:0/976095738 >> v1:172.21.15.176:6789/0 conn(0x7f932c06ccd0 legacy=0x7f932c163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f933147d640 1 -- 172.21.15.67:0/976095738 >> v1:172.21.15.67:6789/0 conn(0x7f932c161710 legacy=0x7f932c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.368+0000 7f933147d640 1 -- 172.21.15.67:0/976095738 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c160f10 con 0x7f932c15e080 2024-04-25T15:26:01.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.369+0000 7f933147d640 1 --2- 172.21.15.67:0/976095738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c15e080 0x7f932c1646b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f931c0097d0 tx=0x7f931c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c584bcea0519809 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.369+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c037070 con 0x7f932c15e080 2024-04-25T15:26:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.369+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f931c02ab90 con 0x7f932c15e080 2024-04-25T15:26:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.369+0000 7f932b7fe640 1 -- 172.21.15.67:0/976095738 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9324001bc0 con 0x7f932c15e080 2024-04-25T15:26:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.369+0000 7f9332ee0640 1 -- 172.21.15.67:0/976095738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c15e080 msgr2=0x7f932c1646b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.369+0000 7f9332ee0640 1 --2- 172.21.15.67:0/976095738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c15e080 0x7f932c1646b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f931c0097d0 tx=0x7f931c02a100 comp rx=0 tx=0).stop 2024-04-25T15:26:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.370+0000 7f9332ee0640 1 -- 172.21.15.67:0/976095738 shutdown_connections 2024-04-25T15:26:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.370+0000 7f9332ee0640 1 --2- 172.21.15.67:0/976095738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c15e080 0x7f932c1646b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.370+0000 7f9332ee0640 1 -- 172.21.15.67:0/976095738 >> 172.21.15.67:0/976095738 conn(0x7f932c0d5d90 msgr2=0x7f932c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:01.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.370+0000 7f9332ee0640 1 -- 172.21.15.67:0/976095738 shutdown_connections 2024-04-25T15:26:01.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.370+0000 7f9332ee0640 1 -- 172.21.15.67:0/976095738 wait complete. 2024-04-25T15:26:01.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.371+0000 7f9332ee0640 1 Processor -- start 2024-04-25T15:26:01.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.371+0000 7f9332ee0640 1 -- start start 2024-04-25T15:26:01.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f9332ee0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f932c15e080 0x7f932c15a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f9332ee0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c161710 0x7f932c15f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f9332ee0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c0b68b0 0x7f932c15f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f9332ee0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f932c16b280 con 0x7f932c161710 2024-04-25T15:26:01.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f9332ee0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f932c16b100 con 0x7f932c15e080 2024-04-25T15:26:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f9332ee0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f932c16b400 con 0x7f932c0b68b0 2024-04-25T15:26:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c161710 0x7f932c15f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c161710 0x7f932c15f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60152/0 (socket says 172.21.15.67:60152) 2024-04-25T15:26:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 -- 172.21.15.67:0/56294633 learned_addr learned my addr 172.21.15.67:0/56294633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f9330c7c640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f932c15e080 0x7f932c15a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f933147d640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c0b68b0 0x7f932c15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 -- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c0b68b0 msgr2=0x7f932c15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c0b68b0 0x7f932c15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 -- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f932c15e080 msgr2=0x7f932c15a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f932c15e080 0x7f932c15a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 -- 172.21.15.67:0/56294633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c0ccb90 con 0x7f932c161710 2024-04-25T15:26:01.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.373+0000 7f932bfff640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c161710 0x7f932c15f230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f931400c430 tx=0x7f931400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.374+0000 7f93297fa640 1 -- 172.21.15.67:0/56294633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314017070 con 0x7f932c161710 2024-04-25T15:26:01.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.374+0000 7f93297fa640 1 -- 172.21.15.67:0/56294633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93140052f0 con 0x7f932c161710 2024-04-25T15:26:01.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.374+0000 7f93297fa640 1 -- 172.21.15.67:0/56294633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314003830 con 0x7f932c161710 2024-04-25T15:26:01.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.374+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932c0ca260 con 0x7f932c161710 2024-04-25T15:26:01.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.374+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932c0ca500 con 0x7f932c161710 2024-04-25T15:26:01.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.375+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f932c0c9b20 con 0x7f932c161710 2024-04-25T15:26:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.375+0000 7f93297fa640 1 -- 172.21.15.67:0/56294633 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f931401a070 con 0x7f932c161710 2024-04-25T15:26:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.376+0000 7f93297fa640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f930c0744b0 0x7f930c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.376+0000 7f93297fa640 1 -- 172.21.15.67:0/56294633 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9314094660 con 0x7f932c161710 2024-04-25T15:26:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.376+0000 7f9330c7c640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f930c0744b0 0x7f930c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.377+0000 7f9330c7c640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f930c0744b0 0x7f930c0769c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9324001890 tx=0x7f9324009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.381+0000 7f93297fa640 1 -- 172.21.15.67:0/56294633 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f931405e720 con 0x7f932c161710 2024-04-25T15:26:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.569+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f932c050740 con 0x7f932c161710 2024-04-25T15:26:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.569+0000 7f93297fa640 1 -- 172.21.15.67:0/56294633 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f9314062a20 con 0x7f932c161710 2024-04-25T15:26:01.570 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f930c0744b0 msgr2=0x7f930c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f930c0744b0 0x7f930c0769c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9324001890 tx=0x7f9324009040 comp rx=0 tx=0).stop 2024-04-25T15:26:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c161710 msgr2=0x7f932c15f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c161710 0x7f932c15f230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f931400c430 tx=0x7f931400c8f0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 shutdown_connections 2024-04-25T15:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f930c0744b0 0x7f930c0769c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f932c0b68b0 0x7f932c15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f932c161710 0x7f932c15f230 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 --2- 172.21.15.67:0/56294633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f932c15e080 0x7f932c15a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 >> 172.21.15.67:0/56294633 conn(0x7f932c0d5d90 msgr2=0x7f932c16a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 shutdown_connections 2024-04-25T15:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.571+0000 7f9332ee0640 1 -- 172.21.15.67:0/56294633 wait complete. 2024-04-25T15:26:01.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:01.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.698+0000 7fb141dba640 1 Processor -- start 2024-04-25T15:26:01.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.698+0000 7fb141dba640 1 -- start start 2024-04-25T15:26:01.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb141dba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb13c08dec0 0x7fb13c0b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb141dba640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb13c08cf20 0x7fb13c08d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb141dba640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13c08d870 con 0x7fb13c0b0c40 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb141dba640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb13c0b95f0 con 0x7fb13c08dec0 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb141dba640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb13c0d21e0 con 0x7fb13c08cf20 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb13b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb13c08dec0 0x7fb13c0b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb13b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb13c08dec0 0x7fb13c0b90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60164/0 (socket says 172.21.15.67:60164) 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb13b7fe640 1 -- 172.21.15.67:0/2786760893 learned_addr learned my addr 172.21.15.67:0/2786760893 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb13bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb13c0b0c40 0x7fb13c0b1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60184/0 (socket says 172.21.15.67:60184) 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb13affd640 1 --2- 172.21.15.67:0/2786760893 >> v2:172.21.15.176:3300/0 conn(0x7fb13c08cf20 0x7fb13c08d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb13b7fe640 1 -- 172.21.15.67:0/2786760893 >> v2:172.21.15.176:3300/0 conn(0x7fb13c08cf20 msgr2=0x7fb13c08d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.699+0000 7fb13b7fe640 1 --2- 172.21.15.67:0/2786760893 >> v2:172.21.15.176:3300/0 conn(0x7fb13c08cf20 0x7fb13c08d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.700+0000 7fb13b7fe640 1 -- 172.21.15.67:0/2786760893 >> v1:172.21.15.67:6789/0 conn(0x7fb13c0b0c40 legacy=0x7fb13c0b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.700+0000 7fb13b7fe640 1 -- 172.21.15.67:0/2786760893 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c0d1940 con 0x7fb13c08dec0 2024-04-25T15:26:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.700+0000 7fb13a7fc640 1 -- 172.21.15.67:0/2786760893 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 455462375 0 0) 0x7fb134001490 con 0x7fb13c0b0c40 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.700+0000 7fb13b7fe640 1 --2- 172.21.15.67:0/2786760893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c08dec0 0x7fb13c0b90b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0d39c0 tx=0x7fb12c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b260c12a92d359f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.700+0000 7fb13a7fc640 1 -- 172.21.15.67:0/2786760893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c037070 con 0x7fb13c08dec0 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.700+0000 7fb13a7fc640 1 -- 172.21.15.67:0/2786760893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12c032440 con 0x7fb13c08dec0 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.700+0000 7fb13a7fc640 1 -- 172.21.15.67:0/2786760893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c032810 con 0x7fb13c08dec0 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 -- 172.21.15.67:0/2786760893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c08dec0 msgr2=0x7fb13c0b90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 --2- 172.21.15.67:0/2786760893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c08dec0 0x7fb13c0b90b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0d39c0 tx=0x7fb12c02a270 comp rx=0 tx=0).stop 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 -- 172.21.15.67:0/2786760893 shutdown_connections 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 --2- 172.21.15.67:0/2786760893 >> v2:172.21.15.176:3300/0 conn(0x7fb13c08cf20 0x7fb13c08d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 --2- 172.21.15.67:0/2786760893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c08dec0 0x7fb13c0b90b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 -- 172.21.15.67:0/2786760893 >> 172.21.15.67:0/2786760893 conn(0x7fb13c0d0510 msgr2=0x7fb13c0d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 -- 172.21.15.67:0/2786760893 shutdown_connections 2024-04-25T15:26:01.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.701+0000 7fb141dba640 1 -- 172.21.15.67:0/2786760893 wait complete. 2024-04-25T15:26:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.703+0000 7fb141dba640 1 Processor -- start 2024-04-25T15:26:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.703+0000 7fb141dba640 1 -- start start 2024-04-25T15:26:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.704+0000 7fb141dba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb13c08cf20 0x7fb13c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.704+0000 7fb141dba640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb13c08dec0 0x7fb13c1f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.704+0000 7fb141dba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c0b0c40 0x7fb13c1f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.704+0000 7fb141dba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c0b0720 con 0x7fb13c0b0c40 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.704+0000 7fb141dba640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c0d1f00 con 0x7fb13c08dec0 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.704+0000 7fb141dba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb13c0d15d0 con 0x7fb13c08cf20 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.704+0000 7fb13bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c0b0c40 0x7fb13c1f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c0b0c40 0x7fb13c1f5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60178/0 (socket says 172.21.15.67:60178) 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 -- 172.21.15.67:0/2365890650 learned_addr learned my addr 172.21.15.67:0/2365890650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13b7fe640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb13c08cf20 0x7fb13c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13affd640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb13c08dec0 0x7fb13c1f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 -- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb13c08cf20 msgr2=0x7fb13c0c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb13c08cf20 0x7fb13c0c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 -- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb13c08dec0 msgr2=0x7fb13c1f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb13c08dec0 0x7fb13c1f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 -- 172.21.15.67:0/2365890650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c1f4770 con 0x7fb13c0b0c40 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13b7fe640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb13c08cf20 0x7fb13c0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13affd640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb13c08dec0 0x7fb13c1f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:01.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb13bfff640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c0b0c40 0x7fb13c1f5090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c730 tx=0x7fb13400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb123fff640 1 -- 172.21.15.67:0/2365890650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb134018070 con 0x7fb13c0b0c40 2024-04-25T15:26:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb123fff640 1 -- 172.21.15.67:0/2365890650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13400ce40 con 0x7fb13c0b0c40 2024-04-25T15:26:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c0c9b60 con 0x7fb13c0b0c40 2024-04-25T15:26:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb123fff640 1 -- 172.21.15.67:0/2365890650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13400a860 con 0x7fb13c0b0c40 2024-04-25T15:26:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.705+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c0ca2c0 con 0x7fb13c0b0c40 2024-04-25T15:26:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.706+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13c0b2480 con 0x7fb13c0b0c40 2024-04-25T15:26:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.707+0000 7fb123fff640 1 -- 172.21.15.67:0/2365890650 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb13401b070 con 0x7fb13c0b0c40 2024-04-25T15:26:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.707+0000 7fb123fff640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb1280744b0 0x7fb1280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.708+0000 7fb123fff640 1 -- 172.21.15.67:0/2365890650 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb134094790 con 0x7fb13c0b0c40 2024-04-25T15:26:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.708+0000 7fb13b7fe640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb1280744b0 0x7fb1280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.708+0000 7fb13b7fe640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb1280744b0 0x7fb1280769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb12c034000 tx=0x7fb12c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.712+0000 7fb123fff640 1 -- 172.21.15.67:0/2365890650 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb13c0b2480 con 0x7fb13c0b0c40 2024-04-25T15:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.897+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb13c050740 con 0x7fb13c0b0c40 2024-04-25T15:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.897+0000 7fb123fff640 1 -- 172.21.15.67:0/2365890650 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fb13c08d870 con 0x7fb13c0b0c40 2024-04-25T15:26:01.898 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb1280744b0 msgr2=0x7fb1280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb1280744b0 0x7fb1280769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb12c034000 tx=0x7fb12c02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c0b0c40 msgr2=0x7fb13c1f5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c0b0c40 0x7fb13c1f5090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c730 tx=0x7fb13400cbf0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 shutdown_connections 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb1280744b0 0x7fb1280769c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c0b0c40 0x7fb13c1f5090 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb13c08dec0 0x7fb13c1f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 --2- 172.21.15.67:0/2365890650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb13c08cf20 0x7fb13c0c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 >> 172.21.15.67:0/2365890650 conn(0x7fb13c0d0510 msgr2=0x7fb13c0b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 shutdown_connections 2024-04-25T15:26:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:01.899+0000 7fb141dba640 1 -- 172.21.15.67:0/2365890650 wait complete. 2024-04-25T15:26:01.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.024+0000 7fcba0049640 1 Processor -- start 2024-04-25T15:26:02.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.025+0000 7fcba0049640 1 -- start start 2024-04-25T15:26:02.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.026+0000 7fcba0049640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb980b2d60 0x7fcb981649f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.026+0000 7fcba0049640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb980b3170 con 0x7fcb980ce040 2024-04-25T15:26:02.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.026+0000 7fcba0049640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb9806d770 con 0x7fcb9806ea60 2024-04-25T15:26:02.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.026+0000 7fcba0049640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcb9806da50 con 0x7fcb980b2d60 2024-04-25T15:26:02.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.026+0000 7fcb9e5e6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb980b2d60 0x7fcb981649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.026+0000 7fcb9dde5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb980ce040 0x7fcb9806e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60186/0 (socket says 172.21.15.67:60186) 2024-04-25T15:26:02.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.026+0000 7fcb9dde5640 1 -- 172.21.15.67:0/1216752307 learned_addr learned my addr 172.21.15.67:0/1216752307 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:02.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.027+0000 7fcb9e5e6640 1 -- 172.21.15.67:0/1216752307 >> v1:172.21.15.176:6789/0 conn(0x7fcb9806ea60 legacy=0x7fcb980b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:02.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.027+0000 7fcb9e5e6640 1 -- 172.21.15.67:0/1216752307 >> v1:172.21.15.67:6789/0 conn(0x7fcb980ce040 legacy=0x7fcb9806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.027+0000 7fcb9e5e6640 1 -- 172.21.15.67:0/1216752307 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb980d2200 con 0x7fcb980b2d60 2024-04-25T15:26:02.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.027+0000 7fcb9e5e6640 1 --2- 172.21.15.67:0/1216752307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980b2d60 0x7fcb981649f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009d90 tx=0x7fcb8802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253a29b8862a2251 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:02.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.027+0000 7fcb9cde3640 1 -- 172.21.15.67:0/1216752307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031600 con 0x7fcb980b2d60 2024-04-25T15:26:02.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.027+0000 7fcb9cde3640 1 -- 172.21.15.67:0/1216752307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb880317a0 con 0x7fcb980b2d60 2024-04-25T15:26:02.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcb9cde3640 1 -- 172.21.15.67:0/1216752307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031b70 con 0x7fcb980b2d60 2024-04-25T15:26:02.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcba0049640 1 -- 172.21.15.67:0/1216752307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980b2d60 msgr2=0x7fcb981649f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcba0049640 1 --2- 172.21.15.67:0/1216752307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980b2d60 0x7fcb981649f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009d90 tx=0x7fcb8802a610 comp rx=0 tx=0).stop 2024-04-25T15:26:02.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcba0049640 1 -- 172.21.15.67:0/1216752307 shutdown_connections 2024-04-25T15:26:02.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcba0049640 1 --2- 172.21.15.67:0/1216752307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980b2d60 0x7fcb981649f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcba0049640 1 -- 172.21.15.67:0/1216752307 >> 172.21.15.67:0/1216752307 conn(0x7fcb980d3a80 msgr2=0x7fcb980d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:02.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcba0049640 1 -- 172.21.15.67:0/1216752307 shutdown_connections 2024-04-25T15:26:02.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.028+0000 7fcba0049640 1 -- 172.21.15.67:0/1216752307 wait complete. 2024-04-25T15:26:02.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.030+0000 7fcba0049640 1 Processor -- start 2024-04-25T15:26:02.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.030+0000 7fcba0049640 1 -- start start 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.031+0000 7fcba0049640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb980b2d60 0x7fcb981f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.031+0000 7fcba0049640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980ce040 0x7fcb980c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.031+0000 7fcba0049640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb980c8470 0x7fcb980c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.031+0000 7fcba0049640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb980b90a0 con 0x7fcb980ce040 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.031+0000 7fcba0049640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb980d1f10 con 0x7fcb980c8470 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.031+0000 7fcba0049640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb980b9220 con 0x7fcb980b2d60 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980ce040 0x7fcb980c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980ce040 0x7fcb980c7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60198/0 (socket says 172.21.15.67:60198) 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 -- 172.21.15.67:0/1043468634 learned_addr learned my addr 172.21.15.67:0/1043468634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 -- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb980b2d60 msgr2=0x7fcb981f60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9dde5640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb980b2d60 0x7fcb981f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9e5e6640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb980c8470 0x7fcb980c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb980b2d60 0x7fcb981f60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 -- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb980c8470 msgr2=0x7fcb980c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb980c8470 0x7fcb980c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 -- 172.21.15.67:0/1043468634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb981609a0 con 0x7fcb980ce040 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9dde5640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb980b2d60 0x7fcb981f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9e5e6640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb980c8470 0x7fcb980c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb9d5e4640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980ce040 0x7fcb980c7f30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009fd0 tx=0x7fcb8c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb967fc640 1 -- 172.21.15.67:0/1043468634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c015020 con 0x7fcb980ce040 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb967fc640 1 -- 172.21.15.67:0/1043468634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb8c011830 con 0x7fcb980ce040 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb980cac60 con 0x7fcb980ce040 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcb967fc640 1 -- 172.21.15.67:0/1043468634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c011c20 con 0x7fcb980ce040 2024-04-25T15:26:02.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.032+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb980caf00 con 0x7fcb980ce040 2024-04-25T15:26:02.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.034+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb98050740 con 0x7fcb980ce040 2024-04-25T15:26:02.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.034+0000 7fcb967fc640 1 -- 172.21.15.67:0/1043468634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb8c018020 con 0x7fcb980ce040 2024-04-25T15:26:02.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.034+0000 7fcb967fc640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb840744b0 0x7fcb840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.035+0000 7fcb967fc640 1 -- 172.21.15.67:0/1043468634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcb8c0950f0 con 0x7fcb980ce040 2024-04-25T15:26:02.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.035+0000 7fcb9dde5640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb840744b0 0x7fcb840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.035+0000 7fcb9dde5640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb840744b0 0x7fcb840769c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000f60 tx=0x7fcb90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:02.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.039+0000 7fcb967fc640 1 -- 172.21.15.67:0/1043468634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb8c01f020 con 0x7fcb980ce040 2024-04-25T15:26:02.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.226+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcb980d4780 con 0x7fcb980ce040 2024-04-25T15:26:02.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.227+0000 7fcb967fc640 1 -- 172.21.15.67:0/1043468634 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fcb8c05f910 con 0x7fcb980ce040 2024-04-25T15:26:02.228 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:02.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb840744b0 msgr2=0x7fcb840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb840744b0 0x7fcb840769c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000f60 tx=0x7fcb90009040 comp rx=0 tx=0).stop 2024-04-25T15:26:02.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980ce040 msgr2=0x7fcb980c7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980ce040 0x7fcb980c7f30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009fd0 tx=0x7fcb8c00b870 comp rx=0 tx=0).stop 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 shutdown_connections 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcb840744b0 0x7fcb840769c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb980c8470 0x7fcb980c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb980ce040 0x7fcb980c7f30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 --2- 172.21.15.67:0/1043468634 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb980b2d60 0x7fcb981f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 >> 172.21.15.67:0/1043468634 conn(0x7fcb980d3a80 msgr2=0x7fcb9806e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 shutdown_connections 2024-04-25T15:26:02.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.229+0000 7fcba0049640 1 -- 172.21.15.67:0/1043468634 wait complete. 2024-04-25T15:26:02.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:26:02.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.353+0000 7f7f17984640 1 Processor -- start 2024-04-25T15:26:02.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.353+0000 7f7f17984640 1 -- start start 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f17984640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f10164b00 0x7f7f100b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f17984640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f101615f0 con 0x7f7f10160fa0 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f17984640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f10164020 con 0x7f7f1015df80 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f17984640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7f101613b0 con 0x7f7f10164b00 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f15f21640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7f10160fa0 0x7f7f100b4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60198/0 (socket says 172.21.15.67:60198) 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f15f21640 1 -- 172.21.15.67:0/3224545146 learned_addr learned my addr 172.21.15.67:0/3224545146 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f15720640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f10164b00 0x7f7f100b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837202991 0 0) 0x7f7f101615f0 con 0x7f7f10160fa0 2024-04-25T15:26:02.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ef8003520 con 0x7f7f10160fa0 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916317350 0 0) 0x7f7f10164020 con 0x7f7f1015df80 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f101615f0 con 0x7f7f1015df80 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3761677693 0 0) 0x7f7ef8003520 con 0x7f7f10160fa0 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f10164b00 msgr2=0x7f7f100b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f10164b00 0x7f7f100b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 >> v1:172.21.15.176:6789/0 conn(0x7f7f1015df80 legacy=0x7f7f10160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f10164300 con 0x7f7f10160fa0 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f7f000019f0 con 0x7f7f10160fa0 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.355+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 >> v1:172.21.15.67:6789/0 conn(0x7f7f10160fa0 legacy=0x7f7f100b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f0ffff640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ef8003b70 0x7f7ef8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f0ffff640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef8004800 0x7f7ef8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f0ffff640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7ef800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f101613b0 con 0x7f7ef8005150 2024-04-25T15:26:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef800c090 con 0x7f7ef8004800 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef800c290 con 0x7f7ef8003b70 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15720640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f10164b00 0x7f7f100b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15f21640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7ef800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f14f1f640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef8004800 0x7f7ef8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15720640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ef8003b70 0x7f7ef8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15f21640 1 -- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ef8003b70 msgr2=0x7f7ef8003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15f21640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ef8003b70 0x7f7ef8003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15f21640 1 -- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef8004800 msgr2=0x7f7ef8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15f21640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ef8004800 0x7f7ef8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.356+0000 7f7f15f21640 1 -- 172.21.15.67:0/3224545146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef8003520 con 0x7f7ef8005150 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f15f21640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7ef800b9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7f1008f1d0 tx=0x7f7ef8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6273c453b31c4a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0000a410 con 0x7f7ef8005150 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0002f070 con 0x7f7ef8005150 2024-04-25T15:26:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f0ffff640 1 -- 172.21.15.67:0/3224545146 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0002bbe0 con 0x7f7ef8005150 2024-04-25T15:26:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f17984640 1 -- 172.21.15.67:0/3224545146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 msgr2=0x7f7ef800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f17984640 1 --2- 172.21.15.67:0/3224545146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7ef800b9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7f1008f1d0 tx=0x7f7ef8005830 comp rx=0 tx=0).stop 2024-04-25T15:26:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f15f21640 1 -- 172.21.15.67:0/3224545146 reap_dead start 2024-04-25T15:26:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f17984640 1 -- 172.21.15.67:0/3224545146 shutdown_connections 2024-04-25T15:26:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.357+0000 7f7f17984640 1 -- 172.21.15.67:0/3224545146 >> 172.21.15.67:0/3224545146 conn(0x7f7f100d2bc0 msgr2=0x7f7f100d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.358+0000 7f7f17984640 1 -- 172.21.15.67:0/3224545146 shutdown_connections 2024-04-25T15:26:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.358+0000 7f7f17984640 1 -- 172.21.15.67:0/3224545146 wait complete. 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.359+0000 7f7f17984640 1 Processor -- start 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.359+0000 7f7f17984640 1 -- start start 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f17984640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7f101564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f17984640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f101f63c0 0x7f7f101fa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f17984640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f101f67d0 0x7f7f101fa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f17984640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f100b5360 con 0x7f7ef8005150 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f17984640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7f100b51e0 con 0x7f7f101f63c0 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f17984640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7f100b54e0 con 0x7f7f101f67d0 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7f101564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7f101564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60224/0 (socket says 172.21.15.67:60224) 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 -- 172.21.15.67:0/3621781009 learned_addr learned my addr 172.21.15.67:0/3621781009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f14f1f640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f101f63c0 0x7f7f101fa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15f21640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f101f67d0 0x7f7f101fa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 -- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f101f67d0 msgr2=0x7f7f101fa630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f101f67d0 0x7f7f101fa630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 -- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f101f63c0 msgr2=0x7f7f101fa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f101f63c0 0x7f7f101fa0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 -- 172.21.15.67:0/3621781009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f10156a00 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.361+0000 7f7f15720640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7f101564c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7f080115d0 tx=0x7f7f080119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.362+0000 7f7f0dffb640 1 -- 172.21.15.67:0/3621781009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08015870 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.362+0000 7f7f0dffb640 1 -- 172.21.15.67:0/3621781009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f08015a10 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.362+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f101622c0 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.362+0000 7f7f0dffb640 1 -- 172.21.15.67:0/3621781009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08015e00 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.362+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f101625a0 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.363+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f101602e0 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.364+0000 7f7f0dffb640 1 -- 172.21.15.67:0/3621781009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7f0801d400 con 0x7f7ef8005150 2024-04-25T15:26:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.364+0000 7f7f0dffb640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7efc0744e0 0x7f7efc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.364+0000 7f7f0dffb640 1 -- 172.21.15.67:0/3621781009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7f08095aa0 con 0x7f7ef8005150 2024-04-25T15:26:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.364+0000 7f7f14f1f640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7efc0744e0 0x7f7efc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:02.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.365+0000 7f7f14f1f640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7efc0744e0 0x7f7efc0769f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7f101633e0 tx=0x7f7f04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:02.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.369+0000 7f7f0dffb640 1 -- 172.21.15.67:0/3621781009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f101602e0 con 0x7f7ef8005150 2024-04-25T15:26:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.567+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7f1015f510 con 0x7f7ef8005150 2024-04-25T15:26:02.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.567+0000 7f7f0dffb640 1 -- 172.21.15.67:0/3621781009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f7f1015f510 con 0x7f7ef8005150 2024-04-25T15:26:02.568 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:26:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7efc0744e0 msgr2=0x7f7efc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7efc0744e0 0x7f7efc0769f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7f101633e0 tx=0x7f7f04009040 comp rx=0 tx=0).stop 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 msgr2=0x7f7f101564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7f101564c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7f080115d0 tx=0x7f7f080119d0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 shutdown_connections 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7efc0744e0 0x7f7efc0769f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7f101f67d0 0x7f7f101fa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f101f63c0 0x7f7f101fa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 --2- 172.21.15.67:0/3621781009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ef8005150 0x7f7f101564c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 >> 172.21.15.67:0/3621781009 conn(0x7f7f100d2bc0 msgr2=0x7f7f100d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.569+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 shutdown_connections 2024-04-25T15:26:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:02.570+0000 7f7f17984640 1 -- 172.21.15.67:0/3621781009 wait complete. 2024-04-25T15:26:02.581 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:26:02.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:02.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-25T15:26:02.612 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:26:02.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:02.612 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:26:02.890 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:02.890 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:02.906 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:02.906 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:02.963 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:02.963 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:03.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.170+0000 7f7edcdbe640 1 Processor -- start 2024-04-25T15:26:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.170+0000 7f7edcdbe640 1 -- start start 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7edcdbe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed81400a0 0x7f7ed80b6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7edcdbe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ed8142730 0x7f7ed8142b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7edcdbe640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed81431d0 con 0x7f7ed8070270 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7edcdbe640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7ed81434b0 con 0x7f7ed8142730 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7edcdbe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed80b6090 con 0x7f7ed81400a0 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7ecffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ed8142730 0x7f7ed8142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7ecffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ed8142730 0x7f7ed8142b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60232/0 (socket says 172.21.15.67:60232) 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.171+0000 7f7ecffff640 1 -- 172.21.15.67:0/2671086794 learned_addr learned my addr 172.21.15.67:0/2671086794 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.172+0000 7f7ed659c640 1 --2- 172.21.15.67:0/2671086794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed81400a0 0x7f7ed80b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.172+0000 7f7ecffff640 1 -- 172.21.15.67:0/2671086794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed81400a0 msgr2=0x7f7ed80b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.172+0000 7f7ecffff640 1 --2- 172.21.15.67:0/2671086794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed81400a0 0x7f7ed80b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.172+0000 7f7ecffff640 1 -- 172.21.15.67:0/2671086794 >> v1:172.21.15.176:6789/0 conn(0x7f7ed8070270 legacy=0x7f7ed8070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.172+0000 7f7ecffff640 1 -- 172.21.15.67:0/2671086794 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed80b6370 con 0x7f7ed8142730 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.172+0000 7f7ecffff640 1 --2- 172.21.15.67:0/2671086794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8142730 0x7f7ed8142b40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009870 tx=0x7f7ec002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c53ceaff3002a5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7ed5d9b640 1 -- 172.21.15.67:0/2671086794 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0037070 con 0x7f7ed8142730 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7ed5d9b640 1 -- 172.21.15.67:0/2671086794 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec0032440 con 0x7f7ed8142730 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7ed5d9b640 1 -- 172.21.15.67:0/2671086794 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0032810 con 0x7f7ed8142730 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 -- 172.21.15.67:0/2671086794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8142730 msgr2=0x7f7ed8142b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/2671086794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8142730 0x7f7ed8142b40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009870 tx=0x7f7ec002a210 comp rx=0 tx=0).stop 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 -- 172.21.15.67:0/2671086794 shutdown_connections 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/2671086794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8142730 0x7f7ed8142b40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/2671086794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed81400a0 0x7f7ed80b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 -- 172.21.15.67:0/2671086794 >> 172.21.15.67:0/2671086794 conn(0x7f7ed8073050 msgr2=0x7f7ed8073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 -- 172.21.15.67:0/2671086794 shutdown_connections 2024-04-25T15:26:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.173+0000 7f7edcdbe640 1 -- 172.21.15.67:0/2671086794 wait complete. 2024-04-25T15:26:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.175+0000 7f7edcdbe640 1 Processor -- start 2024-04-25T15:26:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.175+0000 7f7edcdbe640 1 -- start start 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7edcdbe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8070270 0x7f7ed80ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7edcdbe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed81400a0 0x7f7ed81f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7edcdbe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed8142730 0x7f7ed80cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7edcdbe640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed80b49a0 con 0x7f7ed8070270 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7edcdbe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed80cddb0 con 0x7f7ed81400a0 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7edcdbe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed80b4b20 con 0x7f7ed8142730 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8070270 0x7f7ed80ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8070270 0x7f7ed80ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60238/0 (socket says 172.21.15.67:60238) 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 -- 172.21.15.67:0/171922613 learned_addr learned my addr 172.21.15.67:0/171922613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ecffff640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed81400a0 0x7f7ed81f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed6d9d640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed8142730 0x7f7ed80cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 -- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed8142730 msgr2=0x7f7ed80cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed8142730 0x7f7ed80cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 -- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed81400a0 msgr2=0x7f7ed81f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed81400a0 0x7f7ed81f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 -- 172.21.15.67:0/171922613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed80cb0b0 con 0x7f7ed8070270 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.177+0000 7f7ed659c640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8070270 0x7f7ed80ca760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800d630 tx=0x7f7ec800daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.178+0000 7f7eceffd640 1 -- 172.21.15.67:0/171922613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8004240 con 0x7f7ed8070270 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.178+0000 7f7eceffd640 1 -- 172.21.15.67:0/171922613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec80043e0 con 0x7f7ed8070270 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.178+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed80cb390 con 0x7f7ed8070270 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.178+0000 7f7eceffd640 1 -- 172.21.15.67:0/171922613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8003830 con 0x7f7ed8070270 2024-04-25T15:26:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.178+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed80c7430 con 0x7f7ed8070270 2024-04-25T15:26:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.179+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed8050740 con 0x7f7ed8070270 2024-04-25T15:26:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.179+0000 7f7eceffd640 1 -- 172.21.15.67:0/171922613 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7ec80039d0 con 0x7f7ed8070270 2024-04-25T15:26:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.180+0000 7f7eceffd640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7eb80744e0 0x7f7eb80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.180+0000 7f7eceffd640 1 -- 172.21.15.67:0/171922613 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7ec8095390 con 0x7f7ed8070270 2024-04-25T15:26:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.180+0000 7f7ecffff640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7eb80744e0 0x7f7eb80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.181+0000 7f7ecffff640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7eb80744e0 0x7f7eb80769f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0034000 tx=0x7f7ec002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.184+0000 7f7eceffd640 1 -- 172.21.15.67:0/171922613 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ec805f450 con 0x7f7ed8070270 2024-04-25T15:26:03.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.371+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ed80cf480 con 0x7f7ed8070270 2024-04-25T15:26:03.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.373+0000 7f7eceffd640 1 -- 172.21.15.67:0/171922613 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f7ec8063750 con 0x7f7ed8070270 2024-04-25T15:26:03.374 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:03.374 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.374+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7eb80744e0 msgr2=0x7f7eb80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.374+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7eb80744e0 0x7f7eb80769f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0034000 tx=0x7f7ec002e000 comp rx=0 tx=0).stop 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.374+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8070270 msgr2=0x7f7ed80ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.374+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8070270 0x7f7ed80ca760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800d630 tx=0x7f7ec800daf0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 shutdown_connections 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7eb80744e0 0x7f7eb80769f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ed8142730 0x7f7ed80cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ed81400a0 0x7f7ed81f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 --2- 172.21.15.67:0/171922613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ed8070270 0x7f7ed80ca760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 >> 172.21.15.67:0/171922613 conn(0x7f7ed8073050 msgr2=0x7f7ed80b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 shutdown_connections 2024-04-25T15:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.375+0000 7f7edcdbe640 1 -- 172.21.15.67:0/171922613 wait complete. 2024-04-25T15:26:03.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:03.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.501+0000 7f1c9100c640 1 Processor -- start 2024-04-25T15:26:03.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.501+0000 7f1c9100c640 1 -- start start 2024-04-25T15:26:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.502+0000 7f1c9100c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1c8c06ccd0 0x7f1c8c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.502+0000 7f1c9100c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c8c0d1850 con 0x7f1c8c0d0e00 2024-04-25T15:26:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.502+0000 7f1c9100c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c8c160de0 con 0x7f1c8c0d4a50 2024-04-25T15:26:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.502+0000 7f1c9100c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1c8c1610c0 con 0x7f1c8c06ccd0 2024-04-25T15:26:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.502+0000 7f1c8ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1c8c0d0e00 0x7f1c8c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60204/0 (socket says 172.21.15.67:60204) 2024-04-25T15:26:03.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.502+0000 7f1c8ad9d640 1 -- 172.21.15.67:0/49457773 learned_addr learned my addr 172.21.15.67:0/49457773 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c8a59c640 1 --2- 172.21.15.67:0/49457773 >> v2:172.21.15.176:3300/0 conn(0x7f1c8c06ccd0 0x7f1c8c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833094924 0 0) 0x7f1c8c0d1850 con 0x7f1c8c0d0e00 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c74003520 con 0x7f1c8c0d0e00 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 214087496 0 0) 0x7f1c74003520 con 0x7f1c8c0d0e00 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 >> v2:172.21.15.176:3300/0 conn(0x7f1c8c06ccd0 msgr2=0x7f1c8c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 --2- 172.21.15.67:0/49457773 >> v2:172.21.15.176:3300/0 conn(0x7f1c8c06ccd0 0x7f1c8c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 >> v1:172.21.15.176:6789/0 conn(0x7f1c8c0d4a50 legacy=0x7f1c8c0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c1613a0 con 0x7f1c8c0d0e00 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f1c7c001670 con 0x7f1c8c0d0e00 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 >> v1:172.21.15.67:6789/0 conn(0x7f1c8c0d0e00 legacy=0x7f1c8c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c740039b0 0x7f1c74003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c74004800 0x7f1c74004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c74005250 0x7f1c7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c1610c0 con 0x7f1c740039b0 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7400c1e0 con 0x7f1c74004800 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c7400c3e0 con 0x7f1c74005250 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c8ad9d640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c740039b0 0x7f1c74003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c8a59c640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c74004800 0x7f1c74004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c8b59e640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c74005250 0x7f1c7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c8ad9d640 1 -- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c74005250 msgr2=0x7f1c7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.503+0000 7f1c8ad9d640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c74005250 0x7f1c7400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c8ad9d640 1 -- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c74004800 msgr2=0x7f1c74004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c8ad9d640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c74004800 0x7f1c74004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c8ad9d640 1 -- 172.21.15.67:0/49457773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c74003520 con 0x7f1c740039b0 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c8b59e640 1 -- 172.21.15.67:0/49457773 reap_dead start 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c8a59c640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c74004800 0x7f1c74004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c8ad9d640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c740039b0 0x7f1c74003e20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c240 tx=0x7f1c74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3e9e5af5262c2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c007cf0 con 0x7f1c740039b0 2024-04-25T15:26:03.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c7c0058d0 con 0x7f1c740039b0 2024-04-25T15:26:03.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.504+0000 7f1c89d9b640 1 -- 172.21.15.67:0/49457773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c005d20 con 0x7f1c740039b0 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.505+0000 7f1c9100c640 1 -- 172.21.15.67:0/49457773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c740039b0 msgr2=0x7f1c74003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.505+0000 7f1c9100c640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c740039b0 0x7f1c74003e20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00c240 tx=0x7f1c74005830 comp rx=0 tx=0).stop 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.505+0000 7f1c9100c640 1 -- 172.21.15.67:0/49457773 shutdown_connections 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.505+0000 7f1c9100c640 1 --2- 172.21.15.67:0/49457773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c740039b0 0x7f1c74003e20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.505+0000 7f1c9100c640 1 -- 172.21.15.67:0/49457773 >> 172.21.15.67:0/49457773 conn(0x7f1c8c0d5d90 msgr2=0x7f1c8c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.505+0000 7f1c9100c640 1 -- 172.21.15.67:0/49457773 shutdown_connections 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.505+0000 7f1c9100c640 1 -- 172.21.15.67:0/49457773 wait complete. 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.507+0000 7f1c9100c640 1 Processor -- start 2024-04-25T15:26:03.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.507+0000 7f1c9100c640 1 -- start start 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.508+0000 7f1c9100c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c740039b0 0x7f1c8c0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.508+0000 7f1c9100c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c8c15a6f0 0x7f1c8c1520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.508+0000 7f1c9100c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c15ab00 0x7f1c8c15b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.508+0000 7f1c9100c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c164e00 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.508+0000 7f1c9100c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c0b5320 con 0x7f1c8c15ab00 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.508+0000 7f1c9100c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c164f80 con 0x7f1c740039b0 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c8c15a6f0 0x7f1c8c1520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c8c15a6f0 0x7f1c8c1520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60248/0 (socket says 172.21.15.67:60248) 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 -- 172.21.15.67:0/3703737573 learned_addr learned my addr 172.21.15.67:0/3703737573 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 -- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c740039b0 msgr2=0x7f1c8c0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c740039b0 0x7f1c8c0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 -- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c15ab00 msgr2=0x7f1c8c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c15ab00 0x7f1c8c15b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 -- 172.21.15.67:0/3703737573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c159650 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c8a59c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c8c15a6f0 0x7f1c8c1520f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1c800114e0 tx=0x7f1c800119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c737fe640 1 -- 172.21.15.67:0/3703737573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80015870 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c737fe640 1 -- 172.21.15.67:0/3703737573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c80015a10 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c737fe640 1 -- 172.21.15.67:0/3703737573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80015e00 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8c159930 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.509+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8c159ef0 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.510+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8c050740 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.511+0000 7f1c737fe640 1 -- 172.21.15.67:0/3703737573 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1c8001d400 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.511+0000 7f1c737fe640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1c780744e0 0x7f1c780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.511+0000 7f1c737fe640 1 -- 172.21.15.67:0/3703737573 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1c80095a10 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.512+0000 7f1c8ad9d640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1c780744e0 0x7f1c780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.512+0000 7f1c8ad9d640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1c780744e0 0x7f1c780769f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c033000 tx=0x7f1c7c007000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.516+0000 7f1c737fe640 1 -- 172.21.15.67:0/3703737573 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1c80019030 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.698+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1c8c0d4320 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.699+0000 7f1c737fe640 1 -- 172.21.15.67:0/3703737573 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f1c800601a0 con 0x7f1c8c15a6f0 2024-04-25T15:26:03.700 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1c780744e0 msgr2=0x7f1c780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1c780744e0 0x7f1c780769f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c033000 tx=0x7f1c7c007000 comp rx=0 tx=0).stop 2024-04-25T15:26:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c8c15a6f0 msgr2=0x7f1c8c1520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c8c15a6f0 0x7f1c8c1520f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1c800114e0 tx=0x7f1c800119a0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 shutdown_connections 2024-04-25T15:26:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1c780744e0 0x7f1c780769f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c15ab00 0x7f1c8c15b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c8c15a6f0 0x7f1c8c1520f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 --2- 172.21.15.67:0/3703737573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c740039b0 0x7f1c8c0ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 >> 172.21.15.67:0/3703737573 conn(0x7f1c8c0d5d90 msgr2=0x7f1c8c0b75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 shutdown_connections 2024-04-25T15:26:03.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.701+0000 7f1c9100c640 1 -- 172.21.15.67:0/3703737573 wait complete. 2024-04-25T15:26:03.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:03.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.827+0000 7f302f692640 1 Processor -- start 2024-04-25T15:26:03.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.828+0000 7f302f692640 1 -- start start 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f302f692640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f30280b77d0 0x7f3028164bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f302f692640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30280cffa0 con 0x7f30280d0d60 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f302f692640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30280d0280 con 0x7f30280d1170 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f302f692640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f30280b7be0 con 0x7f30280b77d0 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f302cc2d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f30280d0d60 0x7f3028160360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60212/0 (socket says 172.21.15.67:60212) 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f302cc2d640 1 -- 172.21.15.67:0/2266092100 learned_addr learned my addr 172.21.15.67:0/2266092100 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f302d42e640 1 --2- 172.21.15.67:0/2266092100 >> v2:172.21.15.176:3300/0 conn(0x7f30280b77d0 0x7f3028164bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592470046 0 0) 0x7f30280cffa0 con 0x7f30280d0d60 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.829+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3010003520 con 0x7f30280d0d60 2024-04-25T15:26:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1714025086 0 0) 0x7f3010003520 con 0x7f30280d0d60 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 >> v2:172.21.15.176:3300/0 conn(0x7f30280b77d0 msgr2=0x7f3028164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 --2- 172.21.15.67:0/2266092100 >> v2:172.21.15.176:3300/0 conn(0x7f30280b77d0 0x7f3028164bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 >> v1:172.21.15.176:6789/0 conn(0x7f30280d1170 legacy=0x7f30280b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30280d0560 con 0x7f30280d0d60 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f301c001670 con 0x7f30280d0d60 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302d42e640 1 --2- 172.21.15.67:0/2266092100 >> v2:172.21.15.176:3300/0 conn(0x7f30280b77d0 0x7f3028164bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 >> v1:172.21.15.67:6789/0 conn(0x7f30280d0d60 legacy=0x7f3028160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30100039b0 0x7f3010003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3010004800 0x7f3010004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30100051c0 0x7f301000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30280b7be0 con 0x7f30100039b0 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f301000c150 con 0x7f3010004800 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f301000c350 con 0x7f30100051c0 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302d42e640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30100039b0 0x7f3010003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302cc2d640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3010004800 0x7f3010004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302dc2f640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30100051c0 0x7f301000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302d42e640 1 -- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30100051c0 msgr2=0x7f301000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302d42e640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30100051c0 0x7f301000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302d42e640 1 -- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3010004800 msgr2=0x7f3010004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302d42e640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3010004800 0x7f3010004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302d42e640 1 -- 172.21.15.67:0/2266092100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3010003520 con 0x7f30100039b0 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.830+0000 7f302dc2f640 1 -- 172.21.15.67:0/2266092100 reap_dead start 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.831+0000 7f302d42e640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30100039b0 0x7f3010003e20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3020009c90 tx=0x7f3010005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acbb12c7a8b5980 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.831+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020036070 con 0x7f30100039b0 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.831+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f302002ae50 con 0x7f30100039b0 2024-04-25T15:26:03.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.831+0000 7f3027fff640 1 -- 172.21.15.67:0/2266092100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020009860 con 0x7f30100039b0 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.832+0000 7f302f692640 1 -- 172.21.15.67:0/2266092100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30100039b0 msgr2=0x7f3010003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.832+0000 7f302f692640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30100039b0 0x7f3010003e20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3020009c90 tx=0x7f3010005830 comp rx=0 tx=0).stop 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.832+0000 7f302f692640 1 -- 172.21.15.67:0/2266092100 shutdown_connections 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.832+0000 7f302f692640 1 --2- 172.21.15.67:0/2266092100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30100039b0 0x7f3010003e20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.832+0000 7f302f692640 1 -- 172.21.15.67:0/2266092100 >> 172.21.15.67:0/2266092100 conn(0x7f302809b7d0 msgr2=0x7f30280b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.832+0000 7f302f692640 1 -- 172.21.15.67:0/2266092100 shutdown_connections 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.832+0000 7f302f692640 1 -- 172.21.15.67:0/2266092100 wait complete. 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.834+0000 7f302f692640 1 Processor -- start 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.834+0000 7f302f692640 1 -- start start 2024-04-25T15:26:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302f692640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30100039b0 0x7f30281547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302f692640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30280ce260 0x7f302815ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302f692640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30280ce670 0x7f302815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302f692640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3028164f60 con 0x7f30280ce260 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302f692640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3028164de0 con 0x7f30100039b0 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302f692640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f30281650e0 con 0x7f30280ce670 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30280ce260 0x7f302815ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30280ce260 0x7f302815ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60260/0 (socket says 172.21.15.67:60260) 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 -- 172.21.15.67:0/2217192925 learned_addr learned my addr 172.21.15.67:0/2217192925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302d42e640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30100039b0 0x7f30281547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302dc2f640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30280ce670 0x7f302815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 -- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30280ce670 msgr2=0x7f302815d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30280ce670 0x7f302815d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 -- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30100039b0 msgr2=0x7f30281547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30100039b0 0x7f30281547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302cc2d640 1 -- 172.21.15.67:0/2217192925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30280a0190 con 0x7f30280ce260 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.835+0000 7f302d42e640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30100039b0 0x7f30281547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.836+0000 7f302cc2d640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30280ce260 0x7f302815ccf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f301c007340 tx=0x7f301c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.836+0000 7f3025ffb640 1 -- 172.21.15.67:0/2217192925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c01a070 con 0x7f30280ce260 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.836+0000 7f3025ffb640 1 -- 172.21.15.67:0/2217192925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f301c00f9e0 con 0x7f30280ce260 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.836+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30281f9b80 con 0x7f30280ce260 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.836+0000 7f3025ffb640 1 -- 172.21.15.67:0/2217192925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c00fdb0 con 0x7f30280ce260 2024-04-25T15:26:03.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.836+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30281fa140 con 0x7f30280ce260 2024-04-25T15:26:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.837+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f302806e2b0 con 0x7f30280ce260 2024-04-25T15:26:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.838+0000 7f3025ffb640 1 -- 172.21.15.67:0/2217192925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f301c01d070 con 0x7f30280ce260 2024-04-25T15:26:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.838+0000 7f3025ffb640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30140744e0 0x7f30140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.838+0000 7f3025ffb640 1 -- 172.21.15.67:0/2217192925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f301c094c30 con 0x7f30280ce260 2024-04-25T15:26:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.839+0000 7f302d42e640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30140744e0 0x7f30140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:03.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.839+0000 7f302d42e640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30140744e0 0x7f30140769f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3020033000 tx=0x7f3020008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:03.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:03.843+0000 7f3025ffb640 1 -- 172.21.15.67:0/2217192925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f301c05f450 con 0x7f30280ce260 2024-04-25T15:26:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.028+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f30281f9910 con 0x7f30280ce260 2024-04-25T15:26:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.029+0000 7f3025ffb640 1 -- 172.21.15.67:0/2217192925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f301c05ed50 con 0x7f30280ce260 2024-04-25T15:26:04.030 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.030+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30140744e0 msgr2=0x7f30140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.030+0000 7f302f692640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30140744e0 0x7f30140769f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3020033000 tx=0x7f3020008030 comp rx=0 tx=0).stop 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.030+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30280ce260 msgr2=0x7f302815ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.030+0000 7f302f692640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30280ce260 0x7f302815ccf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f301c007340 tx=0x7f301c007800 comp rx=0 tx=0).stop 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.030+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 shutdown_connections 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.031+0000 7f302f692640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30140744e0 0x7f30140769f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.031+0000 7f302f692640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30280ce670 0x7f302815d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.031+0000 7f302f692640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30280ce260 0x7f302815ccf0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.031+0000 7f302f692640 1 --2- 172.21.15.67:0/2217192925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30100039b0 0x7f30281547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.031+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 >> 172.21.15.67:0/2217192925 conn(0x7f302809b7d0 msgr2=0x7f3028070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.031+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 shutdown_connections 2024-04-25T15:26:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.031+0000 7f302f692640 1 -- 172.21.15.67:0/2217192925 wait complete. 2024-04-25T15:26:04.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.156+0000 7f6997a42640 1 Processor -- start 2024-04-25T15:26:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.156+0000 7f6997a42640 1 -- start start 2024-04-25T15:26:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6997a42640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f69900d1e20 0x7f69900d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6997a42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6990164020 0x7f6990164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6997a42640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6990164970 con 0x7f69900b2e60 2024-04-25T15:26:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6997a42640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f69900d23f0 con 0x7f69900d1e20 2024-04-25T15:26:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6997a42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f69900d26d0 con 0x7f6990164020 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6994fdd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f69900b2e60 0x7f699008d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60216/0 (socket says 172.21.15.67:60216) 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6994fdd640 1 -- 172.21.15.67:0/1754087834 learned_addr learned my addr 172.21.15.67:0/1754087834 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f69957de640 1 --2- 172.21.15.67:0/1754087834 >> v2:172.21.15.176:3300/0 conn(0x7f69900d1e20 0x7f69900d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f6995fdf640 1 --2- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6990164020 0x7f6990164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f698ffff640 1 -- 172.21.15.67:0/1754087834 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343390807 0 0) 0x7f6984001110 con 0x7f69900b2e60 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f698ffff640 1 -- 172.21.15.67:0/1754087834 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6978003520 con 0x7f69900b2e60 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f698ffff640 1 -- 172.21.15.67:0/1754087834 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3463866229 0 0) 0x7f6978003520 con 0x7f69900b2e60 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f69957de640 1 -- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6990164020 msgr2=0x7f6990164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f69957de640 1 --2- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6990164020 0x7f6990164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f69957de640 1 -- 172.21.15.67:0/1754087834 >> v1:172.21.15.67:6789/0 conn(0x7f69900b2e60 legacy=0x7f699008d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.158+0000 7f69957de640 1 -- 172.21.15.67:0/1754087834 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69900d29b0 con 0x7f69900d1e20 2024-04-25T15:26:04.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.159+0000 7f69957de640 1 --2- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69900d1e20 0x7f69900d3490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f69880097a0 tx=0x7f698802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14bc4996d815e192 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:04.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.159+0000 7f698ffff640 1 -- 172.21.15.67:0/1754087834 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988037070 con 0x7f69900d1e20 2024-04-25T15:26:04.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.159+0000 7f698ffff640 1 -- 172.21.15.67:0/1754087834 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698802ac30 con 0x7f69900d1e20 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.159+0000 7f698ffff640 1 -- 172.21.15.67:0/1754087834 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984001840 con 0x7f69900d1e20 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 -- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69900d1e20 msgr2=0x7f69900d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 --2- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69900d1e20 0x7f69900d3490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f69880097a0 tx=0x7f698802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 -- 172.21.15.67:0/1754087834 shutdown_connections 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 --2- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6990164020 0x7f6990164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 --2- 172.21.15.67:0/1754087834 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69900d1e20 0x7f69900d3490 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 -- 172.21.15.67:0/1754087834 >> 172.21.15.67:0/1754087834 conn(0x7f69900d0530 msgr2=0x7f69900d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 -- 172.21.15.67:0/1754087834 shutdown_connections 2024-04-25T15:26:04.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.160+0000 7f6997a42640 1 -- 172.21.15.67:0/1754087834 wait complete. 2024-04-25T15:26:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.162+0000 7f6997a42640 1 Processor -- start 2024-04-25T15:26:04.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.162+0000 7f6997a42640 1 -- start start 2024-04-25T15:26:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f6997a42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69900b2e60 0x7f6990160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f6997a42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d1e20 0x7f69900ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f6997a42640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6990164020 0x7f69900cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f6997a42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69900ce070 con 0x7f69900b2e60 2024-04-25T15:26:04.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f6997a42640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f69900d2f70 con 0x7f6990164020 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f6997a42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f69900ce1f0 con 0x7f69900d1e20 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f69957de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69900b2e60 0x7f6990160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f69957de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69900b2e60 0x7f6990160ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60266/0 (socket says 172.21.15.67:60266) 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.163+0000 7f69957de640 1 -- 172.21.15.67:0/3596523049 learned_addr learned my addr 172.21.15.67:0/3596523049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f6994fdd640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d1e20 0x7f69900ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f6995fdf640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6990164020 0x7f69900cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f69957de640 1 -- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d1e20 msgr2=0x7f69900ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f69957de640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d1e20 0x7f69900ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f69957de640 1 -- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6990164020 msgr2=0x7f69900cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f69957de640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6990164020 0x7f69900cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f69957de640 1 -- 172.21.15.67:0/3596523049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69900b3fa0 con 0x7f69900b2e60 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f69957de640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69900b2e60 0x7f6990160ab0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6988035040 tx=0x7f69880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:04.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f698dffb640 1 -- 172.21.15.67:0/3596523049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988041070 con 0x7f69900b2e60 2024-04-25T15:26:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f698dffb640 1 -- 172.21.15.67:0/3596523049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698803c660 con 0x7f69900b2e60 2024-04-25T15:26:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69900b4280 con 0x7f69900b2e60 2024-04-25T15:26:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.164+0000 7f698dffb640 1 -- 172.21.15.67:0/3596523049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988037040 con 0x7f69900b2e60 2024-04-25T15:26:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.165+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69900b4840 con 0x7f69900b2e60 2024-04-25T15:26:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.166+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6990050740 con 0x7f69900b2e60 2024-04-25T15:26:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.166+0000 7f698dffb640 1 -- 172.21.15.67:0/3596523049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6988044070 con 0x7f69900b2e60 2024-04-25T15:26:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.167+0000 7f698dffb640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f697c0744b0 0x7f697c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.167+0000 7f698dffb640 1 -- 172.21.15.67:0/3596523049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f69880b59a0 con 0x7f69900b2e60 2024-04-25T15:26:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.167+0000 7f6994fdd640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f697c0744b0 0x7f697c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.167+0000 7f6994fdd640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f697c0744b0 0x7f697c0769c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f69900b5180 tx=0x7f6984001230 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:04.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.171+0000 7f698dffb640 1 -- 172.21.15.67:0/3596523049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f698807fae0 con 0x7f69900b2e60 2024-04-25T15:26:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.354+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f699006ebd0 con 0x7f69900b2e60 2024-04-25T15:26:04.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.355+0000 7f698dffb640 1 -- 172.21.15.67:0/3596523049 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f6988083de0 con 0x7f69900b2e60 2024-04-25T15:26:04.356 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f697c0744b0 msgr2=0x7f697c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f697c0744b0 0x7f697c0769c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f69900b5180 tx=0x7f6984001230 comp rx=0 tx=0).stop 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69900b2e60 msgr2=0x7f6990160ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69900b2e60 0x7f6990160ab0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6988035040 tx=0x7f69880046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 shutdown_connections 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f697c0744b0 0x7f697c0769c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6990164020 0x7f69900cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d1e20 0x7f69900ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 --2- 172.21.15.67:0/3596523049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69900b2e60 0x7f6990160ab0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 >> 172.21.15.67:0/3596523049 conn(0x7f69900d0530 msgr2=0x7f69900b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 shutdown_connections 2024-04-25T15:26:04.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.357+0000 7f6997a42640 1 -- 172.21.15.67:0/3596523049 wait complete. 2024-04-25T15:26:04.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:26:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.481+0000 7f40b2f08640 1 Processor -- start 2024-04-25T15:26:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.481+0000 7f40b2f08640 1 -- start start 2024-04-25T15:26:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40b2f08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40ac0b8290 0x7f40ac160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40b2f08640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40ac1615f0 con 0x7f40ac15df80 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40b2f08640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40ac164020 con 0x7f40ac164b00 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40b2f08640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f40ac164670 con 0x7f40ac0b8290 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40abfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40ac0b8290 0x7f40ac160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40abfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40ac0b8290 0x7f40ac160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60280/0 (socket says 172.21.15.67:60280) 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40abfff640 1 -- 172.21.15.67:0/216342258 learned_addr learned my addr 172.21.15.67:0/216342258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.482+0000 7f40abfff640 1 -- 172.21.15.67:0/216342258 >> v1:172.21.15.176:6789/0 conn(0x7f40ac164b00 legacy=0x7f40ac0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.483+0000 7f40abfff640 1 -- 172.21.15.67:0/216342258 >> v1:172.21.15.67:6789/0 conn(0x7f40ac15df80 legacy=0x7f40ac160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.483+0000 7f40ab7fe640 1 -- 172.21.15.67:0/216342258 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507862587 0 0) 0x7f40ac1615f0 con 0x7f40ac15df80 2024-04-25T15:26:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.483+0000 7f40abfff640 1 -- 172.21.15.67:0/216342258 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40ac164300 con 0x7f40ac0b8290 2024-04-25T15:26:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.483+0000 7f40abfff640 1 --2- 172.21.15.67:0/216342258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0b8290 0x7f40ac160890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f40ac1647f0 tx=0x7f409402a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920b90cba2725eac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.483+0000 7f40ab7fe640 1 -- 172.21.15.67:0/216342258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094037070 con 0x7f40ac0b8290 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.483+0000 7f40ab7fe640 1 -- 172.21.15.67:0/216342258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4094032440 con 0x7f40ac0b8290 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.483+0000 7f40ab7fe640 1 -- 172.21.15.67:0/216342258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094032810 con 0x7f40ac0b8290 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.484+0000 7f40b2f08640 1 -- 172.21.15.67:0/216342258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0b8290 msgr2=0x7f40ac160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.484+0000 7f40b2f08640 1 --2- 172.21.15.67:0/216342258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0b8290 0x7f40ac160890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f40ac1647f0 tx=0x7f409402a290 comp rx=0 tx=0).stop 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.484+0000 7f40b2f08640 1 -- 172.21.15.67:0/216342258 shutdown_connections 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.484+0000 7f40b2f08640 1 --2- 172.21.15.67:0/216342258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac0b8290 0x7f40ac160890 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.484+0000 7f40b2f08640 1 -- 172.21.15.67:0/216342258 >> 172.21.15.67:0/216342258 conn(0x7f40ac0d2bc0 msgr2=0x7f40ac0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.484+0000 7f40b2f08640 1 -- 172.21.15.67:0/216342258 shutdown_connections 2024-04-25T15:26:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.484+0000 7f40b2f08640 1 -- 172.21.15.67:0/216342258 wait complete. 2024-04-25T15:26:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.486+0000 7f40b2f08640 1 Processor -- start 2024-04-25T15:26:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.486+0000 7f40b2f08640 1 -- start start 2024-04-25T15:26:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.487+0000 7f40b2f08640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40ac15df80 0x7f40ac1f9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.487+0000 7f40b2f08640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40ac164b00 0x7f40ac147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.487+0000 7f40b2f08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac1636c0 0x7f40ac0d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.487+0000 7f40b2f08640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac0b4c90 con 0x7f40ac1636c0 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.487+0000 7f40b2f08640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac0b4b10 con 0x7f40ac164b00 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.487+0000 7f40b2f08640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f40ac0b4e10 con 0x7f40ac15df80 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac1636c0 0x7f40ac0d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac1636c0 0x7f40ac0d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60290/0 (socket says 172.21.15.67:60290) 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 -- 172.21.15.67:0/3827813151 learned_addr learned my addr 172.21.15.67:0/3827813151 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b0ca4640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40ac15df80 0x7f40ac1f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40abfff640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40ac164b00 0x7f40ac147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 -- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40ac15df80 msgr2=0x7f40ac1f9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40ac15df80 0x7f40ac1f9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 -- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40ac164b00 msgr2=0x7f40ac147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40ac164b00 0x7f40ac147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 -- 172.21.15.67:0/3827813151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40ac1fa170 con 0x7f40ac1636c0 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b14a5640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac1636c0 0x7f40ac0d0350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f40a400c780 tx=0x7f40a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40a97fa640 1 -- 172.21.15.67:0/3827813151 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4003da0 con 0x7f40ac1636c0 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40a97fa640 1 -- 172.21.15.67:0/3827813151 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a400ce30 con 0x7f40ac1636c0 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40a97fa640 1 -- 172.21.15.67:0/3827813151 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a4005290 con 0x7f40ac1636c0 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40ac1fa450 con 0x7f40ac1636c0 2024-04-25T15:26:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.488+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40ac152ad0 con 0x7f40ac1636c0 2024-04-25T15:26:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.489+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40ac050740 con 0x7f40ac1636c0 2024-04-25T15:26:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.490+0000 7f40a97fa640 1 -- 172.21.15.67:0/3827813151 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f40a4005430 con 0x7f40ac1636c0 2024-04-25T15:26:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.491+0000 7f40a97fa640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f408c0744b0 0x7f408c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.491+0000 7f40a97fa640 1 -- 172.21.15.67:0/3827813151 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f40a4094fd0 con 0x7f40ac1636c0 2024-04-25T15:26:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.491+0000 7f40b0ca4640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f408c0744b0 0x7f408c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:04.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.491+0000 7f40b0ca4640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f408c0744b0 0x7f408c0769c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f409c006fd0 tx=0x7f409c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:04.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.495+0000 7f40a97fa640 1 -- 172.21.15.67:0/3827813151 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40a405f110 con 0x7f40ac1636c0 2024-04-25T15:26:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.697+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f40ac0d4ab0 con 0x7f40ac1636c0 2024-04-25T15:26:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.698+0000 7f40a97fa640 1 -- 172.21.15.67:0/3827813151 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f40a4063410 con 0x7f40ac1636c0 2024-04-25T15:26:04.699 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f408c0744b0 msgr2=0x7f408c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f408c0744b0 0x7f408c0769c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f409c006fd0 tx=0x7f409c009060 comp rx=0 tx=0).stop 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac1636c0 msgr2=0x7f40ac0d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac1636c0 0x7f40ac0d0350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f40a400c780 tx=0x7f40a400cc40 comp rx=0 tx=0).stop 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 shutdown_connections 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f408c0744b0 0x7f408c0769c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40ac1636c0 0x7f40ac0d0350 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40ac164b00 0x7f40ac147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 --2- 172.21.15.67:0/3827813151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40ac15df80 0x7f40ac1f9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 >> 172.21.15.67:0/3827813151 conn(0x7f40ac0d2bc0 msgr2=0x7f40ac15f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 shutdown_connections 2024-04-25T15:26:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:04.700+0000 7f40b2f08640 1 -- 172.21.15.67:0/3827813151 wait complete. 2024-04-25T15:26:04.712 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-25T15:26:04.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:04.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-25T15:26:04.738 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:26:04.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:04.738 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T15:26:04.793 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:04.793 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:04.812 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:04.812 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:04.870 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:04.871 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:04.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:05.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.040+0000 7f9c4a84a640 1 Processor -- start 2024-04-25T15:26:05.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.040+0000 7f9c4a84a640 1 -- start start 2024-04-25T15:26:05.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.041+0000 7f9c4a84a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9c440ce800 0x7f9c440d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.041+0000 7f9c4a84a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c44163130 0x7f9c440b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.041+0000 7f9c437fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9c440d1620 0x7f9c44162a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60234/0 (socket says 172.21.15.67:60234) 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.041+0000 7f9c437fe640 1 -- 172.21.15.67:0/1488249406 learned_addr learned my addr 172.21.15.67:0/1488249406 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.041+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c440b0770 con 0x7f9c440d1620 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.041+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9c440b10a0 con 0x7f9c44163130 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.041+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9c440d2100 con 0x7f9c440ce800 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c48de7640 1 --2- 172.21.15.67:0/1488249406 >> v2:172.21.15.67:3300/0 conn(0x7f9c44163130 0x7f9c440b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c43fff640 1 --2- 172.21.15.67:0/1488249406 >> v2:172.21.15.176:3300/0 conn(0x7f9c440ce800 0x7f9c440d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c42ffd640 1 -- 172.21.15.67:0/1488249406 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575731825 0 0) 0x7f9c440b0770 con 0x7f9c440d1620 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c42ffd640 1 -- 172.21.15.67:0/1488249406 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c2c003520 con 0x7f9c440d1620 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c48de7640 1 -- 172.21.15.67:0/1488249406 >> v2:172.21.15.176:3300/0 conn(0x7f9c440ce800 msgr2=0x7f9c440d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c48de7640 1 --2- 172.21.15.67:0/1488249406 >> v2:172.21.15.176:3300/0 conn(0x7f9c440ce800 0x7f9c440d1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c48de7640 1 -- 172.21.15.67:0/1488249406 >> v1:172.21.15.67:6789/0 conn(0x7f9c440d1620 legacy=0x7f9c44162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c48de7640 1 -- 172.21.15.67:0/1488249406 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c440b0a50 con 0x7f9c44163130 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.042+0000 7f9c43fff640 1 --2- 172.21.15.67:0/1488249406 >> v2:172.21.15.176:3300/0 conn(0x7f9c440ce800 0x7f9c440d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.043+0000 7f9c48de7640 1 --2- 172.21.15.67:0/1488249406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c440b2c30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9c4408e040 tx=0x7f9c3c02a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f9779f6a53f15f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.043+0000 7f9c42ffd640 1 -- 172.21.15.67:0/1488249406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c007740 con 0x7f9c44163130 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.043+0000 7f9c42ffd640 1 -- 172.21.15.67:0/1488249406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c3c0078e0 con 0x7f9c44163130 2024-04-25T15:26:05.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.043+0000 7f9c42ffd640 1 -- 172.21.15.67:0/1488249406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c007cb0 con 0x7f9c44163130 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.043+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 msgr2=0x7f9c440b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.044+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/1488249406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c440b2c30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9c4408e040 tx=0x7f9c3c02a560 comp rx=0 tx=0).stop 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.044+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 shutdown_connections 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.044+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/1488249406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c440b2c30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.044+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/1488249406 >> v2:172.21.15.176:3300/0 conn(0x7f9c440ce800 0x7f9c440d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.044+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 >> 172.21.15.67:0/1488249406 conn(0x7f9c440d3a50 msgr2=0x7f9c440d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.044+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 shutdown_connections 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.044+0000 7f9c4a84a640 1 -- 172.21.15.67:0/1488249406 wait complete. 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.045+0000 7f9c4a84a640 1 Processor -- start 2024-04-25T15:26:05.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.046+0000 7f9c4a84a640 1 -- start start 2024-04-25T15:26:05.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c4a84a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c440ce800 0x7f9c4406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c4a84a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c440d1620 0x7f9c4406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c4a84a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c4414ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c4a84a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c44163540 con 0x7f9c44163130 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c4a84a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9c440b51c0 con 0x7f9c440d1620 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c4a84a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9c44154780 con 0x7f9c440ce800 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c4414ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c4414ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60308/0 (socket says 172.21.15.67:60308) 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 -- 172.21.15.67:0/3469342028 learned_addr learned my addr 172.21.15.67:0/3469342028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c437fe640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c440d1620 0x7f9c4406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c43fff640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c440ce800 0x7f9c4406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 -- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c440ce800 msgr2=0x7f9c4406efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c440ce800 0x7f9c4406efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 -- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c440d1620 msgr2=0x7f9c4406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c440d1620 0x7f9c4406e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c48de7640 1 -- 172.21.15.67:0/3469342028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c440b2f20 con 0x7f9c44163130 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.047+0000 7f9c437fe640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c440d1620 0x7f9c4406e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.048+0000 7f9c48de7640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c4414ffb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009e50 tx=0x7f9c3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.048+0000 7f9c40ff9640 1 -- 172.21.15.67:0/3469342028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c03f070 con 0x7f9c44163130 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.048+0000 7f9c40ff9640 1 -- 172.21.15.67:0/3469342028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c3c031c90 con 0x7f9c44163130 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.048+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c440b3200 con 0x7f9c44163130 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.048+0000 7f9c40ff9640 1 -- 172.21.15.67:0/3469342028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c009450 con 0x7f9c44163130 2024-04-25T15:26:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.048+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c440b37c0 con 0x7f9c44163130 2024-04-25T15:26:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.049+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c440b2c80 con 0x7f9c44163130 2024-04-25T15:26:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.049+0000 7f9c40ff9640 1 -- 172.21.15.67:0/3469342028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9c3c043650 con 0x7f9c44163130 2024-04-25T15:26:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.050+0000 7f9c40ff9640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9c300744b0 0x7f9c300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.050+0000 7f9c40ff9640 1 -- 172.21.15.67:0/3469342028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9c3c0b6540 con 0x7f9c44163130 2024-04-25T15:26:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.050+0000 7f9c43fff640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9c300744b0 0x7f9c300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.051+0000 7f9c43fff640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9c300744b0 0x7f9c300769c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400b9f0 tx=0x7f9c34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.055+0000 7f9c40ff9640 1 -- 172.21.15.67:0/3469342028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c3c080df0 con 0x7f9c44163130 2024-04-25T15:26:05.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.242+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c4406f500 con 0x7f9c44163130 2024-04-25T15:26:05.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.244+0000 7f9c40ff9640 1 -- 172.21.15.67:0/3469342028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f9c3c0806f0 con 0x7f9c44163130 2024-04-25T15:26:05.245 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:05.245 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9c300744b0 msgr2=0x7f9c300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9c300744b0 0x7f9c300769c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400b9f0 tx=0x7f9c34008040 comp rx=0 tx=0).stop 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 msgr2=0x7f9c4414ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c4414ffb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009e50 tx=0x7f9c3c004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 shutdown_connections 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9c300744b0 0x7f9c300769c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c44163130 0x7f9c4414ffb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c440d1620 0x7f9c4406e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 --2- 172.21.15.67:0/3469342028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c440ce800 0x7f9c4406efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 >> 172.21.15.67:0/3469342028 conn(0x7f9c440d3a50 msgr2=0x7f9c440d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:05.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 shutdown_connections 2024-04-25T15:26:05.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.246+0000 7f9c4a84a640 1 -- 172.21.15.67:0/3469342028 wait complete. 2024-04-25T15:26:05.259 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:05.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:05.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.375+0000 7fe839955640 1 Processor -- start 2024-04-25T15:26:05.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.375+0000 7fe839955640 1 -- start start 2024-04-25T15:26:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe839955640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe834164b00 0x7fe8340b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe839955640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8341615f0 con 0x7fe83415df80 2024-04-25T15:26:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe839955640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe834164020 con 0x7fe834160fa0 2024-04-25T15:26:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe839955640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe8341613b0 con 0x7fe834164b00 2024-04-25T15:26:05.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe832ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe834164b00 0x7fe8340b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe8327fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe83415df80 0x7fe834160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60244/0 (socket says 172.21.15.67:60244) 2024-04-25T15:26:05.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe832ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe834164b00 0x7fe8340b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60320/0 (socket says 172.21.15.67:60320) 2024-04-25T15:26:05.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe8327fc640 1 -- 172.21.15.67:0/1104364082 learned_addr learned my addr 172.21.15.67:0/1104364082 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:05.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe832ffd640 1 -- 172.21.15.67:0/1104364082 >> v1:172.21.15.176:6789/0 conn(0x7fe834160fa0 legacy=0x7fe8340b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe832ffd640 1 -- 172.21.15.67:0/1104364082 >> v1:172.21.15.67:6789/0 conn(0x7fe83415df80 legacy=0x7fe834160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.376+0000 7fe832ffd640 1 -- 172.21.15.67:0/1104364082 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe834164300 con 0x7fe834164b00 2024-04-25T15:26:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.377+0000 7fe832ffd640 1 --2- 172.21.15.67:0/1104364082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe834164b00 0x7fe8340b7b80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0097a0 tx=0x7fe82c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a4892f1bc458c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.377+0000 7fe831ffb640 1 -- 172.21.15.67:0/1104364082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c037070 con 0x7fe834164b00 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.377+0000 7fe831ffb640 1 -- 172.21.15.67:0/1104364082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c032440 con 0x7fe834164b00 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.377+0000 7fe831ffb640 1 -- 172.21.15.67:0/1104364082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c032810 con 0x7fe834164b00 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.378+0000 7fe839955640 1 -- 172.21.15.67:0/1104364082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe834164b00 msgr2=0x7fe8340b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.378+0000 7fe839955640 1 --2- 172.21.15.67:0/1104364082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe834164b00 0x7fe8340b7b80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0097a0 tx=0x7fe82c02a230 comp rx=0 tx=0).stop 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.378+0000 7fe839955640 1 -- 172.21.15.67:0/1104364082 shutdown_connections 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.378+0000 7fe839955640 1 --2- 172.21.15.67:0/1104364082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe834164b00 0x7fe8340b7b80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.378+0000 7fe839955640 1 -- 172.21.15.67:0/1104364082 >> 172.21.15.67:0/1104364082 conn(0x7fe8340d2bc0 msgr2=0x7fe8340d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.378+0000 7fe839955640 1 -- 172.21.15.67:0/1104364082 shutdown_connections 2024-04-25T15:26:05.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.378+0000 7fe839955640 1 -- 172.21.15.67:0/1104364082 wait complete. 2024-04-25T15:26:05.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.380+0000 7fe839955640 1 Processor -- start 2024-04-25T15:26:05.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.380+0000 7fe839955640 1 -- start start 2024-04-25T15:26:05.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe839955640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe83415df80 0x7fe8340cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe839955640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe834160fa0 0x7fe8340cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe839955640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe834164b00 0x7fe83414c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe839955640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8340b84d0 con 0x7fe83415df80 2024-04-25T15:26:05.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe839955640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8340b8350 con 0x7fe834164b00 2024-04-25T15:26:05.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe839955640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe834164670 con 0x7fe834160fa0 2024-04-25T15:26:05.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe832ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe83415df80 0x7fe8340cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe832ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe83415df80 0x7fe8340cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60336/0 (socket says 172.21.15.67:60336) 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.381+0000 7fe832ffd640 1 -- 172.21.15.67:0/1050165570 learned_addr learned my addr 172.21.15.67:0/1050165570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe8327fc640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe834160fa0 0x7fe8340cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe8337fe640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe834164b00 0x7fe83414c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe832ffd640 1 -- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe834160fa0 msgr2=0x7fe8340cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe832ffd640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe834160fa0 0x7fe8340cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe832ffd640 1 -- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe834164b00 msgr2=0x7fe83414c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe832ffd640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe834164b00 0x7fe83414c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe832ffd640 1 -- 172.21.15.67:0/1050165570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe834149fb0 con 0x7fe83415df80 2024-04-25T15:26:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe832ffd640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe83415df80 0x7fe8340cc920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe82c02a270 tx=0x7fe82c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe82bfff640 1 -- 172.21.15.67:0/1050165570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c037040 con 0x7fe83415df80 2024-04-25T15:26:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe82bfff640 1 -- 172.21.15.67:0/1050165570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c042070 con 0x7fe83415df80 2024-04-25T15:26:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83414a420 con 0x7fe83415df80 2024-04-25T15:26:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe82bfff640 1 -- 172.21.15.67:0/1050165570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c030bb0 con 0x7fe83415df80 2024-04-25T15:26:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.382+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83414a9e0 con 0x7fe83415df80 2024-04-25T15:26:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.383+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe834050740 con 0x7fe83415df80 2024-04-25T15:26:05.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.384+0000 7fe82bfff640 1 -- 172.21.15.67:0/1050165570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe82c047460 con 0x7fe83415df80 2024-04-25T15:26:05.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.384+0000 7fe82bfff640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8140744b0 0x7fe8140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.385+0000 7fe82bfff640 1 -- 172.21.15.67:0/1050165570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe82c0b5bf0 con 0x7fe83415df80 2024-04-25T15:26:05.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.385+0000 7fe8327fc640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8140744b0 0x7fe8140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.385+0000 7fe8327fc640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8140744b0 0x7fe8140769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe834149e80 tx=0x7fe81c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.389+0000 7fe82bfff640 1 -- 172.21.15.67:0/1050165570 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe82c07fd30 con 0x7fe83415df80 2024-04-25T15:26:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.573+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8340d4170 con 0x7fe83415df80 2024-04-25T15:26:05.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.574+0000 7fe82bfff640 1 -- 172.21.15.67:0/1050165570 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fe82c088400 con 0x7fe83415df80 2024-04-25T15:26:05.575 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8140744b0 msgr2=0x7fe8140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8140744b0 0x7fe8140769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe834149e80 tx=0x7fe81c009040 comp rx=0 tx=0).stop 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe83415df80 msgr2=0x7fe8340cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe83415df80 0x7fe8340cc920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe82c02a270 tx=0x7fe82c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 shutdown_connections 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8140744b0 0x7fe8140769c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe834164b00 0x7fe83414c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe834160fa0 0x7fe8340cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 --2- 172.21.15.67:0/1050165570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe83415df80 0x7fe8340cc920 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 >> 172.21.15.67:0/1050165570 conn(0x7fe8340d2bc0 msgr2=0x7fe83415f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.575+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 shutdown_connections 2024-04-25T15:26:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.576+0000 7fe839955640 1 -- 172.21.15.67:0/1050165570 wait complete. 2024-04-25T15:26:05.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:05.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.702+0000 7f14f9906640 1 Processor -- start 2024-04-25T15:26:05.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.702+0000 7f14f9906640 1 -- start start 2024-04-25T15:26:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f9906640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f415e3b0 0x7f14f40cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f9906640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f14f40d28f0 0x7f14f40d2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f9906640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14f40d3240 con 0x7f14f406c5c0 2024-04-25T15:26:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f9906640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f14f40cfe60 con 0x7f14f40d28f0 2024-04-25T15:26:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f9906640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f14f40d0140 con 0x7f14f415e3b0 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f27fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f415e3b0 0x7f14f40cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f37fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f14f40d28f0 0x7f14f40d2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f27fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f415e3b0 0x7f14f40cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52496/0 (socket says 172.21.15.67:52496) 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.703+0000 7f14f27fc640 1 -- 172.21.15.67:0/1409560460 learned_addr learned my addr 172.21.15.67:0/1409560460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.704+0000 7f14f27fc640 1 -- 172.21.15.67:0/1409560460 >> v2:172.21.15.176:3300/0 conn(0x7f14f40d28f0 msgr2=0x7f14f40d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.704+0000 7f14f27fc640 1 --2- 172.21.15.67:0/1409560460 >> v2:172.21.15.176:3300/0 conn(0x7f14f40d28f0 0x7f14f40d2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.704+0000 7f14f27fc640 1 -- 172.21.15.67:0/1409560460 >> v1:172.21.15.176:6789/0 conn(0x7f14f406c5c0 legacy=0x7f14f40cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.704+0000 7f14f27fc640 1 -- 172.21.15.67:0/1409560460 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f40b3bf0 con 0x7f14f415e3b0 2024-04-25T15:26:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.704+0000 7f14f37fe640 1 --2- 172.21.15.67:0/1409560460 >> v2:172.21.15.176:3300/0 conn(0x7f14f40d28f0 0x7f14f40d2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.705+0000 7f14f27fc640 1 --2- 172.21.15.67:0/1409560460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f415e3b0 0x7f14f40cef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009870 tx=0x7f14dc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dfd6740d7167c34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.705+0000 7f14f1ffb640 1 -- 172.21.15.67:0/1409560460 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc037070 con 0x7f14f415e3b0 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.705+0000 7f14f1ffb640 1 -- 172.21.15.67:0/1409560460 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14dc032440 con 0x7f14f415e3b0 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.705+0000 7f14f1ffb640 1 -- 172.21.15.67:0/1409560460 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc032810 con 0x7f14f415e3b0 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.705+0000 7f14f9906640 1 -- 172.21.15.67:0/1409560460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f415e3b0 msgr2=0x7f14f40cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.706+0000 7f14f9906640 1 --2- 172.21.15.67:0/1409560460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f415e3b0 0x7f14f40cef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009870 tx=0x7f14dc02a270 comp rx=0 tx=0).stop 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.706+0000 7f14f9906640 1 -- 172.21.15.67:0/1409560460 shutdown_connections 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.706+0000 7f14f9906640 1 --2- 172.21.15.67:0/1409560460 >> v2:172.21.15.176:3300/0 conn(0x7f14f40d28f0 0x7f14f40d2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.706+0000 7f14f9906640 1 --2- 172.21.15.67:0/1409560460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f415e3b0 0x7f14f40cef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.706+0000 7f14f9906640 1 -- 172.21.15.67:0/1409560460 >> 172.21.15.67:0/1409560460 conn(0x7f14f409b7d0 msgr2=0x7f14f409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.706+0000 7f14f9906640 1 -- 172.21.15.67:0/1409560460 shutdown_connections 2024-04-25T15:26:05.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.706+0000 7f14f9906640 1 -- 172.21.15.67:0/1409560460 wait complete. 2024-04-25T15:26:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.707+0000 7f14f9906640 1 Processor -- start 2024-04-25T15:26:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.708+0000 7f14f9906640 1 -- start start 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f9906640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f406c5c0 0x7f14f40d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f9906640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f40d28f0 0x7f14f41f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f9906640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f415e3b0 0x7f14f4071600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f9906640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f14f40b4570 con 0x7f14f415e3b0 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f9906640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f14f40b43f0 con 0x7f14f406c5c0 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f9906640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f14f40b46f0 con 0x7f14f40d28f0 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f415e3b0 0x7f14f4071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f415e3b0 0x7f14f4071600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60344/0 (socket says 172.21.15.67:60344) 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 -- 172.21.15.67:0/4242624590 learned_addr learned my addr 172.21.15.67:0/4242624590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f2ffd640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f406c5c0 0x7f14f40d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f27fc640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f40d28f0 0x7f14f41f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 -- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f40d28f0 msgr2=0x7f14f41f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f40d28f0 0x7f14f41f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 -- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f406c5c0 msgr2=0x7f14f40d0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f406c5c0 0x7f14f40d0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f37fe640 1 -- 172.21.15.67:0/4242624590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f40d1390 con 0x7f14f415e3b0 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.709+0000 7f14f27fc640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f40d28f0 0x7f14f41f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.710+0000 7f14f37fe640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f415e3b0 0x7f14f4071600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f14e400efc0 tx=0x7f14e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.710+0000 7f14ebfff640 1 -- 172.21.15.67:0/4242624590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4018030 con 0x7f14f415e3b0 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.710+0000 7f14ebfff640 1 -- 172.21.15.67:0/4242624590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e400cd30 con 0x7f14f415e3b0 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.710+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f40d1670 con 0x7f14f415e3b0 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.710+0000 7f14ebfff640 1 -- 172.21.15.67:0/4242624590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4003bb0 con 0x7f14f415e3b0 2024-04-25T15:26:05.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.710+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f40d1c30 con 0x7f14f415e3b0 2024-04-25T15:26:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.711+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f4050740 con 0x7f14f415e3b0 2024-04-25T15:26:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.711+0000 7f14ebfff640 1 -- 172.21.15.67:0/4242624590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f14e401b070 con 0x7f14f415e3b0 2024-04-25T15:26:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.712+0000 7f14ebfff640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f14d40744b0 0x7f14d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:05.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.712+0000 7f14ebfff640 1 -- 172.21.15.67:0/4242624590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f14e4095bb0 con 0x7f14f415e3b0 2024-04-25T15:26:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.712+0000 7f14f2ffd640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f14d40744b0 0x7f14d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:05.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.713+0000 7f14f2ffd640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f14d40744b0 0x7f14d40769c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007fd0 tx=0x7f14ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.717+0000 7f14ebfff640 1 -- 172.21.15.67:0/4242624590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14e4010040 con 0x7f14f415e3b0 2024-04-25T15:26:05.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.901+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f14f4159b80 con 0x7f14f415e3b0 2024-04-25T15:26:05.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.902+0000 7f14ebfff640 1 -- 172.21.15.67:0/4242624590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f14e40602c0 con 0x7f14f415e3b0 2024-04-25T15:26:05.902 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f14d40744b0 msgr2=0x7f14d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f14d40744b0 0x7f14d40769c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007fd0 tx=0x7f14ec009040 comp rx=0 tx=0).stop 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f415e3b0 msgr2=0x7f14f4071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f415e3b0 0x7f14f4071600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f14e400efc0 tx=0x7f14e400c490 comp rx=0 tx=0).stop 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 shutdown_connections 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f14d40744b0 0x7f14d40769c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14f415e3b0 0x7f14f4071600 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f40d28f0 0x7f14f41f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 --2- 172.21.15.67:0/4242624590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f406c5c0 0x7f14f40d0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.903+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 >> 172.21.15.67:0/4242624590 conn(0x7f14f409b7d0 msgr2=0x7f14f406dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.904+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 shutdown_connections 2024-04-25T15:26:05.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:05.904+0000 7f14f9906640 1 -- 172.21.15.67:0/4242624590 wait complete. 2024-04-25T15:26:05.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.029+0000 7f9637e4a640 1 Processor -- start 2024-04-25T15:26:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.029+0000 7f9637e4a640 1 -- start start 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9637e4a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9630164ad0 0x7f96300b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9637e4a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f9630164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9637e4a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9630164880 con 0x7f96300d1f30 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9637e4a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9630164ee0 con 0x7f9630164ad0 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9637e4a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96300b31e0 con 0x7f9630163f30 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9635be6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9630164ad0 0x7f96300b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f96353e5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f9630164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9635be6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9630164ad0 0x7f96300b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:42822/0 (socket says 172.21.15.67:42822) 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f96353e5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f9630164340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52526/0 (socket says 172.21.15.67:52526) 2024-04-25T15:26:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9635be6640 1 -- 172.21.15.67:0/1991309417 learned_addr learned my addr 172.21.15.67:0/1991309417 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9635be6640 1 -- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 msgr2=0x7f9630164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9635be6640 1 --2- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f9630164340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9635be6640 1 -- 172.21.15.67:0/1991309417 >> v1:172.21.15.176:6789/0 conn(0x7f96300d1f30 legacy=0x7f96300d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f9635be6640 1 -- 172.21.15.67:0/1991309417 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96300d3170 con 0x7f9630164ad0 2024-04-25T15:26:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.031+0000 7f96353e5640 1 --2- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f9630164340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.032+0000 7f9635be6640 1 --2- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 0x7f96300b2ca0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9628009870 tx=0x7f962802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb2d65daf641e334 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.032+0000 7f9634be4640 1 -- 172.21.15.67:0/1991309417 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628037070 con 0x7f9630164ad0 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.032+0000 7f9634be4640 1 -- 172.21.15.67:0/1991309417 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9628032440 con 0x7f9630164ad0 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.032+0000 7f9634be4640 1 -- 172.21.15.67:0/1991309417 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628032810 con 0x7f9630164ad0 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 -- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 msgr2=0x7f96300b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 --2- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 0x7f96300b2ca0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9628009870 tx=0x7f962802a270 comp rx=0 tx=0).stop 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 -- 172.21.15.67:0/1991309417 shutdown_connections 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 --2- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f9630164340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 --2- 172.21.15.67:0/1991309417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 0x7f96300b2ca0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 -- 172.21.15.67:0/1991309417 >> 172.21.15.67:0/1991309417 conn(0x7f96300d3aa0 msgr2=0x7f96300d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 -- 172.21.15.67:0/1991309417 shutdown_connections 2024-04-25T15:26:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.033+0000 7f9637e4a640 1 -- 172.21.15.67:0/1991309417 wait complete. 2024-04-25T15:26:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.035+0000 7f9637e4a640 1 Processor -- start 2024-04-25T15:26:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.035+0000 7f9637e4a640 1 -- start start 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9637e4a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96300d1f30 0x7f96301f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9637e4a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f96300cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9637e4a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 0x7f96301ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9637e4a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f963006d9b0 con 0x7f96300d1f30 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9637e4a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f963006d6d0 con 0x7f9630164ad0 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9637e4a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96300b9430 con 0x7f9630163f30 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9635be6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96300d1f30 0x7f96301f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9635be6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96300d1f30 0x7f96301f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60356/0 (socket says 172.21.15.67:60356) 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f9635be6640 1 -- 172.21.15.67:0/2910855704 learned_addr learned my addr 172.21.15.67:0/2910855704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f96363e7640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 0x7f96301ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.036+0000 7f96353e5640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f96300cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9635be6640 1 -- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 msgr2=0x7f96300cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9635be6640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f96300cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9635be6640 1 -- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 msgr2=0x7f96301ebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9635be6640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 0x7f96301ebed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9635be6640 1 -- 172.21.15.67:0/2910855704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96301fa460 con 0x7f96300d1f30 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9635be6640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96300d1f30 0x7f96301f65d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9628000c00 tx=0x7f96280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f962e7fc640 1 -- 172.21.15.67:0/2910855704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628037040 con 0x7f96300d1f30 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f962e7fc640 1 -- 172.21.15.67:0/2910855704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9628042070 con 0x7f96300d1f30 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f962e7fc640 1 -- 172.21.15.67:0/2910855704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628030880 con 0x7f96300d1f30 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96301fa740 con 0x7f96300d1f30 2024-04-25T15:26:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.037+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96301fad00 con 0x7f96300d1f30 2024-04-25T15:26:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.038+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f963015e650 con 0x7f96300d1f30 2024-04-25T15:26:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.039+0000 7f962e7fc640 1 -- 172.21.15.67:0/2910855704 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9628030a20 con 0x7f96300d1f30 2024-04-25T15:26:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.039+0000 7f962e7fc640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f961c0744b0 0x7f961c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.039+0000 7f962e7fc640 1 -- 172.21.15.67:0/2910855704 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f96280b5970 con 0x7f96300d1f30 2024-04-25T15:26:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.040+0000 7f96353e5640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f961c0744b0 0x7f961c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.040+0000 7f96353e5640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f961c0744b0 0x7f961c0769c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f96300732d0 tx=0x7f9624009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.044+0000 7f962e7fc640 1 -- 172.21.15.67:0/2910855704 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f962807fb40 con 0x7f96300d1f30 2024-04-25T15:26:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.228+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f96300d4770 con 0x7f96300d1f30 2024-04-25T15:26:06.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.228+0000 7f962e7fc640 1 -- 172.21.15.67:0/2910855704 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f9628088180 con 0x7f96300d1f30 2024-04-25T15:26:06.270 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f961c0744b0 msgr2=0x7f961c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f961c0744b0 0x7f961c0769c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f96300732d0 tx=0x7f9624009190 comp rx=0 tx=0).stop 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96300d1f30 msgr2=0x7f96301f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96300d1f30 0x7f96301f65d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9628000c00 tx=0x7f96280046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 shutdown_connections 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f961c0744b0 0x7f961c0769c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9630164ad0 0x7f96301ebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9630163f30 0x7f96300cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 --2- 172.21.15.67:0/2910855704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96300d1f30 0x7f96301f65d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 >> 172.21.15.67:0/2910855704 conn(0x7f96300d3aa0 msgr2=0x7f96300d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 shutdown_connections 2024-04-25T15:26:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.230+0000 7f9637e4a640 1 -- 172.21.15.67:0/2910855704 wait complete. 2024-04-25T15:26:06.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:26:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.386+0000 7f53a315f640 1 Processor -- start 2024-04-25T15:26:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.386+0000 7f53a315f640 1 -- start start 2024-04-25T15:26:06.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f53a315f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f539c0d4130 0x7f539c0d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f53a315f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06dcf0 0x7f539c0d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f53a315f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f539c0b3c10 con 0x7f539c06d8e0 2024-04-25T15:26:06.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f53a315f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f539c0b9520 con 0x7f539c0d4130 2024-04-25T15:26:06.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f53a315f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f539c0b4820 con 0x7f539c06dcf0 2024-04-25T15:26:06.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f539bfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f539c0d4130 0x7f539c0d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f53a16fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06dcf0 0x7f539c0d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f539bfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f539c0d4130 0x7f539c0d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:42844/0 (socket says 172.21.15.67:42844) 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f539bfff640 1 -- 172.21.15.67:0/2889195837 learned_addr learned my addr 172.21.15.67:0/2889195837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.388+0000 7f53a0efb640 1 --1- 172.21.15.67:0/2889195837 >> v1:172.21.15.176:6789/0 conn(0x7f539c06d8e0 0x7f539c0d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:50628/0 (socket says 172.21.15.67:50628) 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.389+0000 7f539bfff640 1 -- 172.21.15.67:0/2889195837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06dcf0 msgr2=0x7f539c0d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.389+0000 7f539bfff640 1 --2- 172.21.15.67:0/2889195837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06dcf0 0x7f539c0d22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.389+0000 7f539bfff640 1 -- 172.21.15.67:0/2889195837 >> v1:172.21.15.176:6789/0 conn(0x7f539c06d8e0 legacy=0x7f539c0d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.389+0000 7f539bfff640 1 -- 172.21.15.67:0/2889195837 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c0b3ef0 con 0x7f539c0d4130 2024-04-25T15:26:06.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.390+0000 7f539bfff640 1 --2- 172.21.15.67:0/2889195837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c0d4130 0x7f539c0d1d60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f538c009870 tx=0x7f538c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fea2dce0f224b95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.390+0000 7f539b7fe640 1 -- 172.21.15.67:0/2889195837 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c037070 con 0x7f539c0d4130 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.390+0000 7f539b7fe640 1 -- 172.21.15.67:0/2889195837 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f538c032440 con 0x7f539c0d4130 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.390+0000 7f539b7fe640 1 -- 172.21.15.67:0/2889195837 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c032810 con 0x7f539c0d4130 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.390+0000 7f53a315f640 1 -- 172.21.15.67:0/2889195837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c0d4130 msgr2=0x7f539c0d1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.390+0000 7f53a315f640 1 --2- 172.21.15.67:0/2889195837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c0d4130 0x7f539c0d1d60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f538c009870 tx=0x7f538c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.391+0000 7f53a315f640 1 -- 172.21.15.67:0/2889195837 shutdown_connections 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.391+0000 7f53a315f640 1 --2- 172.21.15.67:0/2889195837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06dcf0 0x7f539c0d22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.391+0000 7f53a315f640 1 --2- 172.21.15.67:0/2889195837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c0d4130 0x7f539c0d1d60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.391+0000 7f53a315f640 1 -- 172.21.15.67:0/2889195837 >> 172.21.15.67:0/2889195837 conn(0x7f539c08d110 msgr2=0x7f539c08d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.391+0000 7f53a315f640 1 -- 172.21.15.67:0/2889195837 shutdown_connections 2024-04-25T15:26:06.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.391+0000 7f53a315f640 1 -- 172.21.15.67:0/2889195837 wait complete. 2024-04-25T15:26:06.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.392+0000 7f53a315f640 1 Processor -- start 2024-04-25T15:26:06.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.392+0000 7f53a315f640 1 -- start start 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a315f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06d8e0 0x7f539c1f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a315f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c06dcf0 0x7f539c15fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a315f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f539c0c9280 0x7f539c0c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a315f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f539c0b38e0 con 0x7f539c0c9280 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a315f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f539c0b44b0 con 0x7f539c06dcf0 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a315f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f539c0d27e0 con 0x7f539c06d8e0 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f539c0c9280 0x7f539c0c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f539c0c9280 0x7f539c0c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60370/0 (socket says 172.21.15.67:60370) 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 -- 172.21.15.67:0/2076302879 learned_addr learned my addr 172.21.15.67:0/2076302879 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f539bfff640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c06dcf0 0x7f539c15fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a0efb640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06d8e0 0x7f539c1f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 -- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06d8e0 msgr2=0x7f539c1f6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06d8e0 0x7f539c1f6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 -- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c06dcf0 msgr2=0x7f539c15fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c06dcf0 0x7f539c15fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 -- 172.21.15.67:0/2076302879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c0c9bd0 con 0x7f539c0c9280 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f539bfff640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c06dcf0 0x7f539c15fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.394+0000 7f53a16fc640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f539c0c9280 0x7f539c0c9690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f538400e7f0 tx=0x7f538400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.395+0000 7f53997fa640 1 -- 172.21.15.67:0/2076302879 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384017070 con 0x7f539c0c9280 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.395+0000 7f53997fa640 1 -- 172.21.15.67:0/2076302879 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f538400ee40 con 0x7f539c0c9280 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.395+0000 7f53997fa640 1 -- 172.21.15.67:0/2076302879 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538400cb80 con 0x7f539c0c9280 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.395+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539c0c7a50 con 0x7f539c0c9280 2024-04-25T15:26:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.395+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f539c0c8010 con 0x7f539c0c9280 2024-04-25T15:26:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.396+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f539c0c77c0 con 0x7f539c0c9280 2024-04-25T15:26:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.396+0000 7f53997fa640 1 -- 172.21.15.67:0/2076302879 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f538401a070 con 0x7f539c0c9280 2024-04-25T15:26:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.397+0000 7f53997fa640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f537c0744b0 0x7f537c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.397+0000 7f53997fa640 1 -- 172.21.15.67:0/2076302879 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5384095130 con 0x7f539c0c9280 2024-04-25T15:26:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.397+0000 7f53a0efb640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f537c0744b0 0x7f537c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.398+0000 7f53a0efb640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f537c0744b0 0x7f537c0769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f53940012e0 tx=0x7f5394005b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.401+0000 7f53997fa640 1 -- 172.21.15.67:0/2076302879 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f539c0c9bd0 con 0x7f539c0c9280 2024-04-25T15:26:06.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.600+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f539c050740 con 0x7f539c0c9280 2024-04-25T15:26:06.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.600+0000 7f53997fa640 1 -- 172.21.15.67:0/2076302879 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f538405f8c0 con 0x7f539c0c9280 2024-04-25T15:26:06.601 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:26:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f537c0744b0 msgr2=0x7f537c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f537c0744b0 0x7f537c0769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f53940012e0 tx=0x7f5394005b70 comp rx=0 tx=0).stop 2024-04-25T15:26:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f539c0c9280 msgr2=0x7f539c0c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f539c0c9280 0x7f539c0c9690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f538400e7f0 tx=0x7f538400ebf0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 shutdown_connections 2024-04-25T15:26:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f537c0744b0 0x7f537c0769c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f539c0c9280 0x7f539c0c9690 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f539c06dcf0 0x7f539c15fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 --2- 172.21.15.67:0/2076302879 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f539c06d8e0 0x7f539c1f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 >> 172.21.15.67:0/2076302879 conn(0x7f539c08d110 msgr2=0x7f539c0d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 shutdown_connections 2024-04-25T15:26:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.602+0000 7f53a315f640 1 -- 172.21.15.67:0/2076302879 wait complete. 2024-04-25T15:26:06.614 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-25T15:26:06.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:06.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-25T15:26:06.646 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:26:06.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:06.646 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T15:26:06.704 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:26:06.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.819+0000 7fa0b76ee640 1 Processor -- start 2024-04-25T15:26:06.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.820+0000 7fa0b76ee640 1 -- start start 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b76ee640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0b00d0e00 0x7fa0b00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b76ee640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa0b00d4a50 0x7fa0b00d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b76ee640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b00d1850 con 0x7fa0b006c5c0 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b76ee640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b0160de0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b76ee640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b01610c0 con 0x7fa0b00d4a50 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b548a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0b00d0e00 0x7fa0b00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b4c89640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0b006c5c0 0x7fa0b006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60250/0 (socket says 172.21.15.67:60250) 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b4c89640 1 -- 172.21.15.67:0/3764848834 learned_addr learned my addr 172.21.15.67:0/3764848834 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.821+0000 7fa0b5c8b640 1 --2- 172.21.15.67:0/3764848834 >> v2:172.21.15.176:3300/0 conn(0x7fa0b00d4a50 0x7fa0b00d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0b548a640 1 -- 172.21.15.67:0/3764848834 >> v2:172.21.15.176:3300/0 conn(0x7fa0b00d4a50 msgr2=0x7fa0b00d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0b548a640 1 --2- 172.21.15.67:0/3764848834 >> v2:172.21.15.176:3300/0 conn(0x7fa0b00d4a50 0x7fa0b00d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0b548a640 1 -- 172.21.15.67:0/3764848834 >> v1:172.21.15.67:6789/0 conn(0x7fa0b006c5c0 legacy=0x7fa0b006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0b548a640 1 -- 172.21.15.67:0/3764848834 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b01613a0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0a7fff640 1 -- 172.21.15.67:0/3764848834 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053809476 0 0) 0x7fa0b00d1850 con 0x7fa0b006c5c0 2024-04-25T15:26:06.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0b5c8b640 1 --2- 172.21.15.67:0/3764848834 >> v2:172.21.15.176:3300/0 conn(0x7fa0b00d4a50 0x7fa0b00d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0b548a640 1 --2- 172.21.15.67:0/3764848834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b00d1210 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00d1750 tx=0x7fa0ac02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867a60517eabe18b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.822+0000 7fa0a7fff640 1 -- 172.21.15.67:0/3764848834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac037070 con 0x7fa0b00d0e00 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0a7fff640 1 -- 172.21.15.67:0/3764848834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0ac032440 con 0x7fa0b00d0e00 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0a7fff640 1 -- 172.21.15.67:0/3764848834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac02ae60 con 0x7fa0b00d0e00 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 -- 172.21.15.67:0/3764848834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 msgr2=0x7fa0b00d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/3764848834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b00d1210 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00d1750 tx=0x7fa0ac02a110 comp rx=0 tx=0).stop 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 -- 172.21.15.67:0/3764848834 shutdown_connections 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/3764848834 >> v2:172.21.15.176:3300/0 conn(0x7fa0b00d4a50 0x7fa0b00d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/3764848834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b00d1210 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 -- 172.21.15.67:0/3764848834 >> 172.21.15.67:0/3764848834 conn(0x7fa0b00d5d90 msgr2=0x7fa0b00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 -- 172.21.15.67:0/3764848834 shutdown_connections 2024-04-25T15:26:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.823+0000 7fa0b76ee640 1 -- 172.21.15.67:0/3764848834 wait complete. 2024-04-25T15:26:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.825+0000 7fa0b76ee640 1 Processor -- start 2024-04-25T15:26:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.825+0000 7fa0b76ee640 1 -- start start 2024-04-25T15:26:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.826+0000 7fa0b76ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b006c5c0 0x7fa0b015fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.826+0000 7fa0b76ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b01f9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.826+0000 7fa0b76ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0b00d4a50 0x7fa0b00ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.826+0000 7fa0b76ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b0164e00 con 0x7fa0b00d0e00 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.826+0000 7fa0b76ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b00b5320 con 0x7fa0b006c5c0 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.826+0000 7fa0b76ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b0164f80 con 0x7fa0b00d4a50 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b01f9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b01f9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60396/0 (socket says 172.21.15.67:60396) 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 -- 172.21.15.67:0/4216260306 learned_addr learned my addr 172.21.15.67:0/4216260306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b548a640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b006c5c0 0x7fa0b015fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b5c8b640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0b00d4a50 0x7fa0b00ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 -- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0b00d4a50 msgr2=0x7fa0b00ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0b00d4a50 0x7fa0b00ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 -- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b006c5c0 msgr2=0x7fa0b015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b006c5c0 0x7fa0b015fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 -- 172.21.15.67:0/4216260306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b01facb0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b5c8b640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0b00d4a50 0x7fa0b00ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b4c89640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b01f9d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000c430 tx=0x7fa0a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0a5ffb640 1 -- 172.21.15.67:0/4216260306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a00074e0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0a5ffb640 1 -- 172.21.15.67:0/4216260306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a00047e0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b01fa9d0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.827+0000 7fa0a5ffb640 1 -- 172.21.15.67:0/4216260306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0004bd0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.828+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b01eb270 con 0x7fa0b00d0e00 2024-04-25T15:26:06.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.829+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b0050740 con 0x7fa0b00d0e00 2024-04-25T15:26:06.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.829+0000 7fa0a5ffb640 1 -- 172.21.15.67:0/4216260306 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa0a0011070 con 0x7fa0b00d0e00 2024-04-25T15:26:06.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.829+0000 7fa0a5ffb640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0940744b0 0x7fa0940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:06.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.830+0000 7fa0a5ffb640 1 -- 172.21.15.67:0/4216260306 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa0a0094da0 con 0x7fa0b00d0e00 2024-04-25T15:26:06.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.830+0000 7fa0b548a640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0940744b0 0x7fa0940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:06.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.830+0000 7fa0b548a640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0940744b0 0x7fa0940769c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac034000 tx=0x7fa0ac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:06.834+0000 7fa0a5ffb640 1 -- 172.21.15.67:0/4216260306 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0a005f5c0 con 0x7fa0b00d0e00 2024-04-25T15:26:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.058+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0b01fa760 con 0x7fa0b00d0e00 2024-04-25T15:26:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.059+0000 7fa0a5ffb640 1 -- 172.21.15.67:0/4216260306 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7fa0a005eec0 con 0x7fa0b00d0e00 2024-04-25T15:26:07.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:07.060 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:26:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:26:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0940744b0 msgr2=0x7fa0940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0940744b0 0x7fa0940769c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac034000 tx=0x7fa0ac02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 msgr2=0x7fa0b01f9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b01f9d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000c430 tx=0x7fa0a000c8f0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 shutdown_connections 2024-04-25T15:26:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0940744b0 0x7fa0940769c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0b00d4a50 0x7fa0b00ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b00d0e00 0x7fa0b01f9d80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 --2- 172.21.15.67:0/4216260306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b006c5c0 0x7fa0b015fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 >> 172.21.15.67:0/4216260306 conn(0x7fa0b00d5d90 msgr2=0x7fa0b01641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 shutdown_connections 2024-04-25T15:26:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.061+0000 7fa0b76ee640 1 -- 172.21.15.67:0/4216260306 wait complete. 2024-04-25T15:26:07.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.183+0000 7f0e0b740640 1 Processor -- start 2024-04-25T15:26:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.183+0000 7f0e0b740640 1 -- start start 2024-04-25T15:26:07.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e0b740640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 0x7f0e040b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e0b740640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0e0415df80 0x7f0e040b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e0b740640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e041615f0 con 0x7f0e04160890 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e0b740640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0e041613b0 con 0x7f0e0415df80 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e0b740640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e04164670 con 0x7f0e04164b00 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e09cdd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e04160890 0x7f0e04160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60256/0 (socket says 172.21.15.67:60256) 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e09cdd640 1 -- 172.21.15.67:0/3782227804 learned_addr learned my addr 172.21.15.67:0/3782227804 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/3782227804 >> v2:172.21.15.176:3300/0 conn(0x7f0e0415df80 0x7f0e040b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e094dc640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 0x7f0e040b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479264468 0 0) 0x7f0e041615f0 con 0x7f0e04160890 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec003520 con 0x7f0e04160890 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 905431388 0 0) 0x7f0dec003520 con 0x7f0e04160890 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 msgr2=0x7f0e040b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 0x7f0e040b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 >> v2:172.21.15.176:3300/0 conn(0x7f0e0415df80 msgr2=0x7f0e040b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 --2- 172.21.15.67:0/3782227804 >> v2:172.21.15.176:3300/0 conn(0x7f0e0415df80 0x7f0e040b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e04164020 con 0x7f0e04160890 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.185+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f0df40019f0 con 0x7f0e04160890 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 >> v1:172.21.15.67:6789/0 conn(0x7f0e04160890 legacy=0x7f0e04160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e03fff640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec0039b0 0x7f0dec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e03fff640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dec004800 0x7f0dec004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e03fff640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 0x7f0dec005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e041613b0 con 0x7f0dec004800 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0e04164670 con 0x7f0dec0039b0 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0dec00b9c0 con 0x7f0e04164b00 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dec004800 0x7f0dec004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e094dc640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec0039b0 0x7f0dec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e09cdd640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 0x7f0dec005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e08cdb640 1 -- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 msgr2=0x7f0dec005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04164b00 0x7f0dec005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e08cdb640 1 -- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec0039b0 msgr2=0x7f0dec003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec0039b0 0x7f0dec003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.186+0000 7f0e08cdb640 1 -- 172.21.15.67:0/3782227804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec003520 con 0x7f0dec004800 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.187+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dec004800 0x7f0dec004c10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0e040d2540 tx=0x7f0dec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db8f9163eef678 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.187+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8036070 con 0x7f0dec004800 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.187+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df8031400 con 0x7f0dec004800 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.187+0000 7f0e03fff640 1 -- 172.21.15.67:0/3782227804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8009690 con 0x7f0dec004800 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.188+0000 7f0e0b740640 1 -- 172.21.15.67:0/3782227804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dec004800 msgr2=0x7f0dec004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.188+0000 7f0e0b740640 1 --2- 172.21.15.67:0/3782227804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dec004800 0x7f0dec004c10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0e040d2540 tx=0x7f0dec005830 comp rx=0 tx=0).stop 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.188+0000 7f0e09cdd640 1 -- 172.21.15.67:0/3782227804 reap_dead start 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.188+0000 7f0e0b740640 1 -- 172.21.15.67:0/3782227804 shutdown_connections 2024-04-25T15:26:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.188+0000 7f0e0b740640 1 -- 172.21.15.67:0/3782227804 >> 172.21.15.67:0/3782227804 conn(0x7f0e040d2bc0 msgr2=0x7f0e040d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:07.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.188+0000 7f0e0b740640 1 -- 172.21.15.67:0/3782227804 shutdown_connections 2024-04-25T15:26:07.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.188+0000 7f0e0b740640 1 -- 172.21.15.67:0/3782227804 wait complete. 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.190+0000 7f0e0b740640 1 Processor -- start 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.190+0000 7f0e0b740640 1 -- start start 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e0b740640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec004800 0x7f0e0415c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e0b740640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e040ccc80 0x7f0e040cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e0b740640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e040cd090 0x7f0e040cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e0b740640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e040b7d00 con 0x7f0e040ccc80 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e0b740640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0e040b7b80 con 0x7f0dec004800 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e0b740640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e040b7e80 con 0x7f0e040cd090 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e040ccc80 0x7f0e040cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e040ccc80 0x7f0e040cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60406/0 (socket says 172.21.15.67:60406) 2024-04-25T15:26:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 -- 172.21.15.67:0/2506533743 learned_addr learned my addr 172.21.15.67:0/2506533743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e094dc640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec004800 0x7f0e0415c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e09cdd640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e040cd090 0x7f0e040cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 -- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e040cd090 msgr2=0x7f0e040cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e040cd090 0x7f0e040cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 -- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec004800 msgr2=0x7f0e0415c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec004800 0x7f0e0415c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 -- 172.21.15.67:0/2506533743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0415d180 con 0x7f0e040ccc80 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.191+0000 7f0e08cdb640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e040ccc80 0x7f0e040cf430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0df802ab10 tx=0x7f0df8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.192+0000 7f0e01ffb640 1 -- 172.21.15.67:0/2506533743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8040020 con 0x7f0e040ccc80 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.192+0000 7f0e01ffb640 1 -- 172.21.15.67:0/2506533743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df8003890 con 0x7f0e040ccc80 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.192+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0415cc20 con 0x7f0e040ccc80 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.192+0000 7f0e01ffb640 1 -- 172.21.15.67:0/2506533743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8036040 con 0x7f0e040ccc80 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.192+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e040cdb40 con 0x7f0e040ccc80 2024-04-25T15:26:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.193+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e040cd5d0 con 0x7f0e040ccc80 2024-04-25T15:26:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.193+0000 7f0e01ffb640 1 -- 172.21.15.67:0/2506533743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0df8043020 con 0x7f0e040ccc80 2024-04-25T15:26:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.194+0000 7f0e01ffb640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0df00744b0 0x7f0df00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.194+0000 7f0e01ffb640 1 -- 172.21.15.67:0/2506533743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0df80b5340 con 0x7f0e040ccc80 2024-04-25T15:26:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.194+0000 7f0e094dc640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0df00744b0 0x7f0df00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.195+0000 7f0e094dc640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0df00744b0 0x7f0df00769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009b20 tx=0x7f0dfc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.199+0000 7f0e01ffb640 1 -- 172.21.15.67:0/2506533743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0df804a070 con 0x7f0e040ccc80 2024-04-25T15:26:07.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.382+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e04050740 con 0x7f0e040ccc80 2024-04-25T15:26:07.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.384+0000 7f0e01ffb640 1 -- 172.21.15.67:0/2506533743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f0e0415d180 con 0x7f0e040ccc80 2024-04-25T15:26:07.385 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:07.385 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0df00744b0 msgr2=0x7f0df00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0df00744b0 0x7f0df00769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009b20 tx=0x7f0dfc009190 comp rx=0 tx=0).stop 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e040ccc80 msgr2=0x7f0e040cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e040ccc80 0x7f0e040cf430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0df802ab10 tx=0x7f0df8004730 comp rx=0 tx=0).stop 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 shutdown_connections 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0df00744b0 0x7f0df00769c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e040cd090 0x7f0e040cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e040ccc80 0x7f0e040cf430 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 --2- 172.21.15.67:0/2506533743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dec004800 0x7f0e0415c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 >> 172.21.15.67:0/2506533743 conn(0x7f0e040d2bc0 msgr2=0x7f0e04160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 shutdown_connections 2024-04-25T15:26:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.386+0000 7f0e0b740640 1 -- 172.21.15.67:0/2506533743 wait complete. 2024-04-25T15:26:07.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:25:27.207015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:25:27.615603+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:07.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.512+0000 7ff949d84640 1 Processor -- start 2024-04-25T15:26:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.513+0000 7ff949d84640 1 -- start start 2024-04-25T15:26:07.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff949d84640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff94406dce0 0x7ff9440704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff949d84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff94415adf0 0x7ff9440cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff949d84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff94406fa20 con 0x7ff9440b0c90 2024-04-25T15:26:07.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff949d84640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff944070fd0 con 0x7ff94415adf0 2024-04-25T15:26:07.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff949d84640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff9440712b0 con 0x7ff94406dce0 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff943fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff94415adf0 0x7ff9440cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff943fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff94415adf0 0x7ff9440cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60410/0 (socket says 172.21.15.67:60410) 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff943fff640 1 -- 172.21.15.67:0/3498859078 learned_addr learned my addr 172.21.15.67:0/3498859078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff942ffd640 1 --2- 172.21.15.67:0/3498859078 >> v2:172.21.15.176:3300/0 conn(0x7ff94406dce0 0x7ff9440704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff943fff640 1 -- 172.21.15.67:0/3498859078 >> v2:172.21.15.176:3300/0 conn(0x7ff94406dce0 msgr2=0x7ff9440704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff943fff640 1 --2- 172.21.15.67:0/3498859078 >> v2:172.21.15.176:3300/0 conn(0x7ff94406dce0 0x7ff9440704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff943fff640 1 -- 172.21.15.67:0/3498859078 >> v1:172.21.15.176:6789/0 conn(0x7ff9440b0c90 legacy=0x7ff94406d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.514+0000 7ff943fff640 1 -- 172.21.15.67:0/3498859078 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94406fcc0 con 0x7ff94415adf0 2024-04-25T15:26:07.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.515+0000 7ff943fff640 1 --2- 172.21.15.67:0/3498859078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff94415adf0 0x7ff9440cec50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff94406eb80 tx=0x7ff93c02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5ce9daaa870700 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.515+0000 7ff9427fc640 1 -- 172.21.15.67:0/3498859078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c036070 con 0x7ff94415adf0 2024-04-25T15:26:07.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.515+0000 7ff9427fc640 1 -- 172.21.15.67:0/3498859078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff93c031600 con 0x7ff94415adf0 2024-04-25T15:26:07.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.515+0000 7ff9427fc640 1 -- 172.21.15.67:0/3498859078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c0319d0 con 0x7ff94415adf0 2024-04-25T15:26:07.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 -- 172.21.15.67:0/3498859078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff94415adf0 msgr2=0x7ff9440cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 --2- 172.21.15.67:0/3498859078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff94415adf0 0x7ff9440cec50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff94406eb80 tx=0x7ff93c02a6d0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 -- 172.21.15.67:0/3498859078 shutdown_connections 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 --2- 172.21.15.67:0/3498859078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff94415adf0 0x7ff9440cec50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 --2- 172.21.15.67:0/3498859078 >> v2:172.21.15.176:3300/0 conn(0x7ff94406dce0 0x7ff9440704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 -- 172.21.15.67:0/3498859078 >> 172.21.15.67:0/3498859078 conn(0x7ff944073090 msgr2=0x7ff9440734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 -- 172.21.15.67:0/3498859078 shutdown_connections 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.516+0000 7ff949d84640 1 -- 172.21.15.67:0/3498859078 wait complete. 2024-04-25T15:26:07.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.518+0000 7ff949d84640 1 Processor -- start 2024-04-25T15:26:07.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.518+0000 7ff949d84640 1 -- start start 2024-04-25T15:26:07.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.519+0000 7ff949d84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9440b0c90 0x7ff944150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.519+0000 7ff949d84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff94415adf0 0x7ff944151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.519+0000 7ff949d84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9441608b0 0x7ff94414d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.519+0000 7ff949d84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff94415d790 con 0x7ff9440b0c90 2024-04-25T15:26:07.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.519+0000 7ff949d84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff944070c80 con 0x7ff9441608b0 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.519+0000 7ff949d84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff94415d910 con 0x7ff94415adf0 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9440b0c90 0x7ff944150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9440b0c90 0x7ff944150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60424/0 (socket says 172.21.15.67:60424) 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 -- 172.21.15.67:0/3512313350 learned_addr learned my addr 172.21.15.67:0/3512313350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff942ffd640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff94415adf0 0x7ff944151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff943fff640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9441608b0 0x7ff94414d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 -- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff94415adf0 msgr2=0x7ff944151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff94415adf0 0x7ff944151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 -- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9441608b0 msgr2=0x7ff94414d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9441608b0 0x7ff94414d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 -- 172.21.15.67:0/3512313350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff944151700 con 0x7ff9440b0c90 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff9437fe640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9440b0c90 0x7ff944150c00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff93400b3c0 tx=0x7ff93400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff92bfff640 1 -- 172.21.15.67:0/3512313350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934011830 con 0x7ff9440b0c90 2024-04-25T15:26:07.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff92bfff640 1 -- 172.21.15.67:0/3512313350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff934016070 con 0x7ff9440b0c90 2024-04-25T15:26:07.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff92bfff640 1 -- 172.21.15.67:0/3512313350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934011c10 con 0x7ff9440b0c90 2024-04-25T15:26:07.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff944150120 con 0x7ff9440b0c90 2024-04-25T15:26:07.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.520+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9441506e0 con 0x7ff9440b0c90 2024-04-25T15:26:07.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.521+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff944050740 con 0x7ff9440b0c90 2024-04-25T15:26:07.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.522+0000 7ff92bfff640 1 -- 172.21.15.67:0/3512313350 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff93401b460 con 0x7ff9440b0c90 2024-04-25T15:26:07.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.523+0000 7ff92bfff640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9300744b0 0x7ff9300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.523+0000 7ff92bfff640 1 -- 172.21.15.67:0/3512313350 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff934094d10 con 0x7ff9440b0c90 2024-04-25T15:26:07.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.523+0000 7ff942ffd640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9300744b0 0x7ff9300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.523+0000 7ff942ffd640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9300744b0 0x7ff9300769c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff938006fd0 tx=0x7ff938008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.527+0000 7ff92bfff640 1 -- 172.21.15.67:0/3512313350 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff934020020 con 0x7ff9440b0c90 2024-04-25T15:26:07.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.710+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff944152e90 con 0x7ff9440b0c90 2024-04-25T15:26:07.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.711+0000 7ff92bfff640 1 -- 172.21.15.67:0/3512313350 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7ff93405f4a0 con 0x7ff9440b0c90 2024-04-25T15:26:07.712 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9300744b0 msgr2=0x7ff9300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9300744b0 0x7ff9300769c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff938006fd0 tx=0x7ff938008040 comp rx=0 tx=0).stop 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9440b0c90 msgr2=0x7ff944150c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9440b0c90 0x7ff944150c00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff93400b3c0 tx=0x7ff93400b880 comp rx=0 tx=0).stop 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 shutdown_connections 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9300744b0 0x7ff9300769c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9441608b0 0x7ff94414d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff94415adf0 0x7ff944151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 --2- 172.21.15.67:0/3512313350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9440b0c90 0x7ff944150c00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 >> 172.21.15.67:0/3512313350 conn(0x7ff944073090 msgr2=0x7ff9441533b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 shutdown_connections 2024-04-25T15:26:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.713+0000 7ff949d84640 1 -- 172.21.15.67:0/3512313350 wait complete. 2024-04-25T15:26:07.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:07.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.839+0000 7f0620df3640 1 Processor -- start 2024-04-25T15:26:07.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.839+0000 7f0620df3640 1 -- start start 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.840+0000 7f0620df3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f061c06ccd0 0x7f061c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.840+0000 7f0620df3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f061c0d4a50 0x7f061c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.840+0000 7f0620df3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061c0d1850 con 0x7f061c0d0e00 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.840+0000 7f0620df3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f061c160de0 con 0x7f061c06ccd0 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.840+0000 7f0620df3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f061c1610c0 con 0x7f061c0d4a50 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f061a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f061c0d0e00 0x7f061c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49698/0 (socket says 172.21.15.67:49698) 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f061a59c640 1 -- 172.21.15.67:0/1856363368 learned_addr learned my addr 172.21.15.67:0/1856363368 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f0619d9b640 1 --2- 172.21.15.67:0/1856363368 >> v2:172.21.15.67:3300/0 conn(0x7f061c06ccd0 0x7f061c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f061ad9d640 1 --2- 172.21.15.67:0/1856363368 >> v2:172.21.15.176:3300/0 conn(0x7f061c0d4a50 0x7f061c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f061959a640 1 -- 172.21.15.67:0/1856363368 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 127155600 0 0) 0x7f061c0d1850 con 0x7f061c0d0e00 2024-04-25T15:26:07.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f061959a640 1 -- 172.21.15.67:0/1856363368 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0608003520 con 0x7f061c0d0e00 2024-04-25T15:26:07.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f0619d9b640 1 -- 172.21.15.67:0/1856363368 >> v2:172.21.15.176:3300/0 conn(0x7f061c0d4a50 msgr2=0x7f061c0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:07.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f0619d9b640 1 --2- 172.21.15.67:0/1856363368 >> v2:172.21.15.176:3300/0 conn(0x7f061c0d4a50 0x7f061c0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f0619d9b640 1 -- 172.21.15.67:0/1856363368 >> v1:172.21.15.67:6789/0 conn(0x7f061c0d0e00 legacy=0x7f061c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.841+0000 7f0619d9b640 1 -- 172.21.15.67:0/1856363368 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c1613a0 con 0x7f061c06ccd0 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.842+0000 7f0619d9b640 1 --2- 172.21.15.67:0/1856363368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c06ccd0 0x7f061c0d1210 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f061c0d1750 tx=0x7f060402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763e2dc5ff674ec4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.842+0000 7f061959a640 1 -- 172.21.15.67:0/1856363368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604037070 con 0x7f061c06ccd0 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.842+0000 7f061959a640 1 -- 172.21.15.67:0/1856363368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0604032440 con 0x7f061c06ccd0 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.842+0000 7f061959a640 1 -- 172.21.15.67:0/1856363368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060402ae60 con 0x7f061c06ccd0 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 -- 172.21.15.67:0/1856363368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c06ccd0 msgr2=0x7f061c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 --2- 172.21.15.67:0/1856363368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c06ccd0 0x7f061c0d1210 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f061c0d1750 tx=0x7f060402a110 comp rx=0 tx=0).stop 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 -- 172.21.15.67:0/1856363368 shutdown_connections 2024-04-25T15:26:07.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 --2- 172.21.15.67:0/1856363368 >> v2:172.21.15.176:3300/0 conn(0x7f061c0d4a50 0x7f061c0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 --2- 172.21.15.67:0/1856363368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c06ccd0 0x7f061c0d1210 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 -- 172.21.15.67:0/1856363368 >> 172.21.15.67:0/1856363368 conn(0x7f061c0d5d90 msgr2=0x7f061c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 -- 172.21.15.67:0/1856363368 shutdown_connections 2024-04-25T15:26:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.843+0000 7f0620df3640 1 -- 172.21.15.67:0/1856363368 wait complete. 2024-04-25T15:26:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.845+0000 7f0620df3640 1 Processor -- start 2024-04-25T15:26:07.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.845+0000 7f0620df3640 1 -- start start 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0620df3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f061c0d0e00 0x7f061c0b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0620df3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0d4a50 0x7f061c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0620df3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f061c0cd870 0x7f061c1f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0620df3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f061c164e00 con 0x7f061c0d4a50 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0620df3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f061c0b5320 con 0x7f061c0cd870 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0620df3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f061c164f80 con 0x7f061c0d0e00 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0d4a50 0x7f061c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0d4a50 0x7f061c0cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49346/0 (socket says 172.21.15.67:49346) 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 -- 172.21.15.67:0/451449442 learned_addr learned my addr 172.21.15.67:0/451449442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f061a59c640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f061c0d0e00 0x7f061c0b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f061ad9d640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f061c0cd870 0x7f061c1f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 -- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f061c0d0e00 msgr2=0x7f061c0b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f061c0d0e00 0x7f061c0b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 -- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f061c0cd870 msgr2=0x7f061c1f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f061c0cd870 0x7f061c1f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.846+0000 7f0619d9b640 1 -- 172.21.15.67:0/451449442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c0ce060 con 0x7f061c0d4a50 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.847+0000 7f0619d9b640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0d4a50 0x7f061c0cd330 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0604000c00 tx=0x7f06040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.847+0000 7f0612ffd640 1 -- 172.21.15.67:0/451449442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604041020 con 0x7f061c0d4a50 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.847+0000 7f0612ffd640 1 -- 172.21.15.67:0/451449442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060403d700 con 0x7f061c0d4a50 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.847+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061c0ce340 con 0x7f061c0d4a50 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.847+0000 7f0612ffd640 1 -- 172.21.15.67:0/451449442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604037040 con 0x7f061c0d4a50 2024-04-25T15:26:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.847+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c0ce900 con 0x7f061c0d4a50 2024-04-25T15:26:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.848+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061c050740 con 0x7f061c0d4a50 2024-04-25T15:26:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.848+0000 7f0612ffd640 1 -- 172.21.15.67:0/451449442 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0604044020 con 0x7f061c0d4a50 2024-04-25T15:26:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.849+0000 7f0612ffd640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.849+0000 7f0612ffd640 1 -- 172.21.15.67:0/451449442 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f06040b5830 con 0x7f061c0d4a50 2024-04-25T15:26:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.849+0000 7f061a59c640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:07.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.850+0000 7f061a59c640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0614005880 tx=0x7f0614009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:07.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:07.854+0000 7f0612ffd640 1 -- 172.21.15.67:0/451449442 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f060404b030 con 0x7f061c0d4a50 2024-04-25T15:26:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.039+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f061c0cddb0 con 0x7f061c0d4a50 2024-04-25T15:26:08.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.039+0000 7f0612ffd640 1 -- 172.21.15.67:0/451449442 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f060407fa10 con 0x7f061c0d4a50 2024-04-25T15:26:08.040 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 msgr2=0x7f05fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0614005880 tx=0x7f0614009040 comp rx=0 tx=0).stop 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0d4a50 msgr2=0x7f061c0cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0d4a50 0x7f061c0cd330 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0604000c00 tx=0x7f06040046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 shutdown_connections 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f061c0cd870 0x7f061c1f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f061c0d4a50 0x7f061c0cd330 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 --2- 172.21.15.67:0/451449442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f061c0d0e00 0x7f061c0b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 >> 172.21.15.67:0/451449442 conn(0x7f061c0d5d90 msgr2=0x7f061c1638a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 shutdown_connections 2024-04-25T15:26:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.041+0000 7f0620df3640 1 -- 172.21.15.67:0/451449442 wait complete. 2024-04-25T15:26:08.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.166+0000 7fb993fba640 1 Processor -- start 2024-04-25T15:26:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.167+0000 7fb993fba640 1 -- start start 2024-04-25T15:26:08.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb993fba640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb98c164b00 0x7fb98c0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb993fba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c15df80 0x7fb98c0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb993fba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb98c160890 0x7fb98c160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb993fba640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb98c164670 con 0x7fb98c160890 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb993fba640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb98c1615f0 con 0x7fb98c164b00 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb993fba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c164020 con 0x7fb98c15df80 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb98c160890 0x7fb98c160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb98c160890 0x7fb98c160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49362/0 (socket says 172.21.15.67:49362) 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 -- 172.21.15.67:0/81267690 learned_addr learned my addr 172.21.15.67:0/81267690 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb991555640 1 --2- 172.21.15.67:0/81267690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c15df80 0x7fb98c0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb991d56640 1 --2- 172.21.15.67:0/81267690 >> v2:172.21.15.176:3300/0 conn(0x7fb98c164b00 0x7fb98c0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 -- 172.21.15.67:0/81267690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c15df80 msgr2=0x7fb98c0b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 --2- 172.21.15.67:0/81267690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c15df80 0x7fb98c0b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 -- 172.21.15.67:0/81267690 >> v2:172.21.15.176:3300/0 conn(0x7fb98c164b00 msgr2=0x7fb98c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 --2- 172.21.15.67:0/81267690 >> v2:172.21.15.176:3300/0 conn(0x7fb98c164b00 0x7fb98c0b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.168+0000 7fb992557640 1 -- 172.21.15.67:0/81267690 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c164300 con 0x7fb98c160890 2024-04-25T15:26:08.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.169+0000 7fb992557640 1 --2- 172.21.15.67:0/81267690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c160890 0x7fb98c160ca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb98c08e410 tx=0x7fb97c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebe52bfe75cdebf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.169+0000 7fb990d54640 1 -- 172.21.15.67:0/81267690 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c036070 con 0x7fb98c160890 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.169+0000 7fb990d54640 1 -- 172.21.15.67:0/81267690 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97c031600 con 0x7fb98c160890 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.169+0000 7fb990d54640 1 -- 172.21.15.67:0/81267690 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c0319d0 con 0x7fb98c160890 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 -- 172.21.15.67:0/81267690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c160890 msgr2=0x7fb98c160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 --2- 172.21.15.67:0/81267690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c160890 0x7fb98c160ca0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb98c08e410 tx=0x7fb97c02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 -- 172.21.15.67:0/81267690 shutdown_connections 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 --2- 172.21.15.67:0/81267690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c160890 0x7fb98c160ca0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 --2- 172.21.15.67:0/81267690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c15df80 0x7fb98c0b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 --2- 172.21.15.67:0/81267690 >> v2:172.21.15.176:3300/0 conn(0x7fb98c164b00 0x7fb98c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 -- 172.21.15.67:0/81267690 >> 172.21.15.67:0/81267690 conn(0x7fb98c0d2bc0 msgr2=0x7fb98c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 -- 172.21.15.67:0/81267690 shutdown_connections 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.170+0000 7fb993fba640 1 -- 172.21.15.67:0/81267690 wait complete. 2024-04-25T15:26:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.172+0000 7fb993fba640 1 Processor -- start 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.172+0000 7fb993fba640 1 -- start start 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb993fba640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb98c15df80 0x7fb98c1f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb993fba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c160890 0x7fb98c158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb993fba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c164b00 0x7fb98c1ea630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb993fba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c0b4a00 con 0x7fb98c164b00 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb993fba640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c0b4880 con 0x7fb98c15df80 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb993fba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c0b4b80 con 0x7fb98c160890 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb992557640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c164b00 0x7fb98c1ea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb992557640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c164b00 0x7fb98c1ea630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49368/0 (socket says 172.21.15.67:49368) 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.173+0000 7fb992557640 1 -- 172.21.15.67:0/1139117364 learned_addr learned my addr 172.21.15.67:0/1139117364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb991555640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c160890 0x7fb98c158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb991d56640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb98c15df80 0x7fb98c1f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb992557640 1 -- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c160890 msgr2=0x7fb98c158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb992557640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c160890 0x7fb98c158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb992557640 1 -- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb98c15df80 msgr2=0x7fb98c1f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb992557640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb98c15df80 0x7fb98c1f5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb992557640 1 -- 172.21.15.67:0/1139117364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c157cf0 con 0x7fb98c164b00 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb992557640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c164b00 0x7fb98c1ea630 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009f40 tx=0x7fb97c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb98a7fc640 1 -- 172.21.15.67:0/1139117364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c03f020 con 0x7fb98c164b00 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb98a7fc640 1 -- 172.21.15.67:0/1139117364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97c037040 con 0x7fb98c164b00 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98c157fd0 con 0x7fb98c164b00 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb98a7fc640 1 -- 172.21.15.67:0/1139117364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c036070 con 0x7fb98c164b00 2024-04-25T15:26:08.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.174+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb98c158590 con 0x7fb98c164b00 2024-04-25T15:26:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.175+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb98c050740 con 0x7fb98c164b00 2024-04-25T15:26:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.176+0000 7fb98a7fc640 1 -- 172.21.15.67:0/1139117364 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb97c042020 con 0x7fb98c164b00 2024-04-25T15:26:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.176+0000 7fb98a7fc640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9780744b0 0x7fb9780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.177+0000 7fb98a7fc640 1 -- 172.21.15.67:0/1139117364 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb97c0b55c0 con 0x7fb98c164b00 2024-04-25T15:26:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.177+0000 7fb991d56640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9780744b0 0x7fb9780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.177+0000 7fb991d56640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9780744b0 0x7fb9780769c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb984009a30 tx=0x7fb984008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.181+0000 7fb98a7fc640 1 -- 172.21.15.67:0/1139117364 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb97c049030 con 0x7fb98c164b00 2024-04-25T15:26:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.372+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb98c160000 con 0x7fb98c164b00 2024-04-25T15:26:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.372+0000 7fb98a7fc640 1 -- 172.21.15.67:0/1139117364 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fb97c07f700 con 0x7fb98c164b00 2024-04-25T15:26:08.373 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9780744b0 msgr2=0x7fb9780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9780744b0 0x7fb9780769c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb984009a30 tx=0x7fb984008040 comp rx=0 tx=0).stop 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c164b00 msgr2=0x7fb98c1ea630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c164b00 0x7fb98c1ea630 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb97c009f40 tx=0x7fb97c004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 shutdown_connections 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9780744b0 0x7fb9780769c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98c164b00 0x7fb98c1ea630 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb98c160890 0x7fb98c158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 --2- 172.21.15.67:0/1139117364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb98c15df80 0x7fb98c1f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.374+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 >> 172.21.15.67:0/1139117364 conn(0x7fb98c0d2bc0 msgr2=0x7fb98c15e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.375+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 shutdown_connections 2024-04-25T15:26:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.375+0000 7fb993fba640 1 -- 172.21.15.67:0/1139117364 wait complete. 2024-04-25T15:26:08.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.498+0000 7fe4b8d9d640 1 Processor -- start 2024-04-25T15:26:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.499+0000 7fe4b8d9d640 1 -- start start 2024-04-25T15:26:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b8d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b41600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b8d9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4b40b5260 0x7fe4b40b9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b8d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4b40b96b0 0x7fe4b40b7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b8d9d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe4b4160620 con 0x7fe4b40b96b0 2024-04-25T15:26:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b8d9d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe4b40d0b30 con 0x7fe4b40b5260 2024-04-25T15:26:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b8d9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe4b40d4420 con 0x7fe4b40b4da0 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4b40b96b0 0x7fe4b40b7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4b40b96b0 0x7fe4b40b7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49374/0 (socket says 172.21.15.67:49374) 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 -- 172.21.15.67:0/1938906119 learned_addr learned my addr 172.21.15.67:0/1938906119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b37fe640 1 --2- 172.21.15.67:0/1938906119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b41600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b2ffd640 1 --2- 172.21.15.67:0/1938906119 >> v2:172.21.15.176:3300/0 conn(0x7fe4b40b5260 0x7fe4b40b9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 -- 172.21.15.67:0/1938906119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 msgr2=0x7fe4b41600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 --2- 172.21.15.67:0/1938906119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b41600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 -- 172.21.15.67:0/1938906119 >> v2:172.21.15.176:3300/0 conn(0x7fe4b40b5260 msgr2=0x7fe4b40b9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 --2- 172.21.15.67:0/1938906119 >> v2:172.21.15.176:3300/0 conn(0x7fe4b40b5260 0x7fe4b40b9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.500+0000 7fe4b3fff640 1 -- 172.21.15.67:0/1938906119 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b40d4700 con 0x7fe4b40b96b0 2024-04-25T15:26:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.501+0000 7fe4b3fff640 1 --2- 172.21.15.67:0/1938906119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40b7bc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40b8860 tx=0x7fe4ac02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64501558fa1465de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.501+0000 7fe4b27fc640 1 -- 172.21.15.67:0/1938906119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac036070 con 0x7fe4b40b96b0 2024-04-25T15:26:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.501+0000 7fe4b27fc640 1 -- 172.21.15.67:0/1938906119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4ac031600 con 0x7fe4b40b96b0 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.501+0000 7fe4b27fc640 1 -- 172.21.15.67:0/1938906119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac0319d0 con 0x7fe4b40b96b0 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/1938906119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 msgr2=0x7fe4b40b7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/1938906119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40b7bc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40b8860 tx=0x7fe4ac02a610 comp rx=0 tx=0).stop 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/1938906119 shutdown_connections 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/1938906119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40b7bc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/1938906119 >> v2:172.21.15.176:3300/0 conn(0x7fe4b40b5260 0x7fe4b40b9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/1938906119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b41600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/1938906119 >> 172.21.15.67:0/1938906119 conn(0x7fe4b40d2bd0 msgr2=0x7fe4b40d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/1938906119 shutdown_connections 2024-04-25T15:26:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.502+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/1938906119 wait complete. 2024-04-25T15:26:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.504+0000 7fe4b8d9d640 1 Processor -- start 2024-04-25T15:26:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.504+0000 7fe4b8d9d640 1 -- start start 2024-04-25T15:26:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.505+0000 7fe4b8d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b415f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.505+0000 7fe4b8d9d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4b40b5260 0x7fe4b41f7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.505+0000 7fe4b8d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b8d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b40d0750 con 0x7fe4b40b96b0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b8d9d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b40d4cc0 con 0x7fe4b40b5260 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b8d9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe4b40d08d0 con 0x7fe4b40b4da0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49388/0 (socket says 172.21.15.67:49388) 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 -- 172.21.15.67:0/4241043615 learned_addr learned my addr 172.21.15.67:0/4241043615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b37fe640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b415f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b2ffd640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4b40b5260 0x7fe4b41f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 -- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 msgr2=0x7fe4b415f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b415f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 -- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4b40b5260 msgr2=0x7fe4b41f7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4b40b5260 0x7fe4b41f7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 -- 172.21.15.67:0/4241043615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b41f75d0 con 0x7fe4b40b96b0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b3fff640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40af320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009f40 tx=0x7fe4ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe49bfff640 1 -- 172.21.15.67:0/4241043615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac036050 con 0x7fe4b40b96b0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe49bfff640 1 -- 172.21.15.67:0/4241043615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4ac042070 con 0x7fe4b40b96b0 2024-04-25T15:26:08.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.506+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b40ce0c0 con 0x7fe4b40b96b0 2024-04-25T15:26:08.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.507+0000 7fe49bfff640 1 -- 172.21.15.67:0/4241043615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac009510 con 0x7fe4b40b96b0 2024-04-25T15:26:08.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.507+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b40ce680 con 0x7fe4b40b96b0 2024-04-25T15:26:08.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.508+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b4050740 con 0x7fe4b40b96b0 2024-04-25T15:26:08.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.508+0000 7fe49bfff640 1 -- 172.21.15.67:0/4241043615 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe4ac007c50 con 0x7fe4b40b96b0 2024-04-25T15:26:08.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.509+0000 7fe49bfff640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4a00744e0 0x7fe4a00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.509+0000 7fe49bfff640 1 -- 172.21.15.67:0/4241043615 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe4ac0b5ff0 con 0x7fe4b40b96b0 2024-04-25T15:26:08.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.509+0000 7fe4b37fe640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4a00744e0 0x7fe4a00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.509+0000 7fe4b37fe640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4a00744e0 0x7fe4a00769f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40079a0 tx=0x7fe4a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.513+0000 7fe49bfff640 1 -- 172.21.15.67:0/4241043615 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4ac0800b0 con 0x7fe4b40b96b0 2024-04-25T15:26:08.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.742+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4b415f800 con 0x7fe4b40b96b0 2024-04-25T15:26:08.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.743+0000 7fe49bfff640 1 -- 172.21.15.67:0/4241043615 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7fe4ac0a1020 con 0x7fe4b40b96b0 2024-04-25T15:26:08.744 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:08.744 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:26:08.744 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:26:08.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4a00744e0 msgr2=0x7fe4a00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4a00744e0 0x7fe4a00769f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40079a0 tx=0x7fe4a4008040 comp rx=0 tx=0).stop 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 msgr2=0x7fe4b40af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40af320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009f40 tx=0x7fe4ac004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 shutdown_connections 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4a00744e0 0x7fe4a00769f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b40b96b0 0x7fe4b40af320 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4b40b5260 0x7fe4b41f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 --2- 172.21.15.67:0/4241043615 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4b40b4da0 0x7fe4b415f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 >> 172.21.15.67:0/4241043615 conn(0x7fe4b40d2bd0 msgr2=0x7fe4b40d65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 shutdown_connections 2024-04-25T15:26:08.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.745+0000 7fe4b8d9d640 1 -- 172.21.15.67:0/4241043615 wait complete. 2024-04-25T15:26:08.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.869+0000 7fcb124f2640 1 Processor -- start 2024-04-25T15:26:08.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.869+0000 7fcb124f2640 1 -- start start 2024-04-25T15:26:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb124f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 0x7fcb0c070480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb124f2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb0c0709c0 con 0x7fcb0c0d0ed0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb124f2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb0c06f3e0 con 0x7fcb0c1612f0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb124f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c1438e0 con 0x7fcb0c1434d0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb0c0d0ed0 0x7fcb0c160be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49712/0 (socket says 172.21.15.67:49712) 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0b7fe640 1 -- 172.21.15.67:0/1052730627 learned_addr learned my addr 172.21.15.67:0/1052730627 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 0x7fcb0c070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788637702 0 0) 0x7fcb0c0709c0 con 0x7fcb0c0d0ed0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaf4003520 con 0x7fcb0c0d0ed0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1024616807 0 0) 0x7fcaf4003520 con 0x7fcb0c0d0ed0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 msgr2=0x7fcb0c070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 0x7fcb0c070480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 >> v1:172.21.15.176:6789/0 conn(0x7fcb0c1612f0 legacy=0x7fcb0c0b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c06f6c0 con 0x7fcb0c0d0ed0 2024-04-25T15:26:08.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.871+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fcb00001670 con 0x7fcb0c0d0ed0 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 >> v1:172.21.15.67:6789/0 conn(0x7fcb0c0d0ed0 legacy=0x7fcb0c160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0affd640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039e0 0x7fcaf4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0affd640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0affd640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 0x7fcaf4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c1438e0 con 0x7fcaf40039e0 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf400baf0 con 0x7fcaf4004800 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf400df00 con 0x7fcb0c1434d0 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039e0 0x7fcaf4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0b7fe640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb10a8f640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 0x7fcaf4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0bfff640 1 -- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 msgr2=0x7fcaf4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 0x7fcaf4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0bfff640 1 -- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf4004800 msgr2=0x7fcaf4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:08.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0bfff640 1 -- 172.21.15.67:0/1052730627 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf4003520 con 0x7fcaf40039e0 2024-04-25T15:26:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb0b7fe640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf4004800 0x7fcaf4004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.872+0000 7fcb10a8f640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c1434d0 0x7fcaf4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039e0 0x7fcaf4003e50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009a90 tx=0x7fcaf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b344123f27064848 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc036070 con 0x7fcaf40039e0 2024-04-25T15:26:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcafc031400 con 0x7fcaf40039e0 2024-04-25T15:26:08.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb0affd640 1 -- 172.21.15.67:0/1052730627 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc009690 con 0x7fcaf40039e0 2024-04-25T15:26:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb124f2640 1 -- 172.21.15.67:0/1052730627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039e0 msgr2=0x7fcaf4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb124f2640 1 --2- 172.21.15.67:0/1052730627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcaf40039e0 0x7fcaf4003e50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009a90 tx=0x7fcaf4005830 comp rx=0 tx=0).stop 2024-04-25T15:26:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb10a8f640 1 -- 172.21.15.67:0/1052730627 reap_dead start 2024-04-25T15:26:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.873+0000 7fcb124f2640 1 -- 172.21.15.67:0/1052730627 shutdown_connections 2024-04-25T15:26:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.874+0000 7fcb124f2640 1 -- 172.21.15.67:0/1052730627 >> 172.21.15.67:0/1052730627 conn(0x7fcb0c0d1ae0 msgr2=0x7fcb0c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.874+0000 7fcb124f2640 1 -- 172.21.15.67:0/1052730627 shutdown_connections 2024-04-25T15:26:08.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.874+0000 7fcb124f2640 1 -- 172.21.15.67:0/1052730627 wait complete. 2024-04-25T15:26:08.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.875+0000 7fcb124f2640 1 Processor -- start 2024-04-25T15:26:08.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.875+0000 7fcb124f2640 1 -- start start 2024-04-25T15:26:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb124f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf40039e0 0x7fcb0c1f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb124f2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c1f6bd0 0x7fcb0c0b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb124f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c0b2be0 0x7fcb0c144d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb124f2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c06f090 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb124f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c06fc80 con 0x7fcaf40039e0 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb124f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c070480 con 0x7fcb0c0b2be0 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c1f6bd0 0x7fcb0c0b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c1f6bd0 0x7fcb0c0b26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49414/0 (socket says 172.21.15.67:49414) 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 -- 172.21.15.67:0/568727787 learned_addr learned my addr 172.21.15.67:0/568727787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf40039e0 0x7fcb0c1f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb10a8f640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c0b2be0 0x7fcb0c144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 -- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c0b2be0 msgr2=0x7fcb0c144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c0b2be0 0x7fcb0c144d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 -- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf40039e0 msgr2=0x7fcb0c1f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf40039e0 0x7fcb0c1f0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 -- 172.21.15.67:0/568727787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0b2ff0 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb10a8f640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c0b2be0 0x7fcb0c144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.877+0000 7fcb0b7fe640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c1f6bd0 0x7fcb0c0b26a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c074210 tx=0x7fcb0000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.878+0000 7fcb08ff9640 1 -- 172.21.15.67:0/568727787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0000ee60 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.878+0000 7fcb08ff9640 1 -- 172.21.15.67:0/568727787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb00018070 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.878+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c15ea10 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.878+0000 7fcb08ff9640 1 -- 172.21.15.67:0/568727787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0000ca40 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.878+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c15efd0 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.879+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0c15e770 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.879+0000 7fcb08ff9640 1 -- 172.21.15.67:0/568727787 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb0000cbe0 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.880+0000 7fcb08ff9640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744b0 0x7fcaf80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.880+0000 7fcb08ff9640 1 -- 172.21.15.67:0/568727787 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcb00095080 con 0x7fcb0c1f6bd0 2024-04-25T15:26:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.880+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744b0 0x7fcaf80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.881+0000 7fcb0bfff640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744b0 0x7fcaf80769c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcafc033000 tx=0x7fcafc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:08.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:08.884+0000 7fcb08ff9640 1 -- 172.21.15.67:0/568727787 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb0c15e770 con 0x7fcb0c1f6bd0 2024-04-25T15:26:09.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.071+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb0c050740 con 0x7fcb0c1f6bd0 2024-04-25T15:26:09.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.072+0000 7fcb08ff9640 1 -- 172.21.15.67:0/568727787 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7fcb0c0b2ff0 con 0x7fcb0c1f6bd0 2024-04-25T15:26:09.073 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:09.073 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:25:30.290402+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.074+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744b0 msgr2=0x7fcaf80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.074+0000 7fcb124f2640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744b0 0x7fcaf80769c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcafc033000 tx=0x7fcafc008030 comp rx=0 tx=0).stop 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.074+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c1f6bd0 msgr2=0x7fcb0c0b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.074+0000 7fcb124f2640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c1f6bd0 0x7fcb0c0b26a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c074210 tx=0x7fcb0000ec10 comp rx=0 tx=0).stop 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 shutdown_connections 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf80744b0 0x7fcaf80769c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c0b2be0 0x7fcb0c144d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c1f6bd0 0x7fcb0c0b26a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 --2- 172.21.15.67:0/568727787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcaf40039e0 0x7fcb0c1f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 >> 172.21.15.67:0/568727787 conn(0x7fcb0c0d1ae0 msgr2=0x7fcb0c06d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 shutdown_connections 2024-04-25T15:26:09.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.075+0000 7fcb124f2640 1 -- 172.21.15.67:0/568727787 wait complete. 2024-04-25T15:26:09.087 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:26:09.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.198+0000 7feefa215640 1 Processor -- start 2024-04-25T15:26:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.198+0000 7feefa215640 1 -- start start 2024-04-25T15:26:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feefa215640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7feef4164b00 0x7feef40b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feefa215640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feef41615f0 con 0x7feef415df80 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feefa215640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feef4164020 con 0x7feef4160fa0 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feefa215640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7feef41613b0 con 0x7feef4164b00 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feef2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feef415df80 0x7feef4160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49724/0 (socket says 172.21.15.67:49724) 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feef2ffd640 1 -- 172.21.15.67:0/994568991 learned_addr learned my addr 172.21.15.67:0/994568991 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feef37fe640 1 --2- 172.21.15.67:0/994568991 >> v2:172.21.15.176:3300/0 conn(0x7feef4164b00 0x7feef40b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feef27fc640 1 -- 172.21.15.67:0/994568991 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138628108 0 0) 0x7feef41615f0 con 0x7feef415df80 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feef27fc640 1 -- 172.21.15.67:0/994568991 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feedc003520 con 0x7feef415df80 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feef27fc640 1 -- 172.21.15.67:0/994568991 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147277356 0 0) 0x7feef4164020 con 0x7feef4160fa0 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.200+0000 7feef27fc640 1 -- 172.21.15.67:0/994568991 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feef41615f0 con 0x7feef4160fa0 2024-04-25T15:26:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.201+0000 7feef37fe640 1 -- 172.21.15.67:0/994568991 >> v1:172.21.15.176:6789/0 conn(0x7feef4160fa0 legacy=0x7feef40b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.201+0000 7feef37fe640 1 -- 172.21.15.67:0/994568991 >> v1:172.21.15.67:6789/0 conn(0x7feef415df80 legacy=0x7feef4160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.201+0000 7feef37fe640 1 -- 172.21.15.67:0/994568991 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef4164300 con 0x7feef4164b00 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.201+0000 7feef37fe640 1 --2- 172.21.15.67:0/994568991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef4164b00 0x7feef40b7b80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feee4009d90 tx=0x7feee402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ac3a626fa95b208 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.201+0000 7feef27fc640 1 -- 172.21.15.67:0/994568991 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4007740 con 0x7feef4164b00 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feef27fc640 1 -- 172.21.15.67:0/994568991 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee40078e0 con 0x7feef4164b00 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feef27fc640 1 -- 172.21.15.67:0/994568991 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4007cb0 con 0x7feef4164b00 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feefa215640 1 -- 172.21.15.67:0/994568991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef4164b00 msgr2=0x7feef40b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feefa215640 1 --2- 172.21.15.67:0/994568991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef4164b00 0x7feef40b7b80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feee4009d90 tx=0x7feee402a6f0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feefa215640 1 -- 172.21.15.67:0/994568991 shutdown_connections 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feefa215640 1 --2- 172.21.15.67:0/994568991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef4164b00 0x7feef40b7b80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feefa215640 1 -- 172.21.15.67:0/994568991 >> 172.21.15.67:0/994568991 conn(0x7feef40d2bc0 msgr2=0x7feef40d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feefa215640 1 -- 172.21.15.67:0/994568991 shutdown_connections 2024-04-25T15:26:09.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.202+0000 7feefa215640 1 -- 172.21.15.67:0/994568991 wait complete. 2024-04-25T15:26:09.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.204+0000 7feefa215640 1 Processor -- start 2024-04-25T15:26:09.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.204+0000 7feefa215640 1 -- start start 2024-04-25T15:26:09.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.205+0000 7feefa215640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef415df80 0x7feef414fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.205+0000 7feefa215640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160fa0 0x7feef41f9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.205+0000 7feefa215640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feef4164b00 0x7feef415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.205+0000 7feefa215640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feef40b84d0 con 0x7feef4160fa0 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.205+0000 7feefa215640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feef40b8350 con 0x7feef415df80 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.205+0000 7feefa215640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feef4164670 con 0x7feef4164b00 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160fa0 0x7feef41f9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160fa0 0x7feef41f9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49424/0 (socket says 172.21.15.67:49424) 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 -- 172.21.15.67:0/2815930015 learned_addr learned my addr 172.21.15.67:0/2815930015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef37fe640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef415df80 0x7feef414fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef3fff640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feef4164b00 0x7feef415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 -- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feef4164b00 msgr2=0x7feef415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feef4164b00 0x7feef415e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 -- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef415df80 msgr2=0x7feef414fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef415df80 0x7feef414fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 -- 172.21.15.67:0/2815930015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef41fac90 con 0x7feef4160fa0 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feef2ffd640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160fa0 0x7feef41f9f60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feee800c410 tx=0x7feee800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feedbfff640 1 -- 172.21.15.67:0/2815930015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8017070 con 0x7feef4160fa0 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef40cb890 con 0x7feef4160fa0 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feedbfff640 1 -- 172.21.15.67:0/2815930015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee8005ce0 con 0x7feef4160fa0 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.206+0000 7feedbfff640 1 -- 172.21.15.67:0/2815930015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee80072d0 con 0x7feef4160fa0 2024-04-25T15:26:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.207+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef40cbe50 con 0x7feef4160fa0 2024-04-25T15:26:09.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.207+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef4050740 con 0x7feef4160fa0 2024-04-25T15:26:09.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.208+0000 7feedbfff640 1 -- 172.21.15.67:0/2815930015 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7feee801a070 con 0x7feef4160fa0 2024-04-25T15:26:09.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.208+0000 7feedbfff640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feee00744b0 0x7feee00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.209+0000 7feedbfff640 1 -- 172.21.15.67:0/2815930015 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7feee80945c0 con 0x7feef4160fa0 2024-04-25T15:26:09.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.209+0000 7feef37fe640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feee00744b0 0x7feee00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.209+0000 7feef37fe640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feee00744b0 0x7feee00769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feee4034000 tx=0x7feee402f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.213+0000 7feedbfff640 1 -- 172.21.15.67:0/2815930015 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feee805e680 con 0x7feef4160fa0 2024-04-25T15:26:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.444+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feef41faa20 con 0x7feef4160fa0 2024-04-25T15:26:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.444+0000 7feedbfff640 1 -- 172.21.15.67:0/2815930015 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7feee8062980 con 0x7feef4160fa0 2024-04-25T15:26:09.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:09.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:26:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.447+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feee00744b0 msgr2=0x7feee00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.447+0000 7feefa215640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feee00744b0 0x7feee00769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feee4034000 tx=0x7feee402f000 comp rx=0 tx=0).stop 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.447+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160fa0 msgr2=0x7feef41f9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.447+0000 7feefa215640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160fa0 0x7feef41f9f60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feee800c410 tx=0x7feee800c8d0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 shutdown_connections 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feee00744b0 0x7feee00769c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feef4164b00 0x7feef415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef4160fa0 0x7feef41f9f60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 --2- 172.21.15.67:0/2815930015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feef415df80 0x7feef414fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 >> 172.21.15.67:0/2815930015 conn(0x7feef40d2bc0 msgr2=0x7feef40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 shutdown_connections 2024-04-25T15:26:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.448+0000 7feefa215640 1 -- 172.21.15.67:0/2815930015 wait complete. 2024-04-25T15:26:09.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:09.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.572+0000 7f53875c5640 1 Processor -- start 2024-04-25T15:26:09.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.572+0000 7f53875c5640 1 -- start start 2024-04-25T15:26:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f53875c5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f53800cf770 0x7f5380162590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f53875c5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5380162ad0 0x7f53800b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f53875c5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53800cfc30 con 0x7f53800b91d0 2024-04-25T15:26:09.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f53875c5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f53800d1ff0 con 0x7f5380162ad0 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f53875c5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f53800d2890 con 0x7f53800cf770 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f5385b62640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5380162ad0 0x7f53800b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f5385b62640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5380162ad0 0x7f53800b0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49434/0 (socket says 172.21.15.67:49434) 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f5385b62640 1 -- 172.21.15.67:0/4184653974 learned_addr learned my addr 172.21.15.67:0/4184653974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.573+0000 7f5384b60640 1 --2- 172.21.15.67:0/4184653974 >> v2:172.21.15.176:3300/0 conn(0x7f53800cf770 0x7f5380162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f5385b62640 1 -- 172.21.15.67:0/4184653974 >> v2:172.21.15.176:3300/0 conn(0x7f53800cf770 msgr2=0x7f5380162590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f5385b62640 1 --2- 172.21.15.67:0/4184653974 >> v2:172.21.15.176:3300/0 conn(0x7f53800cf770 0x7f5380162590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f5385b62640 1 -- 172.21.15.67:0/4184653974 >> v1:172.21.15.176:6789/0 conn(0x7f53800b91d0 legacy=0x7f5380161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f5385b62640 1 -- 172.21.15.67:0/4184653974 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53800d2240 con 0x7f5380162ad0 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f5384b60640 1 --2- 172.21.15.67:0/4184653974 >> v2:172.21.15.176:3300/0 conn(0x7f53800cf770 0x7f5380162590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f5385b62640 1 --2- 172.21.15.67:0/4184653974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5380162ad0 0x7f53800b0690 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5380075fa0 tx=0x7f537002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65aa4c4e8847eab8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f537ffff640 1 -- 172.21.15.67:0/4184653974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370037070 con 0x7f5380162ad0 2024-04-25T15:26:09.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.574+0000 7f537ffff640 1 -- 172.21.15.67:0/4184653974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5370032440 con 0x7f5380162ad0 2024-04-25T15:26:09.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f537ffff640 1 -- 172.21.15.67:0/4184653974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370032810 con 0x7f5380162ad0 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 -- 172.21.15.67:0/4184653974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5380162ad0 msgr2=0x7f53800b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 --2- 172.21.15.67:0/4184653974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5380162ad0 0x7f53800b0690 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5380075fa0 tx=0x7f537002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 -- 172.21.15.67:0/4184653974 shutdown_connections 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 --2- 172.21.15.67:0/4184653974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5380162ad0 0x7f53800b0690 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 --2- 172.21.15.67:0/4184653974 >> v2:172.21.15.176:3300/0 conn(0x7f53800cf770 0x7f5380162590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 -- 172.21.15.67:0/4184653974 >> 172.21.15.67:0/4184653974 conn(0x7f53800d3aa0 msgr2=0x7f53800d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 -- 172.21.15.67:0/4184653974 shutdown_connections 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.575+0000 7f53875c5640 1 -- 172.21.15.67:0/4184653974 wait complete. 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.577+0000 7f53875c5640 1 Processor -- start 2024-04-25T15:26:09.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.577+0000 7f53875c5640 1 -- start start 2024-04-25T15:26:09.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.578+0000 7f53875c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800b91d0 0x7f538015db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.578+0000 7f53875c5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5380162ad0 0x7f5380158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.578+0000 7f53875c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5380159270 0x7f53801fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.578+0000 7f53875c5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53800b24a0 con 0x7f53800b91d0 2024-04-25T15:26:09.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.578+0000 7f53875c5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f53800b2320 con 0x7f5380162ad0 2024-04-25T15:26:09.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.578+0000 7f53875c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f53800b0bd0 con 0x7f5380159270 2024-04-25T15:26:09.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.578+0000 7f5385361640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800b91d0 0x7f538015db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800b91d0 0x7f538015db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49442/0 (socket says 172.21.15.67:49442) 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 -- 172.21.15.67:0/2741423743 learned_addr learned my addr 172.21.15.67:0/2741423743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5384b60640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5380162ad0 0x7f5380158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385b62640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5380159270 0x7f53801fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 -- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5380159270 msgr2=0x7f53801fdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5380159270 0x7f53801fdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 -- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5380162ad0 msgr2=0x7f5380158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5380162ad0 0x7f5380158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 -- 172.21.15.67:0/2741423743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53800cadf0 con 0x7f53800b91d0 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5384b60640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5380162ad0 0x7f5380158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385b62640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5380159270 0x7f53801fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f5385361640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800b91d0 0x7f538015db20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f537800c730 tx=0x7f537800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f537dffb640 1 -- 172.21.15.67:0/2741423743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378016020 con 0x7f53800b91d0 2024-04-25T15:26:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f537dffb640 1 -- 172.21.15.67:0/2741423743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5378006d50 con 0x7f53800b91d0 2024-04-25T15:26:09.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538015f690 con 0x7f53800b91d0 2024-04-25T15:26:09.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.579+0000 7f537dffb640 1 -- 172.21.15.67:0/2741423743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378003bb0 con 0x7f53800b91d0 2024-04-25T15:26:09.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.580+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538015fc50 con 0x7f53800b91d0 2024-04-25T15:26:09.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.581+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f538015f3f0 con 0x7f53800b91d0 2024-04-25T15:26:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.581+0000 7f537dffb640 1 -- 172.21.15.67:0/2741423743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f537800a850 con 0x7f53800b91d0 2024-04-25T15:26:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.582+0000 7f537dffb640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f536c0744e0 0x7f536c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.582+0000 7f537dffb640 1 -- 172.21.15.67:0/2741423743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f5378094e60 con 0x7f53800b91d0 2024-04-25T15:26:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.582+0000 7f5384b60640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f536c0744e0 0x7f536c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.583+0000 7f5384b60640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f536c0744e0 0x7f536c0769f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f53801597a0 tx=0x7f5374008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.587+0000 7f537dffb640 1 -- 172.21.15.67:0/2741423743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5378020020 con 0x7f53800b91d0 2024-04-25T15:26:09.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.810+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5380050740 con 0x7f53800b91d0 2024-04-25T15:26:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.811+0000 7f537dffb640 1 -- 172.21.15.67:0/2741423743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5181 (secure 0 0 0) 0x7f537805f690 con 0x7f53800b91d0 2024-04-25T15:26:09.812 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4259,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3644200964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3644200964},{"type":"v1","addr":"172.21.15.176:6837","nonce":3644200964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1146562263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1146562263},{"type":"v1","addr":"172.21.15.67:6834","nonce":1146562263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2642054233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2642054233},{"type":"v1","addr":"172.21.15.67:6835","nonce":2642054233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:25:28.675855+0000","modified":"2024-04-25T15:25:31.686830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6836/4089268988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4089268988},{"type":"v1","addr":"172.21.15.176:6836","nonce":4089268988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-04-25T15:26:09.812 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-25T15:26:09.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f536c0744e0 msgr2=0x7f536c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f536c0744e0 0x7f536c0769f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f53801597a0 tx=0x7f5374008040 comp rx=0 tx=0).stop 2024-04-25T15:26:09.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800b91d0 msgr2=0x7f538015db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800b91d0 0x7f538015db20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f537800c730 tx=0x7f537800cbf0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 shutdown_connections 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f536c0744e0 0x7f536c0769f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5380159270 0x7f53801fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5380162ad0 0x7f5380158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 --2- 172.21.15.67:0/2741423743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53800b91d0 0x7f538015db20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 >> 172.21.15.67:0/2741423743 conn(0x7f53800d3aa0 msgr2=0x7f53800cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 shutdown_connections 2024-04-25T15:26:09.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.813+0000 7f53875c5640 1 -- 172.21.15.67:0/2741423743 wait complete. 2024-04-25T15:26:09.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:09.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.936+0000 7fced12af640 1 Processor -- start 2024-04-25T15:26:09.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.936+0000 7fced12af640 1 -- start start 2024-04-25T15:26:09.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fced12af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcecc0b8290 0x7fcecc160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fced12af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc15df80 0x7fcecc160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fced12af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcecc1615f0 con 0x7fcecc164b00 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fced12af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcecc164670 con 0x7fcecc0b8290 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fced12af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcecc164020 con 0x7fcecc15df80 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fcec27fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcecc0b8290 0x7fcecc160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fcec27fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcecc0b8290 0x7fcecc160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49446/0 (socket says 172.21.15.67:49446) 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fcec27fc640 1 -- 172.21.15.67:0/2144207290 learned_addr learned my addr 172.21.15.67:0/2144207290 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.937+0000 7fcecb7fe640 1 --2- 172.21.15.67:0/2144207290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc15df80 0x7fcecc160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.938+0000 7fcec27fc640 1 -- 172.21.15.67:0/2144207290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc15df80 msgr2=0x7fcecc160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.938+0000 7fcec27fc640 1 --2- 172.21.15.67:0/2144207290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc15df80 0x7fcecc160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.938+0000 7fcec27fc640 1 -- 172.21.15.67:0/2144207290 >> v1:172.21.15.176:6789/0 conn(0x7fcecc164b00 legacy=0x7fcecc0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.938+0000 7fceca7fc640 1 -- 172.21.15.67:0/2144207290 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1076059136 0 0) 0x7fcecc1615f0 con 0x7fcecc164b00 2024-04-25T15:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.938+0000 7fcec27fc640 1 -- 172.21.15.67:0/2144207290 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcecc164300 con 0x7fcecc0b8290 2024-04-25T15:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.938+0000 7fcec27fc640 1 --2- 172.21.15.67:0/2144207290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc0b8290 0x7fcecc160890 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009870 tx=0x7fceb402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e9a5e383712fb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fceca7fc640 1 -- 172.21.15.67:0/2144207290 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4037070 con 0x7fcecc0b8290 2024-04-25T15:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fceca7fc640 1 -- 172.21.15.67:0/2144207290 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb4032440 con 0x7fcecc0b8290 2024-04-25T15:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fceca7fc640 1 -- 172.21.15.67:0/2144207290 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4032810 con 0x7fcecc0b8290 2024-04-25T15:26:09.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 -- 172.21.15.67:0/2144207290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc0b8290 msgr2=0x7fcecc160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 --2- 172.21.15.67:0/2144207290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc0b8290 0x7fcecc160890 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009870 tx=0x7fceb402a270 comp rx=0 tx=0).stop 2024-04-25T15:26:09.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 -- 172.21.15.67:0/2144207290 shutdown_connections 2024-04-25T15:26:09.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 --2- 172.21.15.67:0/2144207290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc15df80 0x7fcecc160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 --2- 172.21.15.67:0/2144207290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc0b8290 0x7fcecc160890 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 -- 172.21.15.67:0/2144207290 >> 172.21.15.67:0/2144207290 conn(0x7fcecc0d2bc0 msgr2=0x7fcecc0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 -- 172.21.15.67:0/2144207290 shutdown_connections 2024-04-25T15:26:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.939+0000 7fced12af640 1 -- 172.21.15.67:0/2144207290 wait complete. 2024-04-25T15:26:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.941+0000 7fced12af640 1 Processor -- start 2024-04-25T15:26:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.941+0000 7fced12af640 1 -- start start 2024-04-25T15:26:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.942+0000 7fced12af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc15df80 0x7fcecc0d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.942+0000 7fced12af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc164b00 0x7fcecc15aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.942+0000 7fced12af640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcecc15b3e0 0x7fcecc0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.942+0000 7fced12af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc0b4a00 con 0x7fcecc15df80 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fced12af640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcecc0b4880 con 0x7fcecc15b3e0 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fced12af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcecc0b4b80 con 0x7fcecc164b00 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc15df80 0x7fcecc0d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc15df80 0x7fcecc0d4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49448/0 (socket says 172.21.15.67:49448) 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 -- 172.21.15.67:0/1459767297 learned_addr learned my addr 172.21.15.67:0/1459767297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecb7fe640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcecc15b3e0 0x7fcecc0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcec27fc640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc164b00 0x7fcecc15aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 -- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc164b00 msgr2=0x7fcecc15aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc164b00 0x7fcecc15aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 -- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcecc15b3e0 msgr2=0x7fcecc0c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcecc15b3e0 0x7fcecc0c8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 -- 172.21.15.67:0/1459767297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcecc0d4f90 con 0x7fcecc15df80 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcec27fc640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc164b00 0x7fcecc15aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcecaffd640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc15df80 0x7fcecc0d4a50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00c410 tx=0x7fcebc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcec3fff640 1 -- 172.21.15.67:0/1459767297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc017070 con 0x7fcecc15df80 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcec3fff640 1 -- 172.21.15.67:0/1459767297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcebc0035f0 con 0x7fcecc15df80 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcecc156470 con 0x7fcecc15df80 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.943+0000 7fcec3fff640 1 -- 172.21.15.67:0/1459767297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc0039e0 con 0x7fcecc15df80 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.944+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcecc156a30 con 0x7fcecc15df80 2024-04-25T15:26:09.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.945+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcecc069a40 con 0x7fcecc15df80 2024-04-25T15:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.945+0000 7fcec3fff640 1 -- 172.21.15.67:0/1459767297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcebc01a070 con 0x7fcecc15df80 2024-04-25T15:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.945+0000 7fcec3fff640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fceac0744b0 0x7fceac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.946+0000 7fcec3fff640 1 -- 172.21.15.67:0/1459767297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7fcebc0945e0 con 0x7fcecc15df80 2024-04-25T15:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.946+0000 7fcec27fc640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fceac0744b0 0x7fceac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.946+0000 7fcec27fc640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fceac0744b0 0x7fceac0769c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fceb4034000 tx=0x7fceb402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:09.950+0000 7fcec3fff640 1 -- 172.21.15.67:0/1459767297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcecc069a40 con 0x7fcecc15df80 2024-04-25T15:26:10.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.136+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcecc050740 con 0x7fcecc15df80 2024-04-25T15:26:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.137+0000 7fcec3fff640 1 -- 172.21.15.67:0/1459767297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16818 (secure 0 0 0) 0x7fcebc05e730 con 0x7fcecc15df80 2024-04-25T15:26:10.139 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:10.139 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":21,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:09.432268+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:25:27.207015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:25:27.615603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:10.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fceac0744b0 msgr2=0x7fceac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fceac0744b0 0x7fceac0769c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fceb4034000 tx=0x7fceb402e000 comp rx=0 tx=0).stop 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc15df80 msgr2=0x7fcecc0d4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc15df80 0x7fcecc0d4a50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00c410 tx=0x7fcebc00c8d0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 shutdown_connections 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fceac0744b0 0x7fceac0769c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcecc15b3e0 0x7fcecc0c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcecc164b00 0x7fcecc15aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 --2- 172.21.15.67:0/1459767297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcecc15df80 0x7fcecc0d4a50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.139+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 >> 172.21.15.67:0/1459767297 conn(0x7fcecc0d2bc0 msgr2=0x7fcecc15e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.140+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 shutdown_connections 2024-04-25T15:26:10.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.140+0000 7fced12af640 1 -- 172.21.15.67:0/1459767297 wait complete. 2024-04-25T15:26:10.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:26:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.264+0000 7faf5e4d8640 1 Processor -- start 2024-04-25T15:26:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.264+0000 7faf5e4d8640 1 -- start start 2024-04-25T15:26:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5e4d8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d0de0 0x7faf580d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5e4d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf580d4a30 0x7faf580d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5e4d8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf580d1830 con 0x7faf5806c5c0 2024-04-25T15:26:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5e4d8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faf58160dc0 con 0x7faf580d4a30 2024-04-25T15:26:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5e4d8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faf581610a0 con 0x7faf580d0de0 2024-04-25T15:26:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5ca75640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf580d4a30 0x7faf580d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5ca75640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf580d4a30 0x7faf580d4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49454/0 (socket says 172.21.15.67:49454) 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.265+0000 7faf5ca75640 1 -- 172.21.15.67:0/170942760 learned_addr learned my addr 172.21.15.67:0/170942760 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.266+0000 7faf57fff640 1 --2- 172.21.15.67:0/170942760 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d0de0 0x7faf580d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.266+0000 7faf5ca75640 1 -- 172.21.15.67:0/170942760 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d0de0 msgr2=0x7faf580d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.266+0000 7faf5ca75640 1 --2- 172.21.15.67:0/170942760 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d0de0 0x7faf580d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.266+0000 7faf5ca75640 1 -- 172.21.15.67:0/170942760 >> v1:172.21.15.176:6789/0 conn(0x7faf5806c5c0 legacy=0x7faf5806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.266+0000 7faf5ca75640 1 -- 172.21.15.67:0/170942760 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf58161380 con 0x7faf580d4a30 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.266+0000 7faf57fff640 1 --2- 172.21.15.67:0/170942760 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d0de0 0x7faf580d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.266+0000 7faf5ca75640 1 --2- 172.21.15.67:0/170942760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf580d4a30 0x7faf580d4e40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faf50009e50 tx=0x7faf5002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38396ed4245183e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf56ffd640 1 -- 172.21.15.67:0/170942760 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50036070 con 0x7faf580d4a30 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf56ffd640 1 -- 172.21.15.67:0/170942760 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf50031600 con 0x7faf580d4a30 2024-04-25T15:26:10.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf56ffd640 1 -- 172.21.15.67:0/170942760 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf500319d0 con 0x7faf580d4a30 2024-04-25T15:26:10.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 -- 172.21.15.67:0/170942760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf580d4a30 msgr2=0x7faf580d4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/170942760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf580d4a30 0x7faf580d4e40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faf50009e50 tx=0x7faf5002a710 comp rx=0 tx=0).stop 2024-04-25T15:26:10.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 -- 172.21.15.67:0/170942760 shutdown_connections 2024-04-25T15:26:10.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/170942760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf580d4a30 0x7faf580d4e40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/170942760 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d0de0 0x7faf580d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 -- 172.21.15.67:0/170942760 >> 172.21.15.67:0/170942760 conn(0x7faf580d5d70 msgr2=0x7faf580d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:10.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 -- 172.21.15.67:0/170942760 shutdown_connections 2024-04-25T15:26:10.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.267+0000 7faf5e4d8640 1 -- 172.21.15.67:0/170942760 wait complete. 2024-04-25T15:26:10.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.269+0000 7faf5e4d8640 1 Processor -- start 2024-04-25T15:26:10.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.269+0000 7faf5e4d8640 1 -- start start 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5e4d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5806c5c0 0x7faf580cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5e4d8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf580d0de0 0x7faf5814f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5e4d8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d4a30 0x7faf581e9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5e4d8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faf58164de0 con 0x7faf5806c5c0 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5e4d8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faf580b5300 con 0x7faf580d0de0 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5e4d8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faf58164f60 con 0x7faf580d4a30 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5806c5c0 0x7faf580cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5806c5c0 0x7faf580cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49470/0 (socket says 172.21.15.67:49470) 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 -- 172.21.15.67:0/1531319714 learned_addr learned my addr 172.21.15.67:0/1531319714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf577fe640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf580d0de0 0x7faf5814f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5ca75640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d4a30 0x7faf581e9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 -- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d4a30 msgr2=0x7faf581e9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d4a30 0x7faf581e9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 -- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf580d0de0 msgr2=0x7faf5814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf580d0de0 0x7faf5814f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 -- 172.21.15.67:0/1531319714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf580b61f0 con 0x7faf5806c5c0 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf5ca75640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d4a30 0x7faf581e9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.271+0000 7faf57fff640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5806c5c0 0x7faf580cb750 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00d800 tx=0x7faf4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.272+0000 7faf54ff9640 1 -- 172.21.15.67:0/1531319714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c004030 con 0x7faf5806c5c0 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.272+0000 7faf54ff9640 1 -- 172.21.15.67:0/1531319714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c0041d0 con 0x7faf5806c5c0 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.272+0000 7faf54ff9640 1 -- 172.21.15.67:0/1531319714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c0096a0 con 0x7faf5806c5c0 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.272+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf580b64d0 con 0x7faf5806c5c0 2024-04-25T15:26:10.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.272+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf580b8030 con 0x7faf5806c5c0 2024-04-25T15:26:10.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.273+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5815e3b0 con 0x7faf5806c5c0 2024-04-25T15:26:10.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.273+0000 7faf54ff9640 1 -- 172.21.15.67:0/1531319714 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7faf4c009840 con 0x7faf5806c5c0 2024-04-25T15:26:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.274+0000 7faf54ff9640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faf440744b0 0x7faf440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.274+0000 7faf54ff9640 1 -- 172.21.15.67:0/1531319714 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7faf4c0954e0 con 0x7faf5806c5c0 2024-04-25T15:26:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.274+0000 7faf577fe640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faf440744b0 0x7faf440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.275+0000 7faf577fe640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faf440744b0 0x7faf440769c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faf580d57a0 tx=0x7faf48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:10.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.279+0000 7faf54ff9640 1 -- 172.21.15.67:0/1531319714 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf5815e3b0 con 0x7faf5806c5c0 2024-04-25T15:26:10.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.512+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faf58050740 con 0x7faf5806c5c0 2024-04-25T15:26:10.514 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:26:10.513+0000 7f3e02b50640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:26:10.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.515+0000 7faf54ff9640 1 -- 172.21.15.67:0/1531319714 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) ==== 161+0+0 (secure 0 0 0) 0x7faf4c05fbf0 con 0x7faf5806c5c0 2024-04-25T15:26:10.515 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:26:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faf440744b0 msgr2=0x7faf440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faf440744b0 0x7faf440769c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faf580d57a0 tx=0x7faf48009040 comp rx=0 tx=0).stop 2024-04-25T15:26:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5806c5c0 msgr2=0x7faf580cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5806c5c0 0x7faf580cb750 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7faf4c00d800 tx=0x7faf4c00dcc0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 shutdown_connections 2024-04-25T15:26:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faf440744b0 0x7faf440769c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf580d4a30 0x7faf581e9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf580d0de0 0x7faf5814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 --2- 172.21.15.67:0/1531319714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf5806c5c0 0x7faf580cb750 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 >> 172.21.15.67:0/1531319714 conn(0x7faf580d5d70 msgr2=0x7faf58163870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 shutdown_connections 2024-04-25T15:26:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.517+0000 7faf5e4d8640 1 -- 172.21.15.67:0/1531319714 wait complete. 2024-04-25T15:26:10.523 INFO:tasks.ceph.mds.b.smithi176.stderr: -7483> 2024-04-25T15:25:33.591+0000 7efc9845a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:26:10.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:26:10.583 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:26:10.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.643+0000 7ff0d3f08640 1 Processor -- start 2024-04-25T15:26:10.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.644+0000 7ff0d3f08640 1 -- start start 2024-04-25T15:26:10.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d3f08640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff0cc0d0e00 0x7ff0cc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d3f08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0cc0d4a50 0x7ff0cc0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d3f08640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0cc0d1850 con 0x7ff0cc06c5c0 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d3f08640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff0cc160de0 con 0x7ff0cc0d4a50 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d3f08640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff0cc1610c0 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d24a5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0cc0d4a50 0x7ff0cc0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d14a3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff0cc06c5c0 0x7ff0cc06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49738/0 (socket says 172.21.15.67:49738) 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d14a3640 1 -- 172.21.15.67:0/1357212761 learned_addr learned my addr 172.21.15.67:0/1357212761 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d1ca4640 1 --2- 172.21.15.67:0/1357212761 >> v2:172.21.15.176:3300/0 conn(0x7ff0cc0d0e00 0x7ff0cc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.645+0000 7ff0d24a5640 1 -- 172.21.15.67:0/1357212761 >> v2:172.21.15.176:3300/0 conn(0x7ff0cc0d0e00 msgr2=0x7ff0cc0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d24a5640 1 --2- 172.21.15.67:0/1357212761 >> v2:172.21.15.176:3300/0 conn(0x7ff0cc0d0e00 0x7ff0cc0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d24a5640 1 -- 172.21.15.67:0/1357212761 >> v1:172.21.15.67:6789/0 conn(0x7ff0cc06c5c0 legacy=0x7ff0cc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d24a5640 1 -- 172.21.15.67:0/1357212761 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc1613a0 con 0x7ff0cc0d4a50 2024-04-25T15:26:10.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d0ca2640 1 -- 172.21.15.67:0/1357212761 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443498116 0 0) 0x7ff0cc0d1850 con 0x7ff0cc06c5c0 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d24a5640 1 --2- 172.21.15.67:0/1357212761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d4a50 0x7ff0cc0d4e60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0d3e60 tx=0x7ff0bc02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a54ebb3b129ab1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d0ca2640 1 -- 172.21.15.67:0/1357212761 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc007740 con 0x7ff0cc0d4a50 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d0ca2640 1 -- 172.21.15.67:0/1357212761 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0bc0078e0 con 0x7ff0cc0d4a50 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.646+0000 7ff0d0ca2640 1 -- 172.21.15.67:0/1357212761 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc007cb0 con 0x7ff0cc0d4a50 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 -- 172.21.15.67:0/1357212761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d4a50 msgr2=0x7ff0cc0d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/1357212761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d4a50 0x7ff0cc0d4e60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0d3e60 tx=0x7ff0bc02a5f0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 -- 172.21.15.67:0/1357212761 shutdown_connections 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/1357212761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d4a50 0x7ff0cc0d4e60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/1357212761 >> v2:172.21.15.176:3300/0 conn(0x7ff0cc0d0e00 0x7ff0cc0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 -- 172.21.15.67:0/1357212761 >> 172.21.15.67:0/1357212761 conn(0x7ff0cc0d5d90 msgr2=0x7ff0cc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 -- 172.21.15.67:0/1357212761 shutdown_connections 2024-04-25T15:26:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.647+0000 7ff0d3f08640 1 -- 172.21.15.67:0/1357212761 wait complete. 2024-04-25T15:26:10.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.649+0000 7ff0d3f08640 1 Processor -- start 2024-04-25T15:26:10.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.649+0000 7ff0d3f08640 1 -- start start 2024-04-25T15:26:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.650+0000 7ff0d3f08640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc06c5c0 0x7ff0cc1ec350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.650+0000 7ff0d3f08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d0e00 0x7ff0cc0b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.650+0000 7ff0d3f08640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc0d4a50 0x7ff0cc145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.650+0000 7ff0d3f08640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc164e00 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.650+0000 7ff0d3f08640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc0b5320 con 0x7ff0cc06c5c0 2024-04-25T15:26:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.650+0000 7ff0d3f08640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc164f80 con 0x7ff0cc0d4a50 2024-04-25T15:26:10.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.650+0000 7ff0d14a3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d0e00 0x7ff0cc0b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d0e00 0x7ff0cc0b8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49486/0 (socket says 172.21.15.67:49486) 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 -- 172.21.15.67:0/2209725666 learned_addr learned my addr 172.21.15.67:0/2209725666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d1ca4640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc06c5c0 0x7ff0cc1ec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d24a5640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc0d4a50 0x7ff0cc145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 -- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc0d4a50 msgr2=0x7ff0cc145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc0d4a50 0x7ff0cc145eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 -- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc06c5c0 msgr2=0x7ff0cc1ec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc06c5c0 0x7ff0cc1ec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 -- 172.21.15.67:0/2209725666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc0b8650 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d1ca4640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc06c5c0 0x7ff0cc1ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d24a5640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc0d4a50 0x7ff0cc145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d14a3640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d0e00 0x7ff0cc0b8110 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c460 tx=0x7ff0c000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0ca7fc640 1 -- 172.21.15.67:0/2209725666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0018070 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0ca7fc640 1 -- 172.21.15.67:0/2209725666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c0007500 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0cc0cd0b0 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0ca7fc640 1 -- 172.21.15.67:0/2209725666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0004a20 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.651+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0cc1eb870 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.652+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0cc1638d0 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.653+0000 7ff0ca7fc640 1 -- 172.21.15.67:0/2209725666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff0c001b070 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.653+0000 7ff0ca7fc640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0b80744e0 0x7ff0b80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.654+0000 7ff0ca7fc640 1 -- 172.21.15.67:0/2209725666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7ff0c0094590 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.654+0000 7ff0d1ca4640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0b80744e0 0x7ff0b80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.654+0000 7ff0d1ca4640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0b80744e0 0x7ff0b80769f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009af0 tx=0x7ff0c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.658+0000 7ff0ca7fc640 1 -- 172.21.15.67:0/2209725666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0cc1638d0 con 0x7ff0cc0d0e00 2024-04-25T15:26:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:10.890+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff0cc050740 con 0x7ff0cc0d0e00 2024-04-25T15:26:11.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.522+0000 7ff0ca7fc640 1 -- 172.21.15.67:0/2209725666 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) ==== 104+0+0 (secure 0 0 0) 0x7ff0c005ec50 con 0x7ff0cc0d0e00 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0b80744e0 msgr2=0x7ff0b80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0b80744e0 0x7ff0b80769f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009af0 tx=0x7ff0c4008040 comp rx=0 tx=0).stop 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d0e00 msgr2=0x7ff0cc0b8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d0e00 0x7ff0cc0b8110 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000c460 tx=0x7ff0c000c920 comp rx=0 tx=0).stop 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 shutdown_connections 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff0b80744e0 0x7ff0b80769f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc0d4a50 0x7ff0cc145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0cc0d0e00 0x7ff0cc0b8110 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 --2- 172.21.15.67:0/2209725666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc06c5c0 0x7ff0cc1ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 >> 172.21.15.67:0/2209725666 conn(0x7ff0cc0d5d90 msgr2=0x7ff0cc1631c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 shutdown_connections 2024-04-25T15:26:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.525+0000 7ff0d3f08640 1 -- 172.21.15.67:0/2209725666 wait complete. 2024-04-25T15:26:11.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:26:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.655+0000 7f1afb6e0640 1 Processor -- start 2024-04-25T15:26:11.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.655+0000 7f1afb6e0640 1 -- start start 2024-04-25T15:26:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1afb6e0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1af406dce0 0x7f1af415f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1afb6e0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1af40b0af0 0x7f1af415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1afb6e0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af40cdea0 con 0x7f1af40cd9a0 2024-04-25T15:26:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1afb6e0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1af40ceab0 con 0x7f1af406dce0 2024-04-25T15:26:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1afb6e0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1af40ce180 con 0x7f1af40b0af0 2024-04-25T15:26:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af8c7b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1af406dce0 0x7f1af415f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:11.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af8c7b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1af406dce0 0x7f1af415f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49488/0 (socket says 172.21.15.67:49488) 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af8c7b640 1 -- 172.21.15.67:0/2173264392 learned_addr learned my addr 172.21.15.67:0/2173264392 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af9c7d640 1 --2- 172.21.15.67:0/2173264392 >> v2:172.21.15.176:3300/0 conn(0x7f1af40b0af0 0x7f1af415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af8c7b640 1 -- 172.21.15.67:0/2173264392 >> v2:172.21.15.176:3300/0 conn(0x7f1af40b0af0 msgr2=0x7f1af415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af8c7b640 1 --2- 172.21.15.67:0/2173264392 >> v2:172.21.15.176:3300/0 conn(0x7f1af40b0af0 0x7f1af415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af8c7b640 1 -- 172.21.15.67:0/2173264392 >> v1:172.21.15.176:6789/0 conn(0x7f1af40cd9a0 legacy=0x7f1af406d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af8c7b640 1 -- 172.21.15.67:0/2173264392 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af40ce460 con 0x7f1af406dce0 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.657+0000 7f1af9c7d640 1 --2- 172.21.15.67:0/2173264392 >> v2:172.21.15.176:3300/0 conn(0x7f1af40b0af0 0x7f1af415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.658+0000 7f1af8c7b640 1 --2- 172.21.15.67:0/2173264392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af406dce0 0x7f1af415f360 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1af4148020 tx=0x7f1ae802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c20841624fec50b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.658+0000 7f1af3fff640 1 -- 172.21.15.67:0/2173264392 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8037070 con 0x7f1af406dce0 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.658+0000 7f1af3fff640 1 -- 172.21.15.67:0/2173264392 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae8032440 con 0x7f1af406dce0 2024-04-25T15:26:11.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.658+0000 7f1af3fff640 1 -- 172.21.15.67:0/2173264392 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8032810 con 0x7f1af406dce0 2024-04-25T15:26:11.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 -- 172.21.15.67:0/2173264392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af406dce0 msgr2=0x7f1af415f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:11.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/2173264392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af406dce0 0x7f1af415f360 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1af4148020 tx=0x7f1ae802a270 comp rx=0 tx=0).stop 2024-04-25T15:26:11.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 -- 172.21.15.67:0/2173264392 shutdown_connections 2024-04-25T15:26:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/2173264392 >> v2:172.21.15.176:3300/0 conn(0x7f1af40b0af0 0x7f1af415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/2173264392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af406dce0 0x7f1af415f360 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 -- 172.21.15.67:0/2173264392 >> 172.21.15.67:0/2173264392 conn(0x7f1af4073090 msgr2=0x7f1af40734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 -- 172.21.15.67:0/2173264392 shutdown_connections 2024-04-25T15:26:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.659+0000 7f1afb6e0640 1 -- 172.21.15.67:0/2173264392 wait complete. 2024-04-25T15:26:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.661+0000 7f1afb6e0640 1 Processor -- start 2024-04-25T15:26:11.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.661+0000 7f1afb6e0640 1 -- start start 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1afb6e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af40b0af0 0x7f1af413d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1afb6e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1af40cd9a0 0x7f1af4132020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1afb6e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1af413e870 0x7f1af40b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1afb6e0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1af40cef40 con 0x7f1af40b0af0 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1afb6e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1af40b0f00 con 0x7f1af413e870 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1afb6e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1af40cf0c0 con 0x7f1af40cd9a0 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1af947c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af40b0af0 0x7f1af413d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1af947c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af40b0af0 0x7f1af413d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49498/0 (socket says 172.21.15.67:49498) 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.662+0000 7f1af947c640 1 -- 172.21.15.67:0/3909482063 learned_addr learned my addr 172.21.15.67:0/3909482063 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af9c7d640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1af413e870 0x7f1af40b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:11.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af8c7b640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1af40cd9a0 0x7f1af4132020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af947c640 1 -- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1af40cd9a0 msgr2=0x7f1af4132020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af947c640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1af40cd9a0 0x7f1af4132020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af947c640 1 -- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1af413e870 msgr2=0x7f1af40b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af947c640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1af413e870 0x7f1af40b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af947c640 1 -- 172.21.15.67:0/3909482063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af413efc0 con 0x7f1af40b0af0 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af9c7d640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1af413e870 0x7f1af40b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af8c7b640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1af40cd9a0 0x7f1af4132020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af947c640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af40b0af0 0x7f1af413d890 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00c3f0 tx=0x7f1aec00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af1ffb640 1 -- 172.21.15.67:0/3909482063 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0052d0 con 0x7f1af40b0af0 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af1ffb640 1 -- 172.21.15.67:0/3909482063 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aec005490 con 0x7f1af40b0af0 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af413f2a0 con 0x7f1af40b0af0 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1af1ffb640 1 -- 172.21.15.67:0/3909482063 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec007260 con 0x7f1af40b0af0 2024-04-25T15:26:11.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.663+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af413f860 con 0x7f1af40b0af0 2024-04-25T15:26:11.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.664+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af4050740 con 0x7f1af40b0af0 2024-04-25T15:26:11.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.665+0000 7f1af1ffb640 1 -- 172.21.15.67:0/3909482063 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1aec007400 con 0x7f1af40b0af0 2024-04-25T15:26:11.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.665+0000 7f1af1ffb640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1ae00744b0 0x7f1ae00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:11.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.665+0000 7f1af1ffb640 1 -- 172.21.15.67:0/3909482063 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7f1aec0947b0 con 0x7f1af40b0af0 2024-04-25T15:26:11.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.666+0000 7f1af8c7b640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1ae00744b0 0x7f1ae00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:11.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.666+0000 7f1af8c7b640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1ae00744b0 0x7f1ae00769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8034000 tx=0x7f1ae802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.670+0000 7f1af1ffb640 1 -- 172.21.15.67:0/3909482063 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1aec05ee70 con 0x7f1af40b0af0 2024-04-25T15:26:11.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:11.853+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1af41604c0 con 0x7f1af40b0af0 2024-04-25T15:26:12.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.524+0000 7f1af1ffb640 1 -- 172.21.15.67:0/3909482063 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) ==== 181+0+0 (secure 0 0 0) 0x7f1aec05e770 con 0x7f1af40b0af0 2024-04-25T15:26:12.526 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:26:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1ae00744b0 msgr2=0x7f1ae00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1ae00744b0 0x7f1ae00769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8034000 tx=0x7f1ae802e000 comp rx=0 tx=0).stop 2024-04-25T15:26:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af40b0af0 msgr2=0x7f1af413d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af40b0af0 0x7f1af413d890 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00c3f0 tx=0x7f1aec00c8b0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 shutdown_connections 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1ae00744b0 0x7f1ae00769c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1af413e870 0x7f1af40b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1af40cd9a0 0x7f1af4132020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 --2- 172.21.15.67:0/3909482063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1af40b0af0 0x7f1af413d890 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 >> 172.21.15.67:0/3909482063 conn(0x7f1af4073090 msgr2=0x7f1af40b4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 shutdown_connections 2024-04-25T15:26:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.527+0000 7f1afb6e0640 1 -- 172.21.15.67:0/3909482063 wait complete. 2024-04-25T15:26:12.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:26:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.672+0000 7fad9b08c640 1 Processor -- start 2024-04-25T15:26:12.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.672+0000 7fad9b08c640 1 -- start start 2024-04-25T15:26:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.673+0000 7fad9b08c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fad940d0e00 0x7fad940d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad9b08c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad940d1850 con 0x7fad9406c5c0 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad9b08c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad94160de0 con 0x7fad940d4a50 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad9b08c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fad941610c0 con 0x7fad940d0e00 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad93fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fad9406c5c0 0x7fad9406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49746/0 (socket says 172.21.15.67:49746) 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad93fff640 1 -- 172.21.15.67:0/89679238 learned_addr learned my addr 172.21.15.67:0/89679238 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad98e28640 1 --2- 172.21.15.67:0/89679238 >> v2:172.21.15.176:3300/0 conn(0x7fad940d0e00 0x7fad940d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692266546 0 0) 0x7fad940d1850 con 0x7fad9406c5c0 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad84003520 con 0x7fad9406c5c0 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 360712370 0 0) 0x7fad84003520 con 0x7fad9406c5c0 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 >> v2:172.21.15.176:3300/0 conn(0x7fad940d0e00 msgr2=0x7fad940d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 --2- 172.21.15.67:0/89679238 >> v2:172.21.15.176:3300/0 conn(0x7fad940d0e00 0x7fad940d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 >> v1:172.21.15.176:6789/0 conn(0x7fad940d4a50 legacy=0x7fad940d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad941613a0 con 0x7fad9406c5c0 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fad80001670 con 0x7fad9406c5c0 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 >> v1:172.21.15.67:6789/0 conn(0x7fad9406c5c0 legacy=0x7fad9406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad840039b0 0x7fad84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad84004800 0x7fad84004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad84005250 0x7fad8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fad941610c0 con 0x7fad84004800 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fad8400c1e0 con 0x7fad84005250 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.674+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fad8400c3e0 con 0x7fad840039b0 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad93fff640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad84004800 0x7fad84004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad98e28640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad840039b0 0x7fad84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad99629640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad84005250 0x7fad8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad93fff640 1 -- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad840039b0 msgr2=0x7fad84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad93fff640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad840039b0 0x7fad84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad93fff640 1 -- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad84005250 msgr2=0x7fad8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad93fff640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad84005250 0x7fad8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad93fff640 1 -- 172.21.15.67:0/89679238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad84003520 con 0x7fad84004800 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad99629640 1 -- 172.21.15.67:0/89679238 reap_dead start 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.675+0000 7fad98e28640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad840039b0 0x7fad84003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad93fff640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad84004800 0x7fad84004c10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fad8000c240 tx=0x7fad84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebdfa743a1681f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80007cf0 con 0x7fad84004800 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad800058d0 con 0x7fad84004800 2024-04-25T15:26:12.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad937fe640 1 -- 172.21.15.67:0/89679238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80005d00 con 0x7fad84004800 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad9b08c640 1 -- 172.21.15.67:0/89679238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad84004800 msgr2=0x7fad84004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad9b08c640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad84004800 0x7fad84004c10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fad8000c240 tx=0x7fad84005830 comp rx=0 tx=0).stop 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad9b08c640 1 -- 172.21.15.67:0/89679238 shutdown_connections 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad9b08c640 1 --2- 172.21.15.67:0/89679238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad84004800 0x7fad84004c10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.676+0000 7fad9b08c640 1 -- 172.21.15.67:0/89679238 >> 172.21.15.67:0/89679238 conn(0x7fad940d5d90 msgr2=0x7fad940d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.677+0000 7fad9b08c640 1 -- 172.21.15.67:0/89679238 shutdown_connections 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.677+0000 7fad9b08c640 1 -- 172.21.15.67:0/89679238 wait complete. 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.678+0000 7fad9b08c640 1 Processor -- start 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.678+0000 7fad9b08c640 1 -- start start 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad9b08c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad84004800 0x7fad940cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad9b08c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad94147b00 0x7fad94147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad9b08c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad94148450 0x7fad940ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad9b08c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fad94164e00 con 0x7fad94148450 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad9b08c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fad940b5320 con 0x7fad94147b00 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad9b08c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fad94164f80 con 0x7fad84004800 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad94148450 0x7fad940ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad94148450 0x7fad940ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49522/0 (socket says 172.21.15.67:49522) 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 -- 172.21.15.67:0/3747720731 learned_addr learned my addr 172.21.15.67:0/3747720731 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad98e28640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad84004800 0x7fad940cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad93fff640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad94147b00 0x7fad94147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 -- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad84004800 msgr2=0x7fad940cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad84004800 0x7fad940cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 -- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad94147b00 msgr2=0x7fad94147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad94147b00 0x7fad94147f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 -- 172.21.15.67:0/3747720731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad940cde20 con 0x7fad94148450 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.680+0000 7fad99629640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad94148450 0x7fad940ce5e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad940d1e50 tx=0x7fad8c00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.681+0000 7fad917fa640 1 -- 172.21.15.67:0/3747720731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c01a070 con 0x7fad94148450 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.681+0000 7fad917fa640 1 -- 172.21.15.67:0/3747720731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad8c006d60 con 0x7fad94148450 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.681+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad940cc610 con 0x7fad94148450 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.681+0000 7fad917fa640 1 -- 172.21.15.67:0/3747720731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c003bb0 con 0x7fad94148450 2024-04-25T15:26:12.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.681+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad94158a50 con 0x7fad94148450 2024-04-25T15:26:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.682+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad94050740 con 0x7fad94148450 2024-04-25T15:26:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.682+0000 7fad917fa640 1 -- 172.21.15.67:0/3747720731 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fad8c01d070 con 0x7fad94148450 2024-04-25T15:26:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.683+0000 7fad917fa640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad780744b0 0x7fad780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.683+0000 7fad917fa640 1 -- 172.21.15.67:0/3747720731 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5473+0+0 (secure 0 0 0) 0x7fad8c0945a0 con 0x7fad94148450 2024-04-25T15:26:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.683+0000 7fad98e28640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad780744b0 0x7fad780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:12.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.683+0000 7fad98e28640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad780744b0 0x7fad780769c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fad9406b530 tx=0x7fad88008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:12.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.687+0000 7fad917fa640 1 -- 172.21.15.67:0/3747720731 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad8c05e8e0 con 0x7fad94148450 2024-04-25T15:26:12.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:12.873+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fad940d4320 con 0x7fad94148450 2024-04-25T15:26:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.526+0000 7fad917fa640 1 -- 172.21.15.67:0/3747720731 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) ==== 169+0+0 (secure 0 0 0) 0x7fad8c062be0 con 0x7fad94148450 2024-04-25T15:26:13.527 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:26:13.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.530+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad780744b0 msgr2=0x7fad780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:13.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.530+0000 7fad9b08c640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad780744b0 0x7fad780769c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fad9406b530 tx=0x7fad88008040 comp rx=0 tx=0).stop 2024-04-25T15:26:13.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.530+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad94148450 msgr2=0x7fad940ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:13.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.530+0000 7fad9b08c640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad94148450 0x7fad940ce5e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fad940d1e50 tx=0x7fad8c00fc90 comp rx=0 tx=0).stop 2024-04-25T15:26:13.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.534+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 shutdown_connections 2024-04-25T15:26:13.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.534+0000 7fad9b08c640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad780744b0 0x7fad780769c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.534+0000 7fad9b08c640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad94148450 0x7fad940ce5e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.534+0000 7fad9b08c640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad94147b00 0x7fad94147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.534+0000 7fad9b08c640 1 --2- 172.21.15.67:0/3747720731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad84004800 0x7fad940cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.534+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 >> 172.21.15.67:0/3747720731 conn(0x7fad940d5d90 msgr2=0x7fad940bb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:13.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.535+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 shutdown_connections 2024-04-25T15:26:13.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.535+0000 7fad9b08c640 1 -- 172.21.15.67:0/3747720731 wait complete. 2024-04-25T15:26:13.553 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:26:13.553 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:26:13.553 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:26:13.553 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:13.554 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:26:13.554 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:26:13.554 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:26:13.554 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:13.554 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:26:13.554 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:26:13.554 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:26:13.555 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:13.555 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:26:13.555 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:26:13.555 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:26:13.555 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:13.564 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:26:13.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:26:13.565 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:26:13.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:26:13.568 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:26:13.569 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:26:13.571 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:26:13.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:26:13.607 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:26:13.607 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:13.608 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:26:13.608 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:13.613 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:26:13.613 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:13.614 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:26:13.614 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:13.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:26:13.699 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:26:13.699 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:26:13.703 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:26:13.704 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:26:13.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.735+0000 7fe707ce0640 1 Processor -- start 2024-04-25T15:26:13.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.735+0000 7fe707ce0640 1 -- start start 2024-04-25T15:26:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe707ce0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7000d0e00 0x7fe7000d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe707ce0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe70006c5c0 0x7fe70006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe707ce0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe70006cf10 con 0x7fe7000b7810 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe707ce0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe7000d5a60 con 0x7fe70006c5c0 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe707ce0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe7000d1850 con 0x7fe7000d0e00 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe70627d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7000b7810 0x7fe7000b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49776/0 (socket says 172.21.15.67:49776) 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe70627d640 1 -- 172.21.15.67:0/3737858465 learned_addr learned my addr 172.21.15.67:0/3737858465 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe70527b640 1 --2- 172.21.15.67:0/3737858465 >> v2:172.21.15.176:3300/0 conn(0x7fe70006c5c0 0x7fe70006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe705a7c640 1 --2- 172.21.15.67:0/3737858465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7000d0e00 0x7fe7000d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe704a7a640 1 -- 172.21.15.67:0/3737858465 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735105357 0 0) 0x7fe6f8001490 con 0x7fe7000b7810 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe704a7a640 1 -- 172.21.15.67:0/3737858465 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6ec003520 con 0x7fe7000b7810 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe704a7a640 1 -- 172.21.15.67:0/3737858465 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4031048710 0 0) 0x7fe6ec003520 con 0x7fe7000b7810 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe705a7c640 1 -- 172.21.15.67:0/3737858465 >> v2:172.21.15.176:3300/0 conn(0x7fe70006c5c0 msgr2=0x7fe70006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe705a7c640 1 --2- 172.21.15.67:0/3737858465 >> v2:172.21.15.176:3300/0 conn(0x7fe70006c5c0 0x7fe70006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe705a7c640 1 -- 172.21.15.67:0/3737858465 >> v1:172.21.15.67:6789/0 conn(0x7fe7000b7810 legacy=0x7fe7000b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.737+0000 7fe705a7c640 1 -- 172.21.15.67:0/3737858465 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe700160de0 con 0x7fe7000d0e00 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.738+0000 7fe705a7c640 1 --2- 172.21.15.67:0/3737858465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7000d0e00 0x7fe7000d1210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0097a0 tx=0x7fe6fc02a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cde70f90c1c61d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.738+0000 7fe704a7a640 1 -- 172.21.15.67:0/3737858465 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc037070 con 0x7fe7000d0e00 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.738+0000 7fe704a7a640 1 -- 172.21.15.67:0/3737858465 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6fc02aba0 con 0x7fe7000d0e00 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.738+0000 7fe704a7a640 1 -- 172.21.15.67:0/3737858465 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8001bc0 con 0x7fe7000d0e00 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 -- 172.21.15.67:0/3737858465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7000d0e00 msgr2=0x7fe7000d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3737858465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7000d0e00 0x7fe7000d1210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0097a0 tx=0x7fe6fc02a110 comp rx=0 tx=0).stop 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 -- 172.21.15.67:0/3737858465 shutdown_connections 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3737858465 >> v2:172.21.15.176:3300/0 conn(0x7fe70006c5c0 0x7fe70006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3737858465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7000d0e00 0x7fe7000d1210 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 -- 172.21.15.67:0/3737858465 >> 172.21.15.67:0/3737858465 conn(0x7fe7000d5d90 msgr2=0x7fe7000d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 -- 172.21.15.67:0/3737858465 shutdown_connections 2024-04-25T15:26:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.739+0000 7fe707ce0640 1 -- 172.21.15.67:0/3737858465 wait complete. 2024-04-25T15:26:13.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.741+0000 7fe707ce0640 1 Processor -- start 2024-04-25T15:26:13.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.741+0000 7fe707ce0640 1 -- start start 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.742+0000 7fe707ce0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe70006c5c0 0x7fe7001fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.742+0000 7fe707ce0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7000b7810 0x7fe7000c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.742+0000 7fe707ce0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000d0e00 0x7fe70014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.742+0000 7fe707ce0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe700160ad0 con 0x7fe7000b7810 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.742+0000 7fe707ce0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001613a0 con 0x7fe7000d0e00 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.742+0000 7fe707ce0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe7000b5320 con 0x7fe70006c5c0 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.742+0000 7fe70527b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7000b7810 0x7fe7000c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7000b7810 0x7fe7000c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49554/0 (socket says 172.21.15.67:49554) 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 -- 172.21.15.67:0/3122653378 learned_addr learned my addr 172.21.15.67:0/3122653378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe705a7c640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe70006c5c0 0x7fe7001fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70627d640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000d0e00 0x7fe70014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 -- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe70006c5c0 msgr2=0x7fe7001fafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:13.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe70006c5c0 0x7fe7001fafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 -- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000d0e00 msgr2=0x7fe70014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000d0e00 0x7fe70014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 -- 172.21.15.67:0/3122653378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70014e6f0 con 0x7fe7000b7810 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70627d640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000d0e00 0x7fe70014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe705a7c640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe70006c5c0 0x7fe7001fafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe70527b640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7000b7810 0x7fe7000c9830 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000e8a0 tx=0x7fe6f000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe6f67fc640 1 -- 172.21.15.67:0/3122653378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0017070 con 0x7fe7000b7810 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe6f67fc640 1 -- 172.21.15.67:0/3122653378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f000c940 con 0x7fe7000b7810 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.743+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7000c9f60 con 0x7fe7000b7810 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.744+0000 7fe6f67fc640 1 -- 172.21.15.67:0/3122653378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000cd30 con 0x7fe7000b7810 2024-04-25T15:26:13.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.744+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7000ca520 con 0x7fe7000b7810 2024-04-25T15:26:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.745+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe700050740 con 0x7fe7000b7810 2024-04-25T15:26:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.745+0000 7fe6f67fc640 1 -- 172.21.15.67:0/3122653378 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe6f001a070 con 0x7fe7000b7810 2024-04-25T15:26:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.745+0000 7fe6f67fc640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6e40744b0 0x7fe6e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:13.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.746+0000 7fe6f67fc640 1 -- 172.21.15.67:0/3122653378 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 4943+0+0 (secure 0 0 0) 0x7fe6f0095250 con 0x7fe7000b7810 2024-04-25T15:26:13.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.746+0000 7fe705a7c640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6e40744b0 0x7fe6e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:13.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.746+0000 7fe705a7c640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6e40744b0 0x7fe6e40769c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc034000 tx=0x7fe6fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:13.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.750+0000 7fe6f67fc640 1 -- 172.21.15.67:0/3122653378 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe70014e6f0 con 0x7fe7000b7810 2024-04-25T15:26:13.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:13.959+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fe7000b6980 con 0x7fe7000b7810 2024-04-25T15:26:14.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.545+0000 7fe6f67fc640 1 -- 172.21.15.67:0/3122653378 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) ==== 97+0+0 (secure 0 0 0) 0x7fe6f005fce0 con 0x7fe7000b7810 2024-04-25T15:26:14.546 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-25T15:26:14.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6e40744b0 msgr2=0x7fe6e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6e40744b0 0x7fe6e40769c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc034000 tx=0x7fe6fc02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7000b7810 msgr2=0x7fe7000c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7000b7810 0x7fe7000c9830 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000e8a0 tx=0x7fe6f000ed60 comp rx=0 tx=0).stop 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 shutdown_connections 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6e40744b0 0x7fe6e40769c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000d0e00 0x7fe70014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7000b7810 0x7fe7000c9830 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 --2- 172.21.15.67:0/3122653378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe70006c5c0 0x7fe7001fafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.547+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 >> 172.21.15.67:0/3122653378 conn(0x7fe7000d5d90 msgr2=0x7fe7000b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.548+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 shutdown_connections 2024-04-25T15:26:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.548+0000 7fe707ce0640 1 -- 172.21.15.67:0/3122653378 wait complete. 2024-04-25T15:26:14.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:26:14.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.675+0000 7f0b3dc20640 1 Processor -- start 2024-04-25T15:26:14.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.675+0000 7f0b3dc20640 1 -- start start 2024-04-25T15:26:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b3dc20640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0b380d3f70 0x7f0b380b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b3dc20640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b380b5350 con 0x7f0b380b4f40 2024-04-25T15:26:14.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b3dc20640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b380b4180 con 0x7f0b380b2cc0 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b3dc20640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0b380b4ab0 con 0x7f0b380d3f70 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b36ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0b380b4f40 0x7f0b380d3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49786/0 (socket says 172.21.15.67:49786) 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b36ffd640 1 -- 172.21.15.67:0/2730326211 learned_addr learned my addr 172.21.15.67:0/2730326211 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b37fff640 1 --2- 172.21.15.67:0/2730326211 >> v2:172.21.15.176:3300/0 conn(0x7f0b380d3f70 0x7f0b380b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b367fc640 1 -- 172.21.15.67:0/2730326211 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4017669675 0 0) 0x7f0b380b5350 con 0x7f0b380b4f40 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b367fc640 1 -- 172.21.15.67:0/2730326211 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b20003520 con 0x7f0b380b4f40 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b367fc640 1 -- 172.21.15.67:0/2730326211 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2448112863 0 0) 0x7f0b20003520 con 0x7f0b380b4f40 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.677+0000 7f0b37fff640 1 -- 172.21.15.67:0/2730326211 >> v1:172.21.15.176:6789/0 conn(0x7f0b380b2cc0 legacy=0x7f0b380d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.678+0000 7f0b37fff640 1 -- 172.21.15.67:0/2730326211 >> v1:172.21.15.67:6789/0 conn(0x7f0b380b4f40 legacy=0x7f0b380d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.678+0000 7f0b37fff640 1 -- 172.21.15.67:0/2730326211 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b380b4460 con 0x7f0b380d3f70 2024-04-25T15:26:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.678+0000 7f0b37fff640 1 --2- 172.21.15.67:0/2730326211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380d3f70 0x7f0b380b9160 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009d90 tx=0x7f0b3002a510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e874a39ae189abd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.678+0000 7f0b367fc640 1 -- 172.21.15.67:0/2730326211 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30035020 con 0x7f0b380d3f70 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.678+0000 7f0b367fc640 1 -- 172.21.15.67:0/2730326211 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b30007540 con 0x7f0b380d3f70 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b367fc640 1 -- 172.21.15.67:0/2730326211 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c001670 con 0x7f0b380d3f70 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2730326211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380d3f70 msgr2=0x7f0b380b9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2730326211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380d3f70 0x7f0b380b9160 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009d90 tx=0x7f0b3002a510 comp rx=0 tx=0).stop 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2730326211 shutdown_connections 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2730326211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380d3f70 0x7f0b380b9160 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2730326211 >> 172.21.15.67:0/2730326211 conn(0x7f0b3808d110 msgr2=0x7f0b3808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2730326211 shutdown_connections 2024-04-25T15:26:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.679+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2730326211 wait complete. 2024-04-25T15:26:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.681+0000 7f0b3dc20640 1 Processor -- start 2024-04-25T15:26:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.681+0000 7f0b3dc20640 1 -- start start 2024-04-25T15:26:14.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.682+0000 7f0b3dc20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380b2cc0 0x7f0b380cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.682+0000 7f0b3dc20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380b4f40 0x7f0b38161d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.682+0000 7f0b3dc20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b381622b0 0x7f0b380cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.682+0000 7f0b3dc20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b380b0740 con 0x7f0b380b4f40 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.682+0000 7f0b3dc20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0b380b05c0 con 0x7f0b380b2cc0 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.682+0000 7f0b3dc20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0b380b08c0 con 0x7f0b381622b0 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380b4f40 0x7f0b38161d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380b4f40 0x7f0b38161d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49570/0 (socket says 172.21.15.67:49570) 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 -- 172.21.15.67:0/2016776566 learned_addr learned my addr 172.21.15.67:0/2016776566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b37fff640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b381622b0 0x7f0b380cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b377fe640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380b2cc0 0x7f0b380cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 -- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b381622b0 msgr2=0x7f0b380cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b381622b0 0x7f0b380cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 -- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380b2cc0 msgr2=0x7f0b380cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380b2cc0 0x7f0b380cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 -- 172.21.15.67:0/2016776566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b380c7880 con 0x7f0b380b4f40 2024-04-25T15:26:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b36ffd640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380b4f40 0x7f0b38161d70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c550 tx=0x7f0b2c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b1ffff640 1 -- 172.21.15.67:0/2016776566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c018070 con 0x7f0b380b4f40 2024-04-25T15:26:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b1ffff640 1 -- 172.21.15.67:0/2016776566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2c0074e0 con 0x7f0b380b4f40 2024-04-25T15:26:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b1ffff640 1 -- 172.21.15.67:0/2016776566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c004a20 con 0x7f0b380b4f40 2024-04-25T15:26:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b380c7b60 con 0x7f0b380b4f40 2024-04-25T15:26:14.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.683+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b380c8120 con 0x7f0b380b4f40 2024-04-25T15:26:14.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.684+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b380c75f0 con 0x7f0b380b4f40 2024-04-25T15:26:14.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.685+0000 7f0b1ffff640 1 -- 172.21.15.67:0/2016776566 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0b2c004bc0 con 0x7f0b380b4f40 2024-04-25T15:26:14.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.686+0000 7f0b1ffff640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0b240744b0 0x7f0b240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:14.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.686+0000 7f0b1ffff640 1 -- 172.21.15.67:0/2016776566 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f0b2c094250 con 0x7f0b380b4f40 2024-04-25T15:26:14.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.686+0000 7f0b377fe640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0b240744b0 0x7f0b240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:14.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.687+0000 7f0b377fe640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0b240744b0 0x7f0b240769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0b28000f60 tx=0x7f0b28009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:14.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.691+0000 7f0b1ffff640 1 -- 172.21.15.67:0/2016776566 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b2c05e660 con 0x7f0b380b4f40 2024-04-25T15:26:14.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.945+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f0b38050740 con 0x7f0b380b4f40 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.946+0000 7f0b1ffff640 1 -- 172.21.15.67:0/2016776566 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7f0b2c062960 con 0x7f0b380b4f40 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg==", 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.947 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA==", 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q==", 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-25T15:26:14.949 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw==", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg==", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.950 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw==", 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA==", 2024-04-25T15:26:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g==", 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA==", 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA==", 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ==", 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.954 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA==", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q==", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:26:14.955 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ==", 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.956 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg==", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA==", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:26:14.957 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmVIPuGRAAswFIakCxyA7OvvoJIBYUiQ==", 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mds", 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDQdSpm9N8bIxAAMmcs1vQwcTN2n1duisz04Q==", 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mds" 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mgr", 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDQdSpmU+gbIxAAqJrrsys4kKpVuR6VwfvcBw==", 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-osd", 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDQdSpm6PAbIxAANkjcJaTRgIVvPVDRrF13Bw==", 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-osd" 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd", 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDQdSpmavgbIxAACFEmWzl2SaWAN6ircywfvA==", 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.960 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDQdSpmKAIcIxAAC8MlUKkorJGeFjN7etH7OA==", 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rgw", 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDQdSpmIwocIxAA+WVo2ynPPbARr7Hsb5xfyQ==", 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.961 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg==", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag==", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:26:14.962 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:14.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:14.963 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:26:14.963 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:26:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0b240744b0 msgr2=0x7f0b240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0b240744b0 0x7f0b240769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0b28000f60 tx=0x7f0b28009040 comp rx=0 tx=0).stop 2024-04-25T15:26:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380b4f40 msgr2=0x7f0b38161d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380b4f40 0x7f0b38161d70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c550 tx=0x7f0b2c00ca10 comp rx=0 tx=0).stop 2024-04-25T15:26:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 shutdown_connections 2024-04-25T15:26:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0b240744b0 0x7f0b240769c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b381622b0 0x7f0b380cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b380b4f40 0x7f0b38161d70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 --2- 172.21.15.67:0/2016776566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b380b2cc0 0x7f0b380cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:14.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 >> 172.21.15.67:0/2016776566 conn(0x7f0b3808d110 msgr2=0x7f0b380d4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:14.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 shutdown_connections 2024-04-25T15:26:14.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:14.948+0000 7f0b3dc20640 1 -- 172.21.15.67:0/2016776566 wait complete. 2024-04-25T15:26:14.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-25T15:26:15.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.078+0000 7f774ae51640 1 Processor -- start 2024-04-25T15:26:15.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.078+0000 7f774ae51640 1 -- start start 2024-04-25T15:26:15.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f774ae51640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f774406d8e0 0x7f77440d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f774ae51640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77440d2d70 0x7f77440d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f774ae51640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77440b3c10 con 0x7f774406dcf0 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f774ae51640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f77440b9520 con 0x7f774406d8e0 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f774ae51640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77440b4820 con 0x7f77440d2d70 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f77493ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77440d2d70 0x7f77440d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f7748bed640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f774406d8e0 0x7f77440d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f7748bed640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f774406d8e0 0x7f77440d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:44828/0 (socket says 172.21.15.67:44828) 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f7748bed640 1 -- 172.21.15.67:0/2884095448 learned_addr learned my addr 172.21.15.67:0/2884095448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.079+0000 7f77493ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77440d2d70 0x7f77440d3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:35236/0 (socket says 172.21.15.67:35236) 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.080+0000 7f7748bed640 1 -- 172.21.15.67:0/2884095448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77440d2d70 msgr2=0x7f77440d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.080+0000 7f7748bed640 1 --2- 172.21.15.67:0/2884095448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77440d2d70 0x7f77440d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.080+0000 7f7748bed640 1 -- 172.21.15.67:0/2884095448 >> v1:172.21.15.176:6789/0 conn(0x7f774406dcf0 legacy=0x7f77440d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.080+0000 7f7748bed640 1 -- 172.21.15.67:0/2884095448 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77440b3ef0 con 0x7f774406d8e0 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.080+0000 7f7748bed640 1 --2- 172.21.15.67:0/2884095448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406d8e0 0x7f77440d3a20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7738009870 tx=0x7f773802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57be7c2f569d94d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f77437fe640 1 -- 172.21.15.67:0/2884095448 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738037070 con 0x7f774406d8e0 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f77437fe640 1 -- 172.21.15.67:0/2884095448 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7738032440 con 0x7f774406d8e0 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f77437fe640 1 -- 172.21.15.67:0/2884095448 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738032810 con 0x7f774406d8e0 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f774ae51640 1 -- 172.21.15.67:0/2884095448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406d8e0 msgr2=0x7f77440d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f774ae51640 1 --2- 172.21.15.67:0/2884095448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406d8e0 0x7f77440d3a20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7738009870 tx=0x7f773802a1c0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f774ae51640 1 -- 172.21.15.67:0/2884095448 shutdown_connections 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f774ae51640 1 --2- 172.21.15.67:0/2884095448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77440d2d70 0x7f77440d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f774ae51640 1 --2- 172.21.15.67:0/2884095448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406d8e0 0x7f77440d3a20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.081+0000 7f774ae51640 1 -- 172.21.15.67:0/2884095448 >> 172.21.15.67:0/2884095448 conn(0x7f774408d110 msgr2=0x7f774408d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:15.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.082+0000 7f774ae51640 1 -- 172.21.15.67:0/2884095448 shutdown_connections 2024-04-25T15:26:15.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.082+0000 7f774ae51640 1 -- 172.21.15.67:0/2884095448 wait complete. 2024-04-25T15:26:15.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.083+0000 7f774ae51640 1 Processor -- start 2024-04-25T15:26:15.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.083+0000 7f774ae51640 1 -- start start 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f774ae51640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f774406d8e0 0x7f774409d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f774ae51640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406dcf0 0x7f774414c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f774ae51640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440d2d70 0x7f77440b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f774ae51640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77440b38e0 con 0x7f77440d2d70 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f774ae51640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77440b44b0 con 0x7f774406dcf0 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f774ae51640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77440d0bb0 con 0x7f774406d8e0 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440d2d70 0x7f77440b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440d2d70 0x7f77440b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49578/0 (socket says 172.21.15.67:49578) 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 -- 172.21.15.67:0/2062556550 learned_addr learned my addr 172.21.15.67:0/2062556550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f7748bed640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f774406d8e0 0x7f774409d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f7743fff640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406dcf0 0x7f774414c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 -- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f774406d8e0 msgr2=0x7f774409d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f774406d8e0 0x7f774409d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 -- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406dcf0 msgr2=0x7f774414c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406dcf0 0x7f774414c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f77493ee640 1 -- 172.21.15.67:0/2062556550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774414eea0 con 0x7f77440d2d70 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.085+0000 7f7748bed640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f774406d8e0 0x7f774409d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.086+0000 7f77493ee640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440d2d70 0x7f77440b79a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f773c009d30 tx=0x7f773c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.086+0000 7f77417fa640 1 -- 172.21.15.67:0/2062556550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c018020 con 0x7f77440d2d70 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.086+0000 7f77417fa640 1 -- 172.21.15.67:0/2062556550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f773c014480 con 0x7f77440d2d70 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.086+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774414d080 con 0x7f77440d2d70 2024-04-25T15:26:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.086+0000 7f77417fa640 1 -- 172.21.15.67:0/2062556550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773c014870 con 0x7f77440d2d70 2024-04-25T15:26:15.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.086+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774414d340 con 0x7f77440d2d70 2024-04-25T15:26:15.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.087+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7744050740 con 0x7f77440d2d70 2024-04-25T15:26:15.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.088+0000 7f77417fa640 1 -- 172.21.15.67:0/2062556550 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f773c005d10 con 0x7f77440d2d70 2024-04-25T15:26:15.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.088+0000 7f77417fa640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77280744e0 0x7f77280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.088+0000 7f77417fa640 1 -- 172.21.15.67:0/2062556550 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f773c095ac0 con 0x7f77440d2d70 2024-04-25T15:26:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.089+0000 7f7748bed640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77280744e0 0x7f77280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.089+0000 7f7748bed640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77280744e0 0x7f77280769f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7738034000 tx=0x7f773802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.093+0000 7f77417fa640 1 -- 172.21.15.67:0/2062556550 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f773c022030 con 0x7f77440d2d70 2024-04-25T15:26:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.349+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f7744076240 con 0x7f77440d2d70 2024-04-25T15:26:15.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.351+0000 7f77417fa640 1 -- 172.21.15.67:0/2062556550 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f773c010040 con 0x7f77440d2d70 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77280744e0 msgr2=0x7f77280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77280744e0 0x7f77280769f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7738034000 tx=0x7f773802e000 comp rx=0 tx=0).stop 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440d2d70 msgr2=0x7f77440b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440d2d70 0x7f77440b79a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f773c009d30 tx=0x7f773c00c490 comp rx=0 tx=0).stop 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 shutdown_connections 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77280744e0 0x7f77280769f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440d2d70 0x7f77440b79a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f774406dcf0 0x7f774414c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 --2- 172.21.15.67:0/2062556550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f774406d8e0 0x7f774409d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 >> 172.21.15.67:0/2062556550 conn(0x7f774408d110 msgr2=0x7f7744072250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:15.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 shutdown_connections 2024-04-25T15:26:15.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.353+0000 7f774ae51640 1 -- 172.21.15.67:0/2062556550 wait complete. 2024-04-25T15:26:15.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-25T15:26:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.477+0000 7f7b251ee640 1 Processor -- start 2024-04-25T15:26:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.477+0000 7f7b251ee640 1 -- start start 2024-04-25T15:26:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.478+0000 7f7b251ee640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7b2006d5d0 0x7f7b2016f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.478+0000 7f7b251ee640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b200b4af0 con 0x7f7b2014ad60 2024-04-25T15:26:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.478+0000 7f7b251ee640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b200b4dd0 con 0x7f7b200b5e40 2024-04-25T15:26:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.478+0000 7f7b251ee640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7b200b50b0 con 0x7f7b2006d5d0 2024-04-25T15:26:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b2014ad60 0x7f7b20070010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49800/0 (socket says 172.21.15.67:49800) 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/2074066034 learned_addr learned my addr 172.21.15.67:0/2074066034 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202359066 0 0) 0x7f7b200b4af0 con 0x7f7b2014ad60 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1f59e640 1 --2- 172.21.15.67:0/2074066034 >> v2:172.21.15.176:3300/0 conn(0x7f7b2006d5d0 0x7f7b2016f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b0c003520 con 0x7f7b2014ad60 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1468461109 0 0) 0x7f7b0c003520 con 0x7f7b2014ad60 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 >> v2:172.21.15.176:3300/0 conn(0x7f7b2006d5d0 msgr2=0x7f7b2016f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 --2- 172.21.15.67:0/2074066034 >> v2:172.21.15.176:3300/0 conn(0x7f7b2006d5d0 0x7f7b2016f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 >> v1:172.21.15.176:6789/0 conn(0x7f7b200b5e40 legacy=0x7f7b200b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b200b5390 con 0x7f7b2014ad60 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f7b18001690 con 0x7f7b2014ad60 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 >> v1:172.21.15.67:6789/0 conn(0x7f7b2014ad60 legacy=0x7f7b20070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b0c004800 0x7f7b0c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c005250 0x7f7b0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7b200b50b0 con 0x7f7b0c0039b0 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c00c1e0 con 0x7f7b0c005250 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0c00c3e0 con 0x7f7b0c004800 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.479+0000 7f7b1ed9d640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1f59e640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c005250 0x7f7b0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1e59c640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b0c004800 0x7f7b0c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b0c004800 msgr2=0x7f7b0c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1ed9d640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b0c004800 0x7f7b0c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c005250 msgr2=0x7f7b0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1ed9d640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c005250 0x7f7b0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/2074066034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0c003520 con 0x7f7b0c0039b0 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1f59e640 1 -- 172.21.15.67:0/2074066034 reap_dead start 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.480+0000 7f7b1ed9d640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b0c003e20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7b20164c20 tx=0x7f7b0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3669b0f3d176fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18007d10 con 0x7f7b0c0039b0 2024-04-25T15:26:15.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b18005af0 con 0x7f7b0c0039b0 2024-04-25T15:26:15.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b1dd9b640 1 -- 172.21.15.67:0/2074066034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18007070 con 0x7f7b0c0039b0 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b251ee640 1 -- 172.21.15.67:0/2074066034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 msgr2=0x7f7b0c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b251ee640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b0c003e20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7b20164c20 tx=0x7f7b0c005830 comp rx=0 tx=0).stop 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b251ee640 1 -- 172.21.15.67:0/2074066034 shutdown_connections 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b251ee640 1 --2- 172.21.15.67:0/2074066034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b0c003e20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b251ee640 1 -- 172.21.15.67:0/2074066034 >> 172.21.15.67:0/2074066034 conn(0x7f7b200d1ae0 msgr2=0x7f7b200d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b251ee640 1 -- 172.21.15.67:0/2074066034 shutdown_connections 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.481+0000 7f7b251ee640 1 -- 172.21.15.67:0/2074066034 wait complete. 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.483+0000 7f7b251ee640 1 Processor -- start 2024-04-25T15:26:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.483+0000 7f7b251ee640 1 -- start start 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b251ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b201460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b251ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b200d0230 0x7f7b200d0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b251ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b20159e40 0x7f7b2015a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b251ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2006fd00 con 0x7f7b0c0039b0 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b251ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2006fb80 con 0x7f7b20159e40 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b251ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7b200b6250 con 0x7f7b200d0230 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b201460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b201460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49602/0 (socket says 172.21.15.67:49602) 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/3307693712 learned_addr learned my addr 172.21.15.67:0/3307693712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1e59c640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b200d0230 0x7f7b200d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1f59e640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b20159e40 0x7f7b2015a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b200d0230 msgr2=0x7f7b200d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b200d0230 0x7f7b200d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b20159e40 msgr2=0x7f7b2015a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b20159e40 0x7f7b2015a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 -- 172.21.15.67:0/3307693712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b200d0b80 con 0x7f7b0c0039b0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.485+0000 7f7b1ed9d640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b201460a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7b18001990 tx=0x7f7b200b5560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.486+0000 7f7b177fe640 1 -- 172.21.15.67:0/3307693712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18037040 con 0x7f7b0c0039b0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.486+0000 7f7b177fe640 1 -- 172.21.15.67:0/3307693712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b18041070 con 0x7f7b0c0039b0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.486+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b201604d0 con 0x7f7b0c0039b0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.486+0000 7f7b177fe640 1 -- 172.21.15.67:0/3307693712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18001690 con 0x7f7b0c0039b0 2024-04-25T15:26:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.486+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b20160a90 con 0x7f7b0c0039b0 2024-04-25T15:26:15.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.487+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b20050740 con 0x7f7b0c0039b0 2024-04-25T15:26:15.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.487+0000 7f7b177fe640 1 -- 172.21.15.67:0/3307693712 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7b180049e0 con 0x7f7b0c0039b0 2024-04-25T15:26:15.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.488+0000 7f7b177fe640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7b000744e0 0x7f7b000769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.488+0000 7f7b177fe640 1 -- 172.21.15.67:0/3307693712 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f7b180b4cd0 con 0x7f7b0c0039b0 2024-04-25T15:26:15.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.488+0000 7f7b1e59c640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7b000744e0 0x7f7b000769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.489+0000 7f7b1e59c640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7b000744e0 0x7f7b000769f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7b200b6a80 tx=0x7f7b08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.493+0000 7f7b177fe640 1 -- 172.21.15.67:0/3307693712 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7b200d0b80 con 0x7f7b0c0039b0 2024-04-25T15:26:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.761+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f7b200ce0d0 con 0x7f7b0c0039b0 2024-04-25T15:26:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.763+0000 7f7b177fe640 1 -- 172.21.15.67:0/3307693712 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f7b1807f160 con 0x7f7b0c0039b0 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7b000744e0 msgr2=0x7f7b000769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7b000744e0 0x7f7b000769f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7b200b6a80 tx=0x7f7b08009040 comp rx=0 tx=0).stop 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 msgr2=0x7f7b201460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b201460a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7b18001990 tx=0x7f7b200b5560 comp rx=0 tx=0).stop 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 shutdown_connections 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7b000744e0 0x7f7b000769f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b20159e40 0x7f7b2015a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b200d0230 0x7f7b200d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 --2- 172.21.15.67:0/3307693712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0039b0 0x7f7b201460a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.765+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 >> 172.21.15.67:0/3307693712 conn(0x7f7b200d1ae0 msgr2=0x7f7b200b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.766+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 shutdown_connections 2024-04-25T15:26:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.766+0000 7f7b251ee640 1 -- 172.21.15.67:0/3307693712 wait complete. 2024-04-25T15:26:15.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-25T15:26:15.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.889+0000 7f036db7d640 1 Processor -- start 2024-04-25T15:26:15.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.889+0000 7f036db7d640 1 -- start start 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f036db7d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0368164b00 0x7f03680b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f036db7d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03680b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f036db7d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03681615f0 con 0x7f036815df80 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f036db7d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f03681613b0 con 0x7f0368164b00 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f036db7d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0368164670 con 0x7f0368160fa0 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f0367fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03680b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f03677fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0368164b00 0x7f03680b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f0367fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03680b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:35272/0 (socket says 172.21.15.67:35272) 2024-04-25T15:26:15.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.891+0000 7f0367fff640 1 -- 172.21.15.67:0/4193328485 learned_addr learned my addr 172.21.15.67:0/4193328485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.892+0000 7f0367fff640 1 -- 172.21.15.67:0/4193328485 >> v2:172.21.15.176:3300/0 conn(0x7f0368164b00 msgr2=0x7f03680b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.892+0000 7f0367fff640 1 --2- 172.21.15.67:0/4193328485 >> v2:172.21.15.176:3300/0 conn(0x7f0368164b00 0x7f03680b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.892+0000 7f0367fff640 1 -- 172.21.15.67:0/4193328485 >> v1:172.21.15.176:6789/0 conn(0x7f036815df80 legacy=0x7f0368160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.892+0000 7f0367fff640 1 -- 172.21.15.67:0/4193328485 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0368164020 con 0x7f0368160fa0 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.892+0000 7f03677fe640 1 --2- 172.21.15.67:0/4193328485 >> v2:172.21.15.176:3300/0 conn(0x7f0368164b00 0x7f03680b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.892+0000 7f0367fff640 1 --2- 172.21.15.67:0/4193328485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03680b80c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0360009e50 tx=0x7f036002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903e9305b754af97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f03667fc640 1 -- 172.21.15.67:0/4193328485 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360036070 con 0x7f0368160fa0 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f03667fc640 1 -- 172.21.15.67:0/4193328485 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0360031600 con 0x7f0368160fa0 2024-04-25T15:26:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f03667fc640 1 -- 172.21.15.67:0/4193328485 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03600319d0 con 0x7f0368160fa0 2024-04-25T15:26:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 -- 172.21.15.67:0/4193328485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 msgr2=0x7f03680b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 --2- 172.21.15.67:0/4193328485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03680b80c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0360009e50 tx=0x7f036002a710 comp rx=0 tx=0).stop 2024-04-25T15:26:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 -- 172.21.15.67:0/4193328485 shutdown_connections 2024-04-25T15:26:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 --2- 172.21.15.67:0/4193328485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03680b80c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 --2- 172.21.15.67:0/4193328485 >> v2:172.21.15.176:3300/0 conn(0x7f0368164b00 0x7f03680b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 -- 172.21.15.67:0/4193328485 >> 172.21.15.67:0/4193328485 conn(0x7f03680d2bc0 msgr2=0x7f03680d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 -- 172.21.15.67:0/4193328485 shutdown_connections 2024-04-25T15:26:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.893+0000 7f036db7d640 1 -- 172.21.15.67:0/4193328485 wait complete. 2024-04-25T15:26:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.895+0000 7f036db7d640 1 Processor -- start 2024-04-25T15:26:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.895+0000 7f036db7d640 1 -- start start 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f036db7d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036815df80 0x7f03681ea630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f036db7d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03681f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f036db7d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368164b00 0x7f03680cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f036db7d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03680b4a00 con 0x7f036815df80 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f036db7d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f03680b4880 con 0x7f0368164b00 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f036db7d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f03680b4b80 con 0x7f0368160fa0 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036815df80 0x7f03681ea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036815df80 0x7f03681ea630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49610/0 (socket says 172.21.15.67:49610) 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 -- 172.21.15.67:0/1951601426 learned_addr learned my addr 172.21.15.67:0/1951601426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f0366ffd640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03681f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f0367fff640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368164b00 0x7f03680cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 -- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 msgr2=0x7f03681f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03681f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 -- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368164b00 msgr2=0x7f03680cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368164b00 0x7f03680cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 -- 172.21.15.67:0/1951601426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036814e980 con 0x7f036815df80 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f0366ffd640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03681f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.897+0000 7f03677fe640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036815df80 0x7f03681ea630 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f035800e960 tx=0x7f035800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.898+0000 7f034ffff640 1 -- 172.21.15.67:0/1951601426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0358017070 con 0x7f036815df80 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.898+0000 7f034ffff640 1 -- 172.21.15.67:0/1951601426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f035800ca00 con 0x7f036815df80 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.898+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03681eb1a0 con 0x7f036815df80 2024-04-25T15:26:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.898+0000 7f034ffff640 1 -- 172.21.15.67:0/1951601426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035800cdd0 con 0x7f036815df80 2024-04-25T15:26:15.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.898+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03680c9830 con 0x7f036815df80 2024-04-25T15:26:15.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.899+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0368050740 con 0x7f036815df80 2024-04-25T15:26:15.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.899+0000 7f034ffff640 1 -- 172.21.15.67:0/1951601426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f035801a070 con 0x7f036815df80 2024-04-25T15:26:15.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.900+0000 7f034ffff640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03540744b0 0x7f03540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:15.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.900+0000 7f034ffff640 1 -- 172.21.15.67:0/1951601426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f0358094e30 con 0x7f036815df80 2024-04-25T15:26:15.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.900+0000 7f0366ffd640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03540744b0 0x7f03540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:15.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.901+0000 7f0366ffd640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03540744b0 0x7f03540769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f03681eb920 tx=0x7f035c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:15.904+0000 7f034ffff640 1 -- 172.21.15.67:0/1951601426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f035805f910 con 0x7f036815df80 2024-04-25T15:26:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.160+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f03680d4170 con 0x7f036815df80 2024-04-25T15:26:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.163+0000 7f034ffff640 1 -- 172.21.15.67:0/1951601426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f035805f210 con 0x7f036815df80 2024-04-25T15:26:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03540744b0 msgr2=0x7f03540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03540744b0 0x7f03540769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f03681eb920 tx=0x7f035c009040 comp rx=0 tx=0).stop 2024-04-25T15:26:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036815df80 msgr2=0x7f03681ea630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036815df80 0x7f03681ea630 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f035800e960 tx=0x7f035800ee20 comp rx=0 tx=0).stop 2024-04-25T15:26:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 shutdown_connections 2024-04-25T15:26:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03540744b0 0x7f03540769c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368164b00 0x7f03680cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0368160fa0 0x7f03681f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 --2- 172.21.15.67:0/1951601426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036815df80 0x7f03681ea630 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 >> 172.21.15.67:0/1951601426 conn(0x7f03680d2bc0 msgr2=0x7f03680d5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 shutdown_connections 2024-04-25T15:26:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.165+0000 7f036db7d640 1 -- 172.21.15.67:0/1951601426 wait complete. 2024-04-25T15:26:16.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-25T15:26:16.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.290+0000 7f8c330e0640 1 Processor -- start 2024-04-25T15:26:16.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.290+0000 7f8c330e0640 1 -- start start 2024-04-25T15:26:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c330e0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8c2c0b8290 0x7f8c2c160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c330e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c15df80 0x7f8c2c160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c330e0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c2c1615f0 con 0x7f8c2c164b00 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c330e0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8c2c164670 con 0x7f8c2c0b8290 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c330e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c2c164020 con 0x7f8c2c15df80 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c30e7c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c2c164b00 0x7f8c2c0b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49814/0 (socket says 172.21.15.67:49814) 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c30e7c640 1 -- 172.21.15.67:0/3951568518 learned_addr learned my addr 172.21.15.67:0/3951568518 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c2bfff640 1 --2- 172.21.15.67:0/3951568518 >> v2:172.21.15.176:3300/0 conn(0x7f8c2c0b8290 0x7f8c2c160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c3167d640 1 --2- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c15df80 0x7f8c2c160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c2b7fe640 1 -- 172.21.15.67:0/3951568518 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1563980297 0 0) 0x7f8c2c1615f0 con 0x7f8c2c164b00 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c2b7fe640 1 -- 172.21.15.67:0/3951568518 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c1c003520 con 0x7f8c2c164b00 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c2bfff640 1 -- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c15df80 msgr2=0x7f8c2c160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.292+0000 7f8c2bfff640 1 --2- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c15df80 0x7f8c2c160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.293+0000 7f8c2bfff640 1 -- 172.21.15.67:0/3951568518 >> v1:172.21.15.67:6789/0 conn(0x7f8c2c164b00 legacy=0x7f8c2c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.293+0000 7f8c2bfff640 1 -- 172.21.15.67:0/3951568518 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c164300 con 0x7f8c2c0b8290 2024-04-25T15:26:16.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.293+0000 7f8c3167d640 1 --2- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c15df80 0x7f8c2c160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.293+0000 7f8c2bfff640 1 --2- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c0b8290 0x7f8c2c160890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8c180097a0 tx=0x7f8c1802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac30adc62f014b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.293+0000 7f8c2b7fe640 1 -- 172.21.15.67:0/3951568518 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c18037070 con 0x7f8c2c0b8290 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c2b7fe640 1 -- 172.21.15.67:0/3951568518 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c18032440 con 0x7f8c2c0b8290 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c2b7fe640 1 -- 172.21.15.67:0/3951568518 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c18032810 con 0x7f8c2c0b8290 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 -- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c0b8290 msgr2=0x7f8c2c160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 --2- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c0b8290 0x7f8c2c160890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8c180097a0 tx=0x7f8c1802a180 comp rx=0 tx=0).stop 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 -- 172.21.15.67:0/3951568518 shutdown_connections 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 --2- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c15df80 0x7f8c2c160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 --2- 172.21.15.67:0/3951568518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c0b8290 0x7f8c2c160890 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 -- 172.21.15.67:0/3951568518 >> 172.21.15.67:0/3951568518 conn(0x7f8c2c0d2bc0 msgr2=0x7f8c2c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 -- 172.21.15.67:0/3951568518 shutdown_connections 2024-04-25T15:26:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.294+0000 7f8c330e0640 1 -- 172.21.15.67:0/3951568518 wait complete. 2024-04-25T15:26:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.296+0000 7f8c330e0640 1 Processor -- start 2024-04-25T15:26:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.296+0000 7f8c330e0640 1 -- start start 2024-04-25T15:26:16.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.297+0000 7f8c330e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c15df80 0x7f8c2c0b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.297+0000 7f8c330e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c164b00 0x7f8c2c157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.297+0000 7f8c330e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c148480 0x7f8c2c1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.297+0000 7f8c330e0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c0b4a00 con 0x7f8c2c148480 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.297+0000 7f8c330e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c0b4880 con 0x7f8c2c15df80 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.297+0000 7f8c330e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c2c0b4b80 con 0x7f8c2c164b00 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c148480 0x7f8c2c1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c148480 0x7f8c2c1535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49624/0 (socket says 172.21.15.67:49624) 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 -- 172.21.15.67:0/1948801495 learned_addr learned my addr 172.21.15.67:0/1948801495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c30e7c640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c15df80 0x7f8c2c0b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c2bfff640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c164b00 0x7f8c2c157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 -- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c164b00 msgr2=0x7f8c2c157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c164b00 0x7f8c2c157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 -- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c15df80 msgr2=0x7f8c2c0b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c15df80 0x7f8c2c0b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 -- 172.21.15.67:0/1948801495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c1565b0 con 0x7f8c2c148480 2024-04-25T15:26:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c3167d640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c148480 0x7f8c2c1535b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400efc0 tx=0x7f8c2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c297fa640 1 -- 172.21.15.67:0/1948801495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24018020 con 0x7f8c2c148480 2024-04-25T15:26:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c297fa640 1 -- 172.21.15.67:0/1948801495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c24014480 con 0x7f8c2c148480 2024-04-25T15:26:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2c156890 con 0x7f8c2c148480 2024-04-25T15:26:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c297fa640 1 -- 172.21.15.67:0/1948801495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24014870 con 0x7f8c2c148480 2024-04-25T15:26:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.298+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c2c156e50 con 0x7f8c2c148480 2024-04-25T15:26:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.300+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2c050740 con 0x7f8c2c148480 2024-04-25T15:26:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.300+0000 7f8c297fa640 1 -- 172.21.15.67:0/1948801495 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8c24005d10 con 0x7f8c2c148480 2024-04-25T15:26:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.301+0000 7f8c297fa640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8c100744b0 0x7f8c100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.301+0000 7f8c297fa640 1 -- 172.21.15.67:0/1948801495 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f8c24095ab0 con 0x7f8c2c148480 2024-04-25T15:26:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.301+0000 7f8c30e7c640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8c100744b0 0x7f8c100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.302+0000 7f8c30e7c640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8c100744b0 0x7f8c100769c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8c20005880 tx=0x7f8c20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:16.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.306+0000 7f8c297fa640 1 -- 172.21.15.67:0/1948801495 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c24022030 con 0x7f8c2c148480 2024-04-25T15:26:16.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.563+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f8c2c0d4170 con 0x7f8c2c148480 2024-04-25T15:26:16.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.566+0000 7f8c297fa640 1 -- 172.21.15.67:0/1948801495 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f8c24010040 con 0x7f8c2c148480 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8c100744b0 msgr2=0x7f8c100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8c100744b0 0x7f8c100769c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8c20005880 tx=0x7f8c20009040 comp rx=0 tx=0).stop 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c148480 msgr2=0x7f8c2c1535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c148480 0x7f8c2c1535b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400efc0 tx=0x7f8c2400c490 comp rx=0 tx=0).stop 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 shutdown_connections 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8c100744b0 0x7f8c100769c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c2c148480 0x7f8c2c1535b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c2c164b00 0x7f8c2c157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 --2- 172.21.15.67:0/1948801495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c2c15df80 0x7f8c2c0b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 >> 172.21.15.67:0/1948801495 conn(0x7f8c2c0d2bc0 msgr2=0x7f8c2c15eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 shutdown_connections 2024-04-25T15:26:16.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.568+0000 7f8c330e0640 1 -- 172.21.15.67:0/1948801495 wait complete. 2024-04-25T15:26:16.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-25T15:26:16.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.695+0000 7f06c47ce640 1 Processor -- start 2024-04-25T15:26:16.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.695+0000 7f06c47ce640 1 -- start start 2024-04-25T15:26:16.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c47ce640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06bc0d4570 0x7f06bc0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c47ce640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f06bc06e8f0 0x7f06bc0d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c47ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc0b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c47ce640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f06bc06e460 con 0x7f06bc0d4570 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c47ce640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f06bc06d870 con 0x7f06bc06e8f0 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c47ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f06bc06db10 con 0x7f06bc0d09f0 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06bc0d4570 0x7f06bc0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06bc0d4570 0x7f06bc0b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49626/0 (socket says 172.21.15.67:49626) 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 -- 172.21.15.67:0/3080680780 learned_addr learned my addr 172.21.15.67:0/3080680780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c2d6b640 1 --2- 172.21.15.67:0/3080680780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc0b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c1d69640 1 --2- 172.21.15.67:0/3080680780 >> v2:172.21.15.176:3300/0 conn(0x7f06bc06e8f0 0x7f06bc0d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 -- 172.21.15.67:0/3080680780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 msgr2=0x7f06bc0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 --2- 172.21.15.67:0/3080680780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc0b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 -- 172.21.15.67:0/3080680780 >> v2:172.21.15.176:3300/0 conn(0x7f06bc06e8f0 msgr2=0x7f06bc0d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 --2- 172.21.15.67:0/3080680780 >> v2:172.21.15.176:3300/0 conn(0x7f06bc06e8f0 0x7f06bc0d04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.697+0000 7f06c256a640 1 -- 172.21.15.67:0/3080680780 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc06e0f0 con 0x7f06bc0d4570 2024-04-25T15:26:16.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.698+0000 7f06c256a640 1 --2- 172.21.15.67:0/3080680780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc0d4570 0x7f06bc0b4830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f06bc070c30 tx=0x7f06b402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4053c62c76a1154c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:16.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.698+0000 7f06c1568640 1 -- 172.21.15.67:0/3080680780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4036070 con 0x7f06bc0d4570 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.698+0000 7f06c1568640 1 -- 172.21.15.67:0/3080680780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b4031600 con 0x7f06bc0d4570 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.698+0000 7f06c1568640 1 -- 172.21.15.67:0/3080680780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b40319d0 con 0x7f06bc0d4570 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 -- 172.21.15.67:0/3080680780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc0d4570 msgr2=0x7f06bc0b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 --2- 172.21.15.67:0/3080680780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc0d4570 0x7f06bc0b4830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f06bc070c30 tx=0x7f06b402a660 comp rx=0 tx=0).stop 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 -- 172.21.15.67:0/3080680780 shutdown_connections 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 --2- 172.21.15.67:0/3080680780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 --2- 172.21.15.67:0/3080680780 >> v2:172.21.15.176:3300/0 conn(0x7f06bc06e8f0 0x7f06bc0d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 --2- 172.21.15.67:0/3080680780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc0d4570 0x7f06bc0b4830 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 -- 172.21.15.67:0/3080680780 >> 172.21.15.67:0/3080680780 conn(0x7f06bc0d3a50 msgr2=0x7f06bc0d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 -- 172.21.15.67:0/3080680780 shutdown_connections 2024-04-25T15:26:16.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.699+0000 7f06c47ce640 1 -- 172.21.15.67:0/3080680780 wait complete. 2024-04-25T15:26:16.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.701+0000 7f06c47ce640 1 Processor -- start 2024-04-25T15:26:16.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.701+0000 7f06c47ce640 1 -- start start 2024-04-25T15:26:16.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c47ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc06e8f0 0x7f06bc15fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c47ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc14b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c47ce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06bc0d4570 0x7f06bc1fa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c47ce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc0b3300 con 0x7f06bc06e8f0 2024-04-25T15:26:16.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c47ce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc06ed00 con 0x7f06bc0d4570 2024-04-25T15:26:16.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c47ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f06bc0b3480 con 0x7f06bc0d09f0 2024-04-25T15:26:16.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc06e8f0 0x7f06bc15fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc06e8f0 0x7f06bc15fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49632/0 (socket says 172.21.15.67:49632) 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 -- 172.21.15.67:0/1757316676 learned_addr learned my addr 172.21.15.67:0/1757316676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c2d6b640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06bc0d4570 0x7f06bc1fa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c1d69640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc14b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 -- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 msgr2=0x7f06bc14b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc14b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 -- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06bc0d4570 msgr2=0x7f06bc1fa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06bc0d4570 0x7f06bc1fa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.702+0000 7f06c256a640 1 -- 172.21.15.67:0/1757316676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc14bed0 con 0x7f06bc06e8f0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.703+0000 7f06c256a640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc06e8f0 0x7f06bc15fb40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009f90 tx=0x7f06b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.703+0000 7f06baffd640 1 -- 172.21.15.67:0/1757316676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b403f020 con 0x7f06bc06e8f0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.703+0000 7f06baffd640 1 -- 172.21.15.67:0/1757316676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b4037040 con 0x7f06bc06e8f0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.703+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06bc15bd70 con 0x7f06bc06e8f0 2024-04-25T15:26:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.703+0000 7f06baffd640 1 -- 172.21.15.67:0/1757316676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4036070 con 0x7f06bc06e8f0 2024-04-25T15:26:16.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.703+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06bc15e410 con 0x7f06bc06e8f0 2024-04-25T15:26:16.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.704+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06bc050740 con 0x7f06bc06e8f0 2024-04-25T15:26:16.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.705+0000 7f06baffd640 1 -- 172.21.15.67:0/1757316676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f06b4042020 con 0x7f06bc06e8f0 2024-04-25T15:26:16.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.705+0000 7f06baffd640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06a80744e0 0x7f06a80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:16.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.705+0000 7f06baffd640 1 -- 172.21.15.67:0/1757316676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f06b40b5450 con 0x7f06bc06e8f0 2024-04-25T15:26:16.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.705+0000 7f06c1d69640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06a80744e0 0x7f06a80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:16.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.706+0000 7f06c1d69640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06a80744e0 0x7f06a80769f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006fd0 tx=0x7f06b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:16.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.710+0000 7f06baffd640 1 -- 172.21.15.67:0/1757316676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06b4049030 con 0x7f06bc06e8f0 2024-04-25T15:26:16.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.973+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f06bc0d1220 con 0x7f06bc06e8f0 2024-04-25T15:26:16.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.976+0000 7f06baffd640 1 -- 172.21.15.67:0/1757316676 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7f06b407ff30 con 0x7f06bc06e8f0 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06a80744e0 msgr2=0x7f06a80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06a80744e0 0x7f06a80769f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006fd0 tx=0x7f06b0008040 comp rx=0 tx=0).stop 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc06e8f0 msgr2=0x7f06bc15fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc06e8f0 0x7f06bc15fb40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009f90 tx=0x7f06b4004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 shutdown_connections 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06a80744e0 0x7f06a80769f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06bc0d4570 0x7f06bc1fa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06bc0d09f0 0x7f06bc14b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 --2- 172.21.15.67:0/1757316676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06bc06e8f0 0x7f06bc15fb40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 >> 172.21.15.67:0/1757316676 conn(0x7f06bc0d3a50 msgr2=0x7f06bc0712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 shutdown_connections 2024-04-25T15:26:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:16.978+0000 7f06c47ce640 1 -- 172.21.15.67:0/1757316676 wait complete. 2024-04-25T15:26:16.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-25T15:26:17.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.101+0000 7f37ad213640 1 Processor -- start 2024-04-25T15:26:17.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.102+0000 7f37ad213640 1 -- start start 2024-04-25T15:26:17.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37ad213640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37a80d0e00 0x7f37a80d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37ad213640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a80d1850 con 0x7f37a806c5c0 2024-04-25T15:26:17.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37ad213640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a8160de0 con 0x7f37a80d4a50 2024-04-25T15:26:17.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37ad213640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f37a81610c0 con 0x7f37a80d0e00 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a6d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37a80d0e00 0x7f37a80d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a6d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37a80d0e00 0x7f37a80d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49646/0 (socket says 172.21.15.67:49646) 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a6d9d640 1 -- 172.21.15.67:0/685631590 learned_addr learned my addr 172.21.15.67:0/685631590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a659c640 1 --1- 172.21.15.67:0/685631590 >> v1:172.21.15.67:6789/0 conn(0x7f37a806c5c0 0x7f37a806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49820/0 (socket says 172.21.15.67:49820) 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a6d9d640 1 -- 172.21.15.67:0/685631590 >> v1:172.21.15.176:6789/0 conn(0x7f37a80d4a50 legacy=0x7f37a80d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a6d9d640 1 -- 172.21.15.67:0/685631590 >> v1:172.21.15.67:6789/0 conn(0x7f37a806c5c0 legacy=0x7f37a806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a6d9d640 1 -- 172.21.15.67:0/685631590 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a81613a0 con 0x7f37a80d0e00 2024-04-25T15:26:17.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.103+0000 7f37a5d9b640 1 -- 172.21.15.67:0/685631590 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276109369 0 0) 0x7f37a80d1850 con 0x7f37a806c5c0 2024-04-25T15:26:17.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.104+0000 7f37a6d9d640 1 --2- 172.21.15.67:0/685631590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d0e00 0x7f37a80d1210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009870 tx=0x7f37a002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6c75edd7b59eda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.104+0000 7f37a5d9b640 1 -- 172.21.15.67:0/685631590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0037070 con 0x7f37a80d0e00 2024-04-25T15:26:17.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.104+0000 7f37a5d9b640 1 -- 172.21.15.67:0/685631590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a0032440 con 0x7f37a80d0e00 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.104+0000 7f37a5d9b640 1 -- 172.21.15.67:0/685631590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0032810 con 0x7f37a80d0e00 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.105+0000 7f37ad213640 1 -- 172.21.15.67:0/685631590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d0e00 msgr2=0x7f37a80d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.105+0000 7f37ad213640 1 --2- 172.21.15.67:0/685631590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d0e00 0x7f37a80d1210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009870 tx=0x7f37a002a210 comp rx=0 tx=0).stop 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.105+0000 7f37ad213640 1 -- 172.21.15.67:0/685631590 shutdown_connections 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.105+0000 7f37ad213640 1 --2- 172.21.15.67:0/685631590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d0e00 0x7f37a80d1210 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.105+0000 7f37ad213640 1 -- 172.21.15.67:0/685631590 >> 172.21.15.67:0/685631590 conn(0x7f37a80d5d90 msgr2=0x7f37a80d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.105+0000 7f37ad213640 1 -- 172.21.15.67:0/685631590 shutdown_connections 2024-04-25T15:26:17.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.105+0000 7f37ad213640 1 -- 172.21.15.67:0/685631590 wait complete. 2024-04-25T15:26:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.107+0000 7f37ad213640 1 Processor -- start 2024-04-25T15:26:17.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.107+0000 7f37ad213640 1 -- start start 2024-04-25T15:26:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.108+0000 7f37ad213640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37a806c5c0 0x7f37a814ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.108+0000 7f37ad213640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37a80d0e00 0x7f37a80c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.108+0000 7f37ad213640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d4a50 0x7f37a81f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.108+0000 7f37ad213640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8164e00 con 0x7f37a80d4a50 2024-04-25T15:26:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.108+0000 7f37ad213640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f37a80b5320 con 0x7f37a80d0e00 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.108+0000 7f37ad213640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f37a8164f80 con 0x7f37a806c5c0 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d4a50 0x7f37a81f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d4a50 0x7f37a81f9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49654/0 (socket says 172.21.15.67:49654) 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 -- 172.21.15.67:0/1348773814 learned_addr learned my addr 172.21.15.67:0/1348773814 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a659c640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37a80d0e00 0x7f37a80c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a6d9d640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37a806c5c0 0x7f37a814ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 -- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37a806c5c0 msgr2=0x7f37a814ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37a806c5c0 0x7f37a814ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 -- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37a80d0e00 msgr2=0x7f37a80c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37a80d0e00 0x7f37a80c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 -- 172.21.15.67:0/1348773814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a80cc4d0 con 0x7f37a80d4a50 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a6d9d640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37a806c5c0 0x7f37a814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37a759e640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d4a50 0x7f37a81f9e10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f379800b9a0 tx=0x7f379800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f379f7fe640 1 -- 172.21.15.67:0/1348773814 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798019070 con 0x7f37a80d4a50 2024-04-25T15:26:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f379f7fe640 1 -- 172.21.15.67:0/1348773814 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f379800cb40 con 0x7f37a80d4a50 2024-04-25T15:26:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a80cc7b0 con 0x7f37a80d4a50 2024-04-25T15:26:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f379f7fe640 1 -- 172.21.15.67:0/1348773814 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798014440 con 0x7f37a80d4a50 2024-04-25T15:26:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.109+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a80cd830 con 0x7f37a80d4a50 2024-04-25T15:26:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.110+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37a8050740 con 0x7f37a80d4a50 2024-04-25T15:26:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.111+0000 7f379f7fe640 1 -- 172.21.15.67:0/1348773814 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f379801c070 con 0x7f37a80d4a50 2024-04-25T15:26:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.111+0000 7f379f7fe640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f37880744e0 0x7f37880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.112+0000 7f379f7fe640 1 -- 172.21.15.67:0/1348773814 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f3798094aa0 con 0x7f37a80d4a50 2024-04-25T15:26:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.112+0000 7f37a6d9d640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f37880744e0 0x7f37880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.113+0000 7f37a6d9d640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f37880744e0 0x7f37880769f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37a0034000 tx=0x7f37a002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.116+0000 7f379f7fe640 1 -- 172.21.15.67:0/1348773814 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f379805f610 con 0x7f37a80d4a50 2024-04-25T15:26:17.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.372+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f37a80d4320 con 0x7f37a80d4a50 2024-04-25T15:26:17.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.375+0000 7f379f7fe640 1 -- 172.21.15.67:0/1348773814 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7f379805ef10 con 0x7f37a80d4a50 2024-04-25T15:26:17.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f37880744e0 msgr2=0x7f37880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f37880744e0 0x7f37880769f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37a0034000 tx=0x7f37a002e000 comp rx=0 tx=0).stop 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d4a50 msgr2=0x7f37a81f9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d4a50 0x7f37a81f9e10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f379800b9a0 tx=0x7f379800be60 comp rx=0 tx=0).stop 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 shutdown_connections 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f37880744e0 0x7f37880769f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37a80d4a50 0x7f37a81f9e10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37a80d0e00 0x7f37a80c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 --2- 172.21.15.67:0/1348773814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37a806c5c0 0x7f37a814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 >> 172.21.15.67:0/1348773814 conn(0x7f37a80d5d90 msgr2=0x7f37a80b9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 shutdown_connections 2024-04-25T15:26:17.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.377+0000 7f37ad213640 1 -- 172.21.15.67:0/1348773814 wait complete. 2024-04-25T15:26:17.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:17.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.500+0000 7f6dccb57640 1 Processor -- start 2024-04-25T15:26:17.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.500+0000 7f6dccb57640 1 -- start start 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dccb57640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80d0e00 0x7f6dc80d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dccb57640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6dc806c5c0 0x7f6dc806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dccb57640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dc806cf10 con 0x7f6dc80b7810 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dccb57640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6dc80d5a60 con 0x7f6dc806c5c0 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dccb57640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc80d1850 con 0x7f6dc80d0e00 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dc659c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80d0e00 0x7f6dc80d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dc5d9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6dc806c5c0 0x7f6dc806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dc659c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80d0e00 0x7f6dc80d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:35326/0 (socket says 172.21.15.67:35326) 2024-04-25T15:26:17.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dc6d9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f6dc80b7810 0x7f6dc80b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:35446/0 (socket says 172.21.15.67:35446) 2024-04-25T15:26:17.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.502+0000 7f6dc659c640 1 -- 172.21.15.67:0/3873971364 learned_addr learned my addr 172.21.15.67:0/3873971364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.503+0000 7f6dc659c640 1 -- 172.21.15.67:0/3873971364 >> v2:172.21.15.176:3300/0 conn(0x7f6dc806c5c0 msgr2=0x7f6dc806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.503+0000 7f6dc659c640 1 --2- 172.21.15.67:0/3873971364 >> v2:172.21.15.176:3300/0 conn(0x7f6dc806c5c0 0x7f6dc806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.503+0000 7f6dc659c640 1 -- 172.21.15.67:0/3873971364 >> v1:172.21.15.176:6789/0 conn(0x7f6dc80b7810 legacy=0x7f6dc80b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.503+0000 7f6dc659c640 1 -- 172.21.15.67:0/3873971364 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc8160de0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.504+0000 7f6dc659c640 1 --2- 172.21.15.67:0/3873971364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80d0e00 0x7f6dc80d1210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009e50 tx=0x7f6dc002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b37a1243d8a4d8f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.504+0000 7f6dc559a640 1 -- 172.21.15.67:0/3873971364 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0036070 con 0x7f6dc80d0e00 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.504+0000 7f6dc559a640 1 -- 172.21.15.67:0/3873971364 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dc0031600 con 0x7f6dc80d0e00 2024-04-25T15:26:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.504+0000 7f6dc559a640 1 -- 172.21.15.67:0/3873971364 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00319d0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.504+0000 7f6dccb57640 1 -- 172.21.15.67:0/3873971364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80d0e00 msgr2=0x7f6dc80d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.504+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3873971364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80d0e00 0x7f6dc80d1210 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009e50 tx=0x7f6dc002a710 comp rx=0 tx=0).stop 2024-04-25T15:26:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.505+0000 7f6dccb57640 1 -- 172.21.15.67:0/3873971364 shutdown_connections 2024-04-25T15:26:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.505+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3873971364 >> v2:172.21.15.176:3300/0 conn(0x7f6dc806c5c0 0x7f6dc806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.505+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3873971364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80d0e00 0x7f6dc80d1210 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.505+0000 7f6dccb57640 1 -- 172.21.15.67:0/3873971364 >> 172.21.15.67:0/3873971364 conn(0x7f6dc80d5d90 msgr2=0x7f6dc80d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:17.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.505+0000 7f6dccb57640 1 -- 172.21.15.67:0/3873971364 shutdown_connections 2024-04-25T15:26:17.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.505+0000 7f6dccb57640 1 -- 172.21.15.67:0/3873971364 wait complete. 2024-04-25T15:26:17.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.507+0000 7f6dccb57640 1 Processor -- start 2024-04-25T15:26:17.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.507+0000 7f6dccb57640 1 -- start start 2024-04-25T15:26:17.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dccb57640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dc806c5c0 0x7f6dc815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dccb57640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80b7810 0x7f6dc8157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dccb57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc80d0e00 0x7f6dc8163220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dccb57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8160ad0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dccb57640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81613a0 con 0x7f6dc806c5c0 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dccb57640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc80b5320 con 0x7f6dc80b7810 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc80d0e00 0x7f6dc8163220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc80d0e00 0x7f6dc8163220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49668/0 (socket says 172.21.15.67:49668) 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 -- 172.21.15.67:0/3440888313 learned_addr learned my addr 172.21.15.67:0/3440888313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc5d9b640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80b7810 0x7f6dc8157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc659c640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dc806c5c0 0x7f6dc815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 -- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80b7810 msgr2=0x7f6dc8157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80b7810 0x7f6dc8157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 -- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dc806c5c0 msgr2=0x7f6dc815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dc806c5c0 0x7f6dc815b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.508+0000 7f6dc6d9d640 1 -- 172.21.15.67:0/3440888313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc80ce850 con 0x7f6dc80d0e00 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.509+0000 7f6dc6d9d640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc80d0e00 0x7f6dc8163220 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c3f0 tx=0x7f6db800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.509+0000 7f6db6ffd640 1 -- 172.21.15.67:0/3440888313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db80074e0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.509+0000 7f6db6ffd640 1 -- 172.21.15.67:0/3440888313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db8007680 con 0x7f6dc80d0e00 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.509+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc815b740 con 0x7f6dc80d0e00 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.509+0000 7f6db6ffd640 1 -- 172.21.15.67:0/3440888313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db80049e0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.509+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc815dab0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.510+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc80cea20 con 0x7f6dc80d0e00 2024-04-25T15:26:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.511+0000 7f6db6ffd640 1 -- 172.21.15.67:0/3440888313 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6db801a020 con 0x7f6dc80d0e00 2024-04-25T15:26:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.511+0000 7f6db6ffd640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6da80744e0 0x7f6da80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.511+0000 7f6db6ffd640 1 -- 172.21.15.67:0/3440888313 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f6db80943c0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.511+0000 7f6dc659c640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6da80744e0 0x7f6da80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.512+0000 7f6dc659c640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6da80744e0 0x7f6da80769f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0033000 tx=0x7f6dc0009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.516+0000 7f6db6ffd640 1 -- 172.21.15.67:0/3440888313 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6dc806cf10 con 0x7f6dc80d0e00 2024-04-25T15:26:17.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.705+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6dc80ce3b0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.705+0000 7f6db6ffd640 1 -- 172.21.15.67:0/3440888313 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+12855 (secure 0 0 0) 0x7f6db805e8f0 con 0x7f6dc80d0e00 2024-04-25T15:26:17.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:17.706 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":29,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:14.557936+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:17.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6da80744e0 msgr2=0x7f6da80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6da80744e0 0x7f6da80769f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0033000 tx=0x7f6dc0009000 comp rx=0 tx=0).stop 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc80d0e00 msgr2=0x7f6dc8163220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc80d0e00 0x7f6dc8163220 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6db800c3f0 tx=0x7f6db800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 shutdown_connections 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6da80744e0 0x7f6da80769f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6dc80d0e00 0x7f6dc8163220 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dc80b7810 0x7f6dc8157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.707+0000 7f6dccb57640 1 --2- 172.21.15.67:0/3440888313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dc806c5c0 0x7f6dc815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.708+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 >> 172.21.15.67:0/3440888313 conn(0x7f6dc80d5d90 msgr2=0x7f6dc8160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.708+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 shutdown_connections 2024-04-25T15:26:17.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.708+0000 7f6dccb57640 1 -- 172.21.15.67:0/3440888313 wait complete. 2024-04-25T15:26:17.720 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:17.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.833+0000 7f7cbdf4e640 1 Processor -- start 2024-04-25T15:26:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.834+0000 7f7cbdf4e640 1 -- start start 2024-04-25T15:26:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cbdf4e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7cb80ce040 0x7f7cb80b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cbdf4e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb806ea60 0x7f7cb80b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cbdf4e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb806d770 con 0x7f7cb81649f0 2024-04-25T15:26:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cbdf4e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7cb806da50 con 0x7f7cb80ce040 2024-04-25T15:26:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cbdf4e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb80d2200 con 0x7f7cb806ea60 2024-04-25T15:26:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cb6ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7cb81649f0 0x7f7cb806e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57680/0 (socket says 172.21.15.67:57680) 2024-04-25T15:26:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cb6ffd640 1 -- 172.21.15.67:0/3592830959 learned_addr learned my addr 172.21.15.67:0/3592830959 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb806ea60 0x7f7cb80b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308405401 0 0) 0x7f7cb806d770 con 0x7f7cb81649f0 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ca0003520 con 0x7f7cb81649f0 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.835+0000 7f7cb77fe640 1 --2- 172.21.15.67:0/3592830959 >> v2:172.21.15.176:3300/0 conn(0x7f7cb80ce040 0x7f7cb80b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 26771951 0 0) 0x7f7ca0003520 con 0x7f7cb81649f0 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb806ea60 msgr2=0x7f7cb80b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb806ea60 0x7f7cb80b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 >> v2:172.21.15.176:3300/0 conn(0x7f7cb80ce040 msgr2=0x7f7cb80b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 --2- 172.21.15.67:0/3592830959 >> v2:172.21.15.176:3300/0 conn(0x7f7cb80ce040 0x7f7cb80b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb80d24e0 con 0x7f7cb81649f0 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f7cac001670 con 0x7f7cb81649f0 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 >> v1:172.21.15.67:6789/0 conn(0x7f7cb81649f0 legacy=0x7f7cb806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ca00039b0 0x7f7ca0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ca0004340 0x7f7ca0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca0004ed0 0x7f7ca000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb806da50 con 0x7f7ca0004ed0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb80d2200 con 0x7f7ca0004340 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca000bf00 con 0x7f7ca00039b0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca0004ed0 0x7f7ca000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb77fe640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ca00039b0 0x7f7ca0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb6ffd640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ca0004340 0x7f7ca0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb7fff640 1 -- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ca00039b0 msgr2=0x7f7ca0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ca00039b0 0x7f7ca0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb7fff640 1 -- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ca0004340 msgr2=0x7f7ca0004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ca0004340 0x7f7ca0004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.836+0000 7f7cb7fff640 1 -- 172.21.15.67:0/3592830959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca0003520 con 0x7f7ca0004ed0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.837+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca0004ed0 0x7f7ca000b9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fc0 tx=0x7f7ca0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9a3addc6b6b744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.837+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0034020 con 0x7f7ca0004ed0 2024-04-25T15:26:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.837+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb00305f0 con 0x7f7ca0004ed0 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.837+0000 7f7cb67fc640 1 -- 172.21.15.67:0/3592830959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0009c10 con 0x7f7ca0004ed0 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.838+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/3592830959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca0004ed0 msgr2=0x7f7ca000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.838+0000 7f7cbdf4e640 1 --2- 172.21.15.67:0/3592830959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ca0004ed0 0x7f7ca000b9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fc0 tx=0x7f7ca0005830 comp rx=0 tx=0).stop 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.838+0000 7f7cb7fff640 1 -- 172.21.15.67:0/3592830959 reap_dead start 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.838+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/3592830959 shutdown_connections 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.838+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/3592830959 >> 172.21.15.67:0/3592830959 conn(0x7f7cb80d3a80 msgr2=0x7f7cb80d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.838+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/3592830959 shutdown_connections 2024-04-25T15:26:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.838+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/3592830959 wait complete. 2024-04-25T15:26:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.839+0000 7f7cbdf4e640 1 Processor -- start 2024-04-25T15:26:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.840+0000 7f7cbdf4e640 1 -- start start 2024-04-25T15:26:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cbdf4e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb80b5dd0 0x7f7cb8157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cbdf4e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb8161c80 0x7f7cb80ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cbdf4e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb8162090 0x7f7cb80ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cbdf4e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb806ea60 con 0x7f7cb8162090 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cbdf4e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb8164e00 con 0x7f7cb80b5dd0 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cbdf4e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb806ebe0 con 0x7f7cb8161c80 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb8162090 0x7f7cb80ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb8162090 0x7f7cb80ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41408/0 (socket says 172.21.15.67:41408) 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 -- 172.21.15.67:0/2744054743 learned_addr learned my addr 172.21.15.67:0/2744054743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb6ffd640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb8161c80 0x7f7cb80ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb77fe640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb80b5dd0 0x7f7cb8157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 -- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb8161c80 msgr2=0x7f7cb80ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb8161c80 0x7f7cb80ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 -- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb80b5dd0 msgr2=0x7f7cb8157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb80b5dd0 0x7f7cb8157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:17.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.841+0000 7f7cb7fff640 1 -- 172.21.15.67:0/2744054743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb8158520 con 0x7f7cb8162090 2024-04-25T15:26:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.842+0000 7f7cb7fff640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb8162090 0x7f7cb80ce990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000e230 tx=0x7f7cb0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.842+0000 7f7c9ffff640 1 -- 172.21.15.67:0/2744054743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0040070 con 0x7f7cb8162090 2024-04-25T15:26:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.842+0000 7f7c9ffff640 1 -- 172.21.15.67:0/2744054743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb0003990 con 0x7f7cb8162090 2024-04-25T15:26:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.842+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb80b61e0 con 0x7f7cb8162090 2024-04-25T15:26:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.842+0000 7f7c9ffff640 1 -- 172.21.15.67:0/2744054743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0034040 con 0x7f7cb8162090 2024-04-25T15:26:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.842+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb8162840 con 0x7f7cb8162090 2024-04-25T15:26:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.843+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb8050740 con 0x7f7cb8162090 2024-04-25T15:26:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.844+0000 7f7c9ffff640 1 -- 172.21.15.67:0/2744054743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7cb0037020 con 0x7f7cb8162090 2024-04-25T15:26:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.844+0000 7f7c9ffff640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ca40744e0 0x7f7ca40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.844+0000 7f7c9ffff640 1 -- 172.21.15.67:0/2744054743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f7cb00b5110 con 0x7f7cb8162090 2024-04-25T15:26:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.844+0000 7f7cb77fe640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ca40744e0 0x7f7ca40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.845+0000 7f7cb77fe640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ca40744e0 0x7f7ca40769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8073240 tx=0x7f7ca8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:17.849+0000 7f7c9ffff640 1 -- 172.21.15.67:0/2744054743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7cb007f5a0 con 0x7f7cb8162090 2024-04-25T15:26:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.034+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7cb80d4780 con 0x7f7cb8162090 2024-04-25T15:26:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.034+0000 7f7c9ffff640 1 -- 172.21.15.67:0/2744054743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f7cb00838a0 con 0x7f7cb8162090 2024-04-25T15:26:18.035 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ca40744e0 msgr2=0x7f7ca40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ca40744e0 0x7f7ca40769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8073240 tx=0x7f7ca8009190 comp rx=0 tx=0).stop 2024-04-25T15:26:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb8162090 msgr2=0x7f7cb80ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb8162090 0x7f7cb80ce990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000e230 tx=0x7f7cb0009750 comp rx=0 tx=0).stop 2024-04-25T15:26:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 shutdown_connections 2024-04-25T15:26:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ca40744e0 0x7f7ca40769f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cb8162090 0x7f7cb80ce990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7cb8161c80 0x7f7cb80ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 --2- 172.21.15.67:0/2744054743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb80b5dd0 0x7f7cb8157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 >> 172.21.15.67:0/2744054743 conn(0x7f7cb80d3a80 msgr2=0x7f7cb8071eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 shutdown_connections 2024-04-25T15:26:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.036+0000 7f7cbdf4e640 1 -- 172.21.15.67:0/2744054743 wait complete. 2024-04-25T15:26:18.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:26:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.159+0000 7fdd44caf640 1 Processor -- start 2024-04-25T15:26:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.159+0000 7fdd44caf640 1 -- start start 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.160+0000 7fdd44caf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd400b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.160+0000 7fdd44caf640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd4015df80 0x7fdd400b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.160+0000 7fdd44caf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd40160890 0x7fdd40160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.160+0000 7fdd44caf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdd40164670 con 0x7fdd40160890 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.160+0000 7fdd44caf640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdd401615f0 con 0x7fdd4015df80 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.160+0000 7fdd44caf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd40164020 con 0x7fdd40164b00 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd40160890 0x7fdd40160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd40160890 0x7fdd40160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41422/0 (socket says 172.21.15.67:41422) 2024-04-25T15:26:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 -- 172.21.15.67:0/2717163320 learned_addr learned my addr 172.21.15.67:0/2717163320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3e59c640 1 --2- 172.21.15.67:0/2717163320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd400b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3dd9b640 1 --2- 172.21.15.67:0/2717163320 >> v2:172.21.15.176:3300/0 conn(0x7fdd4015df80 0x7fdd400b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 -- 172.21.15.67:0/2717163320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 msgr2=0x7fdd400b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 --2- 172.21.15.67:0/2717163320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd400b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 -- 172.21.15.67:0/2717163320 >> v2:172.21.15.176:3300/0 conn(0x7fdd4015df80 msgr2=0x7fdd400b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 --2- 172.21.15.67:0/2717163320 >> v2:172.21.15.176:3300/0 conn(0x7fdd4015df80 0x7fdd400b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3ed9d640 1 -- 172.21.15.67:0/2717163320 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd40164300 con 0x7fdd40160890 2024-04-25T15:26:18.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.161+0000 7fdd3dd9b640 1 --2- 172.21.15.67:0/2717163320 >> v2:172.21.15.176:3300/0 conn(0x7fdd4015df80 0x7fdd400b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.162+0000 7fdd3ed9d640 1 --2- 172.21.15.67:0/2717163320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd40160ca0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd4008e440 tx=0x7fdd3002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7558664fa7ea0217 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.162+0000 7fdd3d59a640 1 -- 172.21.15.67:0/2717163320 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30036070 con 0x7fdd40160890 2024-04-25T15:26:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.162+0000 7fdd3d59a640 1 -- 172.21.15.67:0/2717163320 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd30031600 con 0x7fdd40160890 2024-04-25T15:26:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.162+0000 7fdd3d59a640 1 -- 172.21.15.67:0/2717163320 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300319d0 con 0x7fdd40160890 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 -- 172.21.15.67:0/2717163320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 msgr2=0x7fdd40160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2717163320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd40160ca0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdd4008e440 tx=0x7fdd3002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 -- 172.21.15.67:0/2717163320 shutdown_connections 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2717163320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd40160ca0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2717163320 >> v2:172.21.15.176:3300/0 conn(0x7fdd4015df80 0x7fdd400b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2717163320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd400b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 -- 172.21.15.67:0/2717163320 >> 172.21.15.67:0/2717163320 conn(0x7fdd400d2bc0 msgr2=0x7fdd400d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 -- 172.21.15.67:0/2717163320 shutdown_connections 2024-04-25T15:26:18.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.163+0000 7fdd44caf640 1 -- 172.21.15.67:0/2717163320 wait complete. 2024-04-25T15:26:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.165+0000 7fdd44caf640 1 Processor -- start 2024-04-25T15:26:18.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.165+0000 7fdd44caf640 1 -- start start 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd44caf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd4015df80 0x7fdd4014f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd44caf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd401ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd44caf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd401e9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd44caf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd400b4a00 con 0x7fdd40160890 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd44caf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd400b4880 con 0x7fdd4015df80 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd44caf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd400b4b80 con 0x7fdd40164b00 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd401ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd401ebb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41428/0 (socket says 172.21.15.67:41428) 2024-04-25T15:26:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 -- 172.21.15.67:0/2689007396 learned_addr learned my addr 172.21.15.67:0/2689007396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3e59c640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd4015df80 0x7fdd4014f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3ed9d640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd401e9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 -- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 msgr2=0x7fdd401e9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd401e9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 -- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd4015df80 msgr2=0x7fdd4014f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd4015df80 0x7fdd4014f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.166+0000 7fdd3dd9b640 1 -- 172.21.15.67:0/2689007396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd401eabd0 con 0x7fdd40160890 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.167+0000 7fdd3dd9b640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd401ebb00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800d550 tx=0x7fdd2800da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.167+0000 7fdd36ffd640 1 -- 172.21.15.67:0/2689007396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28004030 con 0x7fdd40160890 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.167+0000 7fdd36ffd640 1 -- 172.21.15.67:0/2689007396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd280041d0 con 0x7fdd40160890 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.167+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd401eaeb0 con 0x7fdd40160890 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.167+0000 7fdd36ffd640 1 -- 172.21.15.67:0/2689007396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28014620 con 0x7fdd40160890 2024-04-25T15:26:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.167+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd40162f60 con 0x7fdd40160890 2024-04-25T15:26:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.168+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd40161b20 con 0x7fdd40160890 2024-04-25T15:26:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.168+0000 7fdd36ffd640 1 -- 172.21.15.67:0/2689007396 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdd280147c0 con 0x7fdd40160890 2024-04-25T15:26:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.169+0000 7fdd36ffd640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd200744b0 0x7fdd200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.169+0000 7fdd36ffd640 1 -- 172.21.15.67:0/2689007396 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fdd280953a0 con 0x7fdd40160890 2024-04-25T15:26:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.169+0000 7fdd3e59c640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd200744b0 0x7fdd200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.170+0000 7fdd3e59c640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd200744b0 0x7fdd200769c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009aa0 tx=0x7fdd38009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:18.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.173+0000 7fdd36ffd640 1 -- 172.21.15.67:0/2689007396 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd2805f8d0 con 0x7fdd40160890 2024-04-25T15:26:18.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.358+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdd40050740 con 0x7fdd40160890 2024-04-25T15:26:18.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.358+0000 7fdd36ffd640 1 -- 172.21.15.67:0/2689007396 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) ==== 84+0+65 (secure 0 0 0) 0x7fdd28063bd0 con 0x7fdd40160890 2024-04-25T15:26:18.359 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:18.359 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:26:18.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:18.360 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:26:18.360 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:26:18.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:18.360 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:26:18.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd200744b0 msgr2=0x7fdd200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd200744b0 0x7fdd200769c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009aa0 tx=0x7fdd38009190 comp rx=0 tx=0).stop 2024-04-25T15:26:18.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 msgr2=0x7fdd401ebb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd401ebb00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800d550 tx=0x7fdd2800da10 comp rx=0 tx=0).stop 2024-04-25T15:26:18.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 shutdown_connections 2024-04-25T15:26:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd200744b0 0x7fdd200769c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd40164b00 0x7fdd401e9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40160890 0x7fdd401ebb00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 --2- 172.21.15.67:0/2689007396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd4015df80 0x7fdd4014f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 >> 172.21.15.67:0/2689007396 conn(0x7fdd400d2bc0 msgr2=0x7fdd400d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 shutdown_connections 2024-04-25T15:26:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.360+0000 7fdd44caf640 1 -- 172.21.15.67:0/2689007396 wait complete. 2024-04-25T15:26:18.372 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:26:18.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:26:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.488+0000 7f74187c7640 1 Processor -- start 2024-04-25T15:26:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.488+0000 7f74187c7640 1 -- start start 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.489+0000 7f74187c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f741006d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.489+0000 7f74187c7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f741006db10 0x7f7410070270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.489+0000 7f74187c7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f741006df20 con 0x7f74100af710 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.489+0000 7f74187c7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f74100afb20 con 0x7f741006db10 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.489+0000 7f74187c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f741006f440 con 0x7f7410160390 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416d64640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f741006db10 0x7f7410070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416d64640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f741006db10 0x7f7410070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41434/0 (socket says 172.21.15.67:41434) 2024-04-25T15:26:18.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416d64640 1 -- 172.21.15.67:0/940695345 learned_addr learned my addr 172.21.15.67:0/940695345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416563640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74100af710 0x7f7410164a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57682/0 (socket says 172.21.15.67:57682) 2024-04-25T15:26:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7415d62640 1 --2- 172.21.15.67:0/940695345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f741006d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416d64640 1 -- 172.21.15.67:0/940695345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 msgr2=0x7f741006d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416d64640 1 --2- 172.21.15.67:0/940695345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f741006d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416d64640 1 -- 172.21.15.67:0/940695345 >> v1:172.21.15.67:6789/0 conn(0x7f74100af710 legacy=0x7f7410164a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.490+0000 7f7416d64640 1 -- 172.21.15.67:0/940695345 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741006f720 con 0x7f741006db10 2024-04-25T15:26:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.491+0000 7f7416d64640 1 --2- 172.21.15.67:0/940695345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741006db10 0x7f7410070270 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7408009e50 tx=0x7f740802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29429e3b046eec0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.491+0000 7f7415561640 1 -- 172.21.15.67:0/940695345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7408036070 con 0x7f741006db10 2024-04-25T15:26:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.491+0000 7f7415561640 1 -- 172.21.15.67:0/940695345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7408031600 con 0x7f741006db10 2024-04-25T15:26:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.491+0000 7f7415561640 1 -- 172.21.15.67:0/940695345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74080319d0 con 0x7f741006db10 2024-04-25T15:26:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 -- 172.21.15.67:0/940695345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741006db10 msgr2=0x7f7410070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 --2- 172.21.15.67:0/940695345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741006db10 0x7f7410070270 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7408009e50 tx=0x7f740802a710 comp rx=0 tx=0).stop 2024-04-25T15:26:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 -- 172.21.15.67:0/940695345 shutdown_connections 2024-04-25T15:26:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 --2- 172.21.15.67:0/940695345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741006db10 0x7f7410070270 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 --2- 172.21.15.67:0/940695345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f741006d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 -- 172.21.15.67:0/940695345 >> 172.21.15.67:0/940695345 conn(0x7f7410073090 msgr2=0x7f74100734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 -- 172.21.15.67:0/940695345 shutdown_connections 2024-04-25T15:26:18.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.492+0000 7f74187c7640 1 -- 172.21.15.67:0/940695345 wait complete. 2024-04-25T15:26:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.494+0000 7f74187c7640 1 Processor -- start 2024-04-25T15:26:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.494+0000 7f74187c7640 1 -- start start 2024-04-25T15:26:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f74187c7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f741006db10 0x7f7410161170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f74187c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74100af710 0x7f7410159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f74187c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f7410163140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f74187c7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f741006ffc0 con 0x7f74100af710 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f74187c7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f741006fd60 con 0x7f741006db10 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f74187c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f741006f0d0 con 0x7f7410160390 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74100af710 0x7f7410159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74100af710 0x7f7410159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41450/0 (socket says 172.21.15.67:41450) 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 -- 172.21.15.67:0/4106016881 learned_addr learned my addr 172.21.15.67:0/4106016881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7416563640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f741006db10 0x7f7410161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7416d64640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f7410163140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 -- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 msgr2=0x7f7410163140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f7410163140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 -- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f741006db10 msgr2=0x7f7410161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f741006db10 0x7f7410161170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.495+0000 7f7415d62640 1 -- 172.21.15.67:0/4106016881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74101626d0 con 0x7f74100af710 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.496+0000 7f7415d62640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74100af710 0x7f7410159720 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d5f0 tx=0x7f740c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.496+0000 7f7406ffd640 1 -- 172.21.15.67:0/4106016881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c0125f0 con 0x7f74100af710 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.496+0000 7f7406ffd640 1 -- 172.21.15.67:0/4106016881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f740c019070 con 0x7f74100af710 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.496+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7410161b50 con 0x7f74100af710 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.496+0000 7f7406ffd640 1 -- 172.21.15.67:0/4106016881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c0129d0 con 0x7f74100af710 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.496+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7410162110 con 0x7f74100af710 2024-04-25T15:26:18.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.497+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74100734a0 con 0x7f74100af710 2024-04-25T15:26:18.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.498+0000 7f7406ffd640 1 -- 172.21.15.67:0/4106016881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f740c012b70 con 0x7f74100af710 2024-04-25T15:26:18.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.498+0000 7f7406ffd640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f73f40744e0 0x7f73f40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:18.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.498+0000 7f7406ffd640 1 -- 172.21.15.67:0/4106016881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f740c0952d0 con 0x7f74100af710 2024-04-25T15:26:18.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.498+0000 7f7416563640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f73f40744e0 0x7f73f40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:18.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.499+0000 7f7416563640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f73f40744e0 0x7f73f40769f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7400000f60 tx=0x7f7400009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:18.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.508+0000 7f7406ffd640 1 -- 172.21.15.67:0/4106016881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74100734a0 con 0x7f74100af710 2024-04-25T15:26:18.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:18.697+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f74100b6990 con 0x7f74100af710 2024-04-25T15:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.574+0000 7f7406ffd640 1 -- 172.21.15.67:0/4106016881 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) ==== 138+0+0 (secure 0 0 0) 0x7f74100b6990 con 0x7f74100af710 2024-04-25T15:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f73f40744e0 msgr2=0x7f73f40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f73f40744e0 0x7f73f40769f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7400000f60 tx=0x7f7400009040 comp rx=0 tx=0).stop 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74100af710 msgr2=0x7f7410159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74100af710 0x7f7410159720 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d5f0 tx=0x7f740c00dab0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 shutdown_connections 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f73f40744e0 0x7f73f40769f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7410160390 0x7f7410163140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74100af710 0x7f7410159720 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 --2- 172.21.15.67:0/4106016881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f741006db10 0x7f7410161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 >> 172.21.15.67:0/4106016881 conn(0x7f7410073090 msgr2=0x7f74100b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 shutdown_connections 2024-04-25T15:26:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.577+0000 7f74187c7640 1 -- 172.21.15.67:0/4106016881 wait complete. 2024-04-25T15:26:19.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:26:19.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.712+0000 7f02b5cd8640 1 Processor -- start 2024-04-25T15:26:19.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.713+0000 7f02b5cd8640 1 -- start start 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02b5cd8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f02b00b77d0 0x7f02b0164bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02b5cd8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b00d0d60 0x7f02b0160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02b5cd8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b00cffa0 con 0x7f02b01608a0 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02b5cd8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f02b00b7be0 con 0x7f02b00b77d0 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02b5cd8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f02b0165110 con 0x7f02b00d0d60 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02aeffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b00d0d60 0x7f02b0160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02af7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f02b00b77d0 0x7f02b0164bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02aeffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b00d0d60 0x7f02b0160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:37080/0 (socket says 172.21.15.67:37080) 2024-04-25T15:26:19.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.714+0000 7f02aeffd640 1 -- 172.21.15.67:0/1848197125 learned_addr learned my addr 172.21.15.67:0/1848197125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.715+0000 7f02aeffd640 1 -- 172.21.15.67:0/1848197125 >> v2:172.21.15.176:3300/0 conn(0x7f02b00b77d0 msgr2=0x7f02b0164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.715+0000 7f02aeffd640 1 --2- 172.21.15.67:0/1848197125 >> v2:172.21.15.176:3300/0 conn(0x7f02b00b77d0 0x7f02b0164bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.715+0000 7f02aeffd640 1 -- 172.21.15.67:0/1848197125 >> v1:172.21.15.176:6789/0 conn(0x7f02b01608a0 legacy=0x7f02b00b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:19.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.715+0000 7f02aeffd640 1 -- 172.21.15.67:0/1848197125 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b00d0280 con 0x7f02b00d0d60 2024-04-25T15:26:19.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.715+0000 7f02aeffd640 1 --2- 172.21.15.67:0/1848197125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b00d0d60 0x7f02b0160360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009870 tx=0x7f02a402a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7d80b00923f2a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:19.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02ae7fc640 1 -- 172.21.15.67:0/1848197125 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4037070 con 0x7f02b00d0d60 2024-04-25T15:26:19.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02ae7fc640 1 -- 172.21.15.67:0/1848197125 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a4032640 con 0x7f02b00d0d60 2024-04-25T15:26:19.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02ae7fc640 1 -- 172.21.15.67:0/1848197125 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4032a10 con 0x7f02b00d0d60 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1848197125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b00d0d60 msgr2=0x7f02b0160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1848197125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b00d0d60 0x7f02b0160360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009870 tx=0x7f02a402a2e0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1848197125 shutdown_connections 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1848197125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b00d0d60 0x7f02b0160360 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1848197125 >> v2:172.21.15.176:3300/0 conn(0x7f02b00b77d0 0x7f02b0164bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1848197125 >> 172.21.15.67:0/1848197125 conn(0x7f02b009b7d0 msgr2=0x7f02b00b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1848197125 shutdown_connections 2024-04-25T15:26:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.716+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1848197125 wait complete. 2024-04-25T15:26:19.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.718+0000 7f02b5cd8640 1 Processor -- start 2024-04-25T15:26:19.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.718+0000 7f02b5cd8640 1 -- start start 2024-04-25T15:26:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.719+0000 7f02b5cd8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b00b77d0 0x7f02b015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.719+0000 7f02b5cd8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00d0d60 0x7f02b00ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.719+0000 7f02b5cd8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b01f97c0 0x7f02b01f9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02b5cd8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f02b00d1170 con 0x7f02b00d0d60 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02b5cd8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02b00d08d0 con 0x7f02b00b77d0 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02b5cd8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f02b00d12f0 con 0x7f02b01f97c0 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00d0d60 0x7f02b00ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00d0d60 0x7f02b00ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41452/0 (socket says 172.21.15.67:41452) 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 -- 172.21.15.67:0/1726363299 learned_addr learned my addr 172.21.15.67:0/1726363299 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02af7fe640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b00b77d0 0x7f02b015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02affff640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b01f97c0 0x7f02b01f9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 -- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b01f97c0 msgr2=0x7f02b01f9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b01f97c0 0x7f02b01f9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 -- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b00b77d0 msgr2=0x7f02b015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b00b77d0 0x7f02b015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 -- 172.21.15.67:0/1726363299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b01fa110 con 0x7f02b00d0d60 2024-04-25T15:26:19.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02aeffd640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00d0d60 0x7f02b00ce2c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f02a4000c00 tx=0x7f02a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:19.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f0297fff640 1 -- 172.21.15.67:0/1726363299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4037040 con 0x7f02b00d0d60 2024-04-25T15:26:19.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f0297fff640 1 -- 172.21.15.67:0/1726363299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a4042070 con 0x7f02b00d0d60 2024-04-25T15:26:19.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b00cf1f0 con 0x7f02b00d0d60 2024-04-25T15:26:19.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.720+0000 7f0297fff640 1 -- 172.21.15.67:0/1726363299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a40309b0 con 0x7f02b00d0d60 2024-04-25T15:26:19.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.721+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b00cef10 con 0x7f02b00d0d60 2024-04-25T15:26:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.722+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b00d2e30 con 0x7f02b00d0d60 2024-04-25T15:26:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.722+0000 7f0297fff640 1 -- 172.21.15.67:0/1726363299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f02a4030b50 con 0x7f02b00d0d60 2024-04-25T15:26:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.723+0000 7f0297fff640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f029c0744e0 0x7f029c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.723+0000 7f0297fff640 1 -- 172.21.15.67:0/1726363299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5227+0+0 (secure 0 0 0) 0x7f02a40b56f0 con 0x7f02b00d0d60 2024-04-25T15:26:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.723+0000 7f02af7fe640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f029c0744e0 0x7f029c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.724+0000 7f02af7fe640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f029c0744e0 0x7f029c0769f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009aa0 tx=0x7f02a0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:19.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.728+0000 7f0297fff640 1 -- 172.21.15.67:0/1726363299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f02b01fa110 con 0x7f02b00d0d60 2024-04-25T15:26:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:19.914+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f02b0076220 con 0x7f02b00d0d60 2024-04-25T15:26:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.577+0000 7f0297fff640 1 -- 172.21.15.67:0/1726363299 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) ==== 170+0+0 (secure 0 0 0) 0x7f02a407faa0 con 0x7f02b00d0d60 2024-04-25T15:26:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:26:20.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.581+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f029c0744e0 msgr2=0x7f029c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.581+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f029c0744e0 0x7f029c0769f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009aa0 tx=0x7f02a0009190 comp rx=0 tx=0).stop 2024-04-25T15:26:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.581+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00d0d60 msgr2=0x7f02b00ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.581+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00d0d60 0x7f02b00ce2c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f02a4000c00 tx=0x7f02a40046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.582+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 shutdown_connections 2024-04-25T15:26:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.582+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f029c0744e0 0x7f029c0769f0 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009aa0 tx=0x7f02a0009190 comp rx=0 tx=0).stop 2024-04-25T15:26:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.582+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b01f97c0 0x7f02b01f9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.582+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00d0d60 0x7f02b00ce2c0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f02a4000c00 tx=0x7f02a40046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.582+0000 7f02b5cd8640 1 --2- 172.21.15.67:0/1726363299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b00b77d0 0x7f02b015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.582+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 >> 172.21.15.67:0/1726363299 conn(0x7f02b009b7d0 msgr2=0x7f02b0070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.584+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 shutdown_connections 2024-04-25T15:26:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.584+0000 7f02b5cd8640 1 -- 172.21.15.67:0/1726363299 wait complete. 2024-04-25T15:26:20.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:26:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.732+0000 7fa264220640 1 Processor -- start 2024-04-25T15:26:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.732+0000 7fa264220640 1 -- start start 2024-04-25T15:26:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.733+0000 7fa264220640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa25c06ccd0 0x7fa25c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.733+0000 7fa264220640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25c1609b0 con 0x7fa25c163fa0 2024-04-25T15:26:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.733+0000 7fa264220640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa25c164cd0 con 0x7fa25c161710 2024-04-25T15:26:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.733+0000 7fa264220640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa25c160c30 con 0x7fa25c06ccd0 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.733+0000 7fa2627bd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa25c163fa0 0x7fa25c1643b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57684/0 (socket says 172.21.15.67:57684) 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.733+0000 7fa2627bd640 1 -- 172.21.15.67:0/1848107169 learned_addr learned my addr 172.21.15.67:0/1848107169 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa2617bb640 1 --2- 172.21.15.67:0/1848107169 >> v2:172.21.15.176:3300/0 conn(0x7fa25c06ccd0 0x7fa25c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2601788448 0 0) 0x7fa254001490 con 0x7fa25c163fa0 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa248003520 con 0x7fa25c163fa0 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1694109112 0 0) 0x7fa25c164cd0 con 0x7fa25c161710 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa254001490 con 0x7fa25c161710 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2318650273 0 0) 0x7fa248003520 con 0x7fa25c163fa0 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 >> v2:172.21.15.176:3300/0 conn(0x7fa25c06ccd0 msgr2=0x7fa25c15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 --2- 172.21.15.67:0/1848107169 >> v2:172.21.15.176:3300/0 conn(0x7fa25c06ccd0 0x7fa25c15e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 >> v1:172.21.15.176:6789/0 conn(0x7fa25c161710 legacy=0x7fa25c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25c160f10 con 0x7fa25c163fa0 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fa254001bc0 con 0x7fa25c163fa0 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 >> v1:172.21.15.67:6789/0 conn(0x7fa25c163fa0 legacy=0x7fa25c1643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa248003b40 0x7fa248003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa248004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa248005260 0x7fa24800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c160c30 con 0x7fa248004800 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa24800c1c0 con 0x7fa248005260 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa24800c3c0 con 0x7fa248003b40 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa2617bb640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa248004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa261fbc640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa248003b40 0x7fa248003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.734+0000 7fa2627bd640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa248005260 0x7fa24800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa2617bb640 1 -- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa248003b40 msgr2=0x7fa248003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa2617bb640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa248003b40 0x7fa248003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa2617bb640 1 -- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa248005260 msgr2=0x7fa24800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa2617bb640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa248005260 0x7fa24800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa2617bb640 1 -- 172.21.15.67:0/1848107169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa248003520 con 0x7fa248004800 2024-04-25T15:26:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa2627bd640 1 -- 172.21.15.67:0/1848107169 reap_dead start 2024-04-25T15:26:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa2617bb640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa248004c10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009ac0 tx=0x7fa248005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33ec3104df94ddb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c036070 con 0x7fa248004800 2024-04-25T15:26:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa24c031400 con 0x7fa248004800 2024-04-25T15:26:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.735+0000 7fa260fba640 1 -- 172.21.15.67:0/1848107169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c009690 con 0x7fa248004800 2024-04-25T15:26:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.736+0000 7fa264220640 1 -- 172.21.15.67:0/1848107169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 msgr2=0x7fa248004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.736+0000 7fa264220640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa248004c10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009ac0 tx=0x7fa248005830 comp rx=0 tx=0).stop 2024-04-25T15:26:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.736+0000 7fa264220640 1 -- 172.21.15.67:0/1848107169 shutdown_connections 2024-04-25T15:26:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.736+0000 7fa264220640 1 --2- 172.21.15.67:0/1848107169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa248004c10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.736+0000 7fa264220640 1 -- 172.21.15.67:0/1848107169 >> 172.21.15.67:0/1848107169 conn(0x7fa25c0d5d90 msgr2=0x7fa25c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.736+0000 7fa264220640 1 -- 172.21.15.67:0/1848107169 shutdown_connections 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.736+0000 7fa264220640 1 -- 172.21.15.67:0/1848107169 wait complete. 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.738+0000 7fa264220640 1 Processor -- start 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.738+0000 7fa264220640 1 -- start start 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa264220640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa25c0b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa264220640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa25c14a7b0 0x7fa25c158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa264220640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa25c157380 0x7fa25c157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa264220640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c15e200 con 0x7fa248004800 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa264220640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa25c15e080 con 0x7fa25c14a7b0 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa264220640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa25c15e380 con 0x7fa25c157380 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa261fbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa25c0b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa261fbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa25c0b7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41462/0 (socket says 172.21.15.67:41462) 2024-04-25T15:26:20.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.739+0000 7fa261fbc640 1 -- 172.21.15.67:0/2448956604 learned_addr learned my addr 172.21.15.67:0/2448956604 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa2627bd640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa25c157380 0x7fa25c157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa2617bb640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa25c14a7b0 0x7fa25c158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa261fbc640 1 -- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa25c157380 msgr2=0x7fa25c157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa261fbc640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa25c157380 0x7fa25c157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa261fbc640 1 -- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa25c14a7b0 msgr2=0x7fa25c158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa261fbc640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa25c14a7b0 0x7fa25c158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa261fbc640 1 -- 172.21.15.67:0/2448956604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25c0b72f0 con 0x7fa248004800 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa2627bd640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa25c157380 0x7fa25c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa261fbc640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa25c0b7d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa25800f7a0 tx=0x7fa25800fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa2527fc640 1 -- 172.21.15.67:0/2448956604 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25800fe50 con 0x7fa248004800 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa2527fc640 1 -- 172.21.15.67:0/2448956604 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa258018070 con 0x7fa248004800 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25c0cfda0 con 0x7fa248004800 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa2527fc640 1 -- 172.21.15.67:0/2448956604 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25800d730 con 0x7fa248004800 2024-04-25T15:26:20.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.740+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa25c0d0080 con 0x7fa248004800 2024-04-25T15:26:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.741+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa25c050740 con 0x7fa248004800 2024-04-25T15:26:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.742+0000 7fa2527fc640 1 -- 172.21.15.67:0/2448956604 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa25800d8d0 con 0x7fa248004800 2024-04-25T15:26:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.743+0000 7fa2527fc640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2400744b0 0x7fa2400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.743+0000 7fa2527fc640 1 -- 172.21.15.67:0/2448956604 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5609+0+0 (secure 0 0 0) 0x7fa2580950d0 con 0x7fa248004800 2024-04-25T15:26:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.743+0000 7fa2617bb640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2400744b0 0x7fa2400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.743+0000 7fa2617bb640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2400744b0 0x7fa2400769c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa24c033000 tx=0x7fa24c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:20.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.748+0000 7fa2527fc640 1 -- 172.21.15.67:0/2448956604 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa25805f8b0 con 0x7fa248004800 2024-04-25T15:26:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:20.973+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa25c15cd80 con 0x7fa248004800 2024-04-25T15:26:21.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.595+0000 7fa2527fc640 1 -- 172.21.15.67:0/2448956604 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) ==== 172+0+0 (secure 0 0 0) 0x7fa25805f1b0 con 0x7fa248004800 2024-04-25T15:26:21.596 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 4 and data pool 5 2024-04-25T15:26:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2400744b0 msgr2=0x7fa2400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2400744b0 0x7fa2400769c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa24c033000 tx=0x7fa24c008030 comp rx=0 tx=0).stop 2024-04-25T15:26:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 msgr2=0x7fa25c0b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa25c0b7d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa25800f7a0 tx=0x7fa25800fc60 comp rx=0 tx=0).stop 2024-04-25T15:26:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 shutdown_connections 2024-04-25T15:26:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2400744b0 0x7fa2400769c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa25c157380 0x7fa25c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa25c14a7b0 0x7fa25c158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.603+0000 7fa264220640 1 --2- 172.21.15.67:0/2448956604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa248004800 0x7fa25c0b7d90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.604+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 >> 172.21.15.67:0/2448956604 conn(0x7fa25c0d5d90 msgr2=0x7fa25c15e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.605+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 shutdown_connections 2024-04-25T15:26:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.606+0000 7fa264220640 1 -- 172.21.15.67:0/2448956604 wait complete. 2024-04-25T15:26:21.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.747+0000 7f4c79468640 1 Processor -- start 2024-04-25T15:26:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.747+0000 7f4c79468640 1 -- start start 2024-04-25T15:26:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c79468640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 0x7f4c740d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c79468640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c740d1850 con 0x7f4c7406c5c0 2024-04-25T15:26:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c79468640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c74160de0 con 0x7f4c740d4a50 2024-04-25T15:26:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c79468640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c741610c0 con 0x7f4c740d0e00 2024-04-25T15:26:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c727fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c7406c5c0 0x7f4c7406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57694/0 (socket says 172.21.15.67:57694) 2024-04-25T15:26:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c727fc640 1 -- 172.21.15.67:0/2309398886 learned_addr learned my addr 172.21.15.67:0/2309398886 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 0x7f4c740d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4123689268 0 0) 0x7f4c740d1850 con 0x7f4c7406c5c0 2024-04-25T15:26:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c60003520 con 0x7f4c7406c5c0 2024-04-25T15:26:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3703680025 0 0) 0x7f4c60003520 con 0x7f4c7406c5c0 2024-04-25T15:26:21.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 msgr2=0x7f4c740d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 0x7f4c740d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 >> v1:172.21.15.176:6789/0 conn(0x7f4c740d4a50 legacy=0x7f4c740d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c741613a0 con 0x7f4c7406c5c0 2024-04-25T15:26:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.749+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f4c5c001670 con 0x7f4c7406c5c0 2024-04-25T15:26:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 >> v1:172.21.15.67:6789/0 conn(0x7f4c7406c5c0 legacy=0x7f4c7406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c71ffb640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600039e0 0x7f4c60003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c71ffb640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 0x7f4c60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c71ffb640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c60004d40 0x7f4c60005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c741610c0 con 0x7f4c600039e0 2024-04-25T15:26:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6000ba70 con 0x7f4c60004d40 2024-04-25T15:26:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c6000bc70 con 0x7f4c740d0e00 2024-04-25T15:26:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600039e0 0x7f4c60003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c727fc640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 0x7f4c60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c737fe640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c60004d40 0x7f4c60005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c72ffd640 1 -- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 msgr2=0x7f4c60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c740d0e00 0x7f4c60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c72ffd640 1 -- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c60004d40 msgr2=0x7f4c60005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c60004d40 0x7f4c60005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.750+0000 7f4c72ffd640 1 -- 172.21.15.67:0/2309398886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c60003520 con 0x7f4c600039e0 2024-04-25T15:26:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.751+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600039e0 0x7f4c60003e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009fc0 tx=0x7f4c60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b205d4e2f1a2766a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.751+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64034020 con 0x7f4c600039e0 2024-04-25T15:26:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.751+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c640305f0 con 0x7f4c600039e0 2024-04-25T15:26:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.751+0000 7f4c71ffb640 1 -- 172.21.15.67:0/2309398886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64009c10 con 0x7f4c600039e0 2024-04-25T15:26:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.751+0000 7f4c79468640 1 -- 172.21.15.67:0/2309398886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600039e0 msgr2=0x7f4c60003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.751+0000 7f4c79468640 1 --2- 172.21.15.67:0/2309398886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c600039e0 0x7f4c60003e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009fc0 tx=0x7f4c60005830 comp rx=0 tx=0).stop 2024-04-25T15:26:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.751+0000 7f4c737fe640 1 -- 172.21.15.67:0/2309398886 reap_dead start 2024-04-25T15:26:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.752+0000 7f4c79468640 1 -- 172.21.15.67:0/2309398886 shutdown_connections 2024-04-25T15:26:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.752+0000 7f4c79468640 1 -- 172.21.15.67:0/2309398886 >> 172.21.15.67:0/2309398886 conn(0x7f4c740d5d90 msgr2=0x7f4c740d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.752+0000 7f4c79468640 1 -- 172.21.15.67:0/2309398886 shutdown_connections 2024-04-25T15:26:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.752+0000 7f4c79468640 1 -- 172.21.15.67:0/2309398886 wait complete. 2024-04-25T15:26:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.753+0000 7f4c79468640 1 Processor -- start 2024-04-25T15:26:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.754+0000 7f4c79468640 1 -- start start 2024-04-25T15:26:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c79468640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c600039e0 0x7f4c740cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c79468640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c74159280 0x7f4c741fa350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c79468640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c741fa890 0x7f4c74148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c79468640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c74164e00 con 0x7f4c741fa890 2024-04-25T15:26:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c79468640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4c740b5320 con 0x7f4c74159280 2024-04-25T15:26:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c79468640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c74164f80 con 0x7f4c600039e0 2024-04-25T15:26:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c741fa890 0x7f4c74148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c741fa890 0x7f4c74148cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41480/0 (socket says 172.21.15.67:41480) 2024-04-25T15:26:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 -- 172.21.15.67:0/3767558853 learned_addr learned my addr 172.21.15.67:0/3767558853 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c600039e0 0x7f4c740cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c727fc640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c74159280 0x7f4c741fa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 -- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c600039e0 msgr2=0x7f4c740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c600039e0 0x7f4c740cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 -- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c74159280 msgr2=0x7f4c741fa350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c74159280 0x7f4c741fa350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 -- 172.21.15.67:0/3767558853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7415c200 con 0x7f4c741fa890 2024-04-25T15:26:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.755+0000 7f4c737fe640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c741fa890 0x7f4c74148cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4c740d1780 tx=0x7f4c6c00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.756+0000 7f4c6b7fe640 1 -- 172.21.15.67:0/3767558853 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c019070 con 0x7f4c741fa890 2024-04-25T15:26:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.756+0000 7f4c6b7fe640 1 -- 172.21.15.67:0/3767558853 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c6c005030 con 0x7f4c741fa890 2024-04-25T15:26:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.756+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7415c4e0 con 0x7f4c741fa890 2024-04-25T15:26:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.756+0000 7f4c6b7fe640 1 -- 172.21.15.67:0/3767558853 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c005420 con 0x7f4c741fa890 2024-04-25T15:26:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.756+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c740cedd0 con 0x7f4c741fa890 2024-04-25T15:26:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.757+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c74050740 con 0x7f4c741fa890 2024-04-25T15:26:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.757+0000 7f4c6b7fe640 1 -- 172.21.15.67:0/3767558853 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4c6c01c070 con 0x7f4c741fa890 2024-04-25T15:26:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.758+0000 7f4c6b7fe640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c540744b0 0x7f4c540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.758+0000 7f4c6b7fe640 1 -- 172.21.15.67:0/3767558853 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f4c6c0940d0 con 0x7f4c741fa890 2024-04-25T15:26:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.758+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c540744b0 0x7f4c540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.759+0000 7f4c72ffd640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c540744b0 0x7f4c540769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4c64035000 tx=0x7f4c6402f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:21.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.763+0000 7f4c6b7fe640 1 -- 172.21.15.67:0/3767558853 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c6c05e210 con 0x7f4c741fa890 2024-04-25T15:26:21.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.951+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c7415bf90 con 0x7f4c741fa890 2024-04-25T15:26:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.953+0000 7f4c6b7fe640 1 -- 172.21.15.67:0/3767558853 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16646 (secure 0 0 0) 0x7f4c6c062510 con 0x7f4c741fa890 2024-04-25T15:26:21.954 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:21.954 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:20.974309+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c540744b0 msgr2=0x7f4c540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c540744b0 0x7f4c540769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4c64035000 tx=0x7f4c6402f000 comp rx=0 tx=0).stop 2024-04-25T15:26:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c741fa890 msgr2=0x7f4c74148cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c741fa890 0x7f4c74148cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4c740d1780 tx=0x7f4c6c00fc00 comp rx=0 tx=0).stop 2024-04-25T15:26:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 shutdown_connections 2024-04-25T15:26:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c540744b0 0x7f4c540769c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c741fa890 0x7f4c74148cc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c74159280 0x7f4c741fa350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 --2- 172.21.15.67:0/3767558853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c600039e0 0x7f4c740cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 >> 172.21.15.67:0/3767558853 conn(0x7f4c740d5d90 msgr2=0x7f4c740bb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 shutdown_connections 2024-04-25T15:26:21.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:21.955+0000 7f4c79468640 1 -- 172.21.15.67:0/3767558853 wait complete. 2024-04-25T15:26:21.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:22.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.080+0000 7f04f31d7640 1 Processor -- start 2024-04-25T15:26:22.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.080+0000 7f04f31d7640 1 -- start start 2024-04-25T15:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04f31d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cfd40 0x7f04ec0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04f31d7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f04ec0b7f50 0x7f04ec0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04f31d7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ec0d1c10 con 0x7f04ec0b3e00 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04f31d7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f04ec0d27e0 con 0x7f04ec0b7f50 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04f31d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec0d1eb0 con 0x7f04ec0cfd40 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04ebfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f04ec0b3e00 0x7f04ec0b7840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57704/0 (socket says 172.21.15.67:57704) 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04ebfff640 1 -- 172.21.15.67:0/1558313025 learned_addr learned my addr 172.21.15.67:0/1558313025 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04f0f73640 1 --2- 172.21.15.67:0/1558313025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cfd40 0x7f04ec0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04f1774640 1 --2- 172.21.15.67:0/1558313025 >> v2:172.21.15.176:3300/0 conn(0x7f04ec0b7f50 0x7f04ec0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04eb7fe640 1 -- 172.21.15.67:0/1558313025 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994818102 0 0) 0x7f04ec0d1c10 con 0x7f04ec0b3e00 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.081+0000 7f04eb7fe640 1 -- 172.21.15.67:0/1558313025 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04d4003520 con 0x7f04ec0b3e00 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.082+0000 7f04f0f73640 1 -- 172.21.15.67:0/1558313025 >> v2:172.21.15.176:3300/0 conn(0x7f04ec0b7f50 msgr2=0x7f04ec0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.082+0000 7f04f0f73640 1 --2- 172.21.15.67:0/1558313025 >> v2:172.21.15.176:3300/0 conn(0x7f04ec0b7f50 0x7f04ec0ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.082+0000 7f04f0f73640 1 -- 172.21.15.67:0/1558313025 >> v1:172.21.15.67:6789/0 conn(0x7f04ec0b3e00 legacy=0x7f04ec0b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.082+0000 7f04f0f73640 1 -- 172.21.15.67:0/1558313025 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec0d2190 con 0x7f04ec0cfd40 2024-04-25T15:26:22.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.082+0000 7f04f1774640 1 --2- 172.21.15.67:0/1558313025 >> v2:172.21.15.176:3300/0 conn(0x7f04ec0b7f50 0x7f04ec0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:22.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.082+0000 7f04f0f73640 1 --2- 172.21.15.67:0/1558313025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cfd40 0x7f04ec0b21a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0d2960 tx=0x7f04e402a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91f3b22be4a065f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:22.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04eb7fe640 1 -- 172.21.15.67:0/1558313025 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e402ae60 con 0x7f04ec0cfd40 2024-04-25T15:26:22.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04eb7fe640 1 -- 172.21.15.67:0/1558313025 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04e4007740 con 0x7f04ec0cfd40 2024-04-25T15:26:22.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04eb7fe640 1 -- 172.21.15.67:0/1558313025 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4007af0 con 0x7f04ec0cfd40 2024-04-25T15:26:22.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 -- 172.21.15.67:0/1558313025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cfd40 msgr2=0x7f04ec0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 --2- 172.21.15.67:0/1558313025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cfd40 0x7f04ec0b21a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0d2960 tx=0x7f04e402a640 comp rx=0 tx=0).stop 2024-04-25T15:26:22.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 -- 172.21.15.67:0/1558313025 shutdown_connections 2024-04-25T15:26:22.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 --2- 172.21.15.67:0/1558313025 >> v2:172.21.15.176:3300/0 conn(0x7f04ec0b7f50 0x7f04ec0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 --2- 172.21.15.67:0/1558313025 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cfd40 0x7f04ec0b21a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 -- 172.21.15.67:0/1558313025 >> 172.21.15.67:0/1558313025 conn(0x7f04ec09b7d0 msgr2=0x7f04ec09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 -- 172.21.15.67:0/1558313025 shutdown_connections 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.083+0000 7f04f31d7640 1 -- 172.21.15.67:0/1558313025 wait complete. 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.085+0000 7f04f31d7640 1 Processor -- start 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.085+0000 7f04f31d7640 1 -- start start 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f31d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0b3e00 0x7f04ec0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f31d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04ec0cfd40 0x7f04ec0b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f31d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cb800 0x7f04ec09f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f31d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec0b4210 con 0x7f04ec0b3e00 2024-04-25T15:26:22.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f31d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec0ce650 con 0x7f04ec0cfd40 2024-04-25T15:26:22.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f31d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec0b4390 con 0x7f04ec0cb800 2024-04-25T15:26:22.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0b3e00 0x7f04ec0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0b3e00 0x7f04ec0b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41492/0 (socket says 172.21.15.67:41492) 2024-04-25T15:26:22.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 -- 172.21.15.67:0/2247566297 learned_addr learned my addr 172.21.15.67:0/2247566297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:22.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f1774640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cb800 0x7f04ec09f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04ebfff640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04ec0cfd40 0x7f04ec0b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 -- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cb800 msgr2=0x7f04ec09f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cb800 0x7f04ec09f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 -- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04ec0cfd40 msgr2=0x7f04ec0b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04ec0cfd40 0x7f04ec0b1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 -- 172.21.15.67:0/2247566297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec0cbee0 con 0x7f04ec0b3e00 2024-04-25T15:26:22.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04ebfff640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04ec0cfd40 0x7f04ec0b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:22.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.087+0000 7f04f0f73640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0b3e00 0x7f04ec0b19b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f04e4000f80 tx=0x7f04e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:22.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.088+0000 7f04e97fa640 1 -- 172.21.15.67:0/2247566297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4003e10 con 0x7f04ec0b3e00 2024-04-25T15:26:22.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.088+0000 7f04e97fa640 1 -- 172.21.15.67:0/2247566297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04e4004250 con 0x7f04ec0b3e00 2024-04-25T15:26:22.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.088+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec0cc1c0 con 0x7f04ec0b3e00 2024-04-25T15:26:22.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.088+0000 7f04e97fa640 1 -- 172.21.15.67:0/2247566297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4004400 con 0x7f04ec0b3e00 2024-04-25T15:26:22.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.088+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec0cc780 con 0x7f04ec0b3e00 2024-04-25T15:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.089+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec070370 con 0x7f04ec0b3e00 2024-04-25T15:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.089+0000 7f04e97fa640 1 -- 172.21.15.67:0/2247566297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f04e40045a0 con 0x7f04ec0b3e00 2024-04-25T15:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.090+0000 7f04e97fa640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04d80744b0 0x7f04d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.090+0000 7f04e97fa640 1 -- 172.21.15.67:0/2247566297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f04e40b65d0 con 0x7f04ec0b3e00 2024-04-25T15:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.090+0000 7f04ebfff640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04d80744b0 0x7f04d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.090+0000 7f04ebfff640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04d80744b0 0x7f04d80769c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f04e0007fd0 tx=0x7f04e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.094+0000 7f04e97fa640 1 -- 172.21.15.67:0/2247566297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04ec070370 con 0x7f04ec0b3e00 2024-04-25T15:26:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.282+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04ec050740 con 0x7f04ec0b3e00 2024-04-25T15:26:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.283+0000 7f04e97fa640 1 -- 172.21.15.67:0/2247566297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16646 (secure 0 0 0) 0x7f04e403f030 con 0x7f04ec0b3e00 2024-04-25T15:26:22.284 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:22.285 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:20.974309+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04d80744b0 msgr2=0x7f04d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04d80744b0 0x7f04d80769c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f04e0007fd0 tx=0x7f04e0009040 comp rx=0 tx=0).stop 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0b3e00 msgr2=0x7f04ec0b19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0b3e00 0x7f04ec0b19b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f04e4000f80 tx=0x7f04e4004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 shutdown_connections 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04d80744b0 0x7f04d80769c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04ec0cb800 0x7f04ec09f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04ec0cfd40 0x7f04ec0b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 --2- 172.21.15.67:0/2247566297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0b3e00 0x7f04ec0b19b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 >> 172.21.15.67:0/2247566297 conn(0x7f04ec09b7d0 msgr2=0x7f04ec0b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 shutdown_connections 2024-04-25T15:26:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.285+0000 7f04f31d7640 1 -- 172.21.15.67:0/2247566297 wait complete. 2024-04-25T15:26:22.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:26:22.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.413+0000 7f2b21e13640 1 Processor -- start 2024-04-25T15:26:22.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.414+0000 7f2b21e13640 1 -- start start 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b21e13640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2b1c0d1e20 0x7f2b1c0d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b21e13640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c164020 0x7f2b1c164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b21e13640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c164970 con 0x7f2b1c0b2e60 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b21e13640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2b1c0d23f0 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b21e13640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c0d26d0 con 0x7f2b1c164020 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b1bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c164020 0x7f2b1c164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b1b7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2b1c0d1e20 0x7f2b1c0d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b1bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c164020 0x7f2b1c164430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:37160/0 (socket says 172.21.15.67:37160) 2024-04-25T15:26:22.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b1bfff640 1 -- 172.21.15.67:0/347655659 learned_addr learned my addr 172.21.15.67:0/347655659 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b1bfff640 1 -- 172.21.15.67:0/347655659 >> v2:172.21.15.176:3300/0 conn(0x7f2b1c0d1e20 msgr2=0x7f2b1c0d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b1bfff640 1 --2- 172.21.15.67:0/347655659 >> v2:172.21.15.176:3300/0 conn(0x7f2b1c0d1e20 0x7f2b1c0d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.415+0000 7f2b1bfff640 1 -- 172.21.15.67:0/347655659 >> v1:172.21.15.176:6789/0 conn(0x7f2b1c0b2e60 legacy=0x7f2b1c08d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.416+0000 7f2b1bfff640 1 -- 172.21.15.67:0/347655659 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c0d29b0 con 0x7f2b1c164020 2024-04-25T15:26:22.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.416+0000 7f2b1bfff640 1 --2- 172.21.15.67:0/347655659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c164020 0x7f2b1c164430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009e50 tx=0x7f2b1402a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16334327262eb6b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:22.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b1a7fc640 1 -- 172.21.15.67:0/347655659 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b14036070 con 0x7f2b1c164020 2024-04-25T15:26:22.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b1a7fc640 1 -- 172.21.15.67:0/347655659 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b14031600 con 0x7f2b1c164020 2024-04-25T15:26:22.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b1a7fc640 1 -- 172.21.15.67:0/347655659 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b140319d0 con 0x7f2b1c164020 2024-04-25T15:26:22.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 -- 172.21.15.67:0/347655659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c164020 msgr2=0x7f2b1c164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 --2- 172.21.15.67:0/347655659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c164020 0x7f2b1c164430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009e50 tx=0x7f2b1402a7a0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 -- 172.21.15.67:0/347655659 shutdown_connections 2024-04-25T15:26:22.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 --2- 172.21.15.67:0/347655659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c164020 0x7f2b1c164430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 --2- 172.21.15.67:0/347655659 >> v2:172.21.15.176:3300/0 conn(0x7f2b1c0d1e20 0x7f2b1c0d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 -- 172.21.15.67:0/347655659 >> 172.21.15.67:0/347655659 conn(0x7f2b1c0d0530 msgr2=0x7f2b1c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:22.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 -- 172.21.15.67:0/347655659 shutdown_connections 2024-04-25T15:26:22.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.417+0000 7f2b21e13640 1 -- 172.21.15.67:0/347655659 wait complete. 2024-04-25T15:26:22.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.419+0000 7f2b21e13640 1 Processor -- start 2024-04-25T15:26:22.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.419+0000 7f2b21e13640 1 -- start start 2024-04-25T15:26:22.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.420+0000 7f2b21e13640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c0b2e60 0x7f2b1c06e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.420+0000 7f2b21e13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0d1e20 0x7f2b1c1f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.420+0000 7f2b21e13640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b1c164020 0x7f2b1c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.420+0000 7f2b21e13640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0ce070 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b21e13640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0d2f70 con 0x7f2b1c164020 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b21e13640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c0ce1f0 con 0x7f2b1c0b2e60 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0d1e20 0x7f2b1c1f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0d1e20 0x7f2b1c1f5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41496/0 (socket says 172.21.15.67:41496) 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 -- 172.21.15.67:0/3025272787 learned_addr learned my addr 172.21.15.67:0/3025272787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c0b2e60 0x7f2b1c06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1bfff640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b1c164020 0x7f2b1c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 -- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c0b2e60 msgr2=0x7f2b1c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c0b2e60 0x7f2b1c06e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 -- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b1c164020 msgr2=0x7f2b1c15ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:22.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b1c164020 0x7f2b1c15ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:22.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 -- 172.21.15.67:0/3025272787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c15f630 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b1affd640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0d1e20 0x7f2b1c1f5fe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c3c0 tx=0x7f2b1000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:22.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b03fff640 1 -- 172.21.15.67:0/3025272787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b100052d0 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b03fff640 1 -- 172.21.15.67:0/3025272787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b10005490 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b03fff640 1 -- 172.21.15.67:0/3025272787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10007260 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.421+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c1f9e10 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.422+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c1fa0d0 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.422+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c15f800 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.423+0000 7f2b03fff640 1 -- 172.21.15.67:0/3025272787 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2b10007400 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.424+0000 7f2b03fff640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2b080744b0 0x7f2b080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:22.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.424+0000 7f2b03fff640 1 -- 172.21.15.67:0/3025272787 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f2b100945b0 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.424+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2b080744b0 0x7f2b080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:22.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.424+0000 7f2b1b7fe640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2b080744b0 0x7f2b080769c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009aa0 tx=0x7f2b0c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:22.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.428+0000 7f2b03fff640 1 -- 172.21.15.67:0/3025272787 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b1c15f800 con 0x7f2b1c0d1e20 2024-04-25T15:26:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:22.654+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2b1c050740 con 0x7f2b1c0d1e20 2024-04-25T15:26:23.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.607+0000 7f2b03fff640 1 -- 172.21.15.67:0/3025272787 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f2b1c164970 con 0x7f2b1c0d1e20 2024-04-25T15:26:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2b080744b0 msgr2=0x7f2b080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2b080744b0 0x7f2b080769c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009aa0 tx=0x7f2b0c009190 comp rx=0 tx=0).stop 2024-04-25T15:26:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0d1e20 msgr2=0x7f2b1c1f5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0d1e20 0x7f2b1c1f5fe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c3c0 tx=0x7f2b1000c880 comp rx=0 tx=0).stop 2024-04-25T15:26:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 shutdown_connections 2024-04-25T15:26:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2b080744b0 0x7f2b080769c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b1c164020 0x7f2b1c15ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b1c0d1e20 0x7f2b1c1f5fe0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 --2- 172.21.15.67:0/3025272787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b1c0b2e60 0x7f2b1c06e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 >> 172.21.15.67:0/3025272787 conn(0x7f2b1c0d0530 msgr2=0x7f2b1c0d3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:23.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 shutdown_connections 2024-04-25T15:26:23.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.610+0000 7f2b21e13640 1 -- 172.21.15.67:0/3025272787 wait complete. 2024-04-25T15:26:23.625 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:26:23.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:26:23.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.746+0000 7ff762372640 1 Processor -- start 2024-04-25T15:26:23.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.746+0000 7ff762372640 1 -- start start 2024-04-25T15:26:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff762372640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff75c164ad0 0x7ff75c0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff762372640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c0b0a70 0x7ff75c0b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff762372640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75c0d3170 con 0x7ff75c163f30 2024-04-25T15:26:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff762372640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff75c0d2f40 con 0x7ff75c164ad0 2024-04-25T15:26:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff762372640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff75c164ee0 con 0x7ff75c0b0a70 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff75bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff75c164ad0 0x7ff75c0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff75bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff75c164ad0 0x7ff75c0b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41498/0 (socket says 172.21.15.67:41498) 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff75bfff640 1 -- 172.21.15.67:0/1276808165 learned_addr learned my addr 172.21.15.67:0/1276808165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff76090f640 1 --2- 172.21.15.67:0/1276808165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c0b0a70 0x7ff75c0b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff75bfff640 1 -- 172.21.15.67:0/1276808165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c0b0a70 msgr2=0x7ff75c0b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff75bfff640 1 --2- 172.21.15.67:0/1276808165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c0b0a70 0x7ff75c0b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff75bfff640 1 -- 172.21.15.67:0/1276808165 >> v1:172.21.15.176:6789/0 conn(0x7ff75c163f30 legacy=0x7ff75c164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.748+0000 7ff75bfff640 1 -- 172.21.15.67:0/1276808165 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c0d3450 con 0x7ff75c164ad0 2024-04-25T15:26:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.749+0000 7ff75bfff640 1 --2- 172.21.15.67:0/1276808165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c164ad0 0x7ff75c0b2ca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff750009870 tx=0x7ff75002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1171c937c4dc25bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:23.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.749+0000 7ff75affd640 1 -- 172.21.15.67:0/1276808165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750037070 con 0x7ff75c164ad0 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.749+0000 7ff75affd640 1 -- 172.21.15.67:0/1276808165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff750032440 con 0x7ff75c164ad0 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.749+0000 7ff75affd640 1 -- 172.21.15.67:0/1276808165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750032810 con 0x7ff75c164ad0 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 -- 172.21.15.67:0/1276808165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c164ad0 msgr2=0x7ff75c0b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 --2- 172.21.15.67:0/1276808165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c164ad0 0x7ff75c0b2ca0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff750009870 tx=0x7ff75002a270 comp rx=0 tx=0).stop 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 -- 172.21.15.67:0/1276808165 shutdown_connections 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 --2- 172.21.15.67:0/1276808165 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c0b0a70 0x7ff75c0b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 --2- 172.21.15.67:0/1276808165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c164ad0 0x7ff75c0b2ca0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 -- 172.21.15.67:0/1276808165 >> 172.21.15.67:0/1276808165 conn(0x7ff75c0d3aa0 msgr2=0x7ff75c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 -- 172.21.15.67:0/1276808165 shutdown_connections 2024-04-25T15:26:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.750+0000 7ff762372640 1 -- 172.21.15.67:0/1276808165 wait complete. 2024-04-25T15:26:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.752+0000 7ff762372640 1 Processor -- start 2024-04-25T15:26:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.752+0000 7ff762372640 1 -- start start 2024-04-25T15:26:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff762372640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c0b0a70 0x7ff75c1f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff762372640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff75c163f30 0x7ff75c1fa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff762372640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c164ad0 0x7ff75c1fa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff762372640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c06d9b0 con 0x7ff75c0b0a70 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff762372640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff75c0b31e0 con 0x7ff75c163f30 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff762372640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff75c0b9430 con 0x7ff75c164ad0 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff75bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c0b0a70 0x7ff75c1f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff75bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c0b0a70 0x7ff75c1f6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41514/0 (socket says 172.21.15.67:41514) 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff75bfff640 1 -- 172.21.15.67:0/2400988810 learned_addr learned my addr 172.21.15.67:0/2400988810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff76090f640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c164ad0 0x7ff75c1fa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff75b7fe640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff75c163f30 0x7ff75c1fa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.753+0000 7ff75bfff640 1 -- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c164ad0 msgr2=0x7ff75c1fa8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff75bfff640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c164ad0 0x7ff75c1fa8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff75bfff640 1 -- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff75c163f30 msgr2=0x7ff75c1fa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:23.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff75bfff640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff75c163f30 0x7ff75c1fa360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff75bfff640 1 -- 172.21.15.67:0/2400988810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75c1fb0b0 con 0x7ff75c0b0a70 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff76090f640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c164ad0 0x7ff75c1fa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff75bfff640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c0b0a70 0x7ff75c1f6340 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff750000c00 tx=0x7ff7500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff758ff9640 1 -- 172.21.15.67:0/2400988810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750041020 con 0x7ff75c0b0a70 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff758ff9640 1 -- 172.21.15.67:0/2400988810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7500306c0 con 0x7ff75c0b0a70 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff758ff9640 1 -- 172.21.15.67:0/2400988810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff750037040 con 0x7ff75c0b0a70 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75c0b34a0 con 0x7ff75c0b0a70 2024-04-25T15:26:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.754+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75c0b3a60 con 0x7ff75c0b0a70 2024-04-25T15:26:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.755+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff75c0c8cd0 con 0x7ff75c0b0a70 2024-04-25T15:26:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.756+0000 7ff758ff9640 1 -- 172.21.15.67:0/2400988810 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff750044020 con 0x7ff75c0b0a70 2024-04-25T15:26:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.756+0000 7ff758ff9640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff7480744b0 0x7ff7480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:23.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.757+0000 7ff758ff9640 1 -- 172.21.15.67:0/2400988810 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff7500b5ac0 con 0x7ff75c0b0a70 2024-04-25T15:26:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.757+0000 7ff75b7fe640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff7480744b0 0x7ff7480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.757+0000 7ff75b7fe640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff7480744b0 0x7ff7480769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff74c007fd0 tx=0x7ff74c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.761+0000 7ff758ff9640 1 -- 172.21.15.67:0/2400988810 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff75004b030 con 0x7ff75c0b0a70 2024-04-25T15:26:23.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:23.986+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff75c0d4770 con 0x7ff75c0b0a70 2024-04-25T15:26:24.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.616+0000 7ff758ff9640 1 -- 172.21.15.67:0/2400988810 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7ff750080250 con 0x7ff75c0b0a70 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff7480744b0 msgr2=0x7ff7480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff7480744b0 0x7ff7480769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff74c007fd0 tx=0x7ff74c009040 comp rx=0 tx=0).stop 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c0b0a70 msgr2=0x7ff75c1f6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c0b0a70 0x7ff75c1f6340 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff750000c00 tx=0x7ff7500046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 shutdown_connections 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff7480744b0 0x7ff7480769c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff75c164ad0 0x7ff75c1fa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff75c163f30 0x7ff75c1fa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 --2- 172.21.15.67:0/2400988810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff75c0b0a70 0x7ff75c1f6340 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 >> 172.21.15.67:0/2400988810 conn(0x7ff75c0d3aa0 msgr2=0x7ff75c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:24.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 shutdown_connections 2024-04-25T15:26:24.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.618+0000 7ff762372640 1 -- 172.21.15.67:0/2400988810 wait complete. 2024-04-25T15:26:24.631 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:26:24.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:24.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.743+0000 7f3da7c34640 1 Processor -- start 2024-04-25T15:26:24.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.744+0000 7f3da7c34640 1 -- start start 2024-04-25T15:26:24.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da7c34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da7c34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3da006d870 con 0x7f3da006e8f0 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da7c34640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3da006db10 con 0x7f3da00b2dc0 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da7c34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3da006e460 con 0x7f3da00d4570 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da51cf640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3da006e8f0 0x7f3da00d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57720/0 (socket says 172.21.15.67:57720) 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da51cf640 1 -- 172.21.15.67:0/3857202149 learned_addr learned my addr 172.21.15.67:0/3857202149 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da59d0640 1 --2- 172.21.15.67:0/3857202149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da49ce640 1 -- 172.21.15.67:0/3857202149 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994713964 0 0) 0x7f3da006d870 con 0x7f3da006e8f0 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.745+0000 7f3da49ce640 1 -- 172.21.15.67:0/3857202149 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d88003520 con 0x7f3da006e8f0 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.746+0000 7f3da49ce640 1 -- 172.21.15.67:0/3857202149 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1630366306 0 0) 0x7f3d88003520 con 0x7f3da006e8f0 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.746+0000 7f3da59d0640 1 -- 172.21.15.67:0/3857202149 >> v1:172.21.15.176:6789/0 conn(0x7f3da00b2dc0 legacy=0x7f3da00b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.746+0000 7f3da59d0640 1 -- 172.21.15.67:0/3857202149 >> v1:172.21.15.67:6789/0 conn(0x7f3da006e8f0 legacy=0x7f3da00d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.746+0000 7f3da59d0640 1 -- 172.21.15.67:0/3857202149 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da006e0f0 con 0x7f3da00d4570 2024-04-25T15:26:24.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da59d0640 1 --2- 172.21.15.67:0/3857202149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00b4830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3d980097c0 tx=0x7f3d9802a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a77d9e4bb2531d51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:24.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da49ce640 1 -- 172.21.15.67:0/3857202149 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98037070 con 0x7f3da00d4570 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da49ce640 1 -- 172.21.15.67:0/3857202149 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d9802aaa0 con 0x7f3da00d4570 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da49ce640 1 -- 172.21.15.67:0/3857202149 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94001690 con 0x7f3da00d4570 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da7c34640 1 -- 172.21.15.67:0/3857202149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 msgr2=0x7f3da00b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3857202149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00b4830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3d980097c0 tx=0x7f3d9802a010 comp rx=0 tx=0).stop 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da7c34640 1 -- 172.21.15.67:0/3857202149 shutdown_connections 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3857202149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00b4830 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.747+0000 7f3da7c34640 1 -- 172.21.15.67:0/3857202149 >> 172.21.15.67:0/3857202149 conn(0x7f3da00d3a50 msgr2=0x7f3da00d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.748+0000 7f3da7c34640 1 -- 172.21.15.67:0/3857202149 shutdown_connections 2024-04-25T15:26:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.748+0000 7f3da7c34640 1 -- 172.21.15.67:0/3857202149 wait complete. 2024-04-25T15:26:24.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.749+0000 7f3da7c34640 1 Processor -- start 2024-04-25T15:26:24.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.749+0000 7f3da7c34640 1 -- start start 2024-04-25T15:26:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da7c34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da006e8f0 0x7f3da00c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da7c34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3da00b2dc0 0x7f3da00cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da7c34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da7c34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3da006ed00 con 0x7f3da006e8f0 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da7c34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3da00d0fb0 con 0x7f3da00b2dc0 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da7c34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3da00b38e0 con 0x7f3da00d4570 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da006e8f0 0x7f3da00c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da006e8f0 0x7f3da00c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41516/0 (socket says 172.21.15.67:41516) 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 -- 172.21.15.67:0/3219627098 learned_addr learned my addr 172.21.15.67:0/3219627098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da51cf640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3da00b2dc0 0x7f3da00cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da61d1640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 -- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 msgr2=0x7f3da00cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00cf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 -- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3da00b2dc0 msgr2=0x7f3da00cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3da00b2dc0 0x7f3da00cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 -- 172.21.15.67:0/3219627098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da00c7ac0 con 0x7f3da006e8f0 2024-04-25T15:26:24.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da59d0640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da006e8f0 0x7f3da00c7580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3d98035040 tx=0x7f3d980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3d9e7fc640 1 -- 172.21.15.67:0/3219627098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98037040 con 0x7f3da006e8f0 2024-04-25T15:26:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da009e540 con 0x7f3da006e8f0 2024-04-25T15:26:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.751+0000 7f3d9e7fc640 1 -- 172.21.15.67:0/3219627098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d98032d60 con 0x7f3da006e8f0 2024-04-25T15:26:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.752+0000 7f3d9e7fc640 1 -- 172.21.15.67:0/3219627098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98004260 con 0x7f3da006e8f0 2024-04-25T15:26:24.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.752+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da01531f0 con 0x7f3da006e8f0 2024-04-25T15:26:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.753+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da0068c30 con 0x7f3da006e8f0 2024-04-25T15:26:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.753+0000 7f3d9e7fc640 1 -- 172.21.15.67:0/3219627098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d9803a040 con 0x7f3da006e8f0 2024-04-25T15:26:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.754+0000 7f3d9e7fc640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d8c0744b0 0x7f3d8c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.754+0000 7f3d9e7fc640 1 -- 172.21.15.67:0/3219627098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f3d980b52b0 con 0x7f3da006e8f0 2024-04-25T15:26:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.754+0000 7f3da51cf640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d8c0744b0 0x7f3d8c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:24.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.754+0000 7f3da51cf640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d8c0744b0 0x7f3d8c0769c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3da00a1540 tx=0x7f3d94001d80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:24.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.758+0000 7f3d9e7fc640 1 -- 172.21.15.67:0/3219627098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3da0068c30 con 0x7f3da006e8f0 2024-04-25T15:26:24.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.981+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3da0050740 con 0x7f3da006e8f0 2024-04-25T15:26:24.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.982+0000 7f3d9e7fc640 1 -- 172.21.15.67:0/3219627098 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7f3d9804b030 con 0x7f3da006e8f0 2024-04-25T15:26:24.983 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:24.983 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/4217070487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4217070487},{"type":"v1","addr":"172.21.15.176:6835","nonce":4217070487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4207485404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4207485404},{"type":"v1","addr":"172.21.15.176:6837","nonce":4207485404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5067,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2355578883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2355578883},{"type":"v1","addr":"172.21.15.67:6834","nonce":2355578883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:20.974268+0000","modified":"2024-04-25T15:26:23.988098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2490033101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2490033101},{"type":"v1","addr":"172.21.15.67:6835","nonce":2490033101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-04-25T15:26:24.983 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d8c0744b0 msgr2=0x7f3d8c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d8c0744b0 0x7f3d8c0769c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3da00a1540 tx=0x7f3d94001d80 comp rx=0 tx=0).stop 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da006e8f0 msgr2=0x7f3da00c7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da006e8f0 0x7f3da00c7580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3d98035040 tx=0x7f3d980046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 shutdown_connections 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d8c0744b0 0x7f3d8c0769c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3da00d4570 0x7f3da00cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3da00b2dc0 0x7f3da00cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 --2- 172.21.15.67:0/3219627098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da006e8f0 0x7f3da00c7580 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 >> 172.21.15.67:0/3219627098 conn(0x7f3da00d3a50 msgr2=0x7f3da00ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 shutdown_connections 2024-04-25T15:26:24.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:24.984+0000 7f3da7c34640 1 -- 172.21.15.67:0/3219627098 wait complete. 2024-04-25T15:26:24.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:25.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.106+0000 7fd22ad00640 1 Processor -- start 2024-04-25T15:26:25.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.106+0000 7fd22ad00640 1 -- start start 2024-04-25T15:26:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd22ad00640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd22406d8e0 0x7fd2240d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd22ad00640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd22406dcf0 0x7fd2240d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd22ad00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2240b3c10 con 0x7fd2240d04a0 2024-04-25T15:26:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd22ad00640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd2240b9520 con 0x7fd22406dcf0 2024-04-25T15:26:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd22ad00640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2240b4820 con 0x7fd22406d8e0 2024-04-25T15:26:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd22929d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd2240d04a0 0x7fd2240d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57726/0 (socket says 172.21.15.67:57726) 2024-04-25T15:26:25.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd22929d640 1 -- 172.21.15.67:0/3561375741 learned_addr learned my addr 172.21.15.67:0/3561375741 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd228a9c640 1 --2- 172.21.15.67:0/3561375741 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd22406d8e0 0x7fd2240d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd223fff640 1 --2- 172.21.15.67:0/3561375741 >> v2:172.21.15.176:3300/0 conn(0x7fd22406dcf0 0x7fd2240d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd2237fe640 1 -- 172.21.15.67:0/3561375741 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1252254441 0 0) 0x7fd2240b3c10 con 0x7fd2240d04a0 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd2237fe640 1 -- 172.21.15.67:0/3561375741 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd20c003520 con 0x7fd2240d04a0 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd2237fe640 1 -- 172.21.15.67:0/3561375741 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2445136899 0 0) 0x7fd20c003520 con 0x7fd2240d04a0 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd228a9c640 1 -- 172.21.15.67:0/3561375741 >> v2:172.21.15.176:3300/0 conn(0x7fd22406dcf0 msgr2=0x7fd2240d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd228a9c640 1 --2- 172.21.15.67:0/3561375741 >> v2:172.21.15.176:3300/0 conn(0x7fd22406dcf0 0x7fd2240d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd228a9c640 1 -- 172.21.15.67:0/3561375741 >> v1:172.21.15.67:6789/0 conn(0x7fd2240d04a0 legacy=0x7fd2240d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.108+0000 7fd228a9c640 1 -- 172.21.15.67:0/3561375741 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2240b3ef0 con 0x7fd22406d8e0 2024-04-25T15:26:25.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.109+0000 7fd228a9c640 1 --2- 172.21.15.67:0/3561375741 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd22406d8e0 0x7fd2240d3a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2140097a0 tx=0x7fd21402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82585d096528d0d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd2237fe640 1 -- 172.21.15.67:0/3561375741 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd214037070 con 0x7fd22406d8e0 2024-04-25T15:26:25.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd2237fe640 1 -- 172.21.15.67:0/3561375741 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd21402ab10 con 0x7fd22406d8e0 2024-04-25T15:26:25.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd2237fe640 1 -- 172.21.15.67:0/3561375741 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0019f0 con 0x7fd22406d8e0 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 -- 172.21.15.67:0/3561375741 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd22406d8e0 msgr2=0x7fd2240d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 --2- 172.21.15.67:0/3561375741 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd22406d8e0 0x7fd2240d3a20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2140097a0 tx=0x7fd21402a080 comp rx=0 tx=0).stop 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 -- 172.21.15.67:0/3561375741 shutdown_connections 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 --2- 172.21.15.67:0/3561375741 >> v2:172.21.15.176:3300/0 conn(0x7fd22406dcf0 0x7fd2240d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 --2- 172.21.15.67:0/3561375741 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd22406d8e0 0x7fd2240d3a20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 -- 172.21.15.67:0/3561375741 >> 172.21.15.67:0/3561375741 conn(0x7fd22408d110 msgr2=0x7fd22408d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 -- 172.21.15.67:0/3561375741 shutdown_connections 2024-04-25T15:26:25.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.110+0000 7fd22ad00640 1 -- 172.21.15.67:0/3561375741 wait complete. 2024-04-25T15:26:25.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.112+0000 7fd22ad00640 1 Processor -- start 2024-04-25T15:26:25.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.112+0000 7fd22ad00640 1 -- start start 2024-04-25T15:26:25.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22ad00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22406d8e0 0x7fd2241e7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22ad00640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd22406dcf0 0x7fd2240a1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22ad00640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2240d04a0 0x7fd2241540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22ad00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2240b38e0 con 0x7fd22406d8e0 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22ad00640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2240b44b0 con 0x7fd22406dcf0 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22ad00640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2240d0fc0 con 0x7fd2240d04a0 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22406d8e0 0x7fd2241e7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22406d8e0 0x7fd2241e7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41532/0 (socket says 172.21.15.67:41532) 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 -- 172.21.15.67:0/2605883196 learned_addr learned my addr 172.21.15.67:0/2605883196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd223fff640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd22406dcf0 0x7fd2240a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22929d640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2240d04a0 0x7fd2241540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 -- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2240d04a0 msgr2=0x7fd2241540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2240d04a0 0x7fd2241540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 -- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd22406dcf0 msgr2=0x7fd2240a1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd22406dcf0 0x7fd2240a1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 -- 172.21.15.67:0/2605883196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22409f990 con 0x7fd22406d8e0 2024-04-25T15:26:25.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd22929d640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2240d04a0 0x7fd2241540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:25.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd228a9c640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22406d8e0 0x7fd2241e7e90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd2140098c0 tx=0x7fd2140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd2217fa640 1 -- 172.21.15.67:0/2605883196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd214042020 con 0x7fd22406d8e0 2024-04-25T15:26:25.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.114+0000 7fd2217fa640 1 -- 172.21.15.67:0/2605883196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd214003a40 con 0x7fd22406d8e0 2024-04-25T15:26:25.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.115+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd22409fc70 con 0x7fd22406d8e0 2024-04-25T15:26:25.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.115+0000 7fd2217fa640 1 -- 172.21.15.67:0/2605883196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd214037040 con 0x7fd22406d8e0 2024-04-25T15:26:25.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.115+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2241f7010 con 0x7fd22406d8e0 2024-04-25T15:26:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.115+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2241f7270 con 0x7fd22406d8e0 2024-04-25T15:26:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.116+0000 7fd2217fa640 1 -- 172.21.15.67:0/2605883196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd214045020 con 0x7fd22406d8e0 2024-04-25T15:26:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.117+0000 7fd2217fa640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2100744e0 0x7fd2100769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.117+0000 7fd2217fa640 1 -- 172.21.15.67:0/2605883196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd2140b5770 con 0x7fd22406d8e0 2024-04-25T15:26:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.117+0000 7fd223fff640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2100744e0 0x7fd2100769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.118+0000 7fd223fff640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2100744e0 0x7fd2100769f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd224161570 tx=0x7fd218009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.121+0000 7fd2217fa640 1 -- 172.21.15.67:0/2605883196 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd21404c030 con 0x7fd22406d8e0 2024-04-25T15:26:25.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.307+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd224050740 con 0x7fd22406d8e0 2024-04-25T15:26:25.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.308+0000 7fd2217fa640 1 -- 172.21.15.67:0/2605883196 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16633 (secure 0 0 0) 0x7fd21407ff00 con 0x7fd22406d8e0 2024-04-25T15:26:25.309 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:25.310 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:22.596544+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2100744e0 msgr2=0x7fd2100769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2100744e0 0x7fd2100769f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd224161570 tx=0x7fd218009190 comp rx=0 tx=0).stop 2024-04-25T15:26:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22406d8e0 msgr2=0x7fd2241e7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22406d8e0 0x7fd2241e7e90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd2140098c0 tx=0x7fd2140046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 shutdown_connections 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2100744e0 0x7fd2100769f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2240d04a0 0x7fd2241540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd22406dcf0 0x7fd2240a1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 --2- 172.21.15.67:0/2605883196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd22406d8e0 0x7fd2241e7e90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 >> 172.21.15.67:0/2605883196 conn(0x7fd22408d110 msgr2=0x7fd22406e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.310+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 shutdown_connections 2024-04-25T15:26:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.311+0000 7fd22ad00640 1 -- 172.21.15.67:0/2605883196 wait complete. 2024-04-25T15:26:25.323 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:26:25.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:26:25.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.433+0000 7f3cb29a8640 1 Processor -- start 2024-04-25T15:26:25.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.434+0000 7f3cb29a8640 1 -- start start 2024-04-25T15:26:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb29a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cac06dcf0 0x7f3cac0d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb29a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cac0b3c10 con 0x7f3cac06d8e0 2024-04-25T15:26:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb29a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cac0b3ef0 con 0x7f3cac0d4130 2024-04-25T15:26:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb29a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3cac0b9520 con 0x7f3cac06dcf0 2024-04-25T15:26:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cabfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3cac06d8e0 0x7f3cac0d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57736/0 (socket says 172.21.15.67:57736) 2024-04-25T15:26:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb0f45640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3cac06dcf0 0x7f3cac0d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cabfff640 1 -- 172.21.15.67:0/2260954625 learned_addr learned my addr 172.21.15.67:0/2260954625 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:25.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb0f45640 1 -- 172.21.15.67:0/2260954625 >> v1:172.21.15.176:6789/0 conn(0x7f3cac0d4130 legacy=0x7f3cac0d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb0f45640 1 -- 172.21.15.67:0/2260954625 >> v1:172.21.15.67:6789/0 conn(0x7f3cac06d8e0 legacy=0x7f3cac0d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.435+0000 7f3cb0f45640 1 -- 172.21.15.67:0/2260954625 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cac0b41d0 con 0x7f3cac06dcf0 2024-04-25T15:26:25.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.436+0000 7f3cb0f45640 1 --2- 172.21.15.67:0/2260954625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac06dcf0 0x7f3cac0d2470 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009d90 tx=0x7f3ca402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd96b52fec4bc621 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.436+0000 7f3caaffd640 1 -- 172.21.15.67:0/2260954625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4007740 con 0x7f3cac06dcf0 2024-04-25T15:26:25.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.436+0000 7f3caaffd640 1 -- 172.21.15.67:0/2260954625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca40078e0 con 0x7f3cac06dcf0 2024-04-25T15:26:25.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.436+0000 7f3caaffd640 1 -- 172.21.15.67:0/2260954625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4007cb0 con 0x7f3cac06dcf0 2024-04-25T15:26:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.437+0000 7f3cb29a8640 1 -- 172.21.15.67:0/2260954625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac06dcf0 msgr2=0x7f3cac0d2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.437+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/2260954625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac06dcf0 0x7f3cac0d2470 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009d90 tx=0x7f3ca402a660 comp rx=0 tx=0).stop 2024-04-25T15:26:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.437+0000 7f3cb29a8640 1 -- 172.21.15.67:0/2260954625 shutdown_connections 2024-04-25T15:26:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.437+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/2260954625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac06dcf0 0x7f3cac0d2470 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.437+0000 7f3cb29a8640 1 -- 172.21.15.67:0/2260954625 >> 172.21.15.67:0/2260954625 conn(0x7f3cac08d110 msgr2=0x7f3cac08d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.437+0000 7f3cb29a8640 1 -- 172.21.15.67:0/2260954625 shutdown_connections 2024-04-25T15:26:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.437+0000 7f3cb29a8640 1 -- 172.21.15.67:0/2260954625 wait complete. 2024-04-25T15:26:25.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.439+0000 7f3cb29a8640 1 Processor -- start 2024-04-25T15:26:25.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.439+0000 7f3cb29a8640 1 -- start start 2024-04-25T15:26:25.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.440+0000 7f3cb29a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cac06d8e0 0x7f3cac0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.440+0000 7f3cb29a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3cac06dcf0 0x7f3cac1e9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.440+0000 7f3cb29a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac1e9d40 0x7f3cac09e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.440+0000 7f3cb29a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac0b38e0 con 0x7f3cac1e9d40 2024-04-25T15:26:25.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.440+0000 7f3cb29a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac0b4820 con 0x7f3cac06d8e0 2024-04-25T15:26:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.440+0000 7f3cb29a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3cac0d29b0 con 0x7f3cac06dcf0 2024-04-25T15:26:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac1e9d40 0x7f3cac09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac1e9d40 0x7f3cac09e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41550/0 (socket says 172.21.15.67:41550) 2024-04-25T15:26:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 -- 172.21.15.67:0/1293296853 learned_addr learned my addr 172.21.15.67:0/1293296853 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:25.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cabfff640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cac06d8e0 0x7f3cac0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cab7fe640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3cac06dcf0 0x7f3cac1e9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 -- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3cac06dcf0 msgr2=0x7f3cac1e9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3cac06dcf0 0x7f3cac1e9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 -- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cac06d8e0 msgr2=0x7f3cac0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cac06d8e0 0x7f3cac0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 -- 172.21.15.67:0/1293296853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cac1ea150 con 0x7f3cac1e9d40 2024-04-25T15:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb0f45640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac1e9d40 0x7f3cac09e6b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009eb0 tx=0x7f3ca4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3ca8ff9640 1 -- 172.21.15.67:0/1293296853 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca403f070 con 0x7f3cac1e9d40 2024-04-25T15:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3ca8ff9640 1 -- 172.21.15.67:0/1293296853 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca4031c90 con 0x7f3cac1e9d40 2024-04-25T15:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cac09d610 con 0x7f3cac1e9d40 2024-04-25T15:26:25.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3ca8ff9640 1 -- 172.21.15.67:0/1293296853 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4044680 con 0x7f3cac1e9d40 2024-04-25T15:26:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.441+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cac09dbd0 con 0x7f3cac1e9d40 2024-04-25T15:26:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.442+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cac06ca80 con 0x7f3cac1e9d40 2024-04-25T15:26:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.443+0000 7f3ca8ff9640 1 -- 172.21.15.67:0/1293296853 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3ca403b410 con 0x7f3cac1e9d40 2024-04-25T15:26:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.443+0000 7f3ca8ff9640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c980744e0 0x7f3c980769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.444+0000 7f3ca8ff9640 1 -- 172.21.15.67:0/1293296853 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f3ca40b6aa0 con 0x7f3cac1e9d40 2024-04-25T15:26:25.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.444+0000 7f3cabfff640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c980744e0 0x7f3c980769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.444+0000 7f3cabfff640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c980744e0 0x7f3c980769f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0005880 tx=0x7f3ca0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.448+0000 7f3ca8ff9640 1 -- 172.21.15.67:0/1293296853 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cac06ca80 con 0x7f3cac1e9d40 2024-04-25T15:26:25.552 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-25T15:26:25.551+0000 7fbfacaf6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:26:25.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.631+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3cac050740 con 0x7f3c980744e0 2024-04-25T15:26:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.638+0000 7f3ca8ff9640 1 -- 172.21.15.67:0/1293296853 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251867 (secure 0 0 0) 0x7f3cac050740 con 0x7f3c980744e0 2024-04-25T15:26:25.642 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:25.643 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:26:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c980744e0 msgr2=0x7f3c980769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c980744e0 0x7f3c980769f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0005880 tx=0x7f3ca0009040 comp rx=0 tx=0).stop 2024-04-25T15:26:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac1e9d40 msgr2=0x7f3cac09e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac1e9d40 0x7f3cac09e6b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009eb0 tx=0x7f3ca4004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 shutdown_connections 2024-04-25T15:26:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3c980744e0 0x7f3c980769f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cac1e9d40 0x7f3cac09e6b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3cac06dcf0 0x7f3cac1e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 --2- 172.21.15.67:0/1293296853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cac06d8e0 0x7f3cac0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 >> 172.21.15.67:0/1293296853 conn(0x7f3cac08d110 msgr2=0x7f3cac071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 shutdown_connections 2024-04-25T15:26:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.641+0000 7f3cb29a8640 1 -- 172.21.15.67:0/1293296853 wait complete. 2024-04-25T15:26:25.653 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-25T15:26:24.808786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220052,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754668,"statfs":{"total":773094113280,"available":772868780032,"internally_reserved":0,"allocated":3973120,"data_stored":2249064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.231642"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624549+0000","last_change":"2024-04-25T15:26:21.619518+0000","last_active":"2024-04-25T15:26:21.624549+0000","last_peered":"2024-04-25T15:26:21.624549+0000","last_clean":"2024-04-25T15:26:21.624549+0000","last_became_active":"2024-04-25T15:26:21.619201+0000","last_became_peered":"2024-04-25T15:26:21.619201+0000","last_unstale":"2024-04-25T15:26:21.624549+0000","last_undegraded":"2024-04-25T15:26:21.624549+0000","last_fullsized":"2024-04-25T15:26:21.624549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624617+0000","last_change":"2024-04-25T15:26:21.615948+0000","last_active":"2024-04-25T15:26:21.624617+0000","last_peered":"2024-04-25T15:26:21.624617+0000","last_clean":"2024-04-25T15:26:21.624617+0000","last_became_active":"2024-04-25T15:26:21.614861+0000","last_became_peered":"2024-04-25T15:26:21.614861+0000","last_unstale":"2024-04-25T15:26:21.624617+0000","last_undegraded":"2024-04-25T15:26:21.624617+0000","last_fullsized":"2024-04-25T15:26:21.624617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:02.188532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625043+0000","last_change":"2024-04-25T15:26:21.621376+0000","last_active":"2024-04-25T15:26:21.625043+0000","last_peered":"2024-04-25T15:26:21.625043+0000","last_clean":"2024-04-25T15:26:21.625043+0000","last_became_active":"2024-04-25T15:26:21.621017+0000","last_became_peered":"2024-04-25T15:26:21.621017+0000","last_unstale":"2024-04-25T15:26:21.625043+0000","last_undegraded":"2024-04-25T15:26:21.625043+0000","last_fullsized":"2024-04-25T15:26:21.625043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:50.775371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.617725+0000","last_change":"2024-04-25T15:26:21.615440+0000","last_active":"2024-04-25T15:26:21.617725+0000","last_peered":"2024-04-25T15:26:21.617725+0000","last_clean":"2024-04-25T15:26:21.617725+0000","last_became_active":"2024-04-25T15:26:21.615098+0000","last_became_peered":"2024-04-25T15:26:21.615098+0000","last_unstale":"2024-04-25T15:26:21.617725+0000","last_undegraded":"2024-04-25T15:26:21.617725+0000","last_fullsized":"2024-04-25T15:26:21.617725+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:50.775371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616801+0000","last_change":"2024-04-25T15:26:21.608591+0000","last_active":"2024-04-25T15:26:21.616801+0000","last_peered":"2024-04-25T15:26:21.616801+0000","last_clean":"2024-04-25T15:26:21.616801+0000","last_became_active":"2024-04-25T15:26:21.608233+0000","last_became_peered":"2024-04-25T15:26:21.608233+0000","last_unstale":"2024-04-25T15:26:21.616801+0000","last_undegraded":"2024-04-25T15:26:21.616801+0000","last_fullsized":"2024-04-25T15:26:21.616801+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:41.476011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.617241+0000","last_change":"2024-04-25T15:26:21.614891+0000","last_active":"2024-04-25T15:26:21.617241+0000","last_peered":"2024-04-25T15:26:21.617241+0000","last_clean":"2024-04-25T15:26:21.617241+0000","last_became_active":"2024-04-25T15:26:21.614560+0000","last_became_peered":"2024-04-25T15:26:21.614560+0000","last_unstale":"2024-04-25T15:26:21.617241+0000","last_undegraded":"2024-04-25T15:26:21.617241+0000","last_fullsized":"2024-04-25T15:26:21.617241+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:05.105207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616244+0000","last_change":"2024-04-25T15:26:21.610992+0000","last_active":"2024-04-25T15:26:21.616244+0000","last_peered":"2024-04-25T15:26:21.616244+0000","last_clean":"2024-04-25T15:26:21.616244+0000","last_became_active":"2024-04-25T15:26:21.609746+0000","last_became_peered":"2024-04-25T15:26:21.609746+0000","last_unstale":"2024-04-25T15:26:21.616244+0000","last_undegraded":"2024-04-25T15:26:21.616244+0000","last_fullsized":"2024-04-25T15:26:21.616244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:11.403034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618750+0000","last_change":"2024-04-25T15:26:21.616938+0000","last_active":"2024-04-25T15:26:21.618750+0000","last_peered":"2024-04-25T15:26:21.618750+0000","last_clean":"2024-04-25T15:26:21.618750+0000","last_became_active":"2024-04-25T15:26:21.616674+0000","last_became_peered":"2024-04-25T15:26:21.616674+0000","last_unstale":"2024-04-25T15:26:21.618750+0000","last_undegraded":"2024-04-25T15:26:21.618750+0000","last_fullsized":"2024-04-25T15:26:21.618750+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:56.376686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611174+0000","last_change":"2024-04-25T15:26:21.607747+0000","last_active":"2024-04-25T15:26:21.611174+0000","last_peered":"2024-04-25T15:26:21.611174+0000","last_clean":"2024-04-25T15:26:21.611174+0000","last_became_active":"2024-04-25T15:26:21.607360+0000","last_became_peered":"2024-04-25T15:26:21.607360+0000","last_unstale":"2024-04-25T15:26:21.611174+0000","last_undegraded":"2024-04-25T15:26:21.611174+0000","last_fullsized":"2024-04-25T15:26:21.611174+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:10.789126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612205+0000","last_change":"2024-04-25T15:26:21.605133+0000","last_active":"2024-04-25T15:26:21.612205+0000","last_peered":"2024-04-25T15:26:21.612205+0000","last_clean":"2024-04-25T15:26:21.612205+0000","last_became_active":"2024-04-25T15:26:21.604842+0000","last_became_peered":"2024-04-25T15:26:21.604842+0000","last_unstale":"2024-04-25T15:26:21.612205+0000","last_undegraded":"2024-04-25T15:26:21.612205+0000","last_fullsized":"2024-04-25T15:26:21.612205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618696+0000","last_change":"2024-04-25T15:26:21.615052+0000","last_active":"2024-04-25T15:26:21.618696+0000","last_peered":"2024-04-25T15:26:21.618696+0000","last_clean":"2024-04-25T15:26:21.618696+0000","last_became_active":"2024-04-25T15:26:21.614679+0000","last_became_peered":"2024-04-25T15:26:21.614679+0000","last_unstale":"2024-04-25T15:26:21.618696+0000","last_undegraded":"2024-04-25T15:26:21.618696+0000","last_fullsized":"2024-04-25T15:26:21.618696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:53.994403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625096+0000","last_change":"2024-04-25T15:26:21.622166+0000","last_active":"2024-04-25T15:26:21.625096+0000","last_peered":"2024-04-25T15:26:21.625096+0000","last_clean":"2024-04-25T15:26:21.625096+0000","last_became_active":"2024-04-25T15:26:21.621858+0000","last_became_peered":"2024-04-25T15:26:21.621858+0000","last_unstale":"2024-04-25T15:26:21.625096+0000","last_undegraded":"2024-04-25T15:26:21.625096+0000","last_fullsized":"2024-04-25T15:26:21.625096+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:53.994403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619262+0000","last_change":"2024-04-25T15:26:21.613529+0000","last_active":"2024-04-25T15:26:21.619262+0000","last_peered":"2024-04-25T15:26:21.619262+0000","last_clean":"2024-04-25T15:26:21.619262+0000","last_became_active":"2024-04-25T15:26:21.613215+0000","last_became_peered":"2024-04-25T15:26:21.613215+0000","last_unstale":"2024-04-25T15:26:21.619262+0000","last_undegraded":"2024-04-25T15:26:21.619262+0000","last_fullsized":"2024-04-25T15:26:21.619262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:38.942978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625674+0000","last_change":"2024-04-25T15:26:21.618919+0000","last_active":"2024-04-25T15:26:21.625674+0000","last_peered":"2024-04-25T15:26:21.625674+0000","last_clean":"2024-04-25T15:26:21.625674+0000","last_became_active":"2024-04-25T15:26:21.617169+0000","last_became_peered":"2024-04-25T15:26:21.617169+0000","last_unstale":"2024-04-25T15:26:21.625674+0000","last_undegraded":"2024-04-25T15:26:21.625674+0000","last_fullsized":"2024-04-25T15:26:21.625674+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:52.829405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624274+0000","last_change":"2024-04-25T15:26:21.619759+0000","last_active":"2024-04-25T15:26:21.624274+0000","last_peered":"2024-04-25T15:26:21.624274+0000","last_clean":"2024-04-25T15:26:21.624274+0000","last_became_active":"2024-04-25T15:26:21.619377+0000","last_became_peered":"2024-04-25T15:26:21.619377+0000","last_unstale":"2024-04-25T15:26:21.624274+0000","last_undegraded":"2024-04-25T15:26:21.624274+0000","last_fullsized":"2024-04-25T15:26:21.624274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:20.151135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612971+0000","last_change":"2024-04-25T15:26:21.609797+0000","last_active":"2024-04-25T15:26:21.612971+0000","last_peered":"2024-04-25T15:26:21.612971+0000","last_clean":"2024-04-25T15:26:21.612971+0000","last_became_active":"2024-04-25T15:26:21.609507+0000","last_became_peered":"2024-04-25T15:26:21.609507+0000","last_unstale":"2024-04-25T15:26:21.612971+0000","last_undegraded":"2024-04-25T15:26:21.612971+0000","last_fullsized":"2024-04-25T15:26:21.612971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616335+0000","last_change":"2024-04-25T15:26:21.611172+0000","last_active":"2024-04-25T15:26:21.616335+0000","last_peered":"2024-04-25T15:26:21.616335+0000","last_clean":"2024-04-25T15:26:21.616335+0000","last_became_active":"2024-04-25T15:26:21.610131+0000","last_became_peered":"2024-04-25T15:26:21.610131+0000","last_unstale":"2024-04-25T15:26:21.616335+0000","last_undegraded":"2024-04-25T15:26:21.616335+0000","last_fullsized":"2024-04-25T15:26:21.616335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:37.737709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611998+0000","last_change":"2024-04-25T15:26:21.607940+0000","last_active":"2024-04-25T15:26:21.611998+0000","last_peered":"2024-04-25T15:26:21.611998+0000","last_clean":"2024-04-25T15:26:21.611998+0000","last_became_active":"2024-04-25T15:26:21.607301+0000","last_became_peered":"2024-04-25T15:26:21.607301+0000","last_unstale":"2024-04-25T15:26:21.611998+0000","last_undegraded":"2024-04-25T15:26:21.611998+0000","last_fullsized":"2024-04-25T15:26:21.611998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:02.445672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616515+0000","last_change":"2024-04-25T15:26:21.610318+0000","last_active":"2024-04-25T15:26:21.616515+0000","last_peered":"2024-04-25T15:26:21.616515+0000","last_clean":"2024-04-25T15:26:21.616515+0000","last_became_active":"2024-04-25T15:26:21.609989+0000","last_became_peered":"2024-04-25T15:26:21.609989+0000","last_unstale":"2024-04-25T15:26:21.616515+0000","last_undegraded":"2024-04-25T15:26:21.616515+0000","last_fullsized":"2024-04-25T15:26:21.616515+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:43.820705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619336+0000","last_change":"2024-04-25T15:26:21.615358+0000","last_active":"2024-04-25T15:26:21.619336+0000","last_peered":"2024-04-25T15:26:21.619336+0000","last_clean":"2024-04-25T15:26:21.619336+0000","last_became_active":"2024-04-25T15:26:21.615013+0000","last_became_peered":"2024-04-25T15:26:21.615013+0000","last_unstale":"2024-04-25T15:26:21.619336+0000","last_undegraded":"2024-04-25T15:26:21.619336+0000","last_fullsized":"2024-04-25T15:26:21.619336+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:08.299329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.603983+0000","last_change":"2024-04-25T15:26:21.597722+0000","last_active":"2024-04-25T15:26:21.603983+0000","last_peered":"2024-04-25T15:26:21.603983+0000","last_clean":"2024-04-25T15:26:21.603983+0000","last_became_active":"2024-04-25T15:26:21.597413+0000","last_became_peered":"2024-04-25T15:26:21.597413+0000","last_unstale":"2024-04-25T15:26:21.603983+0000","last_undegraded":"2024-04-25T15:26:21.603983+0000","last_fullsized":"2024-04-25T15:26:21.603983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:41.484204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.604698+0000","last_change":"2024-04-25T15:26:21.597463+0000","last_active":"2024-04-25T15:26:21.604698+0000","last_peered":"2024-04-25T15:26:21.604698+0000","last_clean":"2024-04-25T15:26:21.604698+0000","last_became_active":"2024-04-25T15:26:21.597163+0000","last_became_peered":"2024-04-25T15:26:21.597163+0000","last_unstale":"2024-04-25T15:26:21.604698+0000","last_undegraded":"2024-04-25T15:26:21.604698+0000","last_fullsized":"2024-04-25T15:26:21.604698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:09.195802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610934+0000","last_change":"2024-04-25T15:26:21.606008+0000","last_active":"2024-04-25T15:26:21.610934+0000","last_peered":"2024-04-25T15:26:21.610934+0000","last_clean":"2024-04-25T15:26:21.610934+0000","last_became_active":"2024-04-25T15:26:21.605604+0000","last_became_peered":"2024-04-25T15:26:21.605604+0000","last_unstale":"2024-04-25T15:26:21.610934+0000","last_undegraded":"2024-04-25T15:26:21.610934+0000","last_fullsized":"2024-04-25T15:26:21.610934+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621627+0000","last_change":"2024-04-25T15:26:21.618252+0000","last_active":"2024-04-25T15:26:21.621627+0000","last_peered":"2024-04-25T15:26:21.621627+0000","last_clean":"2024-04-25T15:26:21.621627+0000","last_became_active":"2024-04-25T15:26:21.617908+0000","last_became_peered":"2024-04-25T15:26:21.617908+0000","last_unstale":"2024-04-25T15:26:21.621627+0000","last_undegraded":"2024-04-25T15:26:21.621627+0000","last_fullsized":"2024-04-25T15:26:21.621627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:43.385692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.614064+0000","last_change":"2024-04-25T15:26:21.608061+0000","last_active":"2024-04-25T15:26:21.614064+0000","last_peered":"2024-04-25T15:26:21.614064+0000","last_clean":"2024-04-25T15:26:21.614064+0000","last_became_active":"2024-04-25T15:26:21.607706+0000","last_became_peered":"2024-04-25T15:26:21.607706+0000","last_unstale":"2024-04-25T15:26:21.614064+0000","last_undegraded":"2024-04-25T15:26:21.614064+0000","last_fullsized":"2024-04-25T15:26:21.614064+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:37.737709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612025+0000","last_change":"2024-04-25T15:26:21.608873+0000","last_active":"2024-04-25T15:26:21.612025+0000","last_peered":"2024-04-25T15:26:21.612025+0000","last_clean":"2024-04-25T15:26:21.612025+0000","last_became_active":"2024-04-25T15:26:21.608521+0000","last_became_peered":"2024-04-25T15:26:21.608521+0000","last_unstale":"2024-04-25T15:26:21.612025+0000","last_undegraded":"2024-04-25T15:26:21.612025+0000","last_fullsized":"2024-04-25T15:26:21.612025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:42.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618176+0000","last_change":"2024-04-25T15:26:21.614973+0000","last_active":"2024-04-25T15:26:21.618176+0000","last_peered":"2024-04-25T15:26:21.618176+0000","last_clean":"2024-04-25T15:26:21.618176+0000","last_became_active":"2024-04-25T15:26:21.614667+0000","last_became_peered":"2024-04-25T15:26:21.614667+0000","last_unstale":"2024-04-25T15:26:21.618176+0000","last_undegraded":"2024-04-25T15:26:21.618176+0000","last_fullsized":"2024-04-25T15:26:21.618176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:43.385692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630558+0000","last_change":"2024-04-25T15:26:21.626208+0000","last_active":"2024-04-25T15:26:21.630558+0000","last_peered":"2024-04-25T15:26:21.630558+0000","last_clean":"2024-04-25T15:26:21.630558+0000","last_became_active":"2024-04-25T15:26:21.625871+0000","last_became_peered":"2024-04-25T15:26:21.625871+0000","last_unstale":"2024-04-25T15:26:21.630558+0000","last_undegraded":"2024-04-25T15:26:21.630558+0000","last_fullsized":"2024-04-25T15:26:21.630558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:42.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611384+0000","last_change":"2024-04-25T15:26:21.600350+0000","last_active":"2024-04-25T15:26:21.611384+0000","last_peered":"2024-04-25T15:26:21.611384+0000","last_clean":"2024-04-25T15:26:21.611384+0000","last_became_active":"2024-04-25T15:26:21.600011+0000","last_became_peered":"2024-04-25T15:26:21.600011+0000","last_unstale":"2024-04-25T15:26:21.611384+0000","last_undegraded":"2024-04-25T15:26:21.611384+0000","last_fullsized":"2024-04-25T15:26:21.611384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:21.482642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610855+0000","last_change":"2024-04-25T15:26:21.609147+0000","last_active":"2024-04-25T15:26:21.610855+0000","last_peered":"2024-04-25T15:26:21.610855+0000","last_clean":"2024-04-25T15:26:21.610855+0000","last_became_active":"2024-04-25T15:26:21.608816+0000","last_became_peered":"2024-04-25T15:26:21.608816+0000","last_unstale":"2024-04-25T15:26:21.610855+0000","last_undegraded":"2024-04-25T15:26:21.610855+0000","last_fullsized":"2024-04-25T15:26:21.610855+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618877+0000","last_change":"2024-04-25T15:26:21.615295+0000","last_active":"2024-04-25T15:26:21.618877+0000","last_peered":"2024-04-25T15:26:21.618877+0000","last_clean":"2024-04-25T15:26:21.618877+0000","last_became_active":"2024-04-25T15:26:21.614134+0000","last_became_peered":"2024-04-25T15:26:21.614134+0000","last_unstale":"2024-04-25T15:26:21.618877+0000","last_undegraded":"2024-04-25T15:26:21.618877+0000","last_fullsized":"2024-04-25T15:26:21.618877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616215+0000","last_change":"2024-04-25T15:26:21.612104+0000","last_active":"2024-04-25T15:26:21.616215+0000","last_peered":"2024-04-25T15:26:21.616215+0000","last_clean":"2024-04-25T15:26:21.616215+0000","last_became_active":"2024-04-25T15:26:21.611800+0000","last_became_peered":"2024-04-25T15:26:21.611800+0000","last_unstale":"2024-04-25T15:26:21.616215+0000","last_undegraded":"2024-04-25T15:26:21.616215+0000","last_fullsized":"2024-04-25T15:26:21.616215+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:20.151135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610973+0000","last_change":"2024-04-25T15:26:21.606993+0000","last_active":"2024-04-25T15:26:21.610973+0000","last_peered":"2024-04-25T15:26:21.610973+0000","last_clean":"2024-04-25T15:26:21.610973+0000","last_became_active":"2024-04-25T15:26:21.606051+0000","last_became_peered":"2024-04-25T15:26:21.606051+0000","last_unstale":"2024-04-25T15:26:21.610973+0000","last_undegraded":"2024-04-25T15:26:21.610973+0000","last_fullsized":"2024-04-25T15:26:21.610973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.626942+0000","last_change":"2024-04-25T15:26:20.596541+0000","last_active":"2024-04-25T15:26:21.626942+0000","last_peered":"2024-04-25T15:26:21.626942+0000","last_clean":"2024-04-25T15:26:21.626942+0000","last_became_active":"2024-04-25T15:26:20.596246+0000","last_became_peered":"2024-04-25T15:26:20.596246+0000","last_unstale":"2024-04-25T15:26:21.626942+0000","last_undegraded":"2024-04-25T15:26:21.626942+0000","last_fullsized":"2024-04-25T15:26:21.626942+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:27.738092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618632+0000","last_change":"2024-04-25T15:26:21.612216+0000","last_active":"2024-04-25T15:26:21.618632+0000","last_peered":"2024-04-25T15:26:21.618632+0000","last_clean":"2024-04-25T15:26:21.618632+0000","last_became_active":"2024-04-25T15:26:21.611904+0000","last_became_peered":"2024-04-25T15:26:21.611904+0000","last_unstale":"2024-04-25T15:26:21.618632+0000","last_undegraded":"2024-04-25T15:26:21.618632+0000","last_fullsized":"2024-04-25T15:26:21.618632+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:21.482642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619907+0000","last_change":"2024-04-25T15:26:20.589623+0000","last_active":"2024-04-25T15:26:21.619907+0000","last_peered":"2024-04-25T15:26:21.619907+0000","last_clean":"2024-04-25T15:26:21.619907+0000","last_became_active":"2024-04-25T15:26:20.589260+0000","last_became_peered":"2024-04-25T15:26:20.589260+0000","last_unstale":"2024-04-25T15:26:21.619907+0000","last_undegraded":"2024-04-25T15:26:21.619907+0000","last_fullsized":"2024-04-25T15:26:21.619907+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:40.478696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611344+0000","last_change":"2024-04-25T15:26:21.600341+0000","last_active":"2024-04-25T15:26:21.611344+0000","last_peered":"2024-04-25T15:26:21.611344+0000","last_clean":"2024-04-25T15:26:21.611344+0000","last_became_active":"2024-04-25T15:26:21.599984+0000","last_became_peered":"2024-04-25T15:26:21.599984+0000","last_unstale":"2024-04-25T15:26:21.611344+0000","last_undegraded":"2024-04-25T15:26:21.611344+0000","last_fullsized":"2024-04-25T15:26:21.611344+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:28.743600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.627126+0000","last_change":"2024-04-25T15:26:20.602697+0000","last_active":"2024-04-25T15:26:21.627126+0000","last_peered":"2024-04-25T15:26:21.627126+0000","last_clean":"2024-04-25T15:26:21.627126+0000","last_became_active":"2024-04-25T15:26:20.602306+0000","last_became_peered":"2024-04-25T15:26:20.602306+0000","last_unstale":"2024-04-25T15:26:21.627126+0000","last_undegraded":"2024-04-25T15:26:21.627126+0000","last_fullsized":"2024-04-25T15:26:21.627126+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:19.145627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618202+0000","last_change":"2024-04-25T15:26:21.615092+0000","last_active":"2024-04-25T15:26:21.618202+0000","last_peered":"2024-04-25T15:26:21.618202+0000","last_clean":"2024-04-25T15:26:21.618202+0000","last_became_active":"2024-04-25T15:26:21.614764+0000","last_became_peered":"2024-04-25T15:26:21.614764+0000","last_unstale":"2024-04-25T15:26:21.618202+0000","last_undegraded":"2024-04-25T15:26:21.618202+0000","last_fullsized":"2024-04-25T15:26:21.618202+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:20.151135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.622967+0000","last_change":"2024-04-25T15:26:20.598502+0000","last_active":"2024-04-25T15:26:21.622967+0000","last_peered":"2024-04-25T15:26:21.622967+0000","last_clean":"2024-04-25T15:26:21.622967+0000","last_became_active":"2024-04-25T15:26:20.598167+0000","last_became_peered":"2024-04-25T15:26:20.598167+0000","last_unstale":"2024-04-25T15:26:21.622967+0000","last_undegraded":"2024-04-25T15:26:21.622967+0000","last_fullsized":"2024-04-25T15:26:21.622967+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:28.020892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618734+0000","last_change":"2024-04-25T15:26:21.613803+0000","last_active":"2024-04-25T15:26:21.618734+0000","last_peered":"2024-04-25T15:26:21.618734+0000","last_clean":"2024-04-25T15:26:21.618734+0000","last_became_active":"2024-04-25T15:26:21.613495+0000","last_became_peered":"2024-04-25T15:26:21.613495+0000","last_unstale":"2024-04-25T15:26:21.618734+0000","last_undegraded":"2024-04-25T15:26:21.618734+0000","last_fullsized":"2024-04-25T15:26:21.618734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:50.565418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630809+0000","last_change":"2024-04-25T15:26:20.602555+0000","last_active":"2024-04-25T15:26:21.630809+0000","last_peered":"2024-04-25T15:26:21.630809+0000","last_clean":"2024-04-25T15:26:21.630809+0000","last_became_active":"2024-04-25T15:26:20.602210+0000","last_became_peered":"2024-04-25T15:26:20.602210+0000","last_unstale":"2024-04-25T15:26:21.630809+0000","last_undegraded":"2024-04-25T15:26:21.630809+0000","last_fullsized":"2024-04-25T15:26:21.630809+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:44.619005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630565+0000","last_change":"2024-04-25T15:26:21.626293+0000","last_active":"2024-04-25T15:26:21.630565+0000","last_peered":"2024-04-25T15:26:21.630565+0000","last_clean":"2024-04-25T15:26:21.630565+0000","last_became_active":"2024-04-25T15:26:21.625687+0000","last_became_peered":"2024-04-25T15:26:21.625687+0000","last_unstale":"2024-04-25T15:26:21.630565+0000","last_undegraded":"2024-04-25T15:26:21.630565+0000","last_fullsized":"2024-04-25T15:26:21.630565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:37.980797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630808+0000","last_change":"2024-04-25T15:26:20.598522+0000","last_active":"2024-04-25T15:26:21.630808+0000","last_peered":"2024-04-25T15:26:21.630808+0000","last_clean":"2024-04-25T15:26:21.630808+0000","last_became_active":"2024-04-25T15:26:20.598176+0000","last_became_peered":"2024-04-25T15:26:20.598176+0000","last_unstale":"2024-04-25T15:26:21.630808+0000","last_undegraded":"2024-04-25T15:26:21.630808+0000","last_fullsized":"2024-04-25T15:26:21.630808+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:37.937470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.615501+0000","last_change":"2024-04-25T15:26:21.609992+0000","last_active":"2024-04-25T15:26:21.615501+0000","last_peered":"2024-04-25T15:26:21.615501+0000","last_clean":"2024-04-25T15:26:21.615501+0000","last_became_active":"2024-04-25T15:26:21.609654+0000","last_became_peered":"2024-04-25T15:26:21.609654+0000","last_unstale":"2024-04-25T15:26:21.615501+0000","last_undegraded":"2024-04-25T15:26:21.615501+0000","last_fullsized":"2024-04-25T15:26:21.615501+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593130+0000","last_change":"2024-04-25T15:26:20.603710+0000","last_active":"2024-04-25T15:26:21.593130+0000","last_peered":"2024-04-25T15:26:21.593130+0000","last_clean":"2024-04-25T15:26:21.593130+0000","last_became_active":"2024-04-25T15:26:20.603360+0000","last_became_peered":"2024-04-25T15:26:20.603360+0000","last_unstale":"2024-04-25T15:26:21.593130+0000","last_undegraded":"2024-04-25T15:26:21.593130+0000","last_fullsized":"2024-04-25T15:26:21.593130+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:42.380184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621024+0000","last_change":"2024-04-25T15:26:21.618982+0000","last_active":"2024-04-25T15:26:21.621024+0000","last_peered":"2024-04-25T15:26:21.621024+0000","last_clean":"2024-04-25T15:26:21.621024+0000","last_became_active":"2024-04-25T15:26:21.618667+0000","last_became_peered":"2024-04-25T15:26:21.618667+0000","last_unstale":"2024-04-25T15:26:21.621024+0000","last_undegraded":"2024-04-25T15:26:21.621024+0000","last_fullsized":"2024-04-25T15:26:21.621024+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:50.565418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594764+0000","last_change":"2024-04-25T15:26:20.597982+0000","last_active":"2024-04-25T15:26:21.594764+0000","last_peered":"2024-04-25T15:26:21.594764+0000","last_clean":"2024-04-25T15:26:21.594764+0000","last_became_active":"2024-04-25T15:26:20.596709+0000","last_became_peered":"2024-04-25T15:26:20.596709+0000","last_unstale":"2024-04-25T15:26:21.594764+0000","last_undegraded":"2024-04-25T15:26:21.594764+0000","last_fullsized":"2024-04-25T15:26:21.594764+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:02.951986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.628075+0000","last_change":"2024-04-25T15:26:21.625576+0000","last_active":"2024-04-25T15:26:21.628075+0000","last_peered":"2024-04-25T15:26:21.628075+0000","last_clean":"2024-04-25T15:26:21.628075+0000","last_became_active":"2024-04-25T15:26:21.625169+0000","last_became_peered":"2024-04-25T15:26:21.625169+0000","last_unstale":"2024-04-25T15:26:21.628075+0000","last_undegraded":"2024-04-25T15:26:21.628075+0000","last_fullsized":"2024-04-25T15:26:21.628075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:02.445672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593743+0000","last_change":"2024-04-25T15:26:20.602652+0000","last_active":"2024-04-25T15:26:21.593743+0000","last_peered":"2024-04-25T15:26:21.593743+0000","last_clean":"2024-04-25T15:26:21.593743+0000","last_became_active":"2024-04-25T15:26:20.602042+0000","last_became_peered":"2024-04-25T15:26:20.602042+0000","last_unstale":"2024-04-25T15:26:21.593743+0000","last_undegraded":"2024-04-25T15:26:21.593743+0000","last_fullsized":"2024-04-25T15:26:21.593743+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:49.769863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611944+0000","last_change":"2024-04-25T15:26:21.606926+0000","last_active":"2024-04-25T15:26:21.611944+0000","last_peered":"2024-04-25T15:26:21.611944+0000","last_clean":"2024-04-25T15:26:21.611944+0000","last_became_active":"2024-04-25T15:26:21.606531+0000","last_became_peered":"2024-04-25T15:26:21.606531+0000","last_unstale":"2024-04-25T15:26:21.611944+0000","last_undegraded":"2024-04-25T15:26:21.611944+0000","last_fullsized":"2024-04-25T15:26:21.611944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:56.376686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593743+0000","last_change":"2024-04-25T15:26:20.598360+0000","last_active":"2024-04-25T15:26:21.593743+0000","last_peered":"2024-04-25T15:26:21.593743+0000","last_clean":"2024-04-25T15:26:21.593743+0000","last_became_active":"2024-04-25T15:26:20.598042+0000","last_became_peered":"2024-04-25T15:26:20.598042+0000","last_unstale":"2024-04-25T15:26:21.593743+0000","last_undegraded":"2024-04-25T15:26:21.593743+0000","last_fullsized":"2024-04-25T15:26:21.593743+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:20.477134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621075+0000","last_change":"2024-04-25T15:26:21.609926+0000","last_active":"2024-04-25T15:26:21.621075+0000","last_peered":"2024-04-25T15:26:21.621075+0000","last_clean":"2024-04-25T15:26:21.621075+0000","last_became_active":"2024-04-25T15:26:21.608497+0000","last_became_peered":"2024-04-25T15:26:21.608497+0000","last_unstale":"2024-04-25T15:26:21.621075+0000","last_undegraded":"2024-04-25T15:26:21.621075+0000","last_fullsized":"2024-04-25T15:26:21.621075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:37.980797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624871+0000","last_change":"2024-04-25T15:26:20.591095+0000","last_active":"2024-04-25T15:26:21.624871+0000","last_peered":"2024-04-25T15:26:21.624871+0000","last_clean":"2024-04-25T15:26:21.624871+0000","last_became_active":"2024-04-25T15:26:20.590747+0000","last_became_peered":"2024-04-25T15:26:20.590747+0000","last_unstale":"2024-04-25T15:26:21.624871+0000","last_undegraded":"2024-04-25T15:26:21.624871+0000","last_fullsized":"2024-04-25T15:26:21.624871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:36.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625073+0000","last_change":"2024-04-25T15:26:21.621439+0000","last_active":"2024-04-25T15:26:21.625073+0000","last_peered":"2024-04-25T15:26:21.625073+0000","last_clean":"2024-04-25T15:26:21.625073+0000","last_became_active":"2024-04-25T15:26:21.621101+0000","last_became_peered":"2024-04-25T15:26:21.621101+0000","last_unstale":"2024-04-25T15:26:21.625073+0000","last_undegraded":"2024-04-25T15:26:21.625073+0000","last_fullsized":"2024-04-25T15:26:21.625073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:50.565418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.633229+0000","last_change":"2024-04-25T15:26:20.602796+0000","last_active":"2024-04-25T15:26:21.633229+0000","last_peered":"2024-04-25T15:26:21.633229+0000","last_clean":"2024-04-25T15:26:21.633229+0000","last_became_active":"2024-04-25T15:26:20.602443+0000","last_became_peered":"2024-04-25T15:26:20.602443+0000","last_unstale":"2024-04-25T15:26:21.633229+0000","last_undegraded":"2024-04-25T15:26:21.633229+0000","last_fullsized":"2024-04-25T15:26:21.633229+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:52.988896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621062+0000","last_change":"2024-04-25T15:26:21.619342+0000","last_active":"2024-04-25T15:26:21.621062+0000","last_peered":"2024-04-25T15:26:21.621062+0000","last_clean":"2024-04-25T15:26:21.621062+0000","last_became_active":"2024-04-25T15:26:21.618651+0000","last_became_peered":"2024-04-25T15:26:21.618651+0000","last_unstale":"2024-04-25T15:26:21.621062+0000","last_undegraded":"2024-04-25T15:26:21.621062+0000","last_fullsized":"2024-04-25T15:26:21.621062+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:53.994403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630840+0000","last_change":"2024-04-25T15:26:20.591644+0000","last_active":"2024-04-25T15:26:21.630840+0000","last_peered":"2024-04-25T15:26:21.630840+0000","last_clean":"2024-04-25T15:26:21.630840+0000","last_became_active":"2024-04-25T15:26:20.591346+0000","last_became_peered":"2024-04-25T15:26:20.591346+0000","last_unstale":"2024-04-25T15:26:21.630840+0000","last_undegraded":"2024-04-25T15:26:21.630840+0000","last_fullsized":"2024-04-25T15:26:21.630840+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:19.188291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616899+0000","last_change":"2024-04-25T15:26:21.615068+0000","last_active":"2024-04-25T15:26:21.616899+0000","last_peered":"2024-04-25T15:26:21.616899+0000","last_clean":"2024-04-25T15:26:21.616899+0000","last_became_active":"2024-04-25T15:26:21.614637+0000","last_became_peered":"2024-04-25T15:26:21.614637+0000","last_unstale":"2024-04-25T15:26:21.616899+0000","last_undegraded":"2024-04-25T15:26:21.616899+0000","last_fullsized":"2024-04-25T15:26:21.616899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:29.792437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.602696+0000","last_change":"2024-04-25T15:26:20.595469+0000","last_active":"2024-04-25T15:26:21.602696+0000","last_peered":"2024-04-25T15:26:21.602696+0000","last_clean":"2024-04-25T15:26:21.602696+0000","last_became_active":"2024-04-25T15:26:20.594200+0000","last_became_peered":"2024-04-25T15:26:20.594200+0000","last_unstale":"2024-04-25T15:26:21.602696+0000","last_undegraded":"2024-04-25T15:26:21.602696+0000","last_fullsized":"2024-04-25T15:26:21.602696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:20.477134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618565+0000","last_change":"2024-04-25T15:26:21.612824+0000","last_active":"2024-04-25T15:26:21.618565+0000","last_peered":"2024-04-25T15:26:21.618565+0000","last_clean":"2024-04-25T15:26:21.618565+0000","last_became_active":"2024-04-25T15:26:21.612525+0000","last_became_peered":"2024-04-25T15:26:21.612525+0000","last_unstale":"2024-04-25T15:26:21.618565+0000","last_undegraded":"2024-04-25T15:26:21.618565+0000","last_fullsized":"2024-04-25T15:26:21.618565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:50.775371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.592502+0000","last_change":"2024-04-25T15:26:20.603523+0000","last_active":"2024-04-25T15:26:21.592502+0000","last_peered":"2024-04-25T15:26:21.592502+0000","last_clean":"2024-04-25T15:26:21.592502+0000","last_became_active":"2024-04-25T15:26:20.603084+0000","last_became_peered":"2024-04-25T15:26:20.603084+0000","last_unstale":"2024-04-25T15:26:21.592502+0000","last_undegraded":"2024-04-25T15:26:21.592502+0000","last_fullsized":"2024-04-25T15:26:21.592502+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:36.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618722+0000","last_change":"2024-04-25T15:26:21.613399+0000","last_active":"2024-04-25T15:26:21.618722+0000","last_peered":"2024-04-25T15:26:21.618722+0000","last_clean":"2024-04-25T15:26:21.618722+0000","last_became_active":"2024-04-25T15:26:21.612998+0000","last_became_peered":"2024-04-25T15:26:21.612998+0000","last_unstale":"2024-04-25T15:26:21.618722+0000","last_undegraded":"2024-04-25T15:26:21.618722+0000","last_fullsized":"2024-04-25T15:26:21.618722+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:18.873777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630780+0000","last_change":"2024-04-25T15:26:20.602960+0000","last_active":"2024-04-25T15:26:21.630780+0000","last_peered":"2024-04-25T15:26:21.630780+0000","last_clean":"2024-04-25T15:26:21.630780+0000","last_became_active":"2024-04-25T15:26:20.602435+0000","last_became_peered":"2024-04-25T15:26:20.602435+0000","last_unstale":"2024-04-25T15:26:21.630780+0000","last_undegraded":"2024-04-25T15:26:21.630780+0000","last_fullsized":"2024-04-25T15:26:21.630780+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:49.559910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.622204+0000","last_change":"2024-04-25T15:26:21.619051+0000","last_active":"2024-04-25T15:26:21.622204+0000","last_peered":"2024-04-25T15:26:21.622204+0000","last_clean":"2024-04-25T15:26:21.622204+0000","last_became_active":"2024-04-25T15:26:21.618737+0000","last_became_peered":"2024-04-25T15:26:21.618737+0000","last_unstale":"2024-04-25T15:26:21.622204+0000","last_undegraded":"2024-04-25T15:26:21.622204+0000","last_fullsized":"2024-04-25T15:26:21.622204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624776+0000","last_change":"2024-04-25T15:26:20.604571+0000","last_active":"2024-04-25T15:26:21.624776+0000","last_peered":"2024-04-25T15:26:21.624776+0000","last_clean":"2024-04-25T15:26:21.624776+0000","last_became_active":"2024-04-25T15:26:20.604190+0000","last_became_peered":"2024-04-25T15:26:20.604190+0000","last_unstale":"2024-04-25T15:26:21.624776+0000","last_undegraded":"2024-04-25T15:26:21.624776+0000","last_fullsized":"2024-04-25T15:26:21.624776+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:27.738092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.626063+0000","last_change":"2024-04-25T15:26:21.623052+0000","last_active":"2024-04-25T15:26:21.626063+0000","last_peered":"2024-04-25T15:26:21.626063+0000","last_clean":"2024-04-25T15:26:21.626063+0000","last_became_active":"2024-04-25T15:26:21.622607+0000","last_became_peered":"2024-04-25T15:26:21.622607+0000","last_unstale":"2024-04-25T15:26:21.626063+0000","last_undegraded":"2024-04-25T15:26:21.626063+0000","last_fullsized":"2024-04-25T15:26:21.626063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:56.376686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.628303+0000","last_change":"2024-04-25T15:26:20.599542+0000","last_active":"2024-04-25T15:26:21.628303+0000","last_peered":"2024-04-25T15:26:21.628303+0000","last_clean":"2024-04-25T15:26:21.628303+0000","last_became_active":"2024-04-25T15:26:20.599003+0000","last_became_peered":"2024-04-25T15:26:20.599003+0000","last_unstale":"2024-04-25T15:26:21.628303+0000","last_undegraded":"2024-04-25T15:26:21.628303+0000","last_fullsized":"2024-04-25T15:26:21.628303+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:07.293821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612172+0000","last_change":"2024-04-25T15:26:21.605945+0000","last_active":"2024-04-25T15:26:21.612172+0000","last_peered":"2024-04-25T15:26:21.612172+0000","last_clean":"2024-04-25T15:26:21.612172+0000","last_became_active":"2024-04-25T15:26:21.605632+0000","last_became_peered":"2024-04-25T15:26:21.605632+0000","last_unstale":"2024-04-25T15:26:21.612172+0000","last_undegraded":"2024-04-25T15:26:21.612172+0000","last_fullsized":"2024-04-25T15:26:21.612172+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:11.403034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.628356+0000","last_change":"2024-04-25T15:26:20.590476+0000","last_active":"2024-04-25T15:26:21.628356+0000","last_peered":"2024-04-25T15:26:21.628356+0000","last_clean":"2024-04-25T15:26:21.628356+0000","last_became_active":"2024-04-25T15:26:20.590140+0000","last_became_peered":"2024-04-25T15:26:20.590140+0000","last_unstale":"2024-04-25T15:26:21.628356+0000","last_undegraded":"2024-04-25T15:26:21.628356+0000","last_fullsized":"2024-04-25T15:26:21.628356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:11.381284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616428+0000","last_change":"2024-04-25T15:26:21.608276+0000","last_active":"2024-04-25T15:26:21.616428+0000","last_peered":"2024-04-25T15:26:21.616428+0000","last_clean":"2024-04-25T15:26:21.616428+0000","last_became_active":"2024-04-25T15:26:21.607974+0000","last_became_peered":"2024-04-25T15:26:21.607974+0000","last_unstale":"2024-04-25T15:26:21.616428+0000","last_undegraded":"2024-04-25T15:26:21.616428+0000","last_fullsized":"2024-04-25T15:26:21.616428+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:44:07.121400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594944+0000","last_change":"2024-04-25T15:26:20.603093+0000","last_active":"2024-04-25T15:26:21.594944+0000","last_peered":"2024-04-25T15:26:21.594944+0000","last_clean":"2024-04-25T15:26:21.594944+0000","last_became_active":"2024-04-25T15:26:20.602721+0000","last_became_peered":"2024-04-25T15:26:20.602721+0000","last_unstale":"2024-04-25T15:26:21.594944+0000","last_undegraded":"2024-04-25T15:26:21.594944+0000","last_fullsized":"2024-04-25T15:26:21.594944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:55.907186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619311+0000","last_change":"2024-04-25T15:26:21.613339+0000","last_active":"2024-04-25T15:26:21.619311+0000","last_peered":"2024-04-25T15:26:21.619311+0000","last_clean":"2024-04-25T15:26:21.619311+0000","last_became_active":"2024-04-25T15:26:21.612982+0000","last_became_peered":"2024-04-25T15:26:21.612982+0000","last_unstale":"2024-04-25T15:26:21.619311+0000","last_undegraded":"2024-04-25T15:26:21.619311+0000","last_fullsized":"2024-04-25T15:26:21.619311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:12.386792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619482+0000","last_change":"2024-04-25T15:26:20.588423+0000","last_active":"2024-04-25T15:26:21.619482+0000","last_peered":"2024-04-25T15:26:21.619482+0000","last_clean":"2024-04-25T15:26:21.619482+0000","last_became_active":"2024-04-25T15:26:20.588085+0000","last_became_peered":"2024-04-25T15:26:20.588085+0000","last_unstale":"2024-04-25T15:26:21.619482+0000","last_undegraded":"2024-04-25T15:26:21.619482+0000","last_fullsized":"2024-04-25T15:26:21.619482+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:17.868269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616834+0000","last_change":"2024-04-25T15:26:21.611139+0000","last_active":"2024-04-25T15:26:21.616834+0000","last_peered":"2024-04-25T15:26:21.616834+0000","last_clean":"2024-04-25T15:26:21.616834+0000","last_became_active":"2024-04-25T15:26:21.610717+0000","last_became_peered":"2024-04-25T15:26:21.610717+0000","last_unstale":"2024-04-25T15:26:21.616834+0000","last_undegraded":"2024-04-25T15:26:21.616834+0000","last_fullsized":"2024-04-25T15:26:21.616834+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:42.757403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.604166+0000","last_change":"2024-04-25T15:26:20.598628+0000","last_active":"2024-04-25T15:26:21.604166+0000","last_peered":"2024-04-25T15:26:21.604166+0000","last_clean":"2024-04-25T15:26:21.604166+0000","last_became_active":"2024-04-25T15:26:20.598326+0000","last_became_peered":"2024-04-25T15:26:20.598326+0000","last_unstale":"2024-04-25T15:26:21.604166+0000","last_undegraded":"2024-04-25T15:26:21.604166+0000","last_fullsized":"2024-04-25T15:26:21.604166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:40.478696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616432+0000","last_change":"2024-04-25T15:26:21.610593+0000","last_active":"2024-04-25T15:26:21.616432+0000","last_peered":"2024-04-25T15:26:21.616432+0000","last_clean":"2024-04-25T15:26:21.616432+0000","last_became_active":"2024-04-25T15:26:21.610264+0000","last_became_peered":"2024-04-25T15:26:21.610264+0000","last_unstale":"2024-04-25T15:26:21.616432+0000","last_undegraded":"2024-04-25T15:26:21.616432+0000","last_fullsized":"2024-04-25T15:26:21.616432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:01.950164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594974+0000","last_change":"2024-04-25T15:26:20.592537+0000","last_active":"2024-04-25T15:26:21.594974+0000","last_peered":"2024-04-25T15:26:21.594974+0000","last_clean":"2024-04-25T15:26:21.594974+0000","last_became_active":"2024-04-25T15:26:20.592235+0000","last_became_peered":"2024-04-25T15:26:20.592235+0000","last_unstale":"2024-04-25T15:26:21.594974+0000","last_undegraded":"2024-04-25T15:26:21.594974+0000","last_fullsized":"2024-04-25T15:26:21.594974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:09.521438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616256+0000","last_change":"2024-04-25T15:26:21.611357+0000","last_active":"2024-04-25T15:26:21.616256+0000","last_peered":"2024-04-25T15:26:21.616256+0000","last_clean":"2024-04-25T15:26:21.616256+0000","last_became_active":"2024-04-25T15:26:21.609870+0000","last_became_peered":"2024-04-25T15:26:21.609870+0000","last_unstale":"2024-04-25T15:26:21.616256+0000","last_undegraded":"2024-04-25T15:26:21.616256+0000","last_fullsized":"2024-04-25T15:26:21.616256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:43.385692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.607923+0000","last_change":"2024-04-25T15:26:20.591277+0000","last_active":"2024-04-25T15:26:21.607923+0000","last_peered":"2024-04-25T15:26:21.607923+0000","last_clean":"2024-04-25T15:26:21.607923+0000","last_became_active":"2024-04-25T15:26:20.590894+0000","last_became_peered":"2024-04-25T15:26:20.590894+0000","last_unstale":"2024-04-25T15:26:21.607923+0000","last_undegraded":"2024-04-25T15:26:21.607923+0000","last_fullsized":"2024-04-25T15:26:21.607923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:10.397526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624606+0000","last_change":"2024-04-25T15:26:21.618029+0000","last_active":"2024-04-25T15:26:21.624606+0000","last_peered":"2024-04-25T15:26:21.624606+0000","last_clean":"2024-04-25T15:26:21.624606+0000","last_became_active":"2024-04-25T15:26:21.617705+0000","last_became_peered":"2024-04-25T15:26:21.617705+0000","last_unstale":"2024-04-25T15:26:21.624606+0000","last_undegraded":"2024-04-25T15:26:21.624606+0000","last_fullsized":"2024-04-25T15:26:21.624606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:45.624513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.620539+0000","last_change":"2024-04-25T15:26:20.595689+0000","last_active":"2024-04-25T15:26:21.620539+0000","last_peered":"2024-04-25T15:26:21.620539+0000","last_clean":"2024-04-25T15:26:21.620539+0000","last_became_active":"2024-04-25T15:26:20.595306+0000","last_became_peered":"2024-04-25T15:26:20.595306+0000","last_unstale":"2024-04-25T15:26:21.620539+0000","last_undegraded":"2024-04-25T15:26:21.620539+0000","last_fullsized":"2024-04-25T15:26:21.620539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:17.868269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.626119+0000","last_change":"2024-04-25T15:26:21.623737+0000","last_active":"2024-04-25T15:26:21.626119+0000","last_peered":"2024-04-25T15:26:21.626119+0000","last_clean":"2024-04-25T15:26:21.626119+0000","last_became_active":"2024-04-25T15:26:21.623414+0000","last_became_peered":"2024-04-25T15:26:21.623414+0000","last_unstale":"2024-04-25T15:26:21.626119+0000","last_undegraded":"2024-04-25T15:26:21.626119+0000","last_fullsized":"2024-04-25T15:26:21.626119+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:17.079917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593561+0000","last_change":"2024-04-25T15:26:20.590835+0000","last_active":"2024-04-25T15:26:21.593561+0000","last_peered":"2024-04-25T15:26:21.593561+0000","last_clean":"2024-04-25T15:26:21.593561+0000","last_became_active":"2024-04-25T15:26:20.590404+0000","last_became_peered":"2024-04-25T15:26:20.590404+0000","last_unstale":"2024-04-25T15:26:21.593561+0000","last_undegraded":"2024-04-25T15:26:21.593561+0000","last_fullsized":"2024-04-25T15:26:21.593561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:27.738092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610118+0000","last_change":"2024-04-25T15:26:21.607117+0000","last_active":"2024-04-25T15:26:21.610118+0000","last_peered":"2024-04-25T15:26:21.610118+0000","last_clean":"2024-04-25T15:26:21.610118+0000","last_became_active":"2024-04-25T15:26:21.605587+0000","last_became_peered":"2024-04-25T15:26:21.605587+0000","last_unstale":"2024-04-25T15:26:21.610118+0000","last_undegraded":"2024-04-25T15:26:21.610118+0000","last_fullsized":"2024-04-25T15:26:21.610118+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:11.403034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.603035+0000","last_change":"2024-04-25T15:26:20.593663+0000","last_active":"2024-04-25T15:26:21.603035+0000","last_peered":"2024-04-25T15:26:21.603035+0000","last_clean":"2024-04-25T15:26:21.603035+0000","last_became_active":"2024-04-25T15:26:20.593326+0000","last_became_peered":"2024-04-25T15:26:20.593326+0000","last_unstale":"2024-04-25T15:26:21.603035+0000","last_undegraded":"2024-04-25T15:26:21.603035+0000","last_fullsized":"2024-04-25T15:26:21.603035+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:19.669539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618842+0000","last_change":"2024-04-25T15:26:21.613992+0000","last_active":"2024-04-25T15:26:21.618842+0000","last_peered":"2024-04-25T15:26:21.618842+0000","last_clean":"2024-04-25T15:26:21.618842+0000","last_became_active":"2024-04-25T15:26:21.613631+0000","last_became_peered":"2024-04-25T15:26:21.613631+0000","last_unstale":"2024-04-25T15:26:21.618842+0000","last_undegraded":"2024-04-25T15:26:21.618842+0000","last_fullsized":"2024-04-25T15:26:21.618842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:10.789126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594262+0000","last_change":"2024-04-25T15:26:20.595308+0000","last_active":"2024-04-25T15:26:21.594262+0000","last_peered":"2024-04-25T15:26:21.594262+0000","last_clean":"2024-04-25T15:26:21.594262+0000","last_became_active":"2024-04-25T15:26:20.594950+0000","last_became_peered":"2024-04-25T15:26:20.594950+0000","last_unstale":"2024-04-25T15:26:21.594262+0000","last_undegraded":"2024-04-25T15:26:21.594262+0000","last_fullsized":"2024-04-25T15:26:21.594262+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:09.783618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.613335+0000","last_change":"2024-04-25T15:26:21.609574+0000","last_active":"2024-04-25T15:26:21.613335+0000","last_peered":"2024-04-25T15:26:21.613335+0000","last_clean":"2024-04-25T15:26:21.613335+0000","last_became_active":"2024-04-25T15:26:21.609208+0000","last_became_peered":"2024-04-25T15:26:21.609208+0000","last_unstale":"2024-04-25T15:26:21.613335+0000","last_undegraded":"2024-04-25T15:26:21.613335+0000","last_fullsized":"2024-04-25T15:26:21.613335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:37.737709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.602887+0000","last_change":"2024-04-25T15:26:20.597216+0000","last_active":"2024-04-25T15:26:21.602887+0000","last_peered":"2024-04-25T15:26:21.602887+0000","last_clean":"2024-04-25T15:26:21.602887+0000","last_became_active":"2024-04-25T15:26:20.596769+0000","last_became_peered":"2024-04-25T15:26:20.596769+0000","last_unstale":"2024-04-25T15:26:21.602887+0000","last_undegraded":"2024-04-25T15:26:21.602887+0000","last_fullsized":"2024-04-25T15:26:21.602887+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:09.783618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612165+0000","last_change":"2024-04-25T15:26:21.607539+0000","last_active":"2024-04-25T15:26:21.612165+0000","last_peered":"2024-04-25T15:26:21.612165+0000","last_clean":"2024-04-25T15:26:21.612165+0000","last_became_active":"2024-04-25T15:26:21.607208+0000","last_became_peered":"2024-04-25T15:26:21.607208+0000","last_unstale":"2024-04-25T15:26:21.612165+0000","last_undegraded":"2024-04-25T15:26:21.612165+0000","last_fullsized":"2024-04-25T15:26:21.612165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:17.079917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.595496+0000","last_change":"2024-04-25T15:26:20.596459+0000","last_active":"2024-04-25T15:26:21.595496+0000","last_peered":"2024-04-25T15:26:21.595496+0000","last_clean":"2024-04-25T15:26:21.595496+0000","last_became_active":"2024-04-25T15:26:20.596109+0000","last_became_peered":"2024-04-25T15:26:20.596109+0000","last_unstale":"2024-04-25T15:26:21.595496+0000","last_undegraded":"2024-04-25T15:26:21.595496+0000","last_fullsized":"2024-04-25T15:26:21.595496+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:10.350348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611589+0000","last_change":"2024-04-25T15:26:21.607534+0000","last_active":"2024-04-25T15:26:21.611589+0000","last_peered":"2024-04-25T15:26:21.611589+0000","last_clean":"2024-04-25T15:26:21.611589+0000","last_became_active":"2024-04-25T15:26:21.607176+0000","last_became_peered":"2024-04-25T15:26:21.607176+0000","last_unstale":"2024-04-25T15:26:21.611589+0000","last_undegraded":"2024-04-25T15:26:21.611589+0000","last_fullsized":"2024-04-25T15:26:21.611589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:18.873777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.592448+0000","last_change":"2024-04-25T15:26:20.603675+0000","last_active":"2024-04-25T15:26:21.592448+0000","last_peered":"2024-04-25T15:26:21.592448+0000","last_clean":"2024-04-25T15:26:21.592448+0000","last_became_active":"2024-04-25T15:26:20.603294+0000","last_became_peered":"2024-04-25T15:26:20.603294+0000","last_unstale":"2024-04-25T15:26:21.592448+0000","last_undegraded":"2024-04-25T15:26:21.592448+0000","last_fullsized":"2024-04-25T15:26:21.592448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:10.397526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618792+0000","last_change":"2024-04-25T15:26:21.617026+0000","last_active":"2024-04-25T15:26:21.618792+0000","last_peered":"2024-04-25T15:26:21.618792+0000","last_clean":"2024-04-25T15:26:21.618792+0000","last_became_active":"2024-04-25T15:26:21.616701+0000","last_became_peered":"2024-04-25T15:26:21.616701+0000","last_unstale":"2024-04-25T15:26:21.618792+0000","last_undegraded":"2024-04-25T15:26:21.618792+0000","last_fullsized":"2024-04-25T15:26:21.618792+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:17.079917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.604139+0000","last_change":"2024-04-25T15:26:20.598000+0000","last_active":"2024-04-25T15:26:21.604139+0000","last_peered":"2024-04-25T15:26:21.604139+0000","last_clean":"2024-04-25T15:26:21.604139+0000","last_became_active":"2024-04-25T15:26:20.597435+0000","last_became_peered":"2024-04-25T15:26:20.597435+0000","last_unstale":"2024-04-25T15:26:21.604139+0000","last_undegraded":"2024-04-25T15:26:21.604139+0000","last_fullsized":"2024-04-25T15:26:21.604139+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:08.190294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":131,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.602291+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:26:21.602291+0000","last_peered":"2024-04-25T15:26:21.602291+0000","last_clean":"2024-04-25T15:26:21.602291+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:26:21.602291+0000","last_undegraded":"2024-04-25T15:26:21.602291+0000","last_fullsized":"2024-04-25T15:26:21.602291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:26:25.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:26:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.805+0000 7f080b296640 1 Processor -- start 2024-04-25T15:26:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.805+0000 7f080b296640 1 -- start start 2024-04-25T15:26:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f080b296640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08040d0e00 0x7f08040d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f080b296640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f080406c5c0 0x7f080406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f080b296640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08040b7810 0x7f08040b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f080b296640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f080406cf10 con 0x7f08040d0e00 2024-04-25T15:26:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f080b296640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f08040d1850 con 0x7f080406c5c0 2024-04-25T15:26:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f080b296640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0804160de0 con 0x7f08040b7810 2024-04-25T15:26:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f0809032640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08040d0e00 0x7f08040d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f0809032640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08040d0e00 0x7f08040d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41552/0 (socket says 172.21.15.67:41552) 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.806+0000 7f0809032640 1 -- 172.21.15.67:0/1114371605 learned_addr learned my addr 172.21.15.67:0/1114371605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809833640 1 --2- 172.21.15.67:0/1114371605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08040b7810 0x7f08040b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0808831640 1 --2- 172.21.15.67:0/1114371605 >> v2:172.21.15.176:3300/0 conn(0x7f080406c5c0 0x7f080406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809032640 1 -- 172.21.15.67:0/1114371605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08040b7810 msgr2=0x7f08040b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809032640 1 --2- 172.21.15.67:0/1114371605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08040b7810 0x7f08040b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809032640 1 -- 172.21.15.67:0/1114371605 >> v2:172.21.15.176:3300/0 conn(0x7f080406c5c0 msgr2=0x7f080406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809032640 1 --2- 172.21.15.67:0/1114371605 >> v2:172.21.15.176:3300/0 conn(0x7f080406c5c0 0x7f080406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809032640 1 -- 172.21.15.67:0/1114371605 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08041610c0 con 0x7f08040d0e00 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0808831640 1 --2- 172.21.15.67:0/1114371605 >> v2:172.21.15.176:3300/0 conn(0x7f080406c5c0 0x7f080406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809833640 1 --2- 172.21.15.67:0/1114371605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08040b7810 0x7f08040b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.807+0000 7f0809032640 1 --2- 172.21.15.67:0/1114371605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040d0e00 0x7f08040d1210 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f080408f170 tx=0x7f080002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c63db1b0dfdeea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f07fbfff640 1 -- 172.21.15.67:0/1114371605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0800037070 con 0x7f08040d0e00 2024-04-25T15:26:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f07fbfff640 1 -- 172.21.15.67:0/1114371605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0800032440 con 0x7f08040d0e00 2024-04-25T15:26:25.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f07fbfff640 1 -- 172.21.15.67:0/1114371605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0800032810 con 0x7f08040d0e00 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 -- 172.21.15.67:0/1114371605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040d0e00 msgr2=0x7f08040d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 --2- 172.21.15.67:0/1114371605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040d0e00 0x7f08040d1210 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f080408f170 tx=0x7f080002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 -- 172.21.15.67:0/1114371605 shutdown_connections 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 --2- 172.21.15.67:0/1114371605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08040b7810 0x7f08040b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 --2- 172.21.15.67:0/1114371605 >> v2:172.21.15.176:3300/0 conn(0x7f080406c5c0 0x7f080406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 --2- 172.21.15.67:0/1114371605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040d0e00 0x7f08040d1210 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 -- 172.21.15.67:0/1114371605 >> 172.21.15.67:0/1114371605 conn(0x7f08040d5d90 msgr2=0x7f08040d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:25.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 -- 172.21.15.67:0/1114371605 shutdown_connections 2024-04-25T15:26:25.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.808+0000 7f080b296640 1 -- 172.21.15.67:0/1114371605 wait complete. 2024-04-25T15:26:25.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.810+0000 7f080b296640 1 Processor -- start 2024-04-25T15:26:25.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.810+0000 7f080b296640 1 -- start start 2024-04-25T15:26:25.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.811+0000 7f080b296640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f080406c5c0 0x7f08041f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.811+0000 7f080b296640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040b7810 0x7f080415a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.811+0000 7f080b296640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08040d0e00 0x7f080415af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f080b296640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f08040b5320 con 0x7f08040b7810 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f080b296640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0804160ad0 con 0x7f08040d0e00 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f080b296640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0804164e00 con 0x7f080406c5c0 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040b7810 0x7f080415a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040b7810 0x7f080415a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41568/0 (socket says 172.21.15.67:41568) 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 -- 172.21.15.67:0/1901416336 learned_addr learned my addr 172.21.15.67:0/1901416336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0809032640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f080406c5c0 0x7f08041f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0809833640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08040d0e00 0x7f080415af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 -- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f080406c5c0 msgr2=0x7f08041f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f080406c5c0 0x7f08041f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 -- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08040d0e00 msgr2=0x7f080415af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08040d0e00 0x7f080415af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:25.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 -- 172.21.15.67:0/1901416336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f080415d5d0 con 0x7f08040b7810 2024-04-25T15:26:25.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0809833640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08040d0e00 0x7f080415af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:25.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f0808831640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040b7810 0x7f080415a6c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f07f400c7f0 tx=0x7f07f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f07f9ffb640 1 -- 172.21.15.67:0/1901416336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4015070 con 0x7f08040b7810 2024-04-25T15:26:25.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f07f9ffb640 1 -- 172.21.15.67:0/1901416336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f4007820 con 0x7f08040b7810 2024-04-25T15:26:25.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f080415d8b0 con 0x7f08040b7810 2024-04-25T15:26:25.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.812+0000 7f07f9ffb640 1 -- 172.21.15.67:0/1901416336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f4007c10 con 0x7f08040b7810 2024-04-25T15:26:25.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.813+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f080415de70 con 0x7f08040b7810 2024-04-25T15:26:25.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.814+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08041648e0 con 0x7f08040b7810 2024-04-25T15:26:25.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.814+0000 7f07f9ffb640 1 -- 172.21.15.67:0/1901416336 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f07f4018070 con 0x7f08040b7810 2024-04-25T15:26:25.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.815+0000 7f07f9ffb640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07e80744b0 0x7f07e80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:25.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.815+0000 7f07f9ffb640 1 -- 172.21.15.67:0/1901416336 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f07f4095370 con 0x7f08040b7810 2024-04-25T15:26:25.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.815+0000 7f0809032640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07e80744b0 0x7f07e80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:25.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.816+0000 7f0809032640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07e80744b0 0x7f07e80769c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0800034000 tx=0x7f080002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:25.820+0000 7f07f9ffb640 1 -- 172.21.15.67:0/1901416336 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08041648e0 con 0x7f08040b7810 2024-04-25T15:26:26.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.001+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0804050740 con 0x7f07e80744b0 2024-04-25T15:26:26.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.008+0000 7f07f9ffb640 1 -- 172.21.15.67:0/1901416336 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251867 (secure 0 0 0) 0x7f0804050740 con 0x7f07e80744b0 2024-04-25T15:26:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:26.012 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:26:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07e80744b0 msgr2=0x7f07e80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07e80744b0 0x7f07e80769c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0800034000 tx=0x7f080002e000 comp rx=0 tx=0).stop 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040b7810 msgr2=0x7f080415a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040b7810 0x7f080415a6c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f07f400c7f0 tx=0x7f07f400ccb0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 shutdown_connections 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f07e80744b0 0x7f07e80769c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08040d0e00 0x7f080415af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08040b7810 0x7f080415a6c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 --2- 172.21.15.67:0/1901416336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f080406c5c0 0x7f08041f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 >> 172.21.15.67:0/1901416336 conn(0x7f08040d5d90 msgr2=0x7f08041641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 shutdown_connections 2024-04-25T15:26:26.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.010+0000 7f080b296640 1 -- 172.21.15.67:0/1901416336 wait complete. 2024-04-25T15:26:26.022 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-25T15:26:24.808786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220052,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754668,"statfs":{"total":773094113280,"available":772868780032,"internally_reserved":0,"allocated":3973120,"data_stored":2249064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.231642"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624549+0000","last_change":"2024-04-25T15:26:21.619518+0000","last_active":"2024-04-25T15:26:21.624549+0000","last_peered":"2024-04-25T15:26:21.624549+0000","last_clean":"2024-04-25T15:26:21.624549+0000","last_became_active":"2024-04-25T15:26:21.619201+0000","last_became_peered":"2024-04-25T15:26:21.619201+0000","last_unstale":"2024-04-25T15:26:21.624549+0000","last_undegraded":"2024-04-25T15:26:21.624549+0000","last_fullsized":"2024-04-25T15:26:21.624549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624617+0000","last_change":"2024-04-25T15:26:21.615948+0000","last_active":"2024-04-25T15:26:21.624617+0000","last_peered":"2024-04-25T15:26:21.624617+0000","last_clean":"2024-04-25T15:26:21.624617+0000","last_became_active":"2024-04-25T15:26:21.614861+0000","last_became_peered":"2024-04-25T15:26:21.614861+0000","last_unstale":"2024-04-25T15:26:21.624617+0000","last_undegraded":"2024-04-25T15:26:21.624617+0000","last_fullsized":"2024-04-25T15:26:21.624617+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:02.188532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625043+0000","last_change":"2024-04-25T15:26:21.621376+0000","last_active":"2024-04-25T15:26:21.625043+0000","last_peered":"2024-04-25T15:26:21.625043+0000","last_clean":"2024-04-25T15:26:21.625043+0000","last_became_active":"2024-04-25T15:26:21.621017+0000","last_became_peered":"2024-04-25T15:26:21.621017+0000","last_unstale":"2024-04-25T15:26:21.625043+0000","last_undegraded":"2024-04-25T15:26:21.625043+0000","last_fullsized":"2024-04-25T15:26:21.625043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:50.775371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.617725+0000","last_change":"2024-04-25T15:26:21.615440+0000","last_active":"2024-04-25T15:26:21.617725+0000","last_peered":"2024-04-25T15:26:21.617725+0000","last_clean":"2024-04-25T15:26:21.617725+0000","last_became_active":"2024-04-25T15:26:21.615098+0000","last_became_peered":"2024-04-25T15:26:21.615098+0000","last_unstale":"2024-04-25T15:26:21.617725+0000","last_undegraded":"2024-04-25T15:26:21.617725+0000","last_fullsized":"2024-04-25T15:26:21.617725+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:50.775371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616801+0000","last_change":"2024-04-25T15:26:21.608591+0000","last_active":"2024-04-25T15:26:21.616801+0000","last_peered":"2024-04-25T15:26:21.616801+0000","last_clean":"2024-04-25T15:26:21.616801+0000","last_became_active":"2024-04-25T15:26:21.608233+0000","last_became_peered":"2024-04-25T15:26:21.608233+0000","last_unstale":"2024-04-25T15:26:21.616801+0000","last_undegraded":"2024-04-25T15:26:21.616801+0000","last_fullsized":"2024-04-25T15:26:21.616801+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:41.476011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.617241+0000","last_change":"2024-04-25T15:26:21.614891+0000","last_active":"2024-04-25T15:26:21.617241+0000","last_peered":"2024-04-25T15:26:21.617241+0000","last_clean":"2024-04-25T15:26:21.617241+0000","last_became_active":"2024-04-25T15:26:21.614560+0000","last_became_peered":"2024-04-25T15:26:21.614560+0000","last_unstale":"2024-04-25T15:26:21.617241+0000","last_undegraded":"2024-04-25T15:26:21.617241+0000","last_fullsized":"2024-04-25T15:26:21.617241+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:05.105207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616244+0000","last_change":"2024-04-25T15:26:21.610992+0000","last_active":"2024-04-25T15:26:21.616244+0000","last_peered":"2024-04-25T15:26:21.616244+0000","last_clean":"2024-04-25T15:26:21.616244+0000","last_became_active":"2024-04-25T15:26:21.609746+0000","last_became_peered":"2024-04-25T15:26:21.609746+0000","last_unstale":"2024-04-25T15:26:21.616244+0000","last_undegraded":"2024-04-25T15:26:21.616244+0000","last_fullsized":"2024-04-25T15:26:21.616244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:11.403034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618750+0000","last_change":"2024-04-25T15:26:21.616938+0000","last_active":"2024-04-25T15:26:21.618750+0000","last_peered":"2024-04-25T15:26:21.618750+0000","last_clean":"2024-04-25T15:26:21.618750+0000","last_became_active":"2024-04-25T15:26:21.616674+0000","last_became_peered":"2024-04-25T15:26:21.616674+0000","last_unstale":"2024-04-25T15:26:21.618750+0000","last_undegraded":"2024-04-25T15:26:21.618750+0000","last_fullsized":"2024-04-25T15:26:21.618750+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:56.376686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611174+0000","last_change":"2024-04-25T15:26:21.607747+0000","last_active":"2024-04-25T15:26:21.611174+0000","last_peered":"2024-04-25T15:26:21.611174+0000","last_clean":"2024-04-25T15:26:21.611174+0000","last_became_active":"2024-04-25T15:26:21.607360+0000","last_became_peered":"2024-04-25T15:26:21.607360+0000","last_unstale":"2024-04-25T15:26:21.611174+0000","last_undegraded":"2024-04-25T15:26:21.611174+0000","last_fullsized":"2024-04-25T15:26:21.611174+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:10.789126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612205+0000","last_change":"2024-04-25T15:26:21.605133+0000","last_active":"2024-04-25T15:26:21.612205+0000","last_peered":"2024-04-25T15:26:21.612205+0000","last_clean":"2024-04-25T15:26:21.612205+0000","last_became_active":"2024-04-25T15:26:21.604842+0000","last_became_peered":"2024-04-25T15:26:21.604842+0000","last_unstale":"2024-04-25T15:26:21.612205+0000","last_undegraded":"2024-04-25T15:26:21.612205+0000","last_fullsized":"2024-04-25T15:26:21.612205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618696+0000","last_change":"2024-04-25T15:26:21.615052+0000","last_active":"2024-04-25T15:26:21.618696+0000","last_peered":"2024-04-25T15:26:21.618696+0000","last_clean":"2024-04-25T15:26:21.618696+0000","last_became_active":"2024-04-25T15:26:21.614679+0000","last_became_peered":"2024-04-25T15:26:21.614679+0000","last_unstale":"2024-04-25T15:26:21.618696+0000","last_undegraded":"2024-04-25T15:26:21.618696+0000","last_fullsized":"2024-04-25T15:26:21.618696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:53.994403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625096+0000","last_change":"2024-04-25T15:26:21.622166+0000","last_active":"2024-04-25T15:26:21.625096+0000","last_peered":"2024-04-25T15:26:21.625096+0000","last_clean":"2024-04-25T15:26:21.625096+0000","last_became_active":"2024-04-25T15:26:21.621858+0000","last_became_peered":"2024-04-25T15:26:21.621858+0000","last_unstale":"2024-04-25T15:26:21.625096+0000","last_undegraded":"2024-04-25T15:26:21.625096+0000","last_fullsized":"2024-04-25T15:26:21.625096+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:53.994403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619262+0000","last_change":"2024-04-25T15:26:21.613529+0000","last_active":"2024-04-25T15:26:21.619262+0000","last_peered":"2024-04-25T15:26:21.619262+0000","last_clean":"2024-04-25T15:26:21.619262+0000","last_became_active":"2024-04-25T15:26:21.613215+0000","last_became_peered":"2024-04-25T15:26:21.613215+0000","last_unstale":"2024-04-25T15:26:21.619262+0000","last_undegraded":"2024-04-25T15:26:21.619262+0000","last_fullsized":"2024-04-25T15:26:21.619262+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:38.942978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625674+0000","last_change":"2024-04-25T15:26:21.618919+0000","last_active":"2024-04-25T15:26:21.625674+0000","last_peered":"2024-04-25T15:26:21.625674+0000","last_clean":"2024-04-25T15:26:21.625674+0000","last_became_active":"2024-04-25T15:26:21.617169+0000","last_became_peered":"2024-04-25T15:26:21.617169+0000","last_unstale":"2024-04-25T15:26:21.625674+0000","last_undegraded":"2024-04-25T15:26:21.625674+0000","last_fullsized":"2024-04-25T15:26:21.625674+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:52.829405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624274+0000","last_change":"2024-04-25T15:26:21.619759+0000","last_active":"2024-04-25T15:26:21.624274+0000","last_peered":"2024-04-25T15:26:21.624274+0000","last_clean":"2024-04-25T15:26:21.624274+0000","last_became_active":"2024-04-25T15:26:21.619377+0000","last_became_peered":"2024-04-25T15:26:21.619377+0000","last_unstale":"2024-04-25T15:26:21.624274+0000","last_undegraded":"2024-04-25T15:26:21.624274+0000","last_fullsized":"2024-04-25T15:26:21.624274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:20.151135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612971+0000","last_change":"2024-04-25T15:26:21.609797+0000","last_active":"2024-04-25T15:26:21.612971+0000","last_peered":"2024-04-25T15:26:21.612971+0000","last_clean":"2024-04-25T15:26:21.612971+0000","last_became_active":"2024-04-25T15:26:21.609507+0000","last_became_peered":"2024-04-25T15:26:21.609507+0000","last_unstale":"2024-04-25T15:26:21.612971+0000","last_undegraded":"2024-04-25T15:26:21.612971+0000","last_fullsized":"2024-04-25T15:26:21.612971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616335+0000","last_change":"2024-04-25T15:26:21.611172+0000","last_active":"2024-04-25T15:26:21.616335+0000","last_peered":"2024-04-25T15:26:21.616335+0000","last_clean":"2024-04-25T15:26:21.616335+0000","last_became_active":"2024-04-25T15:26:21.610131+0000","last_became_peered":"2024-04-25T15:26:21.610131+0000","last_unstale":"2024-04-25T15:26:21.616335+0000","last_undegraded":"2024-04-25T15:26:21.616335+0000","last_fullsized":"2024-04-25T15:26:21.616335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:37.737709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611998+0000","last_change":"2024-04-25T15:26:21.607940+0000","last_active":"2024-04-25T15:26:21.611998+0000","last_peered":"2024-04-25T15:26:21.611998+0000","last_clean":"2024-04-25T15:26:21.611998+0000","last_became_active":"2024-04-25T15:26:21.607301+0000","last_became_peered":"2024-04-25T15:26:21.607301+0000","last_unstale":"2024-04-25T15:26:21.611998+0000","last_undegraded":"2024-04-25T15:26:21.611998+0000","last_fullsized":"2024-04-25T15:26:21.611998+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:02.445672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616515+0000","last_change":"2024-04-25T15:26:21.610318+0000","last_active":"2024-04-25T15:26:21.616515+0000","last_peered":"2024-04-25T15:26:21.616515+0000","last_clean":"2024-04-25T15:26:21.616515+0000","last_became_active":"2024-04-25T15:26:21.609989+0000","last_became_peered":"2024-04-25T15:26:21.609989+0000","last_unstale":"2024-04-25T15:26:21.616515+0000","last_undegraded":"2024-04-25T15:26:21.616515+0000","last_fullsized":"2024-04-25T15:26:21.616515+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:43.820705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619336+0000","last_change":"2024-04-25T15:26:21.615358+0000","last_active":"2024-04-25T15:26:21.619336+0000","last_peered":"2024-04-25T15:26:21.619336+0000","last_clean":"2024-04-25T15:26:21.619336+0000","last_became_active":"2024-04-25T15:26:21.615013+0000","last_became_peered":"2024-04-25T15:26:21.615013+0000","last_unstale":"2024-04-25T15:26:21.619336+0000","last_undegraded":"2024-04-25T15:26:21.619336+0000","last_fullsized":"2024-04-25T15:26:21.619336+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:08.299329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.603983+0000","last_change":"2024-04-25T15:26:21.597722+0000","last_active":"2024-04-25T15:26:21.603983+0000","last_peered":"2024-04-25T15:26:21.603983+0000","last_clean":"2024-04-25T15:26:21.603983+0000","last_became_active":"2024-04-25T15:26:21.597413+0000","last_became_peered":"2024-04-25T15:26:21.597413+0000","last_unstale":"2024-04-25T15:26:21.603983+0000","last_undegraded":"2024-04-25T15:26:21.603983+0000","last_fullsized":"2024-04-25T15:26:21.603983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:41.484204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.604698+0000","last_change":"2024-04-25T15:26:21.597463+0000","last_active":"2024-04-25T15:26:21.604698+0000","last_peered":"2024-04-25T15:26:21.604698+0000","last_clean":"2024-04-25T15:26:21.604698+0000","last_became_active":"2024-04-25T15:26:21.597163+0000","last_became_peered":"2024-04-25T15:26:21.597163+0000","last_unstale":"2024-04-25T15:26:21.604698+0000","last_undegraded":"2024-04-25T15:26:21.604698+0000","last_fullsized":"2024-04-25T15:26:21.604698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:09.195802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610934+0000","last_change":"2024-04-25T15:26:21.606008+0000","last_active":"2024-04-25T15:26:21.610934+0000","last_peered":"2024-04-25T15:26:21.610934+0000","last_clean":"2024-04-25T15:26:21.610934+0000","last_became_active":"2024-04-25T15:26:21.605604+0000","last_became_peered":"2024-04-25T15:26:21.605604+0000","last_unstale":"2024-04-25T15:26:21.610934+0000","last_undegraded":"2024-04-25T15:26:21.610934+0000","last_fullsized":"2024-04-25T15:26:21.610934+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621627+0000","last_change":"2024-04-25T15:26:21.618252+0000","last_active":"2024-04-25T15:26:21.621627+0000","last_peered":"2024-04-25T15:26:21.621627+0000","last_clean":"2024-04-25T15:26:21.621627+0000","last_became_active":"2024-04-25T15:26:21.617908+0000","last_became_peered":"2024-04-25T15:26:21.617908+0000","last_unstale":"2024-04-25T15:26:21.621627+0000","last_undegraded":"2024-04-25T15:26:21.621627+0000","last_fullsized":"2024-04-25T15:26:21.621627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:43.385692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.614064+0000","last_change":"2024-04-25T15:26:21.608061+0000","last_active":"2024-04-25T15:26:21.614064+0000","last_peered":"2024-04-25T15:26:21.614064+0000","last_clean":"2024-04-25T15:26:21.614064+0000","last_became_active":"2024-04-25T15:26:21.607706+0000","last_became_peered":"2024-04-25T15:26:21.607706+0000","last_unstale":"2024-04-25T15:26:21.614064+0000","last_undegraded":"2024-04-25T15:26:21.614064+0000","last_fullsized":"2024-04-25T15:26:21.614064+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:37.737709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612025+0000","last_change":"2024-04-25T15:26:21.608873+0000","last_active":"2024-04-25T15:26:21.612025+0000","last_peered":"2024-04-25T15:26:21.612025+0000","last_clean":"2024-04-25T15:26:21.612025+0000","last_became_active":"2024-04-25T15:26:21.608521+0000","last_became_peered":"2024-04-25T15:26:21.608521+0000","last_unstale":"2024-04-25T15:26:21.612025+0000","last_undegraded":"2024-04-25T15:26:21.612025+0000","last_fullsized":"2024-04-25T15:26:21.612025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:42.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618176+0000","last_change":"2024-04-25T15:26:21.614973+0000","last_active":"2024-04-25T15:26:21.618176+0000","last_peered":"2024-04-25T15:26:21.618176+0000","last_clean":"2024-04-25T15:26:21.618176+0000","last_became_active":"2024-04-25T15:26:21.614667+0000","last_became_peered":"2024-04-25T15:26:21.614667+0000","last_unstale":"2024-04-25T15:26:21.618176+0000","last_undegraded":"2024-04-25T15:26:21.618176+0000","last_fullsized":"2024-04-25T15:26:21.618176+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:43.385692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630558+0000","last_change":"2024-04-25T15:26:21.626208+0000","last_active":"2024-04-25T15:26:21.630558+0000","last_peered":"2024-04-25T15:26:21.630558+0000","last_clean":"2024-04-25T15:26:21.630558+0000","last_became_active":"2024-04-25T15:26:21.625871+0000","last_became_peered":"2024-04-25T15:26:21.625871+0000","last_unstale":"2024-04-25T15:26:21.630558+0000","last_undegraded":"2024-04-25T15:26:21.630558+0000","last_fullsized":"2024-04-25T15:26:21.630558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:42.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611384+0000","last_change":"2024-04-25T15:26:21.600350+0000","last_active":"2024-04-25T15:26:21.611384+0000","last_peered":"2024-04-25T15:26:21.611384+0000","last_clean":"2024-04-25T15:26:21.611384+0000","last_became_active":"2024-04-25T15:26:21.600011+0000","last_became_peered":"2024-04-25T15:26:21.600011+0000","last_unstale":"2024-04-25T15:26:21.611384+0000","last_undegraded":"2024-04-25T15:26:21.611384+0000","last_fullsized":"2024-04-25T15:26:21.611384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:21.482642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610855+0000","last_change":"2024-04-25T15:26:21.609147+0000","last_active":"2024-04-25T15:26:21.610855+0000","last_peered":"2024-04-25T15:26:21.610855+0000","last_clean":"2024-04-25T15:26:21.610855+0000","last_became_active":"2024-04-25T15:26:21.608816+0000","last_became_peered":"2024-04-25T15:26:21.608816+0000","last_unstale":"2024-04-25T15:26:21.610855+0000","last_undegraded":"2024-04-25T15:26:21.610855+0000","last_fullsized":"2024-04-25T15:26:21.610855+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618877+0000","last_change":"2024-04-25T15:26:21.615295+0000","last_active":"2024-04-25T15:26:21.618877+0000","last_peered":"2024-04-25T15:26:21.618877+0000","last_clean":"2024-04-25T15:26:21.618877+0000","last_became_active":"2024-04-25T15:26:21.614134+0000","last_became_peered":"2024-04-25T15:26:21.614134+0000","last_unstale":"2024-04-25T15:26:21.618877+0000","last_undegraded":"2024-04-25T15:26:21.618877+0000","last_fullsized":"2024-04-25T15:26:21.618877+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616215+0000","last_change":"2024-04-25T15:26:21.612104+0000","last_active":"2024-04-25T15:26:21.616215+0000","last_peered":"2024-04-25T15:26:21.616215+0000","last_clean":"2024-04-25T15:26:21.616215+0000","last_became_active":"2024-04-25T15:26:21.611800+0000","last_became_peered":"2024-04-25T15:26:21.611800+0000","last_unstale":"2024-04-25T15:26:21.616215+0000","last_undegraded":"2024-04-25T15:26:21.616215+0000","last_fullsized":"2024-04-25T15:26:21.616215+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:20.151135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610973+0000","last_change":"2024-04-25T15:26:21.606993+0000","last_active":"2024-04-25T15:26:21.610973+0000","last_peered":"2024-04-25T15:26:21.610973+0000","last_clean":"2024-04-25T15:26:21.610973+0000","last_became_active":"2024-04-25T15:26:21.606051+0000","last_became_peered":"2024-04-25T15:26:21.606051+0000","last_unstale":"2024-04-25T15:26:21.610973+0000","last_undegraded":"2024-04-25T15:26:21.610973+0000","last_fullsized":"2024-04-25T15:26:21.610973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.626942+0000","last_change":"2024-04-25T15:26:20.596541+0000","last_active":"2024-04-25T15:26:21.626942+0000","last_peered":"2024-04-25T15:26:21.626942+0000","last_clean":"2024-04-25T15:26:21.626942+0000","last_became_active":"2024-04-25T15:26:20.596246+0000","last_became_peered":"2024-04-25T15:26:20.596246+0000","last_unstale":"2024-04-25T15:26:21.626942+0000","last_undegraded":"2024-04-25T15:26:21.626942+0000","last_fullsized":"2024-04-25T15:26:21.626942+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:27.738092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618632+0000","last_change":"2024-04-25T15:26:21.612216+0000","last_active":"2024-04-25T15:26:21.618632+0000","last_peered":"2024-04-25T15:26:21.618632+0000","last_clean":"2024-04-25T15:26:21.618632+0000","last_became_active":"2024-04-25T15:26:21.611904+0000","last_became_peered":"2024-04-25T15:26:21.611904+0000","last_unstale":"2024-04-25T15:26:21.618632+0000","last_undegraded":"2024-04-25T15:26:21.618632+0000","last_fullsized":"2024-04-25T15:26:21.618632+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:21.482642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619907+0000","last_change":"2024-04-25T15:26:20.589623+0000","last_active":"2024-04-25T15:26:21.619907+0000","last_peered":"2024-04-25T15:26:21.619907+0000","last_clean":"2024-04-25T15:26:21.619907+0000","last_became_active":"2024-04-25T15:26:20.589260+0000","last_became_peered":"2024-04-25T15:26:20.589260+0000","last_unstale":"2024-04-25T15:26:21.619907+0000","last_undegraded":"2024-04-25T15:26:21.619907+0000","last_fullsized":"2024-04-25T15:26:21.619907+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:40.478696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611344+0000","last_change":"2024-04-25T15:26:21.600341+0000","last_active":"2024-04-25T15:26:21.611344+0000","last_peered":"2024-04-25T15:26:21.611344+0000","last_clean":"2024-04-25T15:26:21.611344+0000","last_became_active":"2024-04-25T15:26:21.599984+0000","last_became_peered":"2024-04-25T15:26:21.599984+0000","last_unstale":"2024-04-25T15:26:21.611344+0000","last_undegraded":"2024-04-25T15:26:21.611344+0000","last_fullsized":"2024-04-25T15:26:21.611344+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:28.743600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.627126+0000","last_change":"2024-04-25T15:26:20.602697+0000","last_active":"2024-04-25T15:26:21.627126+0000","last_peered":"2024-04-25T15:26:21.627126+0000","last_clean":"2024-04-25T15:26:21.627126+0000","last_became_active":"2024-04-25T15:26:20.602306+0000","last_became_peered":"2024-04-25T15:26:20.602306+0000","last_unstale":"2024-04-25T15:26:21.627126+0000","last_undegraded":"2024-04-25T15:26:21.627126+0000","last_fullsized":"2024-04-25T15:26:21.627126+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:19.145627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618202+0000","last_change":"2024-04-25T15:26:21.615092+0000","last_active":"2024-04-25T15:26:21.618202+0000","last_peered":"2024-04-25T15:26:21.618202+0000","last_clean":"2024-04-25T15:26:21.618202+0000","last_became_active":"2024-04-25T15:26:21.614764+0000","last_became_peered":"2024-04-25T15:26:21.614764+0000","last_unstale":"2024-04-25T15:26:21.618202+0000","last_undegraded":"2024-04-25T15:26:21.618202+0000","last_fullsized":"2024-04-25T15:26:21.618202+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:20.151135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.622967+0000","last_change":"2024-04-25T15:26:20.598502+0000","last_active":"2024-04-25T15:26:21.622967+0000","last_peered":"2024-04-25T15:26:21.622967+0000","last_clean":"2024-04-25T15:26:21.622967+0000","last_became_active":"2024-04-25T15:26:20.598167+0000","last_became_peered":"2024-04-25T15:26:20.598167+0000","last_unstale":"2024-04-25T15:26:21.622967+0000","last_undegraded":"2024-04-25T15:26:21.622967+0000","last_fullsized":"2024-04-25T15:26:21.622967+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:28.020892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618734+0000","last_change":"2024-04-25T15:26:21.613803+0000","last_active":"2024-04-25T15:26:21.618734+0000","last_peered":"2024-04-25T15:26:21.618734+0000","last_clean":"2024-04-25T15:26:21.618734+0000","last_became_active":"2024-04-25T15:26:21.613495+0000","last_became_peered":"2024-04-25T15:26:21.613495+0000","last_unstale":"2024-04-25T15:26:21.618734+0000","last_undegraded":"2024-04-25T15:26:21.618734+0000","last_fullsized":"2024-04-25T15:26:21.618734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:50.565418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630809+0000","last_change":"2024-04-25T15:26:20.602555+0000","last_active":"2024-04-25T15:26:21.630809+0000","last_peered":"2024-04-25T15:26:21.630809+0000","last_clean":"2024-04-25T15:26:21.630809+0000","last_became_active":"2024-04-25T15:26:20.602210+0000","last_became_peered":"2024-04-25T15:26:20.602210+0000","last_unstale":"2024-04-25T15:26:21.630809+0000","last_undegraded":"2024-04-25T15:26:21.630809+0000","last_fullsized":"2024-04-25T15:26:21.630809+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:44.619005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630565+0000","last_change":"2024-04-25T15:26:21.626293+0000","last_active":"2024-04-25T15:26:21.630565+0000","last_peered":"2024-04-25T15:26:21.630565+0000","last_clean":"2024-04-25T15:26:21.630565+0000","last_became_active":"2024-04-25T15:26:21.625687+0000","last_became_peered":"2024-04-25T15:26:21.625687+0000","last_unstale":"2024-04-25T15:26:21.630565+0000","last_undegraded":"2024-04-25T15:26:21.630565+0000","last_fullsized":"2024-04-25T15:26:21.630565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:37.980797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630808+0000","last_change":"2024-04-25T15:26:20.598522+0000","last_active":"2024-04-25T15:26:21.630808+0000","last_peered":"2024-04-25T15:26:21.630808+0000","last_clean":"2024-04-25T15:26:21.630808+0000","last_became_active":"2024-04-25T15:26:20.598176+0000","last_became_peered":"2024-04-25T15:26:20.598176+0000","last_unstale":"2024-04-25T15:26:21.630808+0000","last_undegraded":"2024-04-25T15:26:21.630808+0000","last_fullsized":"2024-04-25T15:26:21.630808+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:37.937470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.615501+0000","last_change":"2024-04-25T15:26:21.609992+0000","last_active":"2024-04-25T15:26:21.615501+0000","last_peered":"2024-04-25T15:26:21.615501+0000","last_clean":"2024-04-25T15:26:21.615501+0000","last_became_active":"2024-04-25T15:26:21.609654+0000","last_became_peered":"2024-04-25T15:26:21.609654+0000","last_unstale":"2024-04-25T15:26:21.615501+0000","last_undegraded":"2024-04-25T15:26:21.615501+0000","last_fullsized":"2024-04-25T15:26:21.615501+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:20.675047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593130+0000","last_change":"2024-04-25T15:26:20.603710+0000","last_active":"2024-04-25T15:26:21.593130+0000","last_peered":"2024-04-25T15:26:21.593130+0000","last_clean":"2024-04-25T15:26:21.593130+0000","last_became_active":"2024-04-25T15:26:20.603360+0000","last_became_peered":"2024-04-25T15:26:20.603360+0000","last_unstale":"2024-04-25T15:26:21.593130+0000","last_undegraded":"2024-04-25T15:26:21.593130+0000","last_fullsized":"2024-04-25T15:26:21.593130+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:42.380184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621024+0000","last_change":"2024-04-25T15:26:21.618982+0000","last_active":"2024-04-25T15:26:21.621024+0000","last_peered":"2024-04-25T15:26:21.621024+0000","last_clean":"2024-04-25T15:26:21.621024+0000","last_became_active":"2024-04-25T15:26:21.618667+0000","last_became_peered":"2024-04-25T15:26:21.618667+0000","last_unstale":"2024-04-25T15:26:21.621024+0000","last_undegraded":"2024-04-25T15:26:21.621024+0000","last_fullsized":"2024-04-25T15:26:21.621024+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:50.565418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594764+0000","last_change":"2024-04-25T15:26:20.597982+0000","last_active":"2024-04-25T15:26:21.594764+0000","last_peered":"2024-04-25T15:26:21.594764+0000","last_clean":"2024-04-25T15:26:21.594764+0000","last_became_active":"2024-04-25T15:26:20.596709+0000","last_became_peered":"2024-04-25T15:26:20.596709+0000","last_unstale":"2024-04-25T15:26:21.594764+0000","last_undegraded":"2024-04-25T15:26:21.594764+0000","last_fullsized":"2024-04-25T15:26:21.594764+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:02.951986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.628075+0000","last_change":"2024-04-25T15:26:21.625576+0000","last_active":"2024-04-25T15:26:21.628075+0000","last_peered":"2024-04-25T15:26:21.628075+0000","last_clean":"2024-04-25T15:26:21.628075+0000","last_became_active":"2024-04-25T15:26:21.625169+0000","last_became_peered":"2024-04-25T15:26:21.625169+0000","last_unstale":"2024-04-25T15:26:21.628075+0000","last_undegraded":"2024-04-25T15:26:21.628075+0000","last_fullsized":"2024-04-25T15:26:21.628075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:02.445672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593743+0000","last_change":"2024-04-25T15:26:20.602652+0000","last_active":"2024-04-25T15:26:21.593743+0000","last_peered":"2024-04-25T15:26:21.593743+0000","last_clean":"2024-04-25T15:26:21.593743+0000","last_became_active":"2024-04-25T15:26:20.602042+0000","last_became_peered":"2024-04-25T15:26:20.602042+0000","last_unstale":"2024-04-25T15:26:21.593743+0000","last_undegraded":"2024-04-25T15:26:21.593743+0000","last_fullsized":"2024-04-25T15:26:21.593743+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:49.769863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611944+0000","last_change":"2024-04-25T15:26:21.606926+0000","last_active":"2024-04-25T15:26:21.611944+0000","last_peered":"2024-04-25T15:26:21.611944+0000","last_clean":"2024-04-25T15:26:21.611944+0000","last_became_active":"2024-04-25T15:26:21.606531+0000","last_became_peered":"2024-04-25T15:26:21.606531+0000","last_unstale":"2024-04-25T15:26:21.611944+0000","last_undegraded":"2024-04-25T15:26:21.611944+0000","last_fullsized":"2024-04-25T15:26:21.611944+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:56.376686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593743+0000","last_change":"2024-04-25T15:26:20.598360+0000","last_active":"2024-04-25T15:26:21.593743+0000","last_peered":"2024-04-25T15:26:21.593743+0000","last_clean":"2024-04-25T15:26:21.593743+0000","last_became_active":"2024-04-25T15:26:20.598042+0000","last_became_peered":"2024-04-25T15:26:20.598042+0000","last_unstale":"2024-04-25T15:26:21.593743+0000","last_undegraded":"2024-04-25T15:26:21.593743+0000","last_fullsized":"2024-04-25T15:26:21.593743+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:20.477134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621075+0000","last_change":"2024-04-25T15:26:21.609926+0000","last_active":"2024-04-25T15:26:21.621075+0000","last_peered":"2024-04-25T15:26:21.621075+0000","last_clean":"2024-04-25T15:26:21.621075+0000","last_became_active":"2024-04-25T15:26:21.608497+0000","last_became_peered":"2024-04-25T15:26:21.608497+0000","last_unstale":"2024-04-25T15:26:21.621075+0000","last_undegraded":"2024-04-25T15:26:21.621075+0000","last_fullsized":"2024-04-25T15:26:21.621075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:37.980797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624871+0000","last_change":"2024-04-25T15:26:20.591095+0000","last_active":"2024-04-25T15:26:21.624871+0000","last_peered":"2024-04-25T15:26:21.624871+0000","last_clean":"2024-04-25T15:26:21.624871+0000","last_became_active":"2024-04-25T15:26:20.590747+0000","last_became_peered":"2024-04-25T15:26:20.590747+0000","last_unstale":"2024-04-25T15:26:21.624871+0000","last_undegraded":"2024-04-25T15:26:21.624871+0000","last_fullsized":"2024-04-25T15:26:21.624871+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:36.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.625073+0000","last_change":"2024-04-25T15:26:21.621439+0000","last_active":"2024-04-25T15:26:21.625073+0000","last_peered":"2024-04-25T15:26:21.625073+0000","last_clean":"2024-04-25T15:26:21.625073+0000","last_became_active":"2024-04-25T15:26:21.621101+0000","last_became_peered":"2024-04-25T15:26:21.621101+0000","last_unstale":"2024-04-25T15:26:21.625073+0000","last_undegraded":"2024-04-25T15:26:21.625073+0000","last_fullsized":"2024-04-25T15:26:21.625073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:50.565418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.633229+0000","last_change":"2024-04-25T15:26:20.602796+0000","last_active":"2024-04-25T15:26:21.633229+0000","last_peered":"2024-04-25T15:26:21.633229+0000","last_clean":"2024-04-25T15:26:21.633229+0000","last_became_active":"2024-04-25T15:26:20.602443+0000","last_became_peered":"2024-04-25T15:26:20.602443+0000","last_unstale":"2024-04-25T15:26:21.633229+0000","last_undegraded":"2024-04-25T15:26:21.633229+0000","last_fullsized":"2024-04-25T15:26:21.633229+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:52.988896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.621062+0000","last_change":"2024-04-25T15:26:21.619342+0000","last_active":"2024-04-25T15:26:21.621062+0000","last_peered":"2024-04-25T15:26:21.621062+0000","last_clean":"2024-04-25T15:26:21.621062+0000","last_became_active":"2024-04-25T15:26:21.618651+0000","last_became_peered":"2024-04-25T15:26:21.618651+0000","last_unstale":"2024-04-25T15:26:21.621062+0000","last_undegraded":"2024-04-25T15:26:21.621062+0000","last_fullsized":"2024-04-25T15:26:21.621062+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:21:53.994403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630840+0000","last_change":"2024-04-25T15:26:20.591644+0000","last_active":"2024-04-25T15:26:21.630840+0000","last_peered":"2024-04-25T15:26:21.630840+0000","last_clean":"2024-04-25T15:26:21.630840+0000","last_became_active":"2024-04-25T15:26:20.591346+0000","last_became_peered":"2024-04-25T15:26:20.591346+0000","last_unstale":"2024-04-25T15:26:21.630840+0000","last_undegraded":"2024-04-25T15:26:21.630840+0000","last_fullsized":"2024-04-25T15:26:21.630840+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:19.188291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616899+0000","last_change":"2024-04-25T15:26:21.615068+0000","last_active":"2024-04-25T15:26:21.616899+0000","last_peered":"2024-04-25T15:26:21.616899+0000","last_clean":"2024-04-25T15:26:21.616899+0000","last_became_active":"2024-04-25T15:26:21.614637+0000","last_became_peered":"2024-04-25T15:26:21.614637+0000","last_unstale":"2024-04-25T15:26:21.616899+0000","last_undegraded":"2024-04-25T15:26:21.616899+0000","last_fullsized":"2024-04-25T15:26:21.616899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:29.792437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.602696+0000","last_change":"2024-04-25T15:26:20.595469+0000","last_active":"2024-04-25T15:26:21.602696+0000","last_peered":"2024-04-25T15:26:21.602696+0000","last_clean":"2024-04-25T15:26:21.602696+0000","last_became_active":"2024-04-25T15:26:20.594200+0000","last_became_peered":"2024-04-25T15:26:20.594200+0000","last_unstale":"2024-04-25T15:26:21.602696+0000","last_undegraded":"2024-04-25T15:26:21.602696+0000","last_fullsized":"2024-04-25T15:26:21.602696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:20.477134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618565+0000","last_change":"2024-04-25T15:26:21.612824+0000","last_active":"2024-04-25T15:26:21.618565+0000","last_peered":"2024-04-25T15:26:21.618565+0000","last_clean":"2024-04-25T15:26:21.618565+0000","last_became_active":"2024-04-25T15:26:21.612525+0000","last_became_peered":"2024-04-25T15:26:21.612525+0000","last_unstale":"2024-04-25T15:26:21.618565+0000","last_undegraded":"2024-04-25T15:26:21.618565+0000","last_fullsized":"2024-04-25T15:26:21.618565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:50.775371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.592502+0000","last_change":"2024-04-25T15:26:20.603523+0000","last_active":"2024-04-25T15:26:21.592502+0000","last_peered":"2024-04-25T15:26:21.592502+0000","last_clean":"2024-04-25T15:26:21.592502+0000","last_became_active":"2024-04-25T15:26:20.603084+0000","last_became_peered":"2024-04-25T15:26:20.603084+0000","last_unstale":"2024-04-25T15:26:21.592502+0000","last_undegraded":"2024-04-25T15:26:21.592502+0000","last_fullsized":"2024-04-25T15:26:21.592502+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:36.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618722+0000","last_change":"2024-04-25T15:26:21.613399+0000","last_active":"2024-04-25T15:26:21.618722+0000","last_peered":"2024-04-25T15:26:21.618722+0000","last_clean":"2024-04-25T15:26:21.618722+0000","last_became_active":"2024-04-25T15:26:21.612998+0000","last_became_peered":"2024-04-25T15:26:21.612998+0000","last_unstale":"2024-04-25T15:26:21.618722+0000","last_undegraded":"2024-04-25T15:26:21.618722+0000","last_fullsized":"2024-04-25T15:26:21.618722+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:18.873777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.630780+0000","last_change":"2024-04-25T15:26:20.602960+0000","last_active":"2024-04-25T15:26:21.630780+0000","last_peered":"2024-04-25T15:26:21.630780+0000","last_clean":"2024-04-25T15:26:21.630780+0000","last_became_active":"2024-04-25T15:26:20.602435+0000","last_became_peered":"2024-04-25T15:26:20.602435+0000","last_unstale":"2024-04-25T15:26:21.630780+0000","last_undegraded":"2024-04-25T15:26:21.630780+0000","last_fullsized":"2024-04-25T15:26:21.630780+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:49.559910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.622204+0000","last_change":"2024-04-25T15:26:21.619051+0000","last_active":"2024-04-25T15:26:21.622204+0000","last_peered":"2024-04-25T15:26:21.622204+0000","last_clean":"2024-04-25T15:26:21.622204+0000","last_became_active":"2024-04-25T15:26:21.618737+0000","last_became_peered":"2024-04-25T15:26:21.618737+0000","last_unstale":"2024-04-25T15:26:21.622204+0000","last_undegraded":"2024-04-25T15:26:21.622204+0000","last_fullsized":"2024-04-25T15:26:21.622204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:29.161584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624776+0000","last_change":"2024-04-25T15:26:20.604571+0000","last_active":"2024-04-25T15:26:21.624776+0000","last_peered":"2024-04-25T15:26:21.624776+0000","last_clean":"2024-04-25T15:26:21.624776+0000","last_became_active":"2024-04-25T15:26:20.604190+0000","last_became_peered":"2024-04-25T15:26:20.604190+0000","last_unstale":"2024-04-25T15:26:21.624776+0000","last_undegraded":"2024-04-25T15:26:21.624776+0000","last_fullsized":"2024-04-25T15:26:21.624776+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:27.738092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.626063+0000","last_change":"2024-04-25T15:26:21.623052+0000","last_active":"2024-04-25T15:26:21.626063+0000","last_peered":"2024-04-25T15:26:21.626063+0000","last_clean":"2024-04-25T15:26:21.626063+0000","last_became_active":"2024-04-25T15:26:21.622607+0000","last_became_peered":"2024-04-25T15:26:21.622607+0000","last_unstale":"2024-04-25T15:26:21.626063+0000","last_undegraded":"2024-04-25T15:26:21.626063+0000","last_fullsized":"2024-04-25T15:26:21.626063+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:56.376686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.628303+0000","last_change":"2024-04-25T15:26:20.599542+0000","last_active":"2024-04-25T15:26:21.628303+0000","last_peered":"2024-04-25T15:26:21.628303+0000","last_clean":"2024-04-25T15:26:21.628303+0000","last_became_active":"2024-04-25T15:26:20.599003+0000","last_became_peered":"2024-04-25T15:26:20.599003+0000","last_unstale":"2024-04-25T15:26:21.628303+0000","last_undegraded":"2024-04-25T15:26:21.628303+0000","last_fullsized":"2024-04-25T15:26:21.628303+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:07.293821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612172+0000","last_change":"2024-04-25T15:26:21.605945+0000","last_active":"2024-04-25T15:26:21.612172+0000","last_peered":"2024-04-25T15:26:21.612172+0000","last_clean":"2024-04-25T15:26:21.612172+0000","last_became_active":"2024-04-25T15:26:21.605632+0000","last_became_peered":"2024-04-25T15:26:21.605632+0000","last_unstale":"2024-04-25T15:26:21.612172+0000","last_undegraded":"2024-04-25T15:26:21.612172+0000","last_fullsized":"2024-04-25T15:26:21.612172+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:11.403034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.628356+0000","last_change":"2024-04-25T15:26:20.590476+0000","last_active":"2024-04-25T15:26:21.628356+0000","last_peered":"2024-04-25T15:26:21.628356+0000","last_clean":"2024-04-25T15:26:21.628356+0000","last_became_active":"2024-04-25T15:26:20.590140+0000","last_became_peered":"2024-04-25T15:26:20.590140+0000","last_unstale":"2024-04-25T15:26:21.628356+0000","last_undegraded":"2024-04-25T15:26:21.628356+0000","last_fullsized":"2024-04-25T15:26:21.628356+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:11.381284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616428+0000","last_change":"2024-04-25T15:26:21.608276+0000","last_active":"2024-04-25T15:26:21.616428+0000","last_peered":"2024-04-25T15:26:21.616428+0000","last_clean":"2024-04-25T15:26:21.616428+0000","last_became_active":"2024-04-25T15:26:21.607974+0000","last_became_peered":"2024-04-25T15:26:21.607974+0000","last_unstale":"2024-04-25T15:26:21.616428+0000","last_undegraded":"2024-04-25T15:26:21.616428+0000","last_fullsized":"2024-04-25T15:26:21.616428+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:44:07.121400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594944+0000","last_change":"2024-04-25T15:26:20.603093+0000","last_active":"2024-04-25T15:26:21.594944+0000","last_peered":"2024-04-25T15:26:21.594944+0000","last_clean":"2024-04-25T15:26:21.594944+0000","last_became_active":"2024-04-25T15:26:20.602721+0000","last_became_peered":"2024-04-25T15:26:20.602721+0000","last_unstale":"2024-04-25T15:26:21.594944+0000","last_undegraded":"2024-04-25T15:26:21.594944+0000","last_fullsized":"2024-04-25T15:26:21.594944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:55.907186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619311+0000","last_change":"2024-04-25T15:26:21.613339+0000","last_active":"2024-04-25T15:26:21.619311+0000","last_peered":"2024-04-25T15:26:21.619311+0000","last_clean":"2024-04-25T15:26:21.619311+0000","last_became_active":"2024-04-25T15:26:21.612982+0000","last_became_peered":"2024-04-25T15:26:21.612982+0000","last_unstale":"2024-04-25T15:26:21.619311+0000","last_undegraded":"2024-04-25T15:26:21.619311+0000","last_fullsized":"2024-04-25T15:26:21.619311+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:12.386792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.619482+0000","last_change":"2024-04-25T15:26:20.588423+0000","last_active":"2024-04-25T15:26:21.619482+0000","last_peered":"2024-04-25T15:26:21.619482+0000","last_clean":"2024-04-25T15:26:21.619482+0000","last_became_active":"2024-04-25T15:26:20.588085+0000","last_became_peered":"2024-04-25T15:26:20.588085+0000","last_unstale":"2024-04-25T15:26:21.619482+0000","last_undegraded":"2024-04-25T15:26:21.619482+0000","last_fullsized":"2024-04-25T15:26:21.619482+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:17.868269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616834+0000","last_change":"2024-04-25T15:26:21.611139+0000","last_active":"2024-04-25T15:26:21.616834+0000","last_peered":"2024-04-25T15:26:21.616834+0000","last_clean":"2024-04-25T15:26:21.616834+0000","last_became_active":"2024-04-25T15:26:21.610717+0000","last_became_peered":"2024-04-25T15:26:21.610717+0000","last_unstale":"2024-04-25T15:26:21.616834+0000","last_undegraded":"2024-04-25T15:26:21.616834+0000","last_fullsized":"2024-04-25T15:26:21.616834+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:26:42.757403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.604166+0000","last_change":"2024-04-25T15:26:20.598628+0000","last_active":"2024-04-25T15:26:21.604166+0000","last_peered":"2024-04-25T15:26:21.604166+0000","last_clean":"2024-04-25T15:26:21.604166+0000","last_became_active":"2024-04-25T15:26:20.598326+0000","last_became_peered":"2024-04-25T15:26:20.598326+0000","last_unstale":"2024-04-25T15:26:21.604166+0000","last_undegraded":"2024-04-25T15:26:21.604166+0000","last_fullsized":"2024-04-25T15:26:21.604166+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:40.478696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616432+0000","last_change":"2024-04-25T15:26:21.610593+0000","last_active":"2024-04-25T15:26:21.616432+0000","last_peered":"2024-04-25T15:26:21.616432+0000","last_clean":"2024-04-25T15:26:21.616432+0000","last_became_active":"2024-04-25T15:26:21.610264+0000","last_became_peered":"2024-04-25T15:26:21.610264+0000","last_unstale":"2024-04-25T15:26:21.616432+0000","last_undegraded":"2024-04-25T15:26:21.616432+0000","last_fullsized":"2024-04-25T15:26:21.616432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:01.950164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594974+0000","last_change":"2024-04-25T15:26:20.592537+0000","last_active":"2024-04-25T15:26:21.594974+0000","last_peered":"2024-04-25T15:26:21.594974+0000","last_clean":"2024-04-25T15:26:21.594974+0000","last_became_active":"2024-04-25T15:26:20.592235+0000","last_became_peered":"2024-04-25T15:26:20.592235+0000","last_unstale":"2024-04-25T15:26:21.594974+0000","last_undegraded":"2024-04-25T15:26:21.594974+0000","last_fullsized":"2024-04-25T15:26:21.594974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:09.521438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.616256+0000","last_change":"2024-04-25T15:26:21.611357+0000","last_active":"2024-04-25T15:26:21.616256+0000","last_peered":"2024-04-25T15:26:21.616256+0000","last_clean":"2024-04-25T15:26:21.616256+0000","last_became_active":"2024-04-25T15:26:21.609870+0000","last_became_peered":"2024-04-25T15:26:21.609870+0000","last_unstale":"2024-04-25T15:26:21.616256+0000","last_undegraded":"2024-04-25T15:26:21.616256+0000","last_fullsized":"2024-04-25T15:26:21.616256+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:43.385692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.607923+0000","last_change":"2024-04-25T15:26:20.591277+0000","last_active":"2024-04-25T15:26:21.607923+0000","last_peered":"2024-04-25T15:26:21.607923+0000","last_clean":"2024-04-25T15:26:21.607923+0000","last_became_active":"2024-04-25T15:26:20.590894+0000","last_became_peered":"2024-04-25T15:26:20.590894+0000","last_unstale":"2024-04-25T15:26:21.607923+0000","last_undegraded":"2024-04-25T15:26:21.607923+0000","last_fullsized":"2024-04-25T15:26:21.607923+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:10.397526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.624606+0000","last_change":"2024-04-25T15:26:21.618029+0000","last_active":"2024-04-25T15:26:21.624606+0000","last_peered":"2024-04-25T15:26:21.624606+0000","last_clean":"2024-04-25T15:26:21.624606+0000","last_became_active":"2024-04-25T15:26:21.617705+0000","last_became_peered":"2024-04-25T15:26:21.617705+0000","last_unstale":"2024-04-25T15:26:21.624606+0000","last_undegraded":"2024-04-25T15:26:21.624606+0000","last_fullsized":"2024-04-25T15:26:21.624606+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:45.624513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.620539+0000","last_change":"2024-04-25T15:26:20.595689+0000","last_active":"2024-04-25T15:26:21.620539+0000","last_peered":"2024-04-25T15:26:21.620539+0000","last_clean":"2024-04-25T15:26:21.620539+0000","last_became_active":"2024-04-25T15:26:20.595306+0000","last_became_peered":"2024-04-25T15:26:20.595306+0000","last_unstale":"2024-04-25T15:26:21.620539+0000","last_undegraded":"2024-04-25T15:26:21.620539+0000","last_fullsized":"2024-04-25T15:26:21.620539+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:17.868269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.626119+0000","last_change":"2024-04-25T15:26:21.623737+0000","last_active":"2024-04-25T15:26:21.626119+0000","last_peered":"2024-04-25T15:26:21.626119+0000","last_clean":"2024-04-25T15:26:21.626119+0000","last_became_active":"2024-04-25T15:26:21.623414+0000","last_became_peered":"2024-04-25T15:26:21.623414+0000","last_unstale":"2024-04-25T15:26:21.626119+0000","last_undegraded":"2024-04-25T15:26:21.626119+0000","last_fullsized":"2024-04-25T15:26:21.626119+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:17.079917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.593561+0000","last_change":"2024-04-25T15:26:20.590835+0000","last_active":"2024-04-25T15:26:21.593561+0000","last_peered":"2024-04-25T15:26:21.593561+0000","last_clean":"2024-04-25T15:26:21.593561+0000","last_became_active":"2024-04-25T15:26:20.590404+0000","last_became_peered":"2024-04-25T15:26:20.590404+0000","last_unstale":"2024-04-25T15:26:21.593561+0000","last_undegraded":"2024-04-25T15:26:21.593561+0000","last_fullsized":"2024-04-25T15:26:21.593561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:27.738092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.610118+0000","last_change":"2024-04-25T15:26:21.607117+0000","last_active":"2024-04-25T15:26:21.610118+0000","last_peered":"2024-04-25T15:26:21.610118+0000","last_clean":"2024-04-25T15:26:21.610118+0000","last_became_active":"2024-04-25T15:26:21.605587+0000","last_became_peered":"2024-04-25T15:26:21.605587+0000","last_unstale":"2024-04-25T15:26:21.610118+0000","last_undegraded":"2024-04-25T15:26:21.610118+0000","last_fullsized":"2024-04-25T15:26:21.610118+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:11.403034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.603035+0000","last_change":"2024-04-25T15:26:20.593663+0000","last_active":"2024-04-25T15:26:21.603035+0000","last_peered":"2024-04-25T15:26:21.603035+0000","last_clean":"2024-04-25T15:26:21.603035+0000","last_became_active":"2024-04-25T15:26:20.593326+0000","last_became_peered":"2024-04-25T15:26:20.593326+0000","last_unstale":"2024-04-25T15:26:21.603035+0000","last_undegraded":"2024-04-25T15:26:21.603035+0000","last_fullsized":"2024-04-25T15:26:21.603035+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:19.669539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618842+0000","last_change":"2024-04-25T15:26:21.613992+0000","last_active":"2024-04-25T15:26:21.618842+0000","last_peered":"2024-04-25T15:26:21.618842+0000","last_clean":"2024-04-25T15:26:21.618842+0000","last_became_active":"2024-04-25T15:26:21.613631+0000","last_became_peered":"2024-04-25T15:26:21.613631+0000","last_unstale":"2024-04-25T15:26:21.618842+0000","last_undegraded":"2024-04-25T15:26:21.618842+0000","last_fullsized":"2024-04-25T15:26:21.618842+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:10.789126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.594262+0000","last_change":"2024-04-25T15:26:20.595308+0000","last_active":"2024-04-25T15:26:21.594262+0000","last_peered":"2024-04-25T15:26:21.594262+0000","last_clean":"2024-04-25T15:26:21.594262+0000","last_became_active":"2024-04-25T15:26:20.594950+0000","last_became_peered":"2024-04-25T15:26:20.594950+0000","last_unstale":"2024-04-25T15:26:21.594262+0000","last_undegraded":"2024-04-25T15:26:21.594262+0000","last_fullsized":"2024-04-25T15:26:21.594262+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:09.783618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.613335+0000","last_change":"2024-04-25T15:26:21.609574+0000","last_active":"2024-04-25T15:26:21.613335+0000","last_peered":"2024-04-25T15:26:21.613335+0000","last_clean":"2024-04-25T15:26:21.613335+0000","last_became_active":"2024-04-25T15:26:21.609208+0000","last_became_peered":"2024-04-25T15:26:21.609208+0000","last_unstale":"2024-04-25T15:26:21.613335+0000","last_undegraded":"2024-04-25T15:26:21.613335+0000","last_fullsized":"2024-04-25T15:26:21.613335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:20:37.737709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.602887+0000","last_change":"2024-04-25T15:26:20.597216+0000","last_active":"2024-04-25T15:26:21.602887+0000","last_peered":"2024-04-25T15:26:21.602887+0000","last_clean":"2024-04-25T15:26:21.602887+0000","last_became_active":"2024-04-25T15:26:20.596769+0000","last_became_peered":"2024-04-25T15:26:20.596769+0000","last_unstale":"2024-04-25T15:26:21.602887+0000","last_undegraded":"2024-04-25T15:26:21.602887+0000","last_fullsized":"2024-04-25T15:26:21.602887+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:09.783618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.612165+0000","last_change":"2024-04-25T15:26:21.607539+0000","last_active":"2024-04-25T15:26:21.612165+0000","last_peered":"2024-04-25T15:26:21.612165+0000","last_clean":"2024-04-25T15:26:21.612165+0000","last_became_active":"2024-04-25T15:26:21.607208+0000","last_became_peered":"2024-04-25T15:26:21.607208+0000","last_unstale":"2024-04-25T15:26:21.612165+0000","last_undegraded":"2024-04-25T15:26:21.612165+0000","last_fullsized":"2024-04-25T15:26:21.612165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:17.079917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.595496+0000","last_change":"2024-04-25T15:26:20.596459+0000","last_active":"2024-04-25T15:26:21.595496+0000","last_peered":"2024-04-25T15:26:21.595496+0000","last_clean":"2024-04-25T15:26:21.595496+0000","last_became_active":"2024-04-25T15:26:20.596109+0000","last_became_peered":"2024-04-25T15:26:20.596109+0000","last_unstale":"2024-04-25T15:26:21.595496+0000","last_undegraded":"2024-04-25T15:26:21.595496+0000","last_fullsized":"2024-04-25T15:26:21.595496+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:10.350348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.611589+0000","last_change":"2024-04-25T15:26:21.607534+0000","last_active":"2024-04-25T15:26:21.611589+0000","last_peered":"2024-04-25T15:26:21.611589+0000","last_clean":"2024-04-25T15:26:21.611589+0000","last_became_active":"2024-04-25T15:26:21.607176+0000","last_became_peered":"2024-04-25T15:26:21.607176+0000","last_unstale":"2024-04-25T15:26:21.611589+0000","last_undegraded":"2024-04-25T15:26:21.611589+0000","last_fullsized":"2024-04-25T15:26:21.611589+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:45:18.873777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.592448+0000","last_change":"2024-04-25T15:26:20.603675+0000","last_active":"2024-04-25T15:26:21.592448+0000","last_peered":"2024-04-25T15:26:21.592448+0000","last_clean":"2024-04-25T15:26:21.592448+0000","last_became_active":"2024-04-25T15:26:20.603294+0000","last_became_peered":"2024-04-25T15:26:20.603294+0000","last_unstale":"2024-04-25T15:26:21.592448+0000","last_undegraded":"2024-04-25T15:26:21.592448+0000","last_fullsized":"2024-04-25T15:26:21.592448+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:10.397526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.618792+0000","last_change":"2024-04-25T15:26:21.617026+0000","last_active":"2024-04-25T15:26:21.618792+0000","last_peered":"2024-04-25T15:26:21.618792+0000","last_clean":"2024-04-25T15:26:21.618792+0000","last_became_active":"2024-04-25T15:26:21.616701+0000","last_became_peered":"2024-04-25T15:26:21.616701+0000","last_unstale":"2024-04-25T15:26:21.618792+0000","last_undegraded":"2024-04-25T15:26:21.618792+0000","last_fullsized":"2024-04-25T15:26:21.618792+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:20.575131+0000","last_clean_scrub_stamp":"2024-04-25T15:26:20.575131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:17.079917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.604139+0000","last_change":"2024-04-25T15:26:20.598000+0000","last_active":"2024-04-25T15:26:21.604139+0000","last_peered":"2024-04-25T15:26:21.604139+0000","last_clean":"2024-04-25T15:26:21.604139+0000","last_became_active":"2024-04-25T15:26:20.597435+0000","last_became_peered":"2024-04-25T15:26:20.597435+0000","last_unstale":"2024-04-25T15:26:21.604139+0000","last_undegraded":"2024-04-25T15:26:21.604139+0000","last_fullsized":"2024-04-25T15:26:21.604139+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:19.569623+0000","last_clean_scrub_stamp":"2024-04-25T15:26:19.569623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:08.190294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":131,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-04-25T15:26:21.602291+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:26:21.602291+0000","last_peered":"2024-04-25T15:26:21.602291+0000","last_clean":"2024-04-25T15:26:21.602291+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:26:21.602291+0000","last_undegraded":"2024-04-25T15:26:21.602291+0000","last_fullsized":"2024-04-25T15:26:21.602291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":133541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":723909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:26:26.026 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:26:26.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:26:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.181+0000 7f8cd3cf9640 1 Processor -- start 2024-04-25T15:26:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.181+0000 7f8cd3cf9640 1 -- start start 2024-04-25T15:26:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd3cf9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8ccc15df80 0x7f8ccc160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd3cf9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ccc1615f0 con 0x7f8ccc164b00 2024-04-25T15:26:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd3cf9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ccc164020 con 0x7f8ccc0b8290 2024-04-25T15:26:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd3cf9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8ccc164670 con 0x7f8ccc15df80 2024-04-25T15:26:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd1a95640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8ccc164b00 0x7f8ccc0b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57746/0 (socket says 172.21.15.67:57746) 2024-04-25T15:26:26.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd1a95640 1 -- 172.21.15.67:0/1367417752 learned_addr learned my addr 172.21.15.67:0/1367417752 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd2296640 1 --2- 172.21.15.67:0/1367417752 >> v2:172.21.15.176:3300/0 conn(0x7f8ccc15df80 0x7f8ccc160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 177911607 0 0) 0x7f8ccc1615f0 con 0x7f8ccc164b00 2024-04-25T15:26:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cb8003520 con 0x7f8ccc164b00 2024-04-25T15:26:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3390793037 0 0) 0x7f8ccc164020 con 0x7f8ccc0b8290 2024-04-25T15:26:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ccc1615f0 con 0x7f8ccc0b8290 2024-04-25T15:26:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 253997739 0 0) 0x7f8cb8003520 con 0x7f8ccc164b00 2024-04-25T15:26:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 >> v2:172.21.15.176:3300/0 conn(0x7f8ccc15df80 msgr2=0x7f8ccc160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 --2- 172.21.15.67:0/1367417752 >> v2:172.21.15.176:3300/0 conn(0x7f8ccc15df80 0x7f8ccc160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 >> v1:172.21.15.176:6789/0 conn(0x7f8ccc0b8290 legacy=0x7f8ccc160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc164300 con 0x7f8ccc164b00 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.183+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8cc8001670 con 0x7f8ccc164b00 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 >> v1:172.21.15.67:6789/0 conn(0x7f8ccc164b00 legacy=0x7f8ccc0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd0a93640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb8003b40 0x7f8cb8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd0a93640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb8004800 0x7f8cb8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd0a93640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8cb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc164670 con 0x7f8cb80051c0 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb800c160 con 0x7f8cb8003b40 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb800c360 con 0x7f8cb8004800 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd2296640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8cb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd1a95640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb8003b40 0x7f8cb8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd1294640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb8004800 0x7f8cb8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd2296640 1 -- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb8004800 msgr2=0x7f8cb8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd2296640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb8004800 0x7f8cb8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd2296640 1 -- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb8003b40 msgr2=0x7f8cb8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd2296640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb8003b40 0x7f8cb8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.184+0000 7f8cd2296640 1 -- 172.21.15.67:0/1367417752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb8003520 con 0x7f8cb80051c0 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.185+0000 7f8cd2296640 1 -- 172.21.15.67:0/1367417752 reap_dead start 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.185+0000 7f8cd2296640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8cb800b9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009f70 tx=0x7f8cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d7becd15016ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.185+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4009bf0 con 0x7f8cb80051c0 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.185+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc4030430 con 0x7f8cb80051c0 2024-04-25T15:26:26.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.185+0000 7f8cd0a93640 1 -- 172.21.15.67:0/1367417752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4030800 con 0x7f8cb80051c0 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.186+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1367417752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 msgr2=0x7f8cb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.186+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8cb800b9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4009f70 tx=0x7f8cb8005830 comp rx=0 tx=0).stop 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.186+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1367417752 shutdown_connections 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.186+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1367417752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8cb800b9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.186+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1367417752 >> 172.21.15.67:0/1367417752 conn(0x7f8ccc0d2bc0 msgr2=0x7f8ccc0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.186+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1367417752 shutdown_connections 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.186+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1367417752 wait complete. 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.188+0000 7f8cd3cf9640 1 Processor -- start 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.188+0000 7f8cd3cf9640 1 -- start start 2024-04-25T15:26:26.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd3cf9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8ccc155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd3cf9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc1f9eb0 0x7f8ccc155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd3cf9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ccc156510 0x7f8ccc1f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd3cf9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc0b7f10 con 0x7f8cb80051c0 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd3cf9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc0b7d90 con 0x7f8ccc1f9eb0 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd3cf9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ccc0b8090 con 0x7f8ccc156510 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8ccc155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8ccc155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41584/0 (socket says 172.21.15.67:41584) 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 -- 172.21.15.67:0/1114790595 learned_addr learned my addr 172.21.15.67:0/1114790595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd2296640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ccc156510 0x7f8ccc1f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1294640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc1f9eb0 0x7f8ccc155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 -- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ccc156510 msgr2=0x7f8ccc1f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ccc156510 0x7f8ccc1f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 -- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc1f9eb0 msgr2=0x7f8ccc155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc1f9eb0 0x7f8ccc155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd1a95640 1 -- 172.21.15.67:0/1114790595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc1f6cd0 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.189+0000 7f8cd2296640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ccc156510 0x7f8ccc1f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.190+0000 7f8cd1a95640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8ccc155910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8007310 tx=0x7f8cc8007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.190+0000 7f8cc27fc640 1 -- 172.21.15.67:0/1114790595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc800f960 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.190+0000 7f8cc27fc640 1 -- 172.21.15.67:0/1114790595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc8018070 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.190+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ccc154e30 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.190+0000 7f8cc27fc640 1 -- 172.21.15.67:0/1114790595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc800fd40 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.190+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ccc156a30 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.191+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ccc050740 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.191+0000 7f8cc27fc640 1 -- 172.21.15.67:0/1114790595 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8cc801d460 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.192+0000 7f8cc27fc640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8cb00744e0 0x7f8cb00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.192+0000 7f8cc27fc640 1 -- 172.21.15.67:0/1114790595 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f8cc80951b0 con 0x7f8cb80051c0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.192+0000 7f8cd1294640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8cb00744e0 0x7f8cb00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.192+0000 7f8cd1294640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8cb00744e0 0x7f8cb00769f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc0c94a0 tx=0x7f8cbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:26.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.196+0000 7f8cc27fc640 1 -- 172.21.15.67:0/1114790595 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8cc805f9d0 con 0x7f8cb80051c0 2024-04-25T15:26:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.453+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8ccc0d4170 con 0x7f8cb80051c0 2024-04-25T15:26:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.456+0000 7f8cc27fc640 1 -- 172.21.15.67:0/1114790595 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f8cc805f2d0 con 0x7f8cb80051c0 2024-04-25T15:26:26.457 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:26:26.457 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:26:26.457 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-25T15:26:26.458 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-25T15:26:26.458 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:26:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8cb00744e0 msgr2=0x7f8cb00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8cb00744e0 0x7f8cb00769f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc0c94a0 tx=0x7f8cbc009040 comp rx=0 tx=0).stop 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 msgr2=0x7f8ccc155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8ccc155910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8007310 tx=0x7f8cc8007710 comp rx=0 tx=0).stop 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 shutdown_connections 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8cb00744e0 0x7f8cb00769f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ccc156510 0x7f8ccc1f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ccc1f9eb0 0x7f8ccc155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 --2- 172.21.15.67:0/1114790595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cb80051c0 0x7f8ccc155910 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.458+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 >> 172.21.15.67:0/1114790595 conn(0x7f8ccc0d2bc0 msgr2=0x7f8ccc15e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:26.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.459+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 shutdown_connections 2024-04-25T15:26:26.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.459+0000 7f8cd3cf9640 1 -- 172.21.15.67:0/1114790595 wait complete. 2024-04-25T15:26:26.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:26.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.582+0000 7fa4094ca640 1 Processor -- start 2024-04-25T15:26:26.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.582+0000 7fa4094ca640 1 -- start start 2024-04-25T15:26:26.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa4094ca640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa40406ccd0 0x7fa4040d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa4094ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa4094ca640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4040d1850 con 0x7fa4040d0e00 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa4094ca640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa404160de0 con 0x7fa40406ccd0 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa4094ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4041610c0 con 0x7fa4040d4a50 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa4027fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa40406ccd0 0x7fa4040d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa402ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa4040d0e00 0x7fa40406c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57748/0 (socket says 172.21.15.67:57748) 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.583+0000 7fa402ffd640 1 -- 172.21.15.67:0/2892530542 learned_addr learned my addr 172.21.15.67:0/2892530542 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.584+0000 7fa4037fe640 1 --2- 172.21.15.67:0/2892530542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.584+0000 7fa4027fc640 1 -- 172.21.15.67:0/2892530542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 msgr2=0x7fa4040d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.584+0000 7fa4027fc640 1 --2- 172.21.15.67:0/2892530542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.584+0000 7fa4027fc640 1 -- 172.21.15.67:0/2892530542 >> v1:172.21.15.67:6789/0 conn(0x7fa4040d0e00 legacy=0x7fa40406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.584+0000 7fa4027fc640 1 -- 172.21.15.67:0/2892530542 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4041613a0 con 0x7fa40406ccd0 2024-04-25T15:26:26.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.584+0000 7fa401ffb640 1 -- 172.21.15.67:0/2892530542 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282938442 0 0) 0x7fa4040d1850 con 0x7fa4040d0e00 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.584+0000 7fa4027fc640 1 --2- 172.21.15.67:0/2892530542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa40406ccd0 0x7fa4040d1210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80097a0 tx=0x7fa3f802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3debbb9f8763e0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa401ffb640 1 -- 172.21.15.67:0/2892530542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8037070 con 0x7fa40406ccd0 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa401ffb640 1 -- 172.21.15.67:0/2892530542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f8032440 con 0x7fa40406ccd0 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa401ffb640 1 -- 172.21.15.67:0/2892530542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8032810 con 0x7fa40406ccd0 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 -- 172.21.15.67:0/2892530542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa40406ccd0 msgr2=0x7fa4040d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 --2- 172.21.15.67:0/2892530542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa40406ccd0 0x7fa4040d1210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80097a0 tx=0x7fa3f802a180 comp rx=0 tx=0).stop 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 -- 172.21.15.67:0/2892530542 shutdown_connections 2024-04-25T15:26:26.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 --2- 172.21.15.67:0/2892530542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 --2- 172.21.15.67:0/2892530542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa40406ccd0 0x7fa4040d1210 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 -- 172.21.15.67:0/2892530542 >> 172.21.15.67:0/2892530542 conn(0x7fa4040d5d90 msgr2=0x7fa4040d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 -- 172.21.15.67:0/2892530542 shutdown_connections 2024-04-25T15:26:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.585+0000 7fa4094ca640 1 -- 172.21.15.67:0/2892530542 wait complete. 2024-04-25T15:26:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.587+0000 7fa4094ca640 1 Processor -- start 2024-04-25T15:26:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.587+0000 7fa4094ca640 1 -- start start 2024-04-25T15:26:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4094ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4040d0e00 0x7fa4040b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4094ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040b74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4094ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4040b79f0 0x7fa4040b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4094ca640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa404164e00 con 0x7fa4040d0e00 2024-04-25T15:26:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4094ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4040b5320 con 0x7fa4040b79f0 2024-04-25T15:26:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4094ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa404164f80 con 0x7fa4040d4a50 2024-04-25T15:26:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4040d0e00 0x7fa4040b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4040d0e00 0x7fa4040b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41614/0 (socket says 172.21.15.67:41614) 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 -- 172.21.15.67:0/3941116852 learned_addr learned my addr 172.21.15.67:0/3941116852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 -- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 msgr2=0x7fa4040b74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4027fc640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040b74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4037fe640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4040b79f0 0x7fa4040b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040b74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 -- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4040b79f0 msgr2=0x7fa4040b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4040b79f0 0x7fa4040b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 -- 172.21.15.67:0/3941116852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4040b5c30 con 0x7fa4040d0e00 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4027fc640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040b74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:26.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa402ffd640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4040d0e00 0x7fa4040b56f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c430 tx=0x7fa3f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa3eb7fe640 1 -- 172.21.15.67:0/3941116852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40074e0 con 0x7fa4040d0e00 2024-04-25T15:26:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa3eb7fe640 1 -- 172.21.15.67:0/3941116852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f40047e0 con 0x7fa4040d0e00 2024-04-25T15:26:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4040ce8d0 con 0x7fa4040d0e00 2024-04-25T15:26:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.589+0000 7fa3eb7fe640 1 -- 172.21.15.67:0/3941116852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4004bd0 con 0x7fa4040d0e00 2024-04-25T15:26:26.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.590+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4040cee90 con 0x7fa4040d0e00 2024-04-25T15:26:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.591+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa404050740 con 0x7fa4040d0e00 2024-04-25T15:26:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.591+0000 7fa3eb7fe640 1 -- 172.21.15.67:0/3941116852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa3f4011070 con 0x7fa4040d0e00 2024-04-25T15:26:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.592+0000 7fa3eb7fe640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3f00744b0 0x7fa3f00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.592+0000 7fa3eb7fe640 1 -- 172.21.15.67:0/3941116852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fa3f4094d60 con 0x7fa4040d0e00 2024-04-25T15:26:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.592+0000 7fa4027fc640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3f00744b0 0x7fa3f00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:26.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.592+0000 7fa4027fc640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3f00744b0 0x7fa3f00769c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8034000 tx=0x7fa3f802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.596+0000 7fa3eb7fe640 1 -- 172.21.15.67:0/3941116852 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3f405f610 con 0x7fa4040d0e00 2024-04-25T15:26:26.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.819+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4040d4320 con 0x7fa4040d0e00 2024-04-25T15:26:26.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.819+0000 7fa3eb7fe640 1 -- 172.21.15.67:0/3941116852 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7fa3f405ef10 con 0x7fa4040d0e00 2024-04-25T15:26:26.820 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:26.820 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/4217070487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4217070487},{"type":"v1","addr":"172.21.15.176:6835","nonce":4217070487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4207485404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4207485404},{"type":"v1","addr":"172.21.15.176:6837","nonce":4207485404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5067,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2355578883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2355578883},{"type":"v1","addr":"172.21.15.67:6834","nonce":2355578883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:20.974268+0000","modified":"2024-04-25T15:26:23.988098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2490033101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2490033101},{"type":"v1","addr":"172.21.15.67:6835","nonce":2490033101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-04-25T15:26:26.821 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-04-25T15:26:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3f00744b0 msgr2=0x7fa3f00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3f00744b0 0x7fa3f00769c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8034000 tx=0x7fa3f802e000 comp rx=0 tx=0).stop 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4040d0e00 msgr2=0x7fa4040b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4040d0e00 0x7fa4040b56f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c430 tx=0x7fa3f400c8f0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 shutdown_connections 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3f00744b0 0x7fa3f00769c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4040b79f0 0x7fa4040b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4040d4a50 0x7fa4040b74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 --2- 172.21.15.67:0/3941116852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4040d0e00 0x7fa4040b56f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 >> 172.21.15.67:0/3941116852 conn(0x7fa4040d5d90 msgr2=0x7fa4041639f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 shutdown_connections 2024-04-25T15:26:26.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:26.821+0000 7fa4094ca640 1 -- 172.21.15.67:0/3941116852 wait complete. 2024-04-25T15:26:27.236 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:26:20.974268+0000', 'modified': '2024-04-25T15:26:23.988098+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5070}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5070': {'gid': 5070, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/2490033101', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 2490033101}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 2490033101}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5070, 'qdb_cluster': [5070]} 2024-04-25T15:26:27.237 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:26:27.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T15:26:27.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.351+0000 7f10534e7640 1 Processor -- start 2024-04-25T15:26:27.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.351+0000 7f10534e7640 1 -- start start 2024-04-25T15:26:27.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.352+0000 7f10534e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 0x7f104c0d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.352+0000 7f10534e7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f104c0b3c10 con 0x7f104c0d4130 2024-04-25T15:26:27.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.352+0000 7f10534e7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f104c0b3ef0 con 0x7f104c06d8e0 2024-04-25T15:26:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.352+0000 7f10534e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f104c0b9520 con 0x7f104c06dcf0 2024-04-25T15:26:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.352+0000 7f1050a82640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f104c0d4130 0x7f104c0d1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57754/0 (socket says 172.21.15.67:57754) 2024-04-25T15:26:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.352+0000 7f1050a82640 1 -- 172.21.15.67:0/957735628 learned_addr learned my addr 172.21.15.67:0/957735628 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1051a84640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 0x7f104c0d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382461322 0 0) 0x7f104c0b3c10 con 0x7f104c0d4130 2024-04-25T15:26:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1038003520 con 0x7f104c0d4130 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4254014206 0 0) 0x7f1038003520 con 0x7f104c0d4130 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 msgr2=0x7f104c0d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 0x7f104c0d2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 >> v1:172.21.15.176:6789/0 conn(0x7f104c06d8e0 legacy=0x7f104c0d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c0b41d0 con 0x7f104c0d4130 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f103c001670 con 0x7f104c0d4130 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 >> v1:172.21.15.67:6789/0 conn(0x7f104c0d4130 legacy=0x7f104c0d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10380039e0 0x7f1038003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1038004800 0x7f1038004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 0x7f1038005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f104c0b9520 con 0x7f1038004800 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f103800baf0 con 0x7f10380039e0 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f103800bcf0 con 0x7f104c06dcf0 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1050a82640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1038004800 0x7f1038004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1051283640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10380039e0 0x7f1038003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1051a84640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 0x7f1038005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1050a82640 1 -- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 msgr2=0x7f1038005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1050a82640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f104c06dcf0 0x7f1038005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1050a82640 1 -- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10380039e0 msgr2=0x7f1038003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.353+0000 7f1050a82640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10380039e0 0x7f1038003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.354+0000 7f1050a82640 1 -- 172.21.15.67:0/957735628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1038003520 con 0x7f1038004800 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.354+0000 7f1051283640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10380039e0 0x7f1038003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.354+0000 7f1050a82640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1038004800 0x7f1038004c10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f103c00c240 tx=0x7f1038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d4dc426a40233d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.354+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c007d10 con 0x7f1038004800 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.354+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c0058d0 con 0x7f1038004800 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.354+0000 7f1043fff640 1 -- 172.21.15.67:0/957735628 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c005ce0 con 0x7f1038004800 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.355+0000 7f10534e7640 1 -- 172.21.15.67:0/957735628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1038004800 msgr2=0x7f1038004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.355+0000 7f10534e7640 1 --2- 172.21.15.67:0/957735628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1038004800 0x7f1038004c10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f103c00c240 tx=0x7f1038005830 comp rx=0 tx=0).stop 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.355+0000 7f1051a84640 1 -- 172.21.15.67:0/957735628 reap_dead start 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.355+0000 7f10534e7640 1 -- 172.21.15.67:0/957735628 shutdown_connections 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.355+0000 7f10534e7640 1 -- 172.21.15.67:0/957735628 >> 172.21.15.67:0/957735628 conn(0x7f104c08d110 msgr2=0x7f104c08d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.355+0000 7f10534e7640 1 -- 172.21.15.67:0/957735628 shutdown_connections 2024-04-25T15:26:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.355+0000 7f10534e7640 1 -- 172.21.15.67:0/957735628 wait complete. 2024-04-25T15:26:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.357+0000 7f10534e7640 1 Processor -- start 2024-04-25T15:26:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.357+0000 7f10534e7640 1 -- start start 2024-04-25T15:26:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f10534e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1038004800 0x7f104c1f9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f10534e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c157370 0x7f104c0b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f10534e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f104c157780 0x7f104c0b5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f10534e7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f104c0b38e0 con 0x7f104c157370 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f10534e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f104c0b4820 con 0x7f104c157780 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f10534e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f104c0d2470 con 0x7f1038004800 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f1050a82640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c157370 0x7f104c0b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f1050a82640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c157370 0x7f104c0b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41630/0 (socket says 172.21.15.67:41630) 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f1050a82640 1 -- 172.21.15.67:0/2162849406 learned_addr learned my addr 172.21.15.67:0/2162849406 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f1051283640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1038004800 0x7f104c1f9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.358+0000 7f1051a84640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f104c157780 0x7f104c0b5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1050a82640 1 -- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1038004800 msgr2=0x7f104c1f9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1050a82640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1038004800 0x7f104c1f9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1050a82640 1 -- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f104c157780 msgr2=0x7f104c0b5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1050a82640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f104c157780 0x7f104c0b5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1050a82640 1 -- 172.21.15.67:0/2162849406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c1f9fc0 con 0x7f104c157370 2024-04-25T15:26:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1050a82640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c157370 0x7f104c0b5560 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f103c001970 tx=0x7f104c0b3de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1041ffb640 1 -- 172.21.15.67:0/2162849406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c038040 con 0x7f104c157370 2024-04-25T15:26:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1041ffb640 1 -- 172.21.15.67:0/2162849406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c007070 con 0x7f104c157370 2024-04-25T15:26:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104c15eab0 con 0x7f104c157370 2024-04-25T15:26:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f1041ffb640 1 -- 172.21.15.67:0/2162849406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c001c50 con 0x7f104c157370 2024-04-25T15:26:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.359+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f104c15f070 con 0x7f104c157370 2024-04-25T15:26:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.361+0000 7f1041ffb640 1 -- 172.21.15.67:0/2162849406 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f103c007440 con 0x7f104c157370 2024-04-25T15:26:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.362+0000 7f1041ffb640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10300744b0 0x7f10300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.362+0000 7f1041ffb640 1 -- 172.21.15.67:0/2162849406 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f103c0b5150 con 0x7f104c157370 2024-04-25T15:26:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.362+0000 7f1051283640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10300744b0 0x7f10300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.363+0000 7f1051283640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10300744b0 0x7f10300769c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f104c0744c0 tx=0x7f1048009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.363+0000 7f105a396740 1 Processor -- start 2024-04-25T15:26:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.363+0000 7f105a396740 1 -- start start 2024-04-25T15:26:27.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f105a396740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cca590 0x55d3b308d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f105a396740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d3b2ff1740 con 0x55d3b2cca590 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55d3b2f8d3a0 con 0x55d3b2cc23b0 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55d3b2fed700 con 0x55d3b2ceb270 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f1051a84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cca590 0x55d3b308d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f1051a84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cca590 0x55d3b308d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41638/0 (socket says 172.21.15.67:41638) 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f1051a84640 1 -- 172.21.15.67:0/1178144775 learned_addr learned my addr 172.21.15.67:0/1178144775 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f1051283640 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.364+0000 7f1050a82640 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1051a84640 1 -- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 msgr2=0x55d3b30b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1051a84640 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1051a84640 1 -- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cc23b0 msgr2=0x55d3b30b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1051a84640 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1051a84640 1 -- 172.21.15.67:0/1178144775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3b2fa9750 con 0x55d3b2cca590 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1050a82640 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1051a84640 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cca590 0x55d3b308d260 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f104c0d38e0 tx=0x7f1044009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647d67b8b8551427 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1036ffd640 1 -- 172.21.15.67:0/1178144775 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044016810 con 0x55d3b2cca590 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1036ffd640 1 -- 172.21.15.67:0/1178144775 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104401f070 con 0x55d3b2cca590 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.365+0000 7f1036ffd640 1 -- 172.21.15.67:0/1178144775 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044016bd0 con 0x55d3b2cca590 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 -- 172.21.15.67:0/1178144775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cca590 msgr2=0x55d3b308d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cca590 0x55d3b308d260 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f104c0d38e0 tx=0x7f1044009750 comp rx=0 tx=0).stop 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 -- 172.21.15.67:0/1178144775 shutdown_connections 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cca590 0x55d3b308d260 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 --2- 172.21.15.67:0/1178144775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30b7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 -- 172.21.15.67:0/1178144775 >> 172.21.15.67:0/1178144775 conn(0x55d3b2cdf3c0 msgr2=0x55d3b308c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 -- 172.21.15.67:0/1178144775 shutdown_connections 2024-04-25T15:26:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.366+0000 7f105a396740 1 -- 172.21.15.67:0/1178144775 wait complete. 2024-04-25T15:26:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.367+0000 7f105a396740 1 -- start start 2024-04-25T15:26:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.367+0000 7f105a396740 1 Processor -- start 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f105a396740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f105a396740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d3b2f78fe0 con 0x55d3b2cc23b0 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55d3b30babf0 con 0x55d3b2cca590 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55d3b30d17e0 con 0x55d3b2ceb270 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f105a396740 10 client.0 _ll_register_callbacks cb 0x55d3b2f8f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:26:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f1051283640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f1051283640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41640/0 (socket says 172.21.15.67:41640) 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.368+0000 7f1051283640 1 -- 172.21.15.67:0/1547604099 learned_addr learned my addr 172.21.15.67:0/1547604099 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051a84640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1050a82640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051283640 1 -- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 msgr2=0x55d3b30d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051283640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051283640 1 -- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 msgr2=0x55d3b30d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051283640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051283640 1 -- 172.21.15.67:0/1547604099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3b2d032d0 con 0x55d3b2cc23b0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051a84640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f1051283640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b0fd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1048015710 tx=0x7f1048007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257f46e6f8135d89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f102bfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f102bfff640 1 -- 172.21.15.67:0/1547604099 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048026020 con 0x55d3b2cc23b0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f104401f070 con 0x55d3b2cc23b0 2024-04-25T15:26:27.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f102bfff640 1 -- 172.21.15.67:0/1547604099 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f1048014820 con 0x55d3b2cc23b0 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f102bfff640 1 -- 172.21.15.67:0/1547604099 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f1048030050 con 0x55d3b2cc23b0 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f105a396740 10 client.5229 fetch_fsmap learned FSMap version 16 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55d3b2dcd6b0 con 0x55d3b2cc23b0 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.369+0000 7f102bfff640 1 -- 172.21.15.67:0/1547604099 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104401f070 con 0x55d3b2cc23b0 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f102bfff640 1 -- 172.21.15.67:0/1547604099 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f1048033020 con 0x55d3b2cc23b0 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f105a396740 10 client.5229 fetch_fsmap finished waiting for FSMap version 16 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f105a396740 10 client.5229 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f105a396740 20 client.5229 populate_metadata read hostname 'smithi067' 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f105a396740 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b2ca9e40 0x55d3b30b9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f105a396740 4 client.5229 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 --> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3b2fff350 con 0x55d3b2ca9e40 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f1050a82640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b2ca9e40 0x55d3b30b9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f102bfff640 10 client.5229 ms_handle_connect on v2:172.21.15.67:6833/2490033101 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.370+0000 7f1050a82640 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b2ca9e40 0x55d3b30b9c60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbbf66cff6f39a05 server_cookie=37ba7e708a9cda7b in_seq=0 out_seq=0 2024-04-25T15:26:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f102bfff640 1 -- 172.21.15.67:0/1547604099 <== mds.0 v2:172.21.15.67:6833/2490033101 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d3b2fff350 con 0x55d3b2ca9e40 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f102bfff640 10 client.5229 handle_command_reply: tid=0 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 client.5229 shutdown 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 msgr2=0x55d3b30b0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b0fd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1048015710 tx=0x7f1048007a30 comp rx=0 tx=0).stop 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 shutdown_connections 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b2ca9e40 0x55d3b30b9c60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f102bfff640 0 client.5229 ms_handle_reset on v2:172.21.15.67:6833/2490033101 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 --2- 172.21.15.67:0/1547604099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b30b0fd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 >> 172.21.15.67:0/1547604099 conn(0x55d3b2cdf3c0 msgr2=0x55d3b308af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 shutdown_connections 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 1 -- 172.21.15.67:0/1547604099 wait complete. 2024-04-25T15:26:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.371+0000 7f105a396740 20 client.5229 trim_cache size 0 max 16384 2024-04-25T15:26:27.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.390+0000 7f105a396740 1 Processor -- start 2024-04-25T15:26:27.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.390+0000 7f105a396740 1 -- start start 2024-04-25T15:26:27.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f105a396740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b308af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f105a396740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f104c201e90 con 0x55d3b2cc23b0 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f104c0a0be0 con 0x55d3b2cca590 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f104c201840 con 0x55d3b2ceb270 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b308af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b308af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41654/0 (socket says 172.21.15.67:41654) 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 -- 172.21.15.67:0/505465721 learned_addr learned my addr 172.21.15.67:0/505465721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 -- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 msgr2=0x55d3b30d5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051a84640 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 -- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 msgr2=0x55d3b30d5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 -- 172.21.15.67:0/505465721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3b30c7a90 con 0x55d3b2cc23b0 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051a84640 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f1051283640 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b308af10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1048014d20 tx=0x7f104801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f76cecfa42cfc86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f10367fc640 1 -- 172.21.15.67:0/505465721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048038020 con 0x55d3b2cc23b0 2024-04-25T15:26:27.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f10367fc640 1 -- 172.21.15.67:0/505465721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1048005020 con 0x55d3b2cc23b0 2024-04-25T15:26:27.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.392+0000 7f10367fc640 1 -- 172.21.15.67:0/505465721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048026070 con 0x55d3b2cc23b0 2024-04-25T15:26:27.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 -- 172.21.15.67:0/505465721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 msgr2=0x55d3b308af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b308af10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1048014d20 tx=0x7f104801b000 comp rx=0 tx=0).stop 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 -- 172.21.15.67:0/505465721 shutdown_connections 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 --2- 172.21.15.67:0/505465721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2cc23b0 0x55d3b308af10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 -- 172.21.15.67:0/505465721 >> 172.21.15.67:0/505465721 conn(0x55d3b2cdf3c0 msgr2=0x55d3b308cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 -- 172.21.15.67:0/505465721 shutdown_connections 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.393+0000 7f105a396740 1 -- 172.21.15.67:0/505465721 wait complete. 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.394+0000 7f105a396740 1 -- start start 2024-04-25T15:26:27.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.394+0000 7f105a396740 1 Processor -- start 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.395+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2cc23b0 0x55d3b314f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.395+0000 7f105a396740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.395+0000 7f105a396740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.395+0000 7f105a396740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d3b30b1680 con 0x55d3b2ceb270 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.395+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55d3b30d13a0 con 0x55d3b2cca590 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.395+0000 7f105a396740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55d3b30b1c10 con 0x55d3b2cc23b0 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.395+0000 7f105a396740 10 client.0 _ll_register_callbacks cb 0x7f1048001d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41660/0 (socket says 172.21.15.67:41660) 2024-04-25T15:26:27.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 -- 172.21.15.67:0/3046627654 learned_addr learned my addr 172.21.15.67:0/3046627654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051283640 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2cc23b0 0x55d3b314f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1050a82640 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 -- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2cc23b0 msgr2=0x55d3b314f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2cc23b0 0x55d3b314f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 -- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 msgr2=0x55d3b30d5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 -- 172.21.15.67:0/3046627654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3b30cfab0 con 0x55d3b2ceb270 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f1051a84640 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2ceb270 0x55d3b30d5660 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f104400b010 tx=0x7f1044004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d91906e48fd0de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f102bfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f102bfff640 1 -- 172.21.15.67:0/3046627654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044016a20 con 0x55d3b2ceb270 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f102bfff640 1 -- 172.21.15.67:0/3046627654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f1044016bc0 con 0x55d3b2ceb270 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1048005020 con 0x55d3b2ceb270 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f102bfff640 1 -- 172.21.15.67:0/3046627654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f1044037050 con 0x55d3b2ceb270 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f105a396740 10 client.5235 fetch_fsmap learned FSMap version 16 2024-04-25T15:26:27.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.396+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55d3b30d24a0 con 0x55d3b2ceb270 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f102bfff640 1 -- 172.21.15.67:0/3046627654 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1048005020 con 0x55d3b2ceb270 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f102bfff640 1 -- 172.21.15.67:0/3046627654 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f104403a020 con 0x55d3b2ceb270 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f105a396740 10 client.5235 fetch_fsmap finished waiting for FSMap version 16 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f105a396740 10 client.5235 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f105a396740 20 client.5235 populate_metadata read hostname 'smithi067' 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f105a396740 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b3153c00 0x55d3b30f41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f105a396740 4 client.5235 mds_command: new command op to 5070 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 --> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] -- command(tid 0: {"prefix": "status"}) -- 0x55d3b2ff91c0 con 0x55d3b3153c00 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f1051283640 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b3153c00 0x55d3b30f41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f102bfff640 10 client.5235 ms_handle_connect on v2:172.21.15.67:6833/2490033101 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.397+0000 7f1051283640 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b3153c00 0x55d3b30f41e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8604e7d0c00d08c8 server_cookie=2747c6eb958aae2a in_seq=0 out_seq=0 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f102bfff640 1 -- 172.21.15.67:0/3046627654 <== mds.0 v2:172.21.15.67:6833/2490033101 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55d3b2ff91c0 con 0x55d3b3153c00 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f102bfff640 10 client.5235 handle_command_reply: tid=0 2024-04-25T15:26:27.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 client.5235 shutdown 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2ceb270 msgr2=0x55d3b30d5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2ceb270 0x55d3b30d5660 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f104400b010 tx=0x7f1044004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 shutdown_connections 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:6833/2490033101,v1:172.21.15.67:6835/2490033101] conn(0x55d3b3153c00 0x55d3b30f41e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d3b2ceb270 0x55d3b30d5660 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55d3b2cca590 0x55d3b30d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f102bfff640 0 client.5235 ms_handle_reset on v2:172.21.15.67:6833/2490033101 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 --2- 172.21.15.67:0/3046627654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55d3b2cc23b0 0x55d3b314f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 >> 172.21.15.67:0/3046627654 conn(0x55d3b2cdf3c0 msgr2=0x55d3b30f5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 shutdown_connections 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 1 -- 172.21.15.67:0/3046627654 wait complete. 2024-04-25T15:26:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.398+0000 7f105a396740 20 client.5235 trim_cache size 0 max 16384 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b43d0acb-252a-4f7a-bf14-34d789b94b95", 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5070, 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 5.8001031359999997, 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 16, 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 33, 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 33, 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.696867873 2024-04-25T15:26:27.404 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:26:27.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10300744b0 msgr2=0x7f10300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10300744b0 0x7f10300769c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f104c0744c0 tx=0x7f1048009040 comp rx=0 tx=0).stop 2024-04-25T15:26:27.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c157370 msgr2=0x7f104c0b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c157370 0x7f104c0b5560 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f103c001970 tx=0x7f104c0b3de0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 shutdown_connections 2024-04-25T15:26:27.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f10300744b0 0x7f10300769c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f104c157780 0x7f104c0b5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f104c157370 0x7f104c0b5560 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 --2- 172.21.15.67:0/2162849406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1038004800 0x7f104c1f9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 >> 172.21.15.67:0/2162849406 conn(0x7f104c08d110 msgr2=0x7f104c161250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 shutdown_connections 2024-04-25T15:26:27.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.399+0000 7f10534e7640 1 -- 172.21.15.67:0/2162849406 wait complete. 2024-04-25T15:26:27.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:26:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.521+0000 7fb76dbc5640 1 Processor -- start 2024-04-25T15:26:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.521+0000 7fb76dbc5640 1 -- start start 2024-04-25T15:26:27.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.522+0000 7fb76dbc5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb76806ccd0 0x7fb7680d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.522+0000 7fb76dbc5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7680d4a50 0x7fb7680d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.522+0000 7fb76dbc5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7680d1850 con 0x7fb7680d0e00 2024-04-25T15:26:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb76dbc5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb768160de0 con 0x7fb7680d4a50 2024-04-25T15:26:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb76dbc5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb7681610c0 con 0x7fb76806ccd0 2024-04-25T15:26:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb767fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7680d4a50 0x7fb7680d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb7677fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7680d0e00 0x7fb76806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57768/0 (socket says 172.21.15.67:57768) 2024-04-25T15:26:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb7677fe640 1 -- 172.21.15.67:0/2600631521 learned_addr learned my addr 172.21.15.67:0/2600631521 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb766ffd640 1 --2- 172.21.15.67:0/2600631521 >> v2:172.21.15.176:3300/0 conn(0x7fb76806ccd0 0x7fb7680d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb767fff640 1 -- 172.21.15.67:0/2600631521 >> v2:172.21.15.176:3300/0 conn(0x7fb76806ccd0 msgr2=0x7fb7680d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb767fff640 1 --2- 172.21.15.67:0/2600631521 >> v2:172.21.15.176:3300/0 conn(0x7fb76806ccd0 0x7fb7680d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb767fff640 1 -- 172.21.15.67:0/2600631521 >> v1:172.21.15.67:6789/0 conn(0x7fb7680d0e00 legacy=0x7fb76806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb767fff640 1 -- 172.21.15.67:0/2600631521 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7681613a0 con 0x7fb7680d4a50 2024-04-25T15:26:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.523+0000 7fb7667fc640 1 -- 172.21.15.67:0/2600631521 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2051685863 0 0) 0x7fb7680d1850 con 0x7fb7680d0e00 2024-04-25T15:26:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.524+0000 7fb767fff640 1 --2- 172.21.15.67:0/2600631521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680d4e60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb76808f170 tx=0x7fb76002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a3845a8400e542 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.524+0000 7fb7667fc640 1 -- 172.21.15.67:0/2600631521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760007740 con 0x7fb7680d4a50 2024-04-25T15:26:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.524+0000 7fb7667fc640 1 -- 172.21.15.67:0/2600631521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7600078e0 con 0x7fb7680d4a50 2024-04-25T15:26:27.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.524+0000 7fb7667fc640 1 -- 172.21.15.67:0/2600631521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760007cb0 con 0x7fb7680d4a50 2024-04-25T15:26:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 -- 172.21.15.67:0/2600631521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 msgr2=0x7fb7680d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/2600631521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680d4e60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb76808f170 tx=0x7fb76002a5f0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 -- 172.21.15.67:0/2600631521 shutdown_connections 2024-04-25T15:26:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/2600631521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680d4e60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/2600631521 >> v2:172.21.15.176:3300/0 conn(0x7fb76806ccd0 0x7fb7680d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 -- 172.21.15.67:0/2600631521 >> 172.21.15.67:0/2600631521 conn(0x7fb7680d5d90 msgr2=0x7fb7680d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 -- 172.21.15.67:0/2600631521 shutdown_connections 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.525+0000 7fb76dbc5640 1 -- 172.21.15.67:0/2600631521 wait complete. 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.526+0000 7fb76dbc5640 1 Processor -- start 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.527+0000 7fb76dbc5640 1 -- start start 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb76dbc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7680d0e00 0x7fb76815e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb76dbc5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb76dbc5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7680cc2f0 0x7fb7680cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb76dbc5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb768164e00 con 0x7fb7680d4a50 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb76dbc5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7680b5320 con 0x7fb7680cc2f0 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb76dbc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb768164f80 con 0x7fb7680d0e00 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41682/0 (socket says 172.21.15.67:41682) 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 -- 172.21.15.67:0/3870049839 learned_addr learned my addr 172.21.15.67:0/3870049839 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb7677fe640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7680d0e00 0x7fb76815e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb767fff640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7680cc2f0 0x7fb7680cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 -- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7680d0e00 msgr2=0x7fb76815e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7680d0e00 0x7fb76815e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 -- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7680cc2f0 msgr2=0x7fb7680cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7680cc2f0 0x7fb7680cf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.528+0000 7fb766ffd640 1 -- 172.21.15.67:0/3870049839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7680ca380 con 0x7fb7680d4a50 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.529+0000 7fb766ffd640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680cbdb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb75c00d620 tx=0x7fb75c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.529+0000 7fb74ffff640 1 -- 172.21.15.67:0/3870049839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00b620 con 0x7fb7680d4a50 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.529+0000 7fb74ffff640 1 -- 172.21.15.67:0/3870049839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb75c019070 con 0x7fb7680d4a50 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.529+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7680caee0 con 0x7fb7680d4a50 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.529+0000 7fb74ffff640 1 -- 172.21.15.67:0/3870049839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00ba00 con 0x7fb7680d4a50 2024-04-25T15:26:27.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.529+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7680ca920 con 0x7fb7680d4a50 2024-04-25T15:26:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.530+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb768050740 con 0x7fb7680d4a50 2024-04-25T15:26:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.530+0000 7fb74ffff640 1 -- 172.21.15.67:0/3870049839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb75c00bba0 con 0x7fb7680d4a50 2024-04-25T15:26:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.531+0000 7fb74ffff640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb7540744b0 0x7fb7540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.531+0000 7fb74ffff640 1 -- 172.21.15.67:0/3870049839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fb75c095520 con 0x7fb7680d4a50 2024-04-25T15:26:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.531+0000 7fb7677fe640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb7540744b0 0x7fb7540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:27.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.532+0000 7fb7677fe640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb7540744b0 0x7fb7540769c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb758007fd0 tx=0x7fb758007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:27.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.536+0000 7fb74ffff640 1 -- 172.21.15.67:0/3870049839 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb75c05f660 con 0x7fb7680d4a50 2024-04-25T15:26:27.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.761+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb76815f830 con 0x7fb7680d4a50 2024-04-25T15:26:27.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.761+0000 7fb74ffff640 1 -- 172.21.15.67:0/3870049839 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7fb75c063960 con 0x7fb7680d4a50 2024-04-25T15:26:27.763 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:26:27.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb7540744b0 msgr2=0x7fb7540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb7540744b0 0x7fb7540769c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb758007fd0 tx=0x7fb758007de0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 msgr2=0x7fb7680cbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:27.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680cbdb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb75c00d620 tx=0x7fb75c00dae0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 shutdown_connections 2024-04-25T15:26:27.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb7540744b0 0x7fb7540769c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7680cc2f0 0x7fb7680cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7680d4a50 0x7fb7680cbdb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 --2- 172.21.15.67:0/3870049839 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7680d0e00 0x7fb76815e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:27.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 >> 172.21.15.67:0/3870049839 conn(0x7fb7680d5d90 msgr2=0x7fb76806c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:27.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 shutdown_connections 2024-04-25T15:26:27.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:27.763+0000 7fb76dbc5640 1 -- 172.21.15.67:0/3870049839 wait complete. 2024-04-25T15:26:27.775 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:26:27.776 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:26:27.776 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:26:27.776 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:26:27.776 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:27.776 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:26:27.776 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:26:27.777 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:26:27.777 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:26:27.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:27.777 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:26:27.793 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:27.794 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42711sec preferred_lft 42711sec 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7482:42ff:fe30:5ffc/64 scope link 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:27.795 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::885:bff:fe04:f6cb/64 scope link 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ca:af:51:1f:d5:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8af:51ff:fe1f:d5b0/64 scope link 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:27.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:27.796 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:26:27.857 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:26:27.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:27.859 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:26:27.914 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:26:27.914 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:26:27.914 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:26:27.914 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:26:27.914 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:26:27.915 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:26:27.915 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:26:27.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:26:27.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:26:27.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:26:27.915 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:26:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:26:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:28.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:28.041 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:26:28.057 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:26:28.057 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:26:28.057 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:26:28.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:28.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:26:28.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:28.122 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:28.177 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:26:28.177 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:28.177 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:28.232 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T15:26:28.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:28.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:28.326 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:28.326 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:28.335 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:28.335 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:28.335 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:26:28.335 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:26:28.335 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:26:28.336 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T15:26:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:26:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:26:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:26:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:26:28.338 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:26:28.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:28.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:28.363 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:26:28.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:26:28.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:28.465 INFO:teuthology.orchestra.run.smithi067.stdout:client5155 2024-04-25T15:26:28.466 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:28.466 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:28.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:26:28.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:26:28.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:26:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.636+0000 7ff721c92640 1 Processor -- start 2024-04-25T15:26:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.636+0000 7ff721c92640 1 -- start start 2024-04-25T15:26:28.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.637+0000 7ff721c92640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff71c0b6a00 0x7ff71c15d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:28.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.637+0000 7ff721c92640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff71c15a2f0 0x7ff71c06ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:28.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.637+0000 7ff721c92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c06f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:28.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.637+0000 7ff721c92640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff71c160d50 con 0x7ff71c15a2f0 2024-04-25T15:26:28.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.637+0000 7ff721c92640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff71c15a700 con 0x7ff71c0b6a00 2024-04-25T15:26:28.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.637+0000 7ff721c92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff71c1611e0 con 0x7ff71c06d390 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff71c15a2f0 0x7ff71c06ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff71c15a2f0 0x7ff71c06ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59186/0 (socket says 172.21.15.67:59186) 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 -- 172.21.15.67:0/1749639848 learned_addr learned my addr 172.21.15.67:0/1749639848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71bfff640 1 --2- 172.21.15.67:0/1749639848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c06f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71b7fe640 1 --2- 172.21.15.67:0/1749639848 >> v2:172.21.15.176:3300/0 conn(0x7ff71c0b6a00 0x7ff71c15d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 -- 172.21.15.67:0/1749639848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 msgr2=0x7ff71c06f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 --2- 172.21.15.67:0/1749639848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c06f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 -- 172.21.15.67:0/1749639848 >> v2:172.21.15.176:3300/0 conn(0x7ff71c0b6a00 msgr2=0x7ff71c15d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 --2- 172.21.15.67:0/1749639848 >> v2:172.21.15.176:3300/0 conn(0x7ff71c0b6a00 0x7ff71c15d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71affd640 1 -- 172.21.15.67:0/1749639848 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c06d7a0 con 0x7ff71c15a2f0 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.638+0000 7ff71b7fe640 1 --2- 172.21.15.67:0/1749639848 >> v2:172.21.15.176:3300/0 conn(0x7ff71c0b6a00 0x7ff71c15d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.639+0000 7ff71affd640 1 --2- 172.21.15.67:0/1749639848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c06ce50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0713e0 tx=0x7ff70402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79c7a13c5b6a160 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.639+0000 7ff71a7fc640 1 -- 172.21.15.67:0/1749639848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704037070 con 0x7ff71c15a2f0 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.639+0000 7ff71a7fc640 1 -- 172.21.15.67:0/1749639848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff704032440 con 0x7ff71c15a2f0 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.639+0000 7ff71a7fc640 1 -- 172.21.15.67:0/1749639848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704032810 con 0x7ff71c15a2f0 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 -- 172.21.15.67:0/1749639848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 msgr2=0x7ff71c06ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 --2- 172.21.15.67:0/1749639848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c06ce50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0713e0 tx=0x7ff70402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 -- 172.21.15.67:0/1749639848 shutdown_connections 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 --2- 172.21.15.67:0/1749639848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c06f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 --2- 172.21.15.67:0/1749639848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c06ce50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 --2- 172.21.15.67:0/1749639848 >> v2:172.21.15.176:3300/0 conn(0x7ff71c0b6a00 0x7ff71c15d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 -- 172.21.15.67:0/1749639848 >> 172.21.15.67:0/1749639848 conn(0x7ff71c0d1ae0 msgr2=0x7ff71c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 -- 172.21.15.67:0/1749639848 shutdown_connections 2024-04-25T15:26:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.640+0000 7ff721c92640 1 -- 172.21.15.67:0/1749639848 wait complete. 2024-04-25T15:26:28.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.641+0000 7ff721c92640 1 Processor -- start 2024-04-25T15:26:28.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.642+0000 7ff721c92640 1 -- start start 2024-04-25T15:26:28.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff721c92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c144230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:28.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff721c92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0b6a00 0x7ff71c15ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:28.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff721c92640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c15f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:28.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff721c92640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c1609e0 con 0x7ff71c15a2f0 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff721c92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c160700 con 0x7ff71c0b6a00 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff721c92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff71c0ce580 con 0x7ff71c06d390 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c15f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c15f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59202/0 (socket says 172.21.15.67:59202) 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 -- 172.21.15.67:0/3831264830 learned_addr learned my addr 172.21.15.67:0/3831264830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71b7fe640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c144230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71affd640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0b6a00 0x7ff71c15ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 -- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 msgr2=0x7ff71c144230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c144230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 -- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0b6a00 msgr2=0x7ff71c15ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0b6a00 0x7ff71c15ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71bfff640 1 -- 172.21.15.67:0/3831264830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c1581c0 con 0x7ff71c15a2f0 2024-04-25T15:26:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71b7fe640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.643+0000 7ff71affd640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0b6a00 0x7ff71c15ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.644+0000 7ff71bfff640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c15f3b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff714004660 tx=0x7ff71400e500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.644+0000 7ff713fff640 1 -- 172.21.15.67:0/3831264830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714004810 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.644+0000 7ff713fff640 1 -- 172.21.15.67:0/3831264830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff71400ec10 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.644+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c153500 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.644+0000 7ff713fff640 1 -- 172.21.15.67:0/3831264830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714014470 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.644+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71c153ac0 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.645+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff71c153250 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.646+0000 7ff713fff640 1 -- 172.21.15.67:0/3831264830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff71401b030 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.646+0000 7ff713fff640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6fc0744b0 0x7ff6fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.646+0000 7ff713fff640 1 -- 172.21.15.67:0/3831264830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff714095650 con 0x7ff71c15a2f0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.646+0000 7ff71b7fe640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6fc0744b0 0x7ff6fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.647+0000 7ff71b7fe640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6fc0744b0 0x7ff6fc0769c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00b540 tx=0x7ff70c00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.651+0000 7ff713fff640 1 -- 172.21.15.67:0/3831264830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff714022020 con 0x7ff71c15a2f0 2024-04-25T15:26:28.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.877+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff71c0705a0 con 0x7ff71c15a2f0 2024-04-25T15:26:28.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.877+0000 7ff713fff640 1 -- 172.21.15.67:0/3831264830 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7ff71405ff00 con 0x7ff71c15a2f0 2024-04-25T15:26:28.878 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6fc0744b0 msgr2=0x7ff6fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6fc0744b0 0x7ff6fc0769c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff70c00b540 tx=0x7ff70c00d040 comp rx=0 tx=0).stop 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 msgr2=0x7ff71c15f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c15f3b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff714004660 tx=0x7ff71400e500 comp rx=0 tx=0).stop 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 shutdown_connections 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6fc0744b0 0x7ff6fc0769c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71c15a2f0 0x7ff71c15f3b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0b6a00 0x7ff71c15ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 --2- 172.21.15.67:0/3831264830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c06d390 0x7ff71c144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.879+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 >> 172.21.15.67:0/3831264830 conn(0x7ff71c0d1ae0 msgr2=0x7ff71c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.880+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 shutdown_connections 2024-04-25T15:26:28.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:28.880+0000 7ff721c92640 1 -- 172.21.15.67:0/3831264830 wait complete. 2024-04-25T15:26:28.891 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:26:28.891 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:26:28.892 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:26:28.892 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:26:28.892 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:28.892 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:26:28.892 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:26:28.892 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:26:28.892 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:26:28.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:28.892 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:26:28.911 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:26:28.911 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:26:28.911 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:26:28.911 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:28.911 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42700sec preferred_lft 42700sec 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:26:28.912 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::248e:2bff:fe69:7f63/64 scope link 2024-04-25T15:26:28.913 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:28.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:28.914 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:26:28.979 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:26:28.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:28.981 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:26:29.038 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:26:29.038 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:26:29.039 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:26:29.039 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:26:29.039 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:26:29.039 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:26:29.039 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:26:29.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:26:29.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:26:29.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:26:29.039 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:26:29.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:26:29.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:29.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.163 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:26:29.178 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:26:29.178 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:26:29.178 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:26:29.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:26:29.243 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:29.243 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:29.299 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:26:29.299 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:29.299 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:29.354 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T15:26:29.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:29.448 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:29.448 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:29.456 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:29.456 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:29.456 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:26:29.456 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:26:29.456 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:26:29.457 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:26:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:26:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:26:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:26:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:26:29.459 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:26:29.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.459 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:29.483 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:26:29.573 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:26:29.575 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:29.581 INFO:teuthology.orchestra.run.smithi176.stdout:client5253 2024-04-25T15:26:29.581 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:29.581 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:29.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.597 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-25T15:26:29.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.618 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 0 ./subdir) 2024-04-25T15:26:29.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.685 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:26:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:29.741 INFO:teuthology.orchestra.run.smithi067.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-04-25T15:26:29.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.742 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1K ./subdir) 2024-04-25T15:26:29.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.802 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:26:29.858 INFO:teuthology.orchestra.run.smithi067.stdout:1024 2024-04-25T15:26:29.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.858 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 0M ./subdir) 2024-04-25T15:26:29.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:29.918 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:26:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:29.974 INFO:teuthology.orchestra.run.smithi067.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-04-25T15:26:29.975 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:26:29.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:26:30.059 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:26:30.098 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:26:30.098 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:26:30.099 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:30.099 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:30.114 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:30.115 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:30.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.321+0000 7f040f2f9640 1 Processor -- start 2024-04-25T15:26:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.321+0000 7f040f2f9640 1 -- start start 2024-04-25T15:26:30.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f040f2f9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f040806e8f0 0x7f04080b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f040f2f9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040806d870 con 0x7f04080b4830 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f040f2f9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f040806db10 con 0x7f04080d4570 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f040f2f9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f04080b4c40 con 0x7f040806e8f0 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f040c894640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f04080b4830 0x7f04080b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53776/0 (socket says 172.21.15.67:53776) 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f040c894640 1 -- 172.21.15.67:0/3255133715 learned_addr learned my addr 172.21.15.67:0/3255133715 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f040d896640 1 --2- 172.21.15.67:0/3255133715 >> v2:172.21.15.176:3300/0 conn(0x7f040806e8f0 0x7f04080b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997378206 0 0) 0x7f040806d870 con 0x7f04080b4830 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03f0003520 con 0x7f04080b4830 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607990629 0 0) 0x7f040806db10 con 0x7f04080d4570 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.323+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f040806d870 con 0x7f04080d4570 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.324+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2792227036 0 0) 0x7f03f0003520 con 0x7f04080b4830 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.324+0000 7f040d896640 1 -- 172.21.15.67:0/3255133715 >> v1:172.21.15.176:6789/0 conn(0x7f04080d4570 legacy=0x7f04080d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.324+0000 7f040d896640 1 -- 172.21.15.67:0/3255133715 >> v1:172.21.15.67:6789/0 conn(0x7f04080b4830 legacy=0x7f04080b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.324+0000 7f040d896640 1 -- 172.21.15.67:0/3255133715 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040806e0f0 con 0x7f040806e8f0 2024-04-25T15:26:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.324+0000 7f040d896640 1 --2- 172.21.15.67:0/3255133715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f040806e8f0 0x7f04080b34d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009d90 tx=0x7f03f802a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a579c89126e0955 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.324+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc001670 con 0x7f040806e8f0 2024-04-25T15:26:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f8007540 con 0x7f040806e8f0 2024-04-25T15:26:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f0407fff640 1 -- 172.21.15.67:0/3255133715 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f8007910 con 0x7f040806e8f0 2024-04-25T15:26:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f040f2f9640 1 -- 172.21.15.67:0/3255133715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f040806e8f0 msgr2=0x7f04080b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f040f2f9640 1 --2- 172.21.15.67:0/3255133715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f040806e8f0 0x7f04080b34d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009d90 tx=0x7f03f802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f040f2f9640 1 -- 172.21.15.67:0/3255133715 shutdown_connections 2024-04-25T15:26:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f040f2f9640 1 --2- 172.21.15.67:0/3255133715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f040806e8f0 0x7f04080b34d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f040f2f9640 1 -- 172.21.15.67:0/3255133715 >> 172.21.15.67:0/3255133715 conn(0x7f04080d3a50 msgr2=0x7f04080d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f040f2f9640 1 -- 172.21.15.67:0/3255133715 shutdown_connections 2024-04-25T15:26:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.325+0000 7f040f2f9640 1 -- 172.21.15.67:0/3255133715 wait complete. 2024-04-25T15:26:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.326+0000 7f040f2f9640 1 Processor -- start 2024-04-25T15:26:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.326+0000 7f040f2f9640 1 -- start start 2024-04-25T15:26:30.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.327+0000 7f040f2f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f040806e8f0 0x7f0408160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.327+0000 7f040f2f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04080b4830 0x7f040815af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.327+0000 7f040f2f9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04080d4570 0x7f040815b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.327+0000 7f040f2f9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f04080d0fb0 con 0x7f04080b4830 2024-04-25T15:26:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040f2f9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f040806e460 con 0x7f04080d4570 2024-04-25T15:26:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040f2f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f040806ed00 con 0x7f040806e8f0 2024-04-25T15:26:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04080b4830 0x7f040815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04080b4830 0x7f040815af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59210/0 (socket says 172.21.15.67:59210) 2024-04-25T15:26:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 -- 172.21.15.67:0/4245271369 learned_addr learned my addr 172.21.15.67:0/4245271369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040d896640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04080d4570 0x7f040815b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040d095640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f040806e8f0 0x7f0408160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 -- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f040806e8f0 msgr2=0x7f0408160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f040806e8f0 0x7f0408160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 -- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04080d4570 msgr2=0x7f040815b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04080d4570 0x7f040815b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 -- 172.21.15.67:0/4245271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040815aa90 con 0x7f04080b4830 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040d896640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04080d4570 0x7f040815b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040c894640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04080b4830 0x7f040815af70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c4e0 tx=0x7f03fc00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f0405ffb640 1 -- 172.21.15.67:0/4245271369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc017070 con 0x7f04080b4830 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f0405ffb640 1 -- 172.21.15.67:0/4245271369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03fc005ce0 con 0x7f04080b4830 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f040815ee20 con 0x7f04080b4830 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.328+0000 7f0405ffb640 1 -- 172.21.15.67:0/4245271369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc007270 con 0x7f04080b4830 2024-04-25T15:26:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.329+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f040815f3e0 con 0x7f04080b4830 2024-04-25T15:26:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.330+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0408050740 con 0x7f04080b4830 2024-04-25T15:26:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.330+0000 7f0405ffb640 1 -- 172.21.15.67:0/4245271369 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f03fc01a070 con 0x7f04080b4830 2024-04-25T15:26:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.330+0000 7f0405ffb640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03f40744e0 0x7f03f40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.331+0000 7f0405ffb640 1 -- 172.21.15.67:0/4245271369 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f03fc0944d0 con 0x7f04080b4830 2024-04-25T15:26:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.331+0000 7f040d095640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03f40744e0 0x7f03f40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.331+0000 7f040d095640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03f40744e0 0x7f03f40769f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0400001230 tx=0x7f0400009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.335+0000 7f0405ffb640 1 -- 172.21.15.67:0/4245271369 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f03fc05e610 con 0x7f04080b4830 2024-04-25T15:26:30.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.521+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04080d1220 con 0x7f04080b4830 2024-04-25T15:26:30.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.522+0000 7f0405ffb640 1 -- 172.21.15.67:0/4245271369 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16633 (secure 0 0 0) 0x7f03fc092e20 con 0x7f04080b4830 2024-04-25T15:26:30.524 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:30.524 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:22.596544+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03f40744e0 msgr2=0x7f03f40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03f40744e0 0x7f03f40769f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0400001230 tx=0x7f0400009040 comp rx=0 tx=0).stop 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04080b4830 msgr2=0x7f040815af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04080b4830 0x7f040815af70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00c4e0 tx=0x7f03fc00c9a0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 shutdown_connections 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f03f40744e0 0x7f03f40769f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04080d4570 0x7f040815b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04080b4830 0x7f040815af70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 --2- 172.21.15.67:0/4245271369 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f040806e8f0 0x7f0408160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 >> 172.21.15.67:0/4245271369 conn(0x7f04080d3a50 msgr2=0x7f04080b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 shutdown_connections 2024-04-25T15:26:30.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.524+0000 7f040f2f9640 1 -- 172.21.15.67:0/4245271369 wait complete. 2024-04-25T15:26:30.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:26:18.702421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:26:19.919128+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:30.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.653+0000 7f89278e0640 1 Processor -- start 2024-04-25T15:26:30.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.653+0000 7f89278e0640 1 -- start start 2024-04-25T15:26:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.654+0000 7f89278e0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f892006ccd0 0x7f89200d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.654+0000 7f89278e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.654+0000 7f89278e0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89200d1850 con 0x7f89200d0e00 2024-04-25T15:26:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.654+0000 7f89278e0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8920160de0 con 0x7f892006ccd0 2024-04-25T15:26:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.654+0000 7f89278e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89201610c0 con 0x7f89200d4a50 2024-04-25T15:26:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.654+0000 7f8924e7b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f892006ccd0 0x7f89200d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8924e7b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f892006ccd0 0x7f89200d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59220/0 (socket says 172.21.15.67:59220) 2024-04-25T15:26:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8924e7b640 1 -- 172.21.15.67:0/2379353995 learned_addr learned my addr 172.21.15.67:0/2379353995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8925e7d640 1 --2- 172.21.15.67:0/2379353995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8924e7b640 1 -- 172.21.15.67:0/2379353995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 msgr2=0x7f89200d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8924e7b640 1 --2- 172.21.15.67:0/2379353995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8924e7b640 1 -- 172.21.15.67:0/2379353995 >> v1:172.21.15.176:6789/0 conn(0x7f89200d0e00 legacy=0x7f892006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8924e7b640 1 -- 172.21.15.67:0/2379353995 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89201613a0 con 0x7f892006ccd0 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8925e7d640 1 --2- 172.21.15.67:0/2379353995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.655+0000 7f8924e7b640 1 --2- 172.21.15.67:0/2379353995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f892006ccd0 0x7f89200d1210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8910009870 tx=0x7f891002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f33f8865d336e82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f8917fff640 1 -- 172.21.15.67:0/2379353995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910037070 con 0x7f892006ccd0 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f8917fff640 1 -- 172.21.15.67:0/2379353995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8910032440 con 0x7f892006ccd0 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f8917fff640 1 -- 172.21.15.67:0/2379353995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910032810 con 0x7f892006ccd0 2024-04-25T15:26:30.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 -- 172.21.15.67:0/2379353995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f892006ccd0 msgr2=0x7f89200d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 --2- 172.21.15.67:0/2379353995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f892006ccd0 0x7f89200d1210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8910009870 tx=0x7f891002a270 comp rx=0 tx=0).stop 2024-04-25T15:26:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 -- 172.21.15.67:0/2379353995 shutdown_connections 2024-04-25T15:26:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 --2- 172.21.15.67:0/2379353995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 --2- 172.21.15.67:0/2379353995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f892006ccd0 0x7f89200d1210 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 -- 172.21.15.67:0/2379353995 >> 172.21.15.67:0/2379353995 conn(0x7f89200d5d90 msgr2=0x7f89200d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 -- 172.21.15.67:0/2379353995 shutdown_connections 2024-04-25T15:26:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.656+0000 7f89278e0640 1 -- 172.21.15.67:0/2379353995 wait complete. 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.658+0000 7f89278e0640 1 Processor -- start 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.658+0000 7f89278e0640 1 -- start start 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.659+0000 7f89278e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89200d0e00 0x7f89200b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.659+0000 7f89278e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.659+0000 7f89278e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200b62e0 0x7f89200cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.659+0000 7f89278e0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8920164e00 con 0x7f89200b62e0 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.659+0000 7f89278e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89200b5320 con 0x7f89200d0e00 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.659+0000 7f89278e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8920164f80 con 0x7f89200d4a50 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200b62e0 0x7f89200cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200b62e0 0x7f89200cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59224/0 (socket says 172.21.15.67:59224) 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 -- 172.21.15.67:0/1170711174 learned_addr learned my addr 172.21.15.67:0/1170711174 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f892567c640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89200d0e00 0x7f89200b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8924e7b640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 -- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 msgr2=0x7f89200b5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200b5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 -- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89200d0e00 msgr2=0x7f89200b6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89200d0e00 0x7f89200b6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 -- 172.21.15.67:0/1170711174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89200b7b40 con 0x7f89200b62e0 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f892567c640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89200d0e00 0x7f89200b6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8925e7d640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200b62e0 0x7f89200cd540 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8918007df0 tx=0x7f891800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8915ffb640 1 -- 172.21.15.67:0/1170711174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891801a070 con 0x7f89200b62e0 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8915ffb640 1 -- 172.21.15.67:0/1170711174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8918003990 con 0x7f89200b62e0 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89200b7e20 con 0x7f89200b62e0 2024-04-25T15:26:30.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.660+0000 7f8915ffb640 1 -- 172.21.15.67:0/1170711174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8918003d80 con 0x7f89200b62e0 2024-04-25T15:26:30.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.661+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89200cd020 con 0x7f89200b62e0 2024-04-25T15:26:30.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.662+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89200ca770 con 0x7f89200b62e0 2024-04-25T15:26:30.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.662+0000 7f8915ffb640 1 -- 172.21.15.67:0/1170711174 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f891801d070 con 0x7f89200b62e0 2024-04-25T15:26:30.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.662+0000 7f8915ffb640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89040744b0 0x7f89040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.663+0000 7f8915ffb640 1 -- 172.21.15.67:0/1170711174 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f89180954e0 con 0x7f89200b62e0 2024-04-25T15:26:30.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.663+0000 7f892567c640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89040744b0 0x7f89040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.663+0000 7f892567c640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89040744b0 0x7f89040769c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f891c0015e0 tx=0x7f891c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.667+0000 7f8915ffb640 1 -- 172.21.15.67:0/1170711174 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89200ca770 con 0x7f89200b62e0 2024-04-25T15:26:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.850+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8920050740 con 0x7f89200b62e0 2024-04-25T15:26:30.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.850+0000 7f8915ffb640 1 -- 172.21.15.67:0/1170711174 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) ==== 93+0+10 (secure 0 0 0) 0x7f8918010070 con 0x7f89200b62e0 2024-04-25T15:26:30.852 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89040744b0 msgr2=0x7f89040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89040744b0 0x7f89040769c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f891c0015e0 tx=0x7f891c009040 comp rx=0 tx=0).stop 2024-04-25T15:26:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200b62e0 msgr2=0x7f89200cd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200b62e0 0x7f89200cd540 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8918007df0 tx=0x7f891800d4a0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 shutdown_connections 2024-04-25T15:26:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89040744b0 0x7f89040769c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89200b62e0 0x7f89200cd540 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89200d4a50 0x7f89200b5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 --2- 172.21.15.67:0/1170711174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89200d0e00 0x7f89200b6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 >> 172.21.15.67:0/1170711174 conn(0x7f89200d5d90 msgr2=0x7f89200d4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 shutdown_connections 2024-04-25T15:26:30.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.853+0000 7f89278e0640 1 -- 172.21.15.67:0/1170711174 wait complete. 2024-04-25T15:26:30.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.979+0000 7f15874bd640 1 Processor -- start 2024-04-25T15:26:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.979+0000 7f15874bd640 1 -- start start 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f15874bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800d0e00 0x7f15800d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f15874bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f158006c5c0 0x7f158006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f15874bd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158006cf10 con 0x7f15800b7810 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f15874bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f15800d5a60 con 0x7f158006c5c0 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f15874bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15800d1850 con 0x7f15800d0e00 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1584a58640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f158006c5c0 0x7f158006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1585a5a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f15800b7810 0x7f15800b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53786/0 (socket says 172.21.15.67:53786) 2024-04-25T15:26:30.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1585a5a640 1 -- 172.21.15.67:0/1164997963 learned_addr learned my addr 172.21.15.67:0/1164997963 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:30.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1585259640 1 --2- 172.21.15.67:0/1164997963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800d0e00 0x7f15800d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1584a58640 1 -- 172.21.15.67:0/1164997963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800d0e00 msgr2=0x7f15800d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:30.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1584a58640 1 --2- 172.21.15.67:0/1164997963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800d0e00 0x7f15800d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1584a58640 1 -- 172.21.15.67:0/1164997963 >> v1:172.21.15.67:6789/0 conn(0x7f15800b7810 legacy=0x7f15800b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1584a58640 1 -- 172.21.15.67:0/1164997963 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1580160de0 con 0x7f158006c5c0 2024-04-25T15:26:30.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f157ffff640 1 -- 172.21.15.67:0/1164997963 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4142919608 0 0) 0x7f1578001110 con 0x7f15800b7810 2024-04-25T15:26:30.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.981+0000 7f1585259640 1 --2- 172.21.15.67:0/1164997963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800d0e00 0x7f15800d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:30.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.982+0000 7f1584a58640 1 --2- 172.21.15.67:0/1164997963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158006c5c0 0x7f158006c9d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f15740097a0 tx=0x7f157402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467b9c18209cb5d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.982+0000 7f157ffff640 1 -- 172.21.15.67:0/1164997963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574037070 con 0x7f158006c5c0 2024-04-25T15:26:30.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.982+0000 7f157ffff640 1 -- 172.21.15.67:0/1164997963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1574032440 con 0x7f158006c5c0 2024-04-25T15:26:30.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.982+0000 7f157ffff640 1 -- 172.21.15.67:0/1164997963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574032810 con 0x7f158006c5c0 2024-04-25T15:26:30.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 -- 172.21.15.67:0/1164997963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158006c5c0 msgr2=0x7f158006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 --2- 172.21.15.67:0/1164997963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158006c5c0 0x7f158006c9d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f15740097a0 tx=0x7f157402a180 comp rx=0 tx=0).stop 2024-04-25T15:26:30.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 -- 172.21.15.67:0/1164997963 shutdown_connections 2024-04-25T15:26:30.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 --2- 172.21.15.67:0/1164997963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158006c5c0 0x7f158006c9d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 --2- 172.21.15.67:0/1164997963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800d0e00 0x7f15800d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 -- 172.21.15.67:0/1164997963 >> 172.21.15.67:0/1164997963 conn(0x7f15800d5d90 msgr2=0x7f15800d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:30.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 -- 172.21.15.67:0/1164997963 shutdown_connections 2024-04-25T15:26:30.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.983+0000 7f15874bd640 1 -- 172.21.15.67:0/1164997963 wait complete. 2024-04-25T15:26:30.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.985+0000 7f15874bd640 1 Processor -- start 2024-04-25T15:26:30.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.985+0000 7f15874bd640 1 -- start start 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f15874bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f158006c5c0 0x7f15801573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f15874bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800b7810 0x7f15800cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f15874bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800d0e00 0x7f15801ea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f15874bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1580160ad0 con 0x7f15800d0e00 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f15874bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15801613a0 con 0x7f158006c5c0 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f15874bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15800b5320 con 0x7f15800b7810 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800d0e00 0x7f15801ea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800d0e00 0x7f15801ea1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59254/0 (socket says 172.21.15.67:59254) 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 -- 172.21.15.67:0/882928860 learned_addr learned my addr 172.21.15.67:0/882928860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1584a58640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800b7810 0x7f15800cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585259640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f158006c5c0 0x7f15801573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 -- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800b7810 msgr2=0x7f15800cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800b7810 0x7f15800cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 -- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f158006c5c0 msgr2=0x7f15801573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f158006c5c0 0x7f15801573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.986+0000 7f1585a5a640 1 -- 172.21.15.67:0/882928860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15800cc4b0 con 0x7f15800d0e00 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.987+0000 7f1585a5a640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800d0e00 0x7f15801ea1d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f157800c3e0 tx=0x7f157800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.987+0000 7f157dffb640 1 -- 172.21.15.67:0/882928860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780074e0 con 0x7f15800d0e00 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.987+0000 7f157dffb640 1 -- 172.21.15.67:0/882928860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15780047e0 con 0x7f15800d0e00 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.987+0000 7f157dffb640 1 -- 172.21.15.67:0/882928860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578004bd0 con 0x7f15800d0e00 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.987+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15800ca970 con 0x7f15800d0e00 2024-04-25T15:26:30.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.987+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15800caf30 con 0x7f15800d0e00 2024-04-25T15:26:30.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.988+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1580050740 con 0x7f15800d0e00 2024-04-25T15:26:30.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.989+0000 7f157dffb640 1 -- 172.21.15.67:0/882928860 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1578011070 con 0x7f15800d0e00 2024-04-25T15:26:30.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.989+0000 7f157dffb640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f156c0744e0 0x7f156c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:30.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.990+0000 7f157dffb640 1 -- 172.21.15.67:0/882928860 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f1578094b70 con 0x7f15800d0e00 2024-04-25T15:26:30.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.990+0000 7f1585259640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f156c0744e0 0x7f156c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:30.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.990+0000 7f1585259640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f156c0744e0 0x7f156c0769f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1570009fd0 tx=0x7f15700097f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:30.994+0000 7f157dffb640 1 -- 172.21.15.67:0/882928860 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f158006cf10 con 0x7f15800d0e00 2024-04-25T15:26:31.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.180+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15800d4320 con 0x7f15800d0e00 2024-04-25T15:26:31.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.180+0000 7f157dffb640 1 -- 172.21.15.67:0/882928860 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) ==== 104+0+11 (secure 0 0 0) 0x7f1578067380 con 0x7f15800d0e00 2024-04-25T15:26:31.182 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.182+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f156c0744e0 msgr2=0x7f156c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.182+0000 7f15874bd640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f156c0744e0 0x7f156c0769f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1570009fd0 tx=0x7f15700097f0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.182+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800d0e00 msgr2=0x7f15801ea1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.182+0000 7f15874bd640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800d0e00 0x7f15801ea1d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f157800c3e0 tx=0x7f157800c8a0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 shutdown_connections 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f156c0744e0 0x7f156c0769f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15800d0e00 0x7f15801ea1d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15800b7810 0x7f15800cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 --2- 172.21.15.67:0/882928860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f158006c5c0 0x7f15801573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 >> 172.21.15.67:0/882928860 conn(0x7f15800d5d90 msgr2=0x7f15800b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 shutdown_connections 2024-04-25T15:26:31.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.183+0000 7f15874bd640 1 -- 172.21.15.67:0/882928860 wait complete. 2024-04-25T15:26:31.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.309+0000 7f8a86514640 1 Processor -- start 2024-04-25T15:26:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.309+0000 7f8a86514640 1 -- start start 2024-04-25T15:26:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a86514640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8a8006dce0 0x7f8a8015f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a86514640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a800cdea0 con 0x7f8a800cd9a0 2024-04-25T15:26:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a86514640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a800ce180 con 0x7f8a800b0af0 2024-04-25T15:26:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a86514640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8a800ceab0 con 0x7f8a8006dce0 2024-04-25T15:26:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a800cd9a0 0x7f8a8006d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53790/0 (socket says 172.21.15.67:53790) 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7ffff640 1 -- 172.21.15.67:0/2372074554 learned_addr learned my addr 172.21.15.67:0/2372074554 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7f7fe640 1 --2- 172.21.15.67:0/2372074554 >> v2:172.21.15.176:3300/0 conn(0x7f8a8006dce0 0x7f8a8015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884418034 0 0) 0x7f8a800cdea0 con 0x7f8a800cd9a0 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a6c003520 con 0x7f8a800cd9a0 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3835340087 0 0) 0x7f8a800ce180 con 0x7f8a800b0af0 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a800cdea0 con 0x7f8a800b0af0 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2771814928 0 0) 0x7f8a6c003520 con 0x7f8a800cd9a0 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7f7fe640 1 -- 172.21.15.67:0/2372074554 >> v1:172.21.15.176:6789/0 conn(0x7f8a800b0af0 legacy=0x7f8a8015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7f7fe640 1 -- 172.21.15.67:0/2372074554 >> v1:172.21.15.67:6789/0 conn(0x7f8a800cd9a0 legacy=0x7f8a8006d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.311+0000 7f8a7f7fe640 1 -- 172.21.15.67:0/2372074554 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a800ce460 con 0x7f8a8006dce0 2024-04-25T15:26:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.312+0000 7f8a7f7fe640 1 --2- 172.21.15.67:0/2372074554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a8006dce0 0x7f8a8015f360 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8a700097a0 tx=0x7f8a7002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e16091ef109b3567 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.312+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70037070 con 0x7f8a8006dce0 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.312+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a7002ac50 con 0x7f8a8006dce0 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.312+0000 7f8a7effd640 1 -- 172.21.15.67:0/2372074554 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68001670 con 0x7f8a8006dce0 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.313+0000 7f8a86514640 1 -- 172.21.15.67:0/2372074554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a8006dce0 msgr2=0x7f8a8015f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.313+0000 7f8a86514640 1 --2- 172.21.15.67:0/2372074554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a8006dce0 0x7f8a8015f360 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8a700097a0 tx=0x7f8a7002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.313+0000 7f8a86514640 1 -- 172.21.15.67:0/2372074554 shutdown_connections 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.313+0000 7f8a86514640 1 --2- 172.21.15.67:0/2372074554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a8006dce0 0x7f8a8015f360 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.313+0000 7f8a86514640 1 -- 172.21.15.67:0/2372074554 >> 172.21.15.67:0/2372074554 conn(0x7f8a80073090 msgr2=0x7f8a800734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.313+0000 7f8a86514640 1 -- 172.21.15.67:0/2372074554 shutdown_connections 2024-04-25T15:26:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.313+0000 7f8a86514640 1 -- 172.21.15.67:0/2372074554 wait complete. 2024-04-25T15:26:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.315+0000 7f8a86514640 1 Processor -- start 2024-04-25T15:26:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.315+0000 7f8a86514640 1 -- start start 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.316+0000 7f8a86514640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800b0af0 0x7f8a8013c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.316+0000 7f8a86514640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a800cd9a0 0x7f8a80147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.316+0000 7f8a86514640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a8015d3b0 0x7f8a8015d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.316+0000 7f8a86514640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a800cef40 con 0x7f8a8015d3b0 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.316+0000 7f8a86514640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a800b0f00 con 0x7f8a800cd9a0 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.316+0000 7f8a86514640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a800cf0c0 con 0x7f8a800b0af0 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.316+0000 7f8a84ab1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a8015d3b0 0x7f8a8015d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a8015d3b0 0x7f8a8015d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59256/0 (socket says 172.21.15.67:59256) 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 -- 172.21.15.67:0/289049377 learned_addr learned my addr 172.21.15.67:0/289049377 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a7f7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a800cd9a0 0x7f8a80147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a7ffff640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800b0af0 0x7f8a8013c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 -- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800b0af0 msgr2=0x7f8a8013c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800b0af0 0x7f8a8013c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 -- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a800cd9a0 msgr2=0x7f8a80147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a800cd9a0 0x7f8a80147620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 -- 172.21.15.67:0/289049377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8013c820 con 0x7f8a8015d3b0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a7f7fe640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a800cd9a0 0x7f8a80147620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a84ab1640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a8015d3b0 0x7f8a8015d7c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800c710 tx=0x7f8a7800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a7cff9640 1 -- 172.21.15.67:0/289049377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78003b70 con 0x7f8a8015d3b0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a7cff9640 1 -- 172.21.15.67:0/289049377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a78003d10 con 0x7f8a8015d3b0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8015dd00 con 0x7f8a8015d3b0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a7cff9640 1 -- 172.21.15.67:0/289049377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800aa50 con 0x7f8a8015d3b0 2024-04-25T15:26:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.317+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a8013df80 con 0x7f8a8015d3b0 2024-04-25T15:26:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.318+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a80050740 con 0x7f8a8015d3b0 2024-04-25T15:26:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.319+0000 7f8a7cff9640 1 -- 172.21.15.67:0/289049377 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8a7800abf0 con 0x7f8a8015d3b0 2024-04-25T15:26:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.319+0000 7f8a7cff9640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a600744e0 0x7f8a600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.320+0000 7f8a7cff9640 1 -- 172.21.15.67:0/289049377 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f8a78094fc0 con 0x7f8a8015d3b0 2024-04-25T15:26:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.320+0000 7f8a7ffff640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a600744e0 0x7f8a600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.320+0000 7f8a7ffff640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a600744e0 0x7f8a600769f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a680017e0 tx=0x7f8a68001610 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.324+0000 7f8a7cff9640 1 -- 172.21.15.67:0/289049377 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a8015dd00 con 0x7f8a8015d3b0 2024-04-25T15:26:31.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.511+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a8006d8f0 con 0x7f8a8015d3b0 2024-04-25T15:26:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.511+0000 7f8a7cff9640 1 -- 172.21.15.67:0/289049377 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) ==== 100+0+11 (secure 0 0 0) 0x7f8a8013c820 con 0x7f8a8015d3b0 2024-04-25T15:26:31.513 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a600744e0 msgr2=0x7f8a600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a600744e0 0x7f8a600769f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a680017e0 tx=0x7f8a68001610 comp rx=0 tx=0).stop 2024-04-25T15:26:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a8015d3b0 msgr2=0x7f8a8015d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a8015d3b0 0x7f8a8015d7c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800c710 tx=0x7f8a7800cbd0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 shutdown_connections 2024-04-25T15:26:31.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a600744e0 0x7f8a600769f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a8015d3b0 0x7f8a8015d7c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a800cd9a0 0x7f8a80147620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 --2- 172.21.15.67:0/289049377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800b0af0 0x7f8a8013c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 >> 172.21.15.67:0/289049377 conn(0x7f8a80073090 msgr2=0x7f8a800b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 shutdown_connections 2024-04-25T15:26:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.513+0000 7f8a86514640 1 -- 172.21.15.67:0/289049377 wait complete. 2024-04-25T15:26:31.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:26:31.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.637+0000 7f037f772640 1 Processor -- start 2024-04-25T15:26:31.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.637+0000 7f037f772640 1 -- start start 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.638+0000 7f037f772640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03780d0de0 0x7f03780d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.638+0000 7f037f772640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037806c5c0 0x7f037806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.638+0000 7f037f772640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f03780b77f0 0x7f03780b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.638+0000 7f037f772640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f037806cf10 con 0x7f03780d0de0 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.638+0000 7f037f772640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f03780d1830 con 0x7f03780b77f0 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.638+0000 7f037f772640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0378160dc0 con 0x7f037806c5c0 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03780d0de0 0x7f03780d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03780d0de0 0x7f03780d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59262/0 (socket says 172.21.15.67:59262) 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 -- 172.21.15.67:0/1506151578 learned_addr learned my addr 172.21.15.67:0/1506151578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037dd0f640 1 --2- 172.21.15.67:0/1506151578 >> v2:172.21.15.176:3300/0 conn(0x7f03780b77f0 0x7f03780b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037cd0d640 1 --2- 172.21.15.67:0/1506151578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037806c5c0 0x7f037806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 -- 172.21.15.67:0/1506151578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037806c5c0 msgr2=0x7f037806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 --2- 172.21.15.67:0/1506151578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037806c5c0 0x7f037806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 -- 172.21.15.67:0/1506151578 >> v2:172.21.15.176:3300/0 conn(0x7f03780b77f0 msgr2=0x7f03780b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 --2- 172.21.15.67:0/1506151578 >> v2:172.21.15.176:3300/0 conn(0x7f03780b77f0 0x7f03780b7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.639+0000 7f037d50e640 1 -- 172.21.15.67:0/1506151578 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03781610a0 con 0x7f03780d0de0 2024-04-25T15:26:31.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.640+0000 7f037d50e640 1 --2- 172.21.15.67:0/1506151578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780d0de0 0x7f03780d11f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f037806d1e0 tx=0x7f037402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9482456f9732849b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.641+0000 7f036ffff640 1 -- 172.21.15.67:0/1506151578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374037070 con 0x7f03780d0de0 2024-04-25T15:26:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.641+0000 7f036ffff640 1 -- 172.21.15.67:0/1506151578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0374032440 con 0x7f03780d0de0 2024-04-25T15:26:31.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.641+0000 7f036ffff640 1 -- 172.21.15.67:0/1506151578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374032810 con 0x7f03780d0de0 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 -- 172.21.15.67:0/1506151578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780d0de0 msgr2=0x7f03780d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 --2- 172.21.15.67:0/1506151578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780d0de0 0x7f03780d11f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f037806d1e0 tx=0x7f037402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 -- 172.21.15.67:0/1506151578 shutdown_connections 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 --2- 172.21.15.67:0/1506151578 >> v2:172.21.15.176:3300/0 conn(0x7f03780b77f0 0x7f03780b7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 --2- 172.21.15.67:0/1506151578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037806c5c0 0x7f037806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 --2- 172.21.15.67:0/1506151578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780d0de0 0x7f03780d11f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 -- 172.21.15.67:0/1506151578 >> 172.21.15.67:0/1506151578 conn(0x7f03780d5d70 msgr2=0x7f03780d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:31.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 -- 172.21.15.67:0/1506151578 shutdown_connections 2024-04-25T15:26:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.642+0000 7f037f772640 1 -- 172.21.15.67:0/1506151578 wait complete. 2024-04-25T15:26:31.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.644+0000 7f037f772640 1 Processor -- start 2024-04-25T15:26:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.644+0000 7f037f772640 1 -- start start 2024-04-25T15:26:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037f772640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037806c5c0 0x7f0378154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037f772640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780b77f0 0x7f037815dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037f772640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03780d0de0 0x7f03780cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037f772640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03780b5300 con 0x7f03780b77f0 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037f772640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0378160ab0 con 0x7f037806c5c0 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037f772640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0378164de0 con 0x7f03780d0de0 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037cd0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780b77f0 0x7f037815dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037cd0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780b77f0 0x7f037815dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59274/0 (socket says 172.21.15.67:59274) 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037cd0d640 1 -- 172.21.15.67:0/3845656196 learned_addr learned my addr 172.21.15.67:0/3845656196 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037dd0f640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03780d0de0 0x7f03780cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.645+0000 7f037d50e640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037806c5c0 0x7f0378154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037cd0d640 1 -- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03780d0de0 msgr2=0x7f03780cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037cd0d640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03780d0de0 0x7f03780cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037cd0d640 1 -- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037806c5c0 msgr2=0x7f0378154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037cd0d640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037806c5c0 0x7f0378154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037cd0d640 1 -- 172.21.15.67:0/3845656196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03781f72a0 con 0x7f03780b77f0 2024-04-25T15:26:31.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037cd0d640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780b77f0 0x7f037815dd90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f036800e870 tx=0x7f036800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f036dffb640 1 -- 172.21.15.67:0/3845656196 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368017070 con 0x7f03780b77f0 2024-04-25T15:26:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f036dffb640 1 -- 172.21.15.67:0/3845656196 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f036800ee60 con 0x7f03780b77f0 2024-04-25T15:26:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f036dffb640 1 -- 172.21.15.67:0/3845656196 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036800cb80 con 0x7f03780b77f0 2024-04-25T15:26:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03781f7580 con 0x7f03780b77f0 2024-04-25T15:26:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.646+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03781f7b40 con 0x7f03780b77f0 2024-04-25T15:26:31.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.647+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03781f7010 con 0x7f03780b77f0 2024-04-25T15:26:31.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.648+0000 7f036dffb640 1 -- 172.21.15.67:0/3845656196 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f036801a070 con 0x7f03780b77f0 2024-04-25T15:26:31.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.648+0000 7f036dffb640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f035c0744b0 0x7f035c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:31.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.648+0000 7f036dffb640 1 -- 172.21.15.67:0/3845656196 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0368095330 con 0x7f03780b77f0 2024-04-25T15:26:31.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.649+0000 7f037d50e640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f035c0744b0 0x7f035c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:31.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.649+0000 7f037d50e640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f035c0744b0 0x7f035c0769c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0374034000 tx=0x7f037402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:31.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.653+0000 7f036dffb640 1 -- 172.21.15.67:0/3845656196 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f036805f500 con 0x7f03780b77f0 2024-04-25T15:26:31.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.849+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0378050740 con 0x7f03780b77f0 2024-04-25T15:26:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.849+0000 7f036dffb640 1 -- 172.21.15.67:0/3845656196 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7f0368063800 con 0x7f03780b77f0 2024-04-25T15:26:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:26:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f035c0744b0 msgr2=0x7f035c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f035c0744b0 0x7f035c0769c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0374034000 tx=0x7f037402e000 comp rx=0 tx=0).stop 2024-04-25T15:26:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780b77f0 msgr2=0x7f037815dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780b77f0 0x7f037815dd90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f036800e870 tx=0x7f036800ed30 comp rx=0 tx=0).stop 2024-04-25T15:26:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 shutdown_connections 2024-04-25T15:26:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f035c0744b0 0x7f035c0769c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03780d0de0 0x7f03780cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03780b77f0 0x7f037815dd90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 --2- 172.21.15.67:0/3845656196 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037806c5c0 0x7f0378154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 >> 172.21.15.67:0/3845656196 conn(0x7f03780d5d70 msgr2=0x7f0378160180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 shutdown_connections 2024-04-25T15:26:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:31.851+0000 7f037f772640 1 -- 172.21.15.67:0/3845656196 wait complete. 2024-04-25T15:26:31.864 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T15:26:31.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:31.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:31.892 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:26:31.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:31.892 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:26:31.949 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:26:31.950 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:31.950 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:31.969 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:31.969 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:32.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:32.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.137+0000 7f24adad1640 1 Processor -- start 2024-04-25T15:26:32.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.137+0000 7f24adad1640 1 -- start start 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.138+0000 7f24adad1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f24a8143440 0x7f24a8160b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.138+0000 7f24adad1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24a81610a0 0x7f24a80d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.138+0000 7f24adad1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a80d0830 0x7f24a815d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.138+0000 7f24adad1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f24a806f4a0 con 0x7f24a81610a0 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.138+0000 7f24adad1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f24a81614b0 con 0x7f24a8143440 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.138+0000 7f24adad1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f24a8143850 con 0x7f24a80d0830 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24a81610a0 0x7f24a80d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24a81610a0 0x7f24a80d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59280/0 (socket says 172.21.15.67:59280) 2024-04-25T15:26:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 -- 172.21.15.67:0/3606167686 learned_addr learned my addr 172.21.15.67:0/3606167686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a7fff640 1 --2- 172.21.15.67:0/3606167686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a80d0830 0x7f24a815d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a77fe640 1 --2- 172.21.15.67:0/3606167686 >> v2:172.21.15.176:3300/0 conn(0x7f24a8143440 0x7f24a8160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 -- 172.21.15.67:0/3606167686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a80d0830 msgr2=0x7f24a815d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 --2- 172.21.15.67:0/3606167686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a80d0830 0x7f24a815d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 -- 172.21.15.67:0/3606167686 >> v2:172.21.15.176:3300/0 conn(0x7f24a8143440 msgr2=0x7f24a8160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 --2- 172.21.15.67:0/3606167686 >> v2:172.21.15.176:3300/0 conn(0x7f24a8143440 0x7f24a8160b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a6ffd640 1 -- 172.21.15.67:0/3606167686 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a80d16e0 con 0x7f24a81610a0 2024-04-25T15:26:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.139+0000 7f24a77fe640 1 --2- 172.21.15.67:0/3606167686 >> v2:172.21.15.176:3300/0 conn(0x7f24a8143440 0x7f24a8160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:32.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.140+0000 7f24a6ffd640 1 --2- 172.21.15.67:0/3606167686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a81610a0 0x7f24a80d02f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f249c009870 tx=0x7f249c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d04ceb2d95384e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.140+0000 7f24a67fc640 1 -- 172.21.15.67:0/3606167686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c037070 con 0x7f24a81610a0 2024-04-25T15:26:32.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.140+0000 7f24a67fc640 1 -- 172.21.15.67:0/3606167686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249c032440 con 0x7f24a81610a0 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.140+0000 7f24a67fc640 1 -- 172.21.15.67:0/3606167686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c032810 con 0x7f24a81610a0 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 -- 172.21.15.67:0/3606167686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a81610a0 msgr2=0x7f24a80d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 --2- 172.21.15.67:0/3606167686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a81610a0 0x7f24a80d02f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f249c009870 tx=0x7f249c02a180 comp rx=0 tx=0).stop 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 -- 172.21.15.67:0/3606167686 shutdown_connections 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 --2- 172.21.15.67:0/3606167686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a80d0830 0x7f24a815d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 --2- 172.21.15.67:0/3606167686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a81610a0 0x7f24a80d02f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 --2- 172.21.15.67:0/3606167686 >> v2:172.21.15.176:3300/0 conn(0x7f24a8143440 0x7f24a8160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 -- 172.21.15.67:0/3606167686 >> 172.21.15.67:0/3606167686 conn(0x7f24a80d1ae0 msgr2=0x7f24a80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 -- 172.21.15.67:0/3606167686 shutdown_connections 2024-04-25T15:26:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.141+0000 7f24adad1640 1 -- 172.21.15.67:0/3606167686 wait complete. 2024-04-25T15:26:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.142+0000 7f24adad1640 1 Processor -- start 2024-04-25T15:26:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.143+0000 7f24adad1640 1 -- start start 2024-04-25T15:26:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24adad1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a80d0830 0x7f24a80cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24adad1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8143440 0x7f24a8152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24adad1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a81610a0 0x7f24a8145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24adad1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24a806f130 con 0x7f24a8143440 2024-04-25T15:26:32.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24adad1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f24a80d1400 con 0x7f24a80d0830 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24adad1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f24a80b4fb0 con 0x7f24a81610a0 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8143440 0x7f24a8152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8143440 0x7f24a8152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59286/0 (socket says 172.21.15.67:59286) 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 -- 172.21.15.67:0/1579936965 learned_addr learned my addr 172.21.15.67:0/1579936965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a7fff640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a81610a0 0x7f24a8145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a77fe640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a80d0830 0x7f24a80cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 -- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a81610a0 msgr2=0x7f24a8145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a81610a0 0x7f24a8145da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 -- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a80d0830 msgr2=0x7f24a80cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a80d0830 0x7f24a80cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a6ffd640 1 -- 172.21.15.67:0/1579936965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a80c7890 con 0x7f24a8143440 2024-04-25T15:26:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.144+0000 7f24a77fe640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a80d0830 0x7f24a80cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.145+0000 7f24a6ffd640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8143440 0x7f24a8152730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f249c02a1c0 tx=0x7f249c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.145+0000 7f248ffff640 1 -- 172.21.15.67:0/1579936965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c041020 con 0x7f24a8143440 2024-04-25T15:26:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.145+0000 7f248ffff640 1 -- 172.21.15.67:0/1579936965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249c0306c0 con 0x7f24a8143440 2024-04-25T15:26:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.145+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a80c7b70 con 0x7f24a8143440 2024-04-25T15:26:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.145+0000 7f248ffff640 1 -- 172.21.15.67:0/1579936965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c037040 con 0x7f24a8143440 2024-04-25T15:26:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.145+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a80c8130 con 0x7f24a8143440 2024-04-25T15:26:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.146+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a80c75b0 con 0x7f24a8143440 2024-04-25T15:26:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.146+0000 7f248ffff640 1 -- 172.21.15.67:0/1579936965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f249c044020 con 0x7f24a8143440 2024-04-25T15:26:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.147+0000 7f248ffff640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24940744e0 0x7f24940769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.147+0000 7f248ffff640 1 -- 172.21.15.67:0/1579936965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f249c0b5a50 con 0x7f24a8143440 2024-04-25T15:26:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.147+0000 7f24a77fe640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24940744e0 0x7f24940769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.148+0000 7f24a77fe640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24940744e0 0x7f24940769f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f24a806dff0 tx=0x7f2498008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.151+0000 7f248ffff640 1 -- 172.21.15.67:0/1579936965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f249c04b030 con 0x7f24a8143440 2024-04-25T15:26:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.336+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24a8160af0 con 0x7f24a8143440 2024-04-25T15:26:32.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.338+0000 7f248ffff640 1 -- 172.21.15.67:0/1579936965 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16633 (secure 0 0 0) 0x7f249c0801e0 con 0x7f24a8143440 2024-04-25T15:26:32.343 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:32.343 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:22.596544+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24940744e0 msgr2=0x7f24940769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24940744e0 0x7f24940769f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f24a806dff0 tx=0x7f2498008040 comp rx=0 tx=0).stop 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8143440 msgr2=0x7f24a8152730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8143440 0x7f24a8152730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f249c02a1c0 tx=0x7f249c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 shutdown_connections 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24940744e0 0x7f24940769f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24a81610a0 0x7f24a8145da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a8143440 0x7f24a8152730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 --2- 172.21.15.67:0/1579936965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24a80d0830 0x7f24a80cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 >> 172.21.15.67:0/1579936965 conn(0x7f24a80d1ae0 msgr2=0x7f24a815de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 shutdown_connections 2024-04-25T15:26:32.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.340+0000 7f24adad1640 1 -- 172.21.15.67:0/1579936965 wait complete. 2024-04-25T15:26:32.354 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:26:18.702421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:26:19.919128+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:32.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.467+0000 7f4ba550d640 1 Processor -- start 2024-04-25T15:26:32.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.467+0000 7f4ba550d640 1 -- start start 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.468+0000 7f4ba550d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ba006e760 0x7f4ba00d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.468+0000 7f4ba550d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4ba00d1270 0x7f4ba00d49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.468+0000 7f4ba550d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba0161970 con 0x7f4ba015e0c0 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.468+0000 7f4ba550d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4ba01614e0 con 0x7f4ba006e760 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.468+0000 7f4ba550d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4ba00d1a40 con 0x7f4ba00d1270 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.468+0000 7f4b97fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ba006e760 0x7f4ba00d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.468+0000 7f4b97fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ba006e760 0x7f4ba00d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59300/0 (socket says 172.21.15.67:59300) 2024-04-25T15:26:32.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.469+0000 7f4b97fff640 1 -- 172.21.15.67:0/4269604082 learned_addr learned my addr 172.21.15.67:0/4269604082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:32.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.469+0000 7f4b9f7fe640 1 --2- 172.21.15.67:0/4269604082 >> v2:172.21.15.176:3300/0 conn(0x7f4ba00d1270 0x7f4ba00d49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.469+0000 7f4b97fff640 1 -- 172.21.15.67:0/4269604082 >> v2:172.21.15.176:3300/0 conn(0x7f4ba00d1270 msgr2=0x7f4ba00d49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:32.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.469+0000 7f4b97fff640 1 --2- 172.21.15.67:0/4269604082 >> v2:172.21.15.176:3300/0 conn(0x7f4ba00d1270 0x7f4ba00d49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.469+0000 7f4b97fff640 1 -- 172.21.15.67:0/4269604082 >> v1:172.21.15.176:6789/0 conn(0x7f4ba015e0c0 legacy=0x7f4ba006e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:32.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.469+0000 7f4b97fff640 1 -- 172.21.15.67:0/4269604082 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba0160bb0 con 0x7f4ba006e760 2024-04-25T15:26:32.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.470+0000 7f4b97fff640 1 --2- 172.21.15.67:0/4269604082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba006e760 0x7f4ba00d0d30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00b5790 tx=0x7f4b8802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183d95ed858639ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.470+0000 7f4b9e7fc640 1 -- 172.21.15.67:0/4269604082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88037070 con 0x7f4ba006e760 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.470+0000 7f4b9e7fc640 1 -- 172.21.15.67:0/4269604082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b88032440 con 0x7f4ba006e760 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.470+0000 7f4b9e7fc640 1 -- 172.21.15.67:0/4269604082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88032810 con 0x7f4ba006e760 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.470+0000 7f4ba550d640 1 -- 172.21.15.67:0/4269604082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba006e760 msgr2=0x7f4ba00d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.470+0000 7f4ba550d640 1 --2- 172.21.15.67:0/4269604082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba006e760 0x7f4ba00d0d30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00b5790 tx=0x7f4b8802a230 comp rx=0 tx=0).stop 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.471+0000 7f4ba550d640 1 -- 172.21.15.67:0/4269604082 shutdown_connections 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.471+0000 7f4ba550d640 1 --2- 172.21.15.67:0/4269604082 >> v2:172.21.15.176:3300/0 conn(0x7f4ba00d1270 0x7f4ba00d49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.471+0000 7f4ba550d640 1 --2- 172.21.15.67:0/4269604082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba006e760 0x7f4ba00d0d30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.471+0000 7f4ba550d640 1 -- 172.21.15.67:0/4269604082 >> 172.21.15.67:0/4269604082 conn(0x7f4ba00d2800 msgr2=0x7f4ba00d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.471+0000 7f4ba550d640 1 -- 172.21.15.67:0/4269604082 shutdown_connections 2024-04-25T15:26:32.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.471+0000 7f4ba550d640 1 -- 172.21.15.67:0/4269604082 wait complete. 2024-04-25T15:26:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.472+0000 7f4ba550d640 1 Processor -- start 2024-04-25T15:26:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.472+0000 7f4ba550d640 1 -- start start 2024-04-25T15:26:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4ba550d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba006e760 0x7f4ba014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4ba550d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba00d1270 0x7f4ba01ea410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4ba550d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba015e0c0 0x7f4ba00ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4ba550d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba00d1680 con 0x7f4ba00d1270 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4ba550d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba0161170 con 0x7f4ba015e0c0 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4ba550d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba00d1800 con 0x7f4ba006e760 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba00d1270 0x7f4ba01ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba00d1270 0x7f4ba01ea410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59316/0 (socket says 172.21.15.67:59316) 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 -- 172.21.15.67:0/3775325871 learned_addr learned my addr 172.21.15.67:0/3775325871 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b9f7fe640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba015e0c0 0x7f4ba00ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b9effd640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba006e760 0x7f4ba014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 -- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba006e760 msgr2=0x7f4ba014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba006e760 0x7f4ba014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 -- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba015e0c0 msgr2=0x7f4ba00ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba015e0c0 0x7f4ba00ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 -- 172.21.15.67:0/3775325871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba01ea950 con 0x7f4ba00d1270 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b9effd640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba006e760 0x7f4ba014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b97fff640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba00d1270 0x7f4ba01ea410 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4b8802a270 tx=0x7f4b880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b977fe640 1 -- 172.21.15.67:0/3775325871 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88041020 con 0x7f4ba00d1270 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.474+0000 7f4b977fe640 1 -- 172.21.15.67:0/3775325871 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b880306c0 con 0x7f4ba00d1270 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.475+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba01eae60 con 0x7f4ba00d1270 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.475+0000 7f4b977fe640 1 -- 172.21.15.67:0/3775325871 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88037040 con 0x7f4ba00d1270 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.475+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba01eb420 con 0x7f4ba00d1270 2024-04-25T15:26:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.476+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba0050740 con 0x7f4ba00d1270 2024-04-25T15:26:32.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.476+0000 7f4b977fe640 1 -- 172.21.15.67:0/3775325871 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4b88044020 con 0x7f4ba00d1270 2024-04-25T15:26:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.477+0000 7f4b977fe640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b800744b0 0x7f4b800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.477+0000 7f4b977fe640 1 -- 172.21.15.67:0/3775325871 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f4b880b59a0 con 0x7f4ba00d1270 2024-04-25T15:26:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.477+0000 7f4b9effd640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b800744b0 0x7f4b800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.477+0000 7f4b9effd640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b800744b0 0x7f4b800769c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b90006fd0 tx=0x7f4b90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.481+0000 7f4b977fe640 1 -- 172.21.15.67:0/3775325871 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b8804b030 con 0x7f4ba00d1270 2024-04-25T15:26:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.664+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4ba00cb060 con 0x7f4ba00d1270 2024-04-25T15:26:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.665+0000 7f4b977fe640 1 -- 172.21.15.67:0/3775325871 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) ==== 93+0+10 (secure 0 0 0) 0x7f4b88080250 con 0x7f4ba00d1270 2024-04-25T15:26:32.667 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:32.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b800744b0 msgr2=0x7f4b800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b800744b0 0x7f4b800769c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b90006fd0 tx=0x7f4b90008040 comp rx=0 tx=0).stop 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba00d1270 msgr2=0x7f4ba01ea410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba00d1270 0x7f4ba01ea410 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4b8802a270 tx=0x7f4b880046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 shutdown_connections 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4b800744b0 0x7f4b800769c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ba015e0c0 0x7f4ba00ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ba00d1270 0x7f4ba01ea410 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 --2- 172.21.15.67:0/3775325871 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ba006e760 0x7f4ba014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.666+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 >> 172.21.15.67:0/3775325871 conn(0x7f4ba00d2800 msgr2=0x7f4ba00d5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.667+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 shutdown_connections 2024-04-25T15:26:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.667+0000 7f4ba550d640 1 -- 172.21.15.67:0/3775325871 wait complete. 2024-04-25T15:26:32.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.793+0000 7f6968a0c640 1 Processor -- start 2024-04-25T15:26:32.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.793+0000 7f6968a0c640 1 -- start start 2024-04-25T15:26:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6968a0c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f69640d0e00 0x7f69640d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6968a0c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69640d4a50 0x7f69640d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6968a0c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69640d1850 con 0x7f696406c5c0 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6968a0c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6964160de0 con 0x7f69640d4a50 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6968a0c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f69641610c0 con 0x7f69640d0e00 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6961d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f696406c5c0 0x7f696406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53796/0 (socket says 172.21.15.67:53796) 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6961d9b640 1 -- 172.21.15.67:0/3511450399 learned_addr learned my addr 172.21.15.67:0/3511450399 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.794+0000 7f6962d9d640 1 --2- 172.21.15.67:0/3511450399 >> v2:172.21.15.67:3300/0 conn(0x7f69640d4a50 0x7f69640d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f696259c640 1 --2- 172.21.15.67:0/3511450399 >> v2:172.21.15.176:3300/0 conn(0x7f69640d0e00 0x7f69640d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f696159a640 1 -- 172.21.15.67:0/3511450399 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258662021 0 0) 0x7f69640d1850 con 0x7f696406c5c0 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f696159a640 1 -- 172.21.15.67:0/3511450399 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f694c003520 con 0x7f696406c5c0 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f6962d9d640 1 -- 172.21.15.67:0/3511450399 >> v2:172.21.15.176:3300/0 conn(0x7f69640d0e00 msgr2=0x7f69640d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f6962d9d640 1 --2- 172.21.15.67:0/3511450399 >> v2:172.21.15.176:3300/0 conn(0x7f69640d0e00 0x7f69640d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f6962d9d640 1 -- 172.21.15.67:0/3511450399 >> v1:172.21.15.67:6789/0 conn(0x7f696406c5c0 legacy=0x7f696406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f6962d9d640 1 -- 172.21.15.67:0/3511450399 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69641613a0 con 0x7f69640d4a50 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.795+0000 7f6962d9d640 1 --2- 172.21.15.67:0/3511450399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69640d4a50 0x7f69640d4e60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f69640d3e60 tx=0x7f695c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985ef5eba34d420f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f696159a640 1 -- 172.21.15.67:0/3511450399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c007740 con 0x7f69640d4a50 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f696159a640 1 -- 172.21.15.67:0/3511450399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c0078e0 con 0x7f69640d4a50 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f696159a640 1 -- 172.21.15.67:0/3511450399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c007cb0 con 0x7f69640d4a50 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 -- 172.21.15.67:0/3511450399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69640d4a50 msgr2=0x7f69640d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 --2- 172.21.15.67:0/3511450399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69640d4a50 0x7f69640d4e60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f69640d3e60 tx=0x7f695c02a5f0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 -- 172.21.15.67:0/3511450399 shutdown_connections 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 --2- 172.21.15.67:0/3511450399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69640d4a50 0x7f69640d4e60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 --2- 172.21.15.67:0/3511450399 >> v2:172.21.15.176:3300/0 conn(0x7f69640d0e00 0x7f69640d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 -- 172.21.15.67:0/3511450399 >> 172.21.15.67:0/3511450399 conn(0x7f69640d5d90 msgr2=0x7f69640d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 -- 172.21.15.67:0/3511450399 shutdown_connections 2024-04-25T15:26:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.796+0000 7f6968a0c640 1 -- 172.21.15.67:0/3511450399 wait complete. 2024-04-25T15:26:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.798+0000 7f6968a0c640 1 Processor -- start 2024-04-25T15:26:32.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.798+0000 7f6968a0c640 1 -- start start 2024-04-25T15:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6968a0c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f696406c5c0 0x7f69640cf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6968a0c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69640d0e00 0x7f69641f96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6968a0c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69640d4a50 0x7f69641f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6968a0c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6964164e00 con 0x7f696406c5c0 2024-04-25T15:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6968a0c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f69640b5320 con 0x7f69640d4a50 2024-04-25T15:26:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6968a0c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6964164f80 con 0x7f69640d0e00 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f696406c5c0 0x7f69640cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f696406c5c0 0x7f69640cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59330/0 (socket says 172.21.15.67:59330) 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 -- 172.21.15.67:0/1234032265 learned_addr learned my addr 172.21.15.67:0/1234032265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6962d9d640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69640d4a50 0x7f69641f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6961d9b640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69640d0e00 0x7f69641f96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 -- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69640d0e00 msgr2=0x7f69641f96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69640d0e00 0x7f69641f96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 -- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69640d4a50 msgr2=0x7f69641f67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69640d4a50 0x7f69641f67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 -- 172.21.15.67:0/1234032265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69640ca820 con 0x7f696406c5c0 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f696259c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f696406c5c0 0x7f69640cf340 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f695400d5f0 tx=0x7f695400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6952ffd640 1 -- 172.21.15.67:0/1234032265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69540125f0 con 0x7f696406c5c0 2024-04-25T15:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6952ffd640 1 -- 172.21.15.67:0/1234032265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6954019070 con 0x7f696406c5c0 2024-04-25T15:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69640cfc70 con 0x7f696406c5c0 2024-04-25T15:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.800+0000 7f6952ffd640 1 -- 172.21.15.67:0/1234032265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69540129d0 con 0x7f696406c5c0 2024-04-25T15:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.801+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69640cada0 con 0x7f696406c5c0 2024-04-25T15:26:32.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.801+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6964050740 con 0x7f696406c5c0 2024-04-25T15:26:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.802+0000 7f6952ffd640 1 -- 172.21.15.67:0/1234032265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6954012b70 con 0x7f696406c5c0 2024-04-25T15:26:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.803+0000 7f6952ffd640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f69440744b0 0x7f69440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.803+0000 7f6952ffd640 1 -- 172.21.15.67:0/1234032265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f69540956b0 con 0x7f696406c5c0 2024-04-25T15:26:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.803+0000 7f6961d9b640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f69440744b0 0x7f69440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.804+0000 7f6961d9b640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f69440744b0 0x7f69440769c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6958007fd0 tx=0x7f6958009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.807+0000 7f6952ffd640 1 -- 172.21.15.67:0/1234032265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f695405fe40 con 0x7f696406c5c0 2024-04-25T15:26:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.994+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f69640d4320 con 0x7f696406c5c0 2024-04-25T15:26:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.995+0000 7f6952ffd640 1 -- 172.21.15.67:0/1234032265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) ==== 104+0+11 (secure 0 0 0) 0x7f695405f740 con 0x7f696406c5c0 2024-04-25T15:26:32.996 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f69440744b0 msgr2=0x7f69440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f69440744b0 0x7f69440769c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6958007fd0 tx=0x7f6958009040 comp rx=0 tx=0).stop 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f696406c5c0 msgr2=0x7f69640cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f696406c5c0 0x7f69640cf340 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f695400d5f0 tx=0x7f695400dab0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 shutdown_connections 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f69440744b0 0x7f69440769c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69640d4a50 0x7f69641f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69640d0e00 0x7f69641f96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 --2- 172.21.15.67:0/1234032265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f696406c5c0 0x7f69640cf340 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 >> 172.21.15.67:0/1234032265 conn(0x7f69640d5d90 msgr2=0x7f69640d4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:32.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 shutdown_connections 2024-04-25T15:26:33.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:32.997+0000 7f6968a0c640 1 -- 172.21.15.67:0/1234032265 wait complete. 2024-04-25T15:26:33.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.122+0000 7fc761714640 1 Processor -- start 2024-04-25T15:26:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.122+0000 7fc761714640 1 -- start start 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc761714640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c164ad0 0x7fc75c0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc761714640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc75c163f30 0x7fc75c164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc761714640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75c164880 con 0x7fc75c0d1f30 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc761714640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc75c164ee0 con 0x7fc75c163f30 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc761714640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc75c0b31e0 con 0x7fc75c164ad0 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc75c163f30 0x7fc75c164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc75c0d1f30 0x7fc75c0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53800/0 (socket says 172.21.15.67:53800) 2024-04-25T15:26:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75b7fe640 1 -- 172.21.15.67:0/2455210047 learned_addr learned my addr 172.21.15.67:0/2455210047 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75affd640 1 --2- 172.21.15.67:0/2455210047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c164ad0 0x7fc75c0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75a7fc640 1 -- 172.21.15.67:0/2455210047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c164ad0 msgr2=0x7fc75c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75a7fc640 1 --2- 172.21.15.67:0/2455210047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c164ad0 0x7fc75c0b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75a7fc640 1 -- 172.21.15.67:0/2455210047 >> v1:172.21.15.67:6789/0 conn(0x7fc75c0d1f30 legacy=0x7fc75c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75a7fc640 1 -- 172.21.15.67:0/2455210047 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c0d3170 con 0x7fc75c163f30 2024-04-25T15:26:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc759ffb640 1 -- 172.21.15.67:0/2455210047 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834788119 0 0) 0x7fc74c001490 con 0x7fc75c0d1f30 2024-04-25T15:26:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.124+0000 7fc75affd640 1 --2- 172.21.15.67:0/2455210047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c164ad0 0x7fc75c0b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.125+0000 7fc75a7fc640 1 --2- 172.21.15.67:0/2455210047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c163f30 0x7fc75c164340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc7440097a0 tx=0x7fc74402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810bb870ccc731ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.125+0000 7fc759ffb640 1 -- 172.21.15.67:0/2455210047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744037070 con 0x7fc75c163f30 2024-04-25T15:26:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.125+0000 7fc759ffb640 1 -- 172.21.15.67:0/2455210047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc744032440 con 0x7fc75c163f30 2024-04-25T15:26:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.125+0000 7fc759ffb640 1 -- 172.21.15.67:0/2455210047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744032810 con 0x7fc75c163f30 2024-04-25T15:26:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 -- 172.21.15.67:0/2455210047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c163f30 msgr2=0x7fc75c164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 --2- 172.21.15.67:0/2455210047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c163f30 0x7fc75c164340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc7440097a0 tx=0x7fc74402a180 comp rx=0 tx=0).stop 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 -- 172.21.15.67:0/2455210047 shutdown_connections 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 --2- 172.21.15.67:0/2455210047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c163f30 0x7fc75c164340 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 --2- 172.21.15.67:0/2455210047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c164ad0 0x7fc75c0b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 -- 172.21.15.67:0/2455210047 >> 172.21.15.67:0/2455210047 conn(0x7fc75c0d3aa0 msgr2=0x7fc75c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 -- 172.21.15.67:0/2455210047 shutdown_connections 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.126+0000 7fc761714640 1 -- 172.21.15.67:0/2455210047 wait complete. 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.128+0000 7fc761714640 1 Processor -- start 2024-04-25T15:26:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.128+0000 7fc761714640 1 -- start start 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc761714640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c0d1f30 0x7fc75c156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc761714640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc75c163f30 0x7fc75c15a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc761714640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c164ad0 0x7fc75c15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc761714640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c06d9b0 con 0x7fc75c164ad0 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc761714640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c06d6d0 con 0x7fc75c163f30 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc761714640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc75c0b9430 con 0x7fc75c0d1f30 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c164ad0 0x7fc75c15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c164ad0 0x7fc75c15ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59340/0 (socket says 172.21.15.67:59340) 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 -- 172.21.15.67:0/2236482175 learned_addr learned my addr 172.21.15.67:0/2236482175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75affd640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c0d1f30 0x7fc75c156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75a7fc640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc75c163f30 0x7fc75c15a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 -- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c0d1f30 msgr2=0x7fc75c156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c0d1f30 0x7fc75c156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 -- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc75c163f30 msgr2=0x7fc75c15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc75c163f30 0x7fc75c15a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.129+0000 7fc75b7fe640 1 -- 172.21.15.67:0/2236482175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c156960 con 0x7fc75c164ad0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.130+0000 7fc75b7fe640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c164ad0 0x7fc75c15ad80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00c730 tx=0x7fc74c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.130+0000 7fc7537fe640 1 -- 172.21.15.67:0/2236482175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c018070 con 0x7fc75c164ad0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.130+0000 7fc7537fe640 1 -- 172.21.15.67:0/2236482175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc74c00ce40 con 0x7fc75c164ad0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.130+0000 7fc7537fe640 1 -- 172.21.15.67:0/2236482175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c00a880 con 0x7fc75c164ad0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.130+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75c15b600 con 0x7fc75c164ad0 2024-04-25T15:26:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.130+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75c15bbc0 con 0x7fc75c164ad0 2024-04-25T15:26:33.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.131+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc75c050740 con 0x7fc75c164ad0 2024-04-25T15:26:33.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.132+0000 7fc7537fe640 1 -- 172.21.15.67:0/2236482175 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc74c01b070 con 0x7fc75c164ad0 2024-04-25T15:26:33.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.132+0000 7fc7537fe640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc73c0744b0 0x7fc73c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.132+0000 7fc7537fe640 1 -- 172.21.15.67:0/2236482175 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc74c094690 con 0x7fc75c164ad0 2024-04-25T15:26:33.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.132+0000 7fc75affd640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc73c0744b0 0x7fc73c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.133+0000 7fc75affd640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc73c0744b0 0x7fc73c0769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc754006fd0 tx=0x7fc754008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.137+0000 7fc7537fe640 1 -- 172.21.15.67:0/2236482175 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc75c164880 con 0x7fc75c164ad0 2024-04-25T15:26:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.321+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc75c06eba0 con 0x7fc75c164ad0 2024-04-25T15:26:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.321+0000 7fc7537fe640 1 -- 172.21.15.67:0/2236482175 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) ==== 100+0+11 (secure 0 0 0) 0x7fc74c05e860 con 0x7fc75c164ad0 2024-04-25T15:26:33.323 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc73c0744b0 msgr2=0x7fc73c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc73c0744b0 0x7fc73c0769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc754006fd0 tx=0x7fc754008040 comp rx=0 tx=0).stop 2024-04-25T15:26:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c164ad0 msgr2=0x7fc75c15ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c164ad0 0x7fc75c15ad80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00c730 tx=0x7fc74c00cbf0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 shutdown_connections 2024-04-25T15:26:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc73c0744b0 0x7fc73c0769c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc75c164ad0 0x7fc75c15ad80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc75c163f30 0x7fc75c15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 --2- 172.21.15.67:0/2236482175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc75c0d1f30 0x7fc75c156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 >> 172.21.15.67:0/2236482175 conn(0x7fc75c0d3aa0 msgr2=0x7fc75c0d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 shutdown_connections 2024-04-25T15:26:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.323+0000 7fc761714640 1 -- 172.21.15.67:0/2236482175 wait complete. 2024-04-25T15:26:33.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:26:33.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.448+0000 7f8055dc5640 1 Processor -- start 2024-04-25T15:26:33.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.448+0000 7f8055dc5640 1 -- start start 2024-04-25T15:26:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.449+0000 7f8055dc5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f80500d4570 0x7f80500b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.449+0000 7f8055dc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 0x7f80500b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.449+0000 7f8055dc5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f805006d870 con 0x7f805006e8f0 2024-04-25T15:26:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.449+0000 7f8055dc5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f805006e460 con 0x7f80500d4570 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.449+0000 7f8055dc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f805006db10 con 0x7f80500b2dc0 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.449+0000 7f804effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f805006e8f0 0x7f80500d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53806/0 (socket says 172.21.15.67:53806) 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.449+0000 7f804effd640 1 -- 172.21.15.67:0/4072242815 learned_addr learned my addr 172.21.15.67:0/4072242815 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804f7fe640 1 --2- 172.21.15.67:0/4072242815 >> v2:172.21.15.176:3300/0 conn(0x7f80500d4570 0x7f80500b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804ffff640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 0x7f80500b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1072412591 0 0) 0x7f805006d870 con 0x7f805006e8f0 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8038003520 con 0x7f805006e8f0 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3835185748 0 0) 0x7f8038003520 con 0x7f805006e8f0 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 msgr2=0x7f80500b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 0x7f80500b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 >> v2:172.21.15.176:3300/0 conn(0x7f80500d4570 msgr2=0x7f80500b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 --2- 172.21.15.67:0/4072242815 >> v2:172.21.15.176:3300/0 conn(0x7f80500d4570 0x7f80500b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805006e0f0 con 0x7f805006e8f0 2024-04-25T15:26:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8040001690 con 0x7f805006e8f0 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 >> v1:172.21.15.67:6789/0 conn(0x7f805006e8f0 legacy=0x7f80500d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80380039b0 0x7f8038003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 0x7f8038004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8038004d40 0x7f8038005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f805006e460 con 0x7f80380039b0 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f805006db10 con 0x7f8038004d40 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f803800ba70 con 0x7f80500b2dc0 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804f7fe640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80380039b0 0x7f8038003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804effd640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 0x7f8038004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804ffff640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8038004d40 0x7f8038005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804f7fe640 1 -- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 msgr2=0x7f8038004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804f7fe640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80500b2dc0 0x7f8038004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804f7fe640 1 -- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8038004d40 msgr2=0x7f8038005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804f7fe640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8038004d40 0x7f8038005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.450+0000 7f804f7fe640 1 -- 172.21.15.67:0/4072242815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8038003520 con 0x7f80380039b0 2024-04-25T15:26:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.451+0000 7f804f7fe640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80380039b0 0x7f8038003e20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8044009ab0 tx=0x7f8038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b645da686ef763dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.451+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044036070 con 0x7f80380039b0 2024-04-25T15:26:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.451+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8044005990 con 0x7f80380039b0 2024-04-25T15:26:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.451+0000 7f804e7fc640 1 -- 172.21.15.67:0/4072242815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044009680 con 0x7f80380039b0 2024-04-25T15:26:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.452+0000 7f8055dc5640 1 -- 172.21.15.67:0/4072242815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80380039b0 msgr2=0x7f8038003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.452+0000 7f8055dc5640 1 --2- 172.21.15.67:0/4072242815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80380039b0 0x7f8038003e20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8044009ab0 tx=0x7f8038005830 comp rx=0 tx=0).stop 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.452+0000 7f804ffff640 1 -- 172.21.15.67:0/4072242815 reap_dead start 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.452+0000 7f8055dc5640 1 -- 172.21.15.67:0/4072242815 shutdown_connections 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.452+0000 7f8055dc5640 1 -- 172.21.15.67:0/4072242815 >> 172.21.15.67:0/4072242815 conn(0x7f80500d3a50 msgr2=0x7f80500d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.452+0000 7f8055dc5640 1 -- 172.21.15.67:0/4072242815 shutdown_connections 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.452+0000 7f8055dc5640 1 -- 172.21.15.67:0/4072242815 wait complete. 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.454+0000 7f8055dc5640 1 Processor -- start 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.454+0000 7f8055dc5640 1 -- start start 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f8055dc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80380039b0 0x7f805009f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f8055dc5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f805009e4a0 0x7f80500ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f8055dc5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80500c7b70 0x7f80500b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f8055dc5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f805006ed00 con 0x7f80500c7b70 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f8055dc5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f80500d0fb0 con 0x7f805009e4a0 2024-04-25T15:26:33.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f8055dc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f80500b31d0 con 0x7f80380039b0 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f804ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80500c7b70 0x7f80500b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f804ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80500c7b70 0x7f80500b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59352/0 (socket says 172.21.15.67:59352) 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.455+0000 7f804ffff640 1 -- 172.21.15.67:0/2040977607 learned_addr learned my addr 172.21.15.67:0/2040977607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804f7fe640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80380039b0 0x7f805009f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804effd640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f805009e4a0 0x7f80500ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804ffff640 1 -- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80380039b0 msgr2=0x7f805009f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804ffff640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80380039b0 0x7f805009f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804ffff640 1 -- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f805009e4a0 msgr2=0x7f80500ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804ffff640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f805009e4a0 0x7f80500ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804ffff640 1 -- 172.21.15.67:0/2040977607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8050154040 con 0x7f80500c7b70 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804f7fe640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80380039b0 0x7f805009f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f804ffff640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80500c7b70 0x7f80500b78f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8050074ab0 tx=0x7f804800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f8037fff640 1 -- 172.21.15.67:0/2040977607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804800fde0 con 0x7f80500c7b70 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f8037fff640 1 -- 172.21.15.67:0/2040977607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f804801d070 con 0x7f80500c7b70 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8050154320 con 0x7f80500c7b70 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f8037fff640 1 -- 172.21.15.67:0/2040977607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048015670 con 0x7f80500c7b70 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.456+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80501548e0 con 0x7f80500c7b70 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.457+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8050153bc0 con 0x7f80500c7b70 2024-04-25T15:26:33.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.458+0000 7f8037fff640 1 -- 172.21.15.67:0/2040977607 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8048015810 con 0x7f80500c7b70 2024-04-25T15:26:33.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.458+0000 7f8037fff640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f803c0744b0 0x7f803c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.458+0000 7f8037fff640 1 -- 172.21.15.67:0/2040977607 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f8048095b40 con 0x7f80500c7b70 2024-04-25T15:26:33.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.459+0000 7f804f7fe640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f803c0744b0 0x7f803c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.459+0000 7f804f7fe640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f803c0744b0 0x7f803c0769c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8044033000 tx=0x7f8044008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.463+0000 7f8037fff640 1 -- 172.21.15.67:0/2040977607 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8050153bc0 con 0x7f80500c7b70 2024-04-25T15:26:33.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.668+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8050050740 con 0x7f80500c7b70 2024-04-25T15:26:33.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.668+0000 7f8037fff640 1 -- 172.21.15.67:0/2040977607 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7f8048010040 con 0x7f80500c7b70 2024-04-25T15:26:33.670 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f803c0744b0 msgr2=0x7f803c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f803c0744b0 0x7f803c0769c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8044033000 tx=0x7f8044008030 comp rx=0 tx=0).stop 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80500c7b70 msgr2=0x7f80500b78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80500c7b70 0x7f80500b78f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8050074ab0 tx=0x7f804800f580 comp rx=0 tx=0).stop 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 shutdown_connections 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f803c0744b0 0x7f803c0769c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80500c7b70 0x7f80500b78f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f805009e4a0 0x7f80500ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 --2- 172.21.15.67:0/2040977607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80380039b0 0x7f805009f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 >> 172.21.15.67:0/2040977607 conn(0x7f80500d3a50 msgr2=0x7f80500b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:33.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 shutdown_connections 2024-04-25T15:26:33.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.671+0000 7f8055dc5640 1 -- 172.21.15.67:0/2040977607 wait complete. 2024-04-25T15:26:33.684 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:26:33.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:33.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:33.716 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:26:33.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:33.716 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:26:33.772 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:26:33.773 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:33.773 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:33.789 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:26:33.789 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:26:33.789 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:33.789 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:33.829 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:26:33.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.939+0000 7f4d943e3640 1 Processor -- start 2024-04-25T15:26:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.939+0000 7f4d943e3640 1 -- start start 2024-04-25T15:26:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d943e3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d8c0cfd40 0x7f4d8c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d943e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c0b7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d943e3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8c0b8190 con 0x7f4d8c0b7d80 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d943e3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4d8c0d27e0 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d943e3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8c0d1c10 con 0x7f4d8c0b3e00 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9217f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d8c0cfd40 0x7f4d8c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9217f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d8c0cfd40 0x7f4d8c0b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59354/0 (socket says 172.21.15.67:59354) 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9217f640 1 -- 172.21.15.67:0/2932649189 learned_addr learned my addr 172.21.15.67:0/2932649189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9197e640 1 --2- 172.21.15.67:0/2932649189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c0b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9217f640 1 -- 172.21.15.67:0/2932649189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 msgr2=0x7f4d8c0b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9217f640 1 --2- 172.21.15.67:0/2932649189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c0b7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9217f640 1 -- 172.21.15.67:0/2932649189 >> v1:172.21.15.176:6789/0 conn(0x7f4d8c0b7d80 legacy=0x7f4d8c0d2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9217f640 1 -- 172.21.15.67:0/2932649189 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8c0d1eb0 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.941+0000 7f4d9197e640 1 --2- 172.21.15.67:0/2932649189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c0b7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.942+0000 7f4d9217f640 1 --2- 172.21.15.67:0/2932649189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c0b21a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009870 tx=0x7f4d8402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff17f6ff005dada server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.942+0000 7f4d9117d640 1 -- 172.21.15.67:0/2932649189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84037070 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.942+0000 7f4d9117d640 1 -- 172.21.15.67:0/2932649189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d84032440 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.942+0000 7f4d9117d640 1 -- 172.21.15.67:0/2932649189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84032810 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 -- 172.21.15.67:0/2932649189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 msgr2=0x7f4d8c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 --2- 172.21.15.67:0/2932649189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c0b21a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009870 tx=0x7f4d8402a210 comp rx=0 tx=0).stop 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 -- 172.21.15.67:0/2932649189 shutdown_connections 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 --2- 172.21.15.67:0/2932649189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c0b7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 --2- 172.21.15.67:0/2932649189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c0b21a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 -- 172.21.15.67:0/2932649189 >> 172.21.15.67:0/2932649189 conn(0x7f4d8c09b7d0 msgr2=0x7f4d8c09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 -- 172.21.15.67:0/2932649189 shutdown_connections 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.943+0000 7f4d943e3640 1 -- 172.21.15.67:0/2932649189 wait complete. 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.945+0000 7f4d943e3640 1 Processor -- start 2024-04-25T15:26:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.945+0000 7f4d943e3640 1 -- start start 2024-04-25T15:26:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d943e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c14bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d943e3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d8c0b7d80 0x7f4d8c15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d943e3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c06ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d943e3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c0b4210 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d943e3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c0d2470 con 0x7f4d8c0b7d80 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d943e3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8c0b4390 con 0x7f4d8c0b3e00 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d92980640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c06ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d92980640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c06ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59360/0 (socket says 172.21.15.67:59360) 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.946+0000 7f4d92980640 1 -- 172.21.15.67:0/566564608 learned_addr learned my addr 172.21.15.67:0/566564608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d9217f640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c14bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d9197e640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d8c0b7d80 0x7f4d8c15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d92980640 1 -- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 msgr2=0x7f4d8c14bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d92980640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c14bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d92980640 1 -- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d8c0b7d80 msgr2=0x7f4d8c15aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d92980640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d8c0b7d80 0x7f4d8c15aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d92980640 1 -- 172.21.15.67:0/566564608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8c14d5d0 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d9217f640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c14bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d92980640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c06ce50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00b9a0 tx=0x7f4d7c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d8affd640 1 -- 172.21.15.67:0/566564608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c019070 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d8affd640 1 -- 172.21.15.67:0/566564608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d7c00cba0 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8c15d9b0 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d8affd640 1 -- 172.21.15.67:0/566564608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c014440 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.947+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8c14caf0 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.948+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d8c050740 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.949+0000 7f4d8affd640 1 -- 172.21.15.67:0/566564608 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4d7c01c070 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.949+0000 7f4d8affd640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d780744e0 0x7f4d780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.949+0000 7f4d8affd640 1 -- 172.21.15.67:0/566564608 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f4d7c094e30 con 0x7f4d8c0cfd40 2024-04-25T15:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.950+0000 7f4d9217f640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d780744e0 0x7f4d780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:33.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.950+0000 7f4d9217f640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d780744e0 0x7f4d780769f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4d84034000 tx=0x7f4d8402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:33.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:33.954+0000 7f4d8affd640 1 -- 172.21.15.67:0/566564608 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d7c05f5c0 con 0x7f4d8c0cfd40 2024-04-25T15:26:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.177+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d8c15b300 con 0x7f4d8c0cfd40 2024-04-25T15:26:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.178+0000 7f4d8affd640 1 -- 172.21.15.67:0/566564608 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7f4d7c05eec0 con 0x7f4d8c0cfd40 2024-04-25T15:26:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:34.180 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/4217070487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4217070487},{"type":"v1","addr":"172.21.15.176:6835","nonce":4217070487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4207485404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4207485404},{"type":"v1","addr":"172.21.15.176:6837","nonce":4207485404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5067,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2355578883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2355578883},{"type":"v1","addr":"172.21.15.67:6834","nonce":2355578883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:20.974268+0000","modified":"2024-04-25T15:26:23.988098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2490033101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2490033101},{"type":"v1","addr":"172.21.15.67:6835","nonce":2490033101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-04-25T15:26:34.180 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-04-25T15:26:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d780744e0 msgr2=0x7f4d780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d780744e0 0x7f4d780769f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4d84034000 tx=0x7f4d8402e000 comp rx=0 tx=0).stop 2024-04-25T15:26:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 msgr2=0x7f4d8c06ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c06ce50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c00b9a0 tx=0x7f4d7c00be60 comp rx=0 tx=0).stop 2024-04-25T15:26:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 shutdown_connections 2024-04-25T15:26:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4d780744e0 0x7f4d780769f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d8c0cfd40 0x7f4d8c06ce50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d8c0b7d80 0x7f4d8c15aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 --2- 172.21.15.67:0/566564608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d8c0b3e00 0x7f4d8c14bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 >> 172.21.15.67:0/566564608 conn(0x7f4d8c09b7d0 msgr2=0x7f4d8c071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 shutdown_connections 2024-04-25T15:26:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.180+0000 7f4d943e3640 1 -- 172.21.15.67:0/566564608 wait complete. 2024-04-25T15:26:34.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:34.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.305+0000 7f4037f6c640 1 Processor -- start 2024-04-25T15:26:34.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.305+0000 7f4037f6c640 1 -- start start 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4037f6c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40300d0e00 0x7f40300d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4037f6c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f40300d4a50 0x7f40300d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4037f6c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40300d1850 con 0x7f403006c5c0 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4037f6c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4030160de0 con 0x7f40300d0e00 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4037f6c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f40301610c0 con 0x7f40300d4a50 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4035d08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40300d0e00 0x7f40300d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4035507640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f403006c5c0 0x7f403006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53822/0 (socket says 172.21.15.67:53822) 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4035d08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40300d0e00 0x7f40300d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59366/0 (socket says 172.21.15.67:59366) 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.306+0000 7f4035507640 1 -- 172.21.15.67:0/3367071573 learned_addr learned my addr 172.21.15.67:0/3367071573 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.307+0000 7f4035d08640 1 -- 172.21.15.67:0/3367071573 >> v2:172.21.15.176:3300/0 conn(0x7f40300d4a50 msgr2=0x7f40300d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.307+0000 7f4036509640 1 --2- 172.21.15.67:0/3367071573 >> v2:172.21.15.176:3300/0 conn(0x7f40300d4a50 0x7f40300d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.307+0000 7f4035d08640 1 --2- 172.21.15.67:0/3367071573 >> v2:172.21.15.176:3300/0 conn(0x7f40300d4a50 0x7f40300d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.307+0000 7f4035d08640 1 -- 172.21.15.67:0/3367071573 >> v1:172.21.15.67:6789/0 conn(0x7f403006c5c0 legacy=0x7f403006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.307+0000 7f4034d06640 1 -- 172.21.15.67:0/3367071573 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 40176443 0 0) 0x7f40300d1850 con 0x7f403006c5c0 2024-04-25T15:26:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.307+0000 7f4035d08640 1 -- 172.21.15.67:0/3367071573 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40301613a0 con 0x7f40300d0e00 2024-04-25T15:26:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.307+0000 7f4036509640 1 --2- 172.21.15.67:0/3367071573 >> v2:172.21.15.176:3300/0 conn(0x7f40300d4a50 0x7f40300d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4035d08640 1 --2- 172.21.15.67:0/3367071573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40300d0e00 0x7f40300d1210 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f40300d1750 tx=0x7f402c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278966f2bd668573 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4034d06640 1 -- 172.21.15.67:0/3367071573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c037070 con 0x7f40300d0e00 2024-04-25T15:26:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4034d06640 1 -- 172.21.15.67:0/3367071573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c032440 con 0x7f40300d0e00 2024-04-25T15:26:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4034d06640 1 -- 172.21.15.67:0/3367071573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c02ae60 con 0x7f40300d0e00 2024-04-25T15:26:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4037f6c640 1 -- 172.21.15.67:0/3367071573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40300d0e00 msgr2=0x7f40300d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3367071573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40300d0e00 0x7f40300d1210 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f40300d1750 tx=0x7f402c02a110 comp rx=0 tx=0).stop 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4037f6c640 1 -- 172.21.15.67:0/3367071573 shutdown_connections 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3367071573 >> v2:172.21.15.176:3300/0 conn(0x7f40300d4a50 0x7f40300d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3367071573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40300d0e00 0x7f40300d1210 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.308+0000 7f4037f6c640 1 -- 172.21.15.67:0/3367071573 >> 172.21.15.67:0/3367071573 conn(0x7f40300d5d90 msgr2=0x7f40300d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.309+0000 7f4037f6c640 1 -- 172.21.15.67:0/3367071573 shutdown_connections 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.309+0000 7f4037f6c640 1 -- 172.21.15.67:0/3367071573 wait complete. 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.310+0000 7f4037f6c640 1 Processor -- start 2024-04-25T15:26:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.310+0000 7f4037f6c640 1 -- start start 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4037f6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403006c5c0 0x7f40300ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4037f6c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40300d0e00 0x7f40300cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4037f6c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40300d4a50 0x7f4030160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4037f6c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4030164e00 con 0x7f403006c5c0 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4037f6c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f40300b5320 con 0x7f40300d4a50 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4037f6c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4030164f80 con 0x7f40300d0e00 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403006c5c0 0x7f40300ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403006c5c0 0x7f40300ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59378/0 (socket says 172.21.15.67:59378) 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 -- 172.21.15.67:0/3609312687 learned_addr learned my addr 172.21.15.67:0/3609312687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 -- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40300d0e00 msgr2=0x7f40300cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035507640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40300d0e00 0x7f40300cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4036509640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40300d4a50 0x7f4030160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40300d0e00 0x7f40300cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 -- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40300d4a50 msgr2=0x7f4030160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40300d4a50 0x7f4030160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 -- 172.21.15.67:0/3609312687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40300cc0d0 con 0x7f403006c5c0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035507640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40300d0e00 0x7f40300cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4036509640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40300d4a50 0x7f4030160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f4035d08640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403006c5c0 0x7f40300ca8e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f402c035040 tx=0x7f402c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f40267fc640 1 -- 172.21.15.67:0/3609312687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c037050 con 0x7f403006c5c0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f40267fc640 1 -- 172.21.15.67:0/3609312687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c038040 con 0x7f403006c5c0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.312+0000 7f40267fc640 1 -- 172.21.15.67:0/3609312687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c004450 con 0x7f403006c5c0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.313+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40300cc3b0 con 0x7f403006c5c0 2024-04-25T15:26:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.313+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40300cc970 con 0x7f403006c5c0 2024-04-25T15:26:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.314+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4030050740 con 0x7f403006c5c0 2024-04-25T15:26:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.314+0000 7f40267fc640 1 -- 172.21.15.67:0/3609312687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f402c030980 con 0x7f403006c5c0 2024-04-25T15:26:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.315+0000 7f40267fc640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40140744e0 0x7f40140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.315+0000 7f40267fc640 1 -- 172.21.15.67:0/3609312687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f402c0b5c00 con 0x7f403006c5c0 2024-04-25T15:26:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.315+0000 7f4035507640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40140744e0 0x7f40140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.316+0000 7f4035507640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40140744e0 0x7f40140769f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4030068930 tx=0x7f4020009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.320+0000 7f40267fc640 1 -- 172.21.15.67:0/3609312687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f402c0bc080 con 0x7f403006c5c0 2024-04-25T15:26:34.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.508+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40300d4320 con 0x7f403006c5c0 2024-04-25T15:26:34.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.509+0000 7f40267fc640 1 -- 172.21.15.67:0/3609312687 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16633 (secure 0 0 0) 0x7f402c080390 con 0x7f403006c5c0 2024-04-25T15:26:34.511 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:34.511 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:22.596544+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40140744e0 msgr2=0x7f40140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40140744e0 0x7f40140769f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4030068930 tx=0x7f4020009040 comp rx=0 tx=0).stop 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403006c5c0 msgr2=0x7f40300ca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403006c5c0 0x7f40300ca8e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f402c035040 tx=0x7f402c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 shutdown_connections 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f40140744e0 0x7f40140769f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40300d4a50 0x7f4030160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40300d0e00 0x7f40300cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 --2- 172.21.15.67:0/3609312687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f403006c5c0 0x7f40300ca8e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 >> 172.21.15.67:0/3609312687 conn(0x7f40300d5d90 msgr2=0x7f40300d4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 shutdown_connections 2024-04-25T15:26:34.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.511+0000 7f4037f6c640 1 -- 172.21.15.67:0/3609312687 wait complete. 2024-04-25T15:26:34.524 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:26:18.702421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:26:19.919128+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:34.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.638+0000 7f9bd6070640 1 Processor -- start 2024-04-25T15:26:34.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.638+0000 7f9bd6070640 1 -- start start 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bd6070640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd006ccd0 0x7f9bd00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bd6070640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bd00d4a50 0x7f9bd00d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bd6070640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd00d1850 con 0x7f9bd00d0e00 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bd6070640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd0160de0 con 0x7f9bd00d4a50 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bd6070640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd01610c0 con 0x7f9bd006ccd0 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bcffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bd00d4a50 0x7f9bd00d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bcf7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9bd00d0e00 0x7f9bd006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53826/0 (socket says 172.21.15.67:53826) 2024-04-25T15:26:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bcf7fe640 1 -- 172.21.15.67:0/1636616147 learned_addr learned my addr 172.21.15.67:0/1636616147 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:34.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bceffd640 1 --2- 172.21.15.67:0/1636616147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd006ccd0 0x7f9bd00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bcffff640 1 -- 172.21.15.67:0/1636616147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd006ccd0 msgr2=0x7f9bd00d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:34.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bcffff640 1 --2- 172.21.15.67:0/1636616147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd006ccd0 0x7f9bd00d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bcffff640 1 -- 172.21.15.67:0/1636616147 >> v1:172.21.15.67:6789/0 conn(0x7f9bd00d0e00 legacy=0x7f9bd006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bce7fc640 1 -- 172.21.15.67:0/1636616147 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3216560959 0 0) 0x7f9bd00d1850 con 0x7f9bd00d0e00 2024-04-25T15:26:34.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.640+0000 7f9bcffff640 1 -- 172.21.15.67:0/1636616147 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd01613a0 con 0x7f9bd00d4a50 2024-04-25T15:26:34.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.641+0000 7f9bcffff640 1 --2- 172.21.15.67:0/1636616147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd00d4a50 0x7f9bd00d4e60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009d90 tx=0x7f9bc802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc7c1b09fdeb009 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.641+0000 7f9bce7fc640 1 -- 172.21.15.67:0/1636616147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8007740 con 0x7f9bd00d4a50 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.641+0000 7f9bce7fc640 1 -- 172.21.15.67:0/1636616147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc80078e0 con 0x7f9bd00d4a50 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.641+0000 7f9bce7fc640 1 -- 172.21.15.67:0/1636616147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8007cb0 con 0x7f9bd00d4a50 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 -- 172.21.15.67:0/1636616147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd00d4a50 msgr2=0x7f9bd00d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 --2- 172.21.15.67:0/1636616147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd00d4a50 0x7f9bd00d4e60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009d90 tx=0x7f9bc802a660 comp rx=0 tx=0).stop 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 -- 172.21.15.67:0/1636616147 shutdown_connections 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 --2- 172.21.15.67:0/1636616147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd00d4a50 0x7f9bd00d4e60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 --2- 172.21.15.67:0/1636616147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd006ccd0 0x7f9bd00d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 -- 172.21.15.67:0/1636616147 >> 172.21.15.67:0/1636616147 conn(0x7f9bd00d5d90 msgr2=0x7f9bd00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 -- 172.21.15.67:0/1636616147 shutdown_connections 2024-04-25T15:26:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.642+0000 7f9bd6070640 1 -- 172.21.15.67:0/1636616147 wait complete. 2024-04-25T15:26:34.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.643+0000 7f9bd6070640 1 Processor -- start 2024-04-25T15:26:34.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.644+0000 7f9bd6070640 1 -- start start 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bd6070640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bd00d0e00 0x7f9bd0148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bd6070640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd00d4a50 0x7f9bd0148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bd6070640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd014e9d0 0x7f9bd014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bd6070640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0164e00 con 0x7f9bd014e9d0 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bd6070640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd00b5320 con 0x7f9bd00d0e00 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bd6070640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd0164f80 con 0x7f9bd00d4a50 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd014e9d0 0x7f9bd014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd014e9d0 0x7f9bd014ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59406/0 (socket says 172.21.15.67:59406) 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 -- 172.21.15.67:0/516812724 learned_addr learned my addr 172.21.15.67:0/516812724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bceffd640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd00d4a50 0x7f9bd0148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcf7fe640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bd00d0e00 0x7f9bd0148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 -- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd00d4a50 msgr2=0x7f9bd0148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd00d4a50 0x7f9bd0148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 -- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bd00d0e00 msgr2=0x7f9bd0148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bd00d0e00 0x7f9bd0148870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bcffff640 1 -- 172.21.15.67:0/516812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd01494c0 con 0x7f9bd014e9d0 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.645+0000 7f9bceffd640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd00d4a50 0x7f9bd0148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.646+0000 7f9bcffff640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd014e9d0 0x7f9bd014ede0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9bc802ac50 tx=0x7f9bc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.646+0000 7f9bb7fff640 1 -- 172.21.15.67:0/516812724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc803f070 con 0x7f9bd014e9d0 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.646+0000 7f9bb7fff640 1 -- 172.21.15.67:0/516812724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc8031bc0 con 0x7f9bd014e9d0 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.646+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd00b5f50 con 0x7f9bd014e9d0 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.646+0000 7f9bb7fff640 1 -- 172.21.15.67:0/516812724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8044460 con 0x7f9bd014e9d0 2024-04-25T15:26:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.646+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd00ca1c0 con 0x7f9bd014e9d0 2024-04-25T15:26:34.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.647+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd00c9f30 con 0x7f9bd014e9d0 2024-04-25T15:26:34.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.650+0000 7f9bb7fff640 1 -- 172.21.15.67:0/516812724 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9bc8007790 con 0x7f9bd014e9d0 2024-04-25T15:26:34.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.650+0000 7f9bb7fff640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bbc0744b0 0x7f9bbc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.651+0000 7f9bb7fff640 1 -- 172.21.15.67:0/516812724 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f9bc80b6be0 con 0x7f9bd014e9d0 2024-04-25T15:26:34.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.651+0000 7f9bcf7fe640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bbc0744b0 0x7f9bbc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.651+0000 7f9bcf7fe640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bbc0744b0 0x7f9bbc0769c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0000f60 tx=0x7f9bc0007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.654+0000 7f9bb7fff640 1 -- 172.21.15.67:0/516812724 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9bc8081570 con 0x7f9bd014e9d0 2024-04-25T15:26:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.842+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9bd0050740 con 0x7f9bd014e9d0 2024-04-25T15:26:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.843+0000 7f9bb7fff640 1 -- 172.21.15.67:0/516812724 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) ==== 93+0+10 (secure 0 0 0) 0x7f9bc8080e70 con 0x7f9bd014e9d0 2024-04-25T15:26:34.845 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bbc0744b0 msgr2=0x7f9bbc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bbc0744b0 0x7f9bbc0769c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0000f60 tx=0x7f9bc0007c00 comp rx=0 tx=0).stop 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd014e9d0 msgr2=0x7f9bd014ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd014e9d0 0x7f9bd014ede0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9bc802ac50 tx=0x7f9bc8004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 shutdown_connections 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9bbc0744b0 0x7f9bbc0769c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bd014e9d0 0x7f9bd014ede0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9bd00d4a50 0x7f9bd0148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 --2- 172.21.15.67:0/516812724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9bd00d0e00 0x7f9bd0148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 >> 172.21.15.67:0/516812724 conn(0x7f9bd00d5d90 msgr2=0x7f9bd0163900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 shutdown_connections 2024-04-25T15:26:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.845+0000 7f9bd6070640 1 -- 172.21.15.67:0/516812724 wait complete. 2024-04-25T15:26:34.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.971+0000 7f44d95b2640 1 Processor -- start 2024-04-25T15:26:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.971+0000 7f44d95b2640 1 -- start start 2024-04-25T15:26:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.972+0000 7f44d95b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44d40d1210 0x7f44d40d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.972+0000 7f44d95b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44d40d1850 con 0x7f44d40d0e00 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.972+0000 7f44d95b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44d4160de0 con 0x7f44d406ccd0 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.972+0000 7f44d95b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f44d40d1620 con 0x7f44d40d1210 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.972+0000 7f44d37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44d40d1210 0x7f44d40d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.972+0000 7f44d2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f44d40d0e00 0x7f44d406c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53830/0 (socket says 172.21.15.67:53830) 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.972+0000 7f44d2ffd640 1 -- 172.21.15.67:0/2194544553 learned_addr learned my addr 172.21.15.67:0/2194544553 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.973+0000 7f44d37fe640 1 -- 172.21.15.67:0/2194544553 >> v1:172.21.15.176:6789/0 conn(0x7f44d406ccd0 legacy=0x7f44d40d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.973+0000 7f44d37fe640 1 -- 172.21.15.67:0/2194544553 >> v1:172.21.15.67:6789/0 conn(0x7f44d40d0e00 legacy=0x7f44d406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.973+0000 7f44d27fc640 1 -- 172.21.15.67:0/2194544553 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967084646 0 0) 0x7f44d40d1850 con 0x7f44d40d0e00 2024-04-25T15:26:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.973+0000 7f44d37fe640 1 -- 172.21.15.67:0/2194544553 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d41610c0 con 0x7f44d40d1210 2024-04-25T15:26:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.974+0000 7f44d37fe640 1 --2- 172.21.15.67:0/2194544553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d1210 0x7f44d40d5160 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009d90 tx=0x7f44cc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b567c7428d7f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.974+0000 7f44d27fc640 1 -- 172.21.15.67:0/2194544553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc031600 con 0x7f44d40d1210 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.974+0000 7f44d27fc640 1 -- 172.21.15.67:0/2194544553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44cc0317a0 con 0x7f44d40d1210 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.974+0000 7f44d27fc640 1 -- 172.21.15.67:0/2194544553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc031b70 con 0x7f44d40d1210 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.974+0000 7f44d95b2640 1 -- 172.21.15.67:0/2194544553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d1210 msgr2=0x7f44d40d5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.974+0000 7f44d95b2640 1 --2- 172.21.15.67:0/2194544553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d1210 0x7f44d40d5160 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009d90 tx=0x7f44cc02a610 comp rx=0 tx=0).stop 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.975+0000 7f44d95b2640 1 -- 172.21.15.67:0/2194544553 shutdown_connections 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.975+0000 7f44d95b2640 1 --2- 172.21.15.67:0/2194544553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d1210 0x7f44d40d5160 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.975+0000 7f44d95b2640 1 -- 172.21.15.67:0/2194544553 >> 172.21.15.67:0/2194544553 conn(0x7f44d40d5d90 msgr2=0x7f44d40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.975+0000 7f44d95b2640 1 -- 172.21.15.67:0/2194544553 shutdown_connections 2024-04-25T15:26:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.975+0000 7f44d95b2640 1 -- 172.21.15.67:0/2194544553 wait complete. 2024-04-25T15:26:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.976+0000 7f44d95b2640 1 Processor -- start 2024-04-25T15:26:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.976+0000 7f44d95b2640 1 -- start start 2024-04-25T15:26:34.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d95b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d0e00 0x7f44d40b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d95b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44d40d1210 0x7f44d415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d95b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44d415afd0 0x7f44d415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d95b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f44d40b5320 con 0x7f44d40d0e00 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d95b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f44d4160ad0 con 0x7f44d415afd0 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d95b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44d4164e00 con 0x7f44d40d1210 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d0e00 0x7f44d40b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d0e00 0x7f44d40b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59418/0 (socket says 172.21.15.67:59418) 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 -- 172.21.15.67:0/1566077435 learned_addr learned my addr 172.21.15.67:0/1566077435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d37fe640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44d415afd0 0x7f44d415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44cbfff640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44d40d1210 0x7f44d415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 -- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44d40d1210 msgr2=0x7f44d415bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44d40d1210 0x7f44d415bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 -- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44d415afd0 msgr2=0x7f44d415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44d415afd0 0x7f44d415b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 -- 172.21.15.67:0/1566077435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d415b920 con 0x7f44d40d0e00 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44cbfff640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44d40d1210 0x7f44d415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d37fe640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44d415afd0 0x7f44d415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44d2ffd640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d0e00 0x7f44d40b7120 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f44c400c430 tx=0x7f44c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.978+0000 7f44cb7fe640 1 -- 172.21.15.67:0/1566077435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c40074e0 con 0x7f44d40d0e00 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.979+0000 7f44cb7fe640 1 -- 172.21.15.67:0/1566077435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c40047e0 con 0x7f44d40d0e00 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.979+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d415c140 con 0x7f44d40d0e00 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.979+0000 7f44cb7fe640 1 -- 172.21.15.67:0/1566077435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4004bd0 con 0x7f44d40d0e00 2024-04-25T15:26:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.979+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d40cbc90 con 0x7f44d40d0e00 2024-04-25T15:26:34.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.979+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d4050740 con 0x7f44d40d0e00 2024-04-25T15:26:34.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.980+0000 7f44cb7fe640 1 -- 172.21.15.67:0/1566077435 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f44c4011070 con 0x7f44d40d0e00 2024-04-25T15:26:34.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.981+0000 7f44cb7fe640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44b40744b0 0x7f44b40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:34.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.981+0000 7f44cb7fe640 1 -- 172.21.15.67:0/1566077435 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f44c4094e40 con 0x7f44d40d0e00 2024-04-25T15:26:34.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.981+0000 7f44cbfff640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44b40744b0 0x7f44b40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:34.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.981+0000 7f44cbfff640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44b40744b0 0x7f44b40769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f44bc006fd0 tx=0x7f44bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:34.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:34.985+0000 7f44cb7fe640 1 -- 172.21.15.67:0/1566077435 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f44d415b920 con 0x7f44d40d0e00 2024-04-25T15:26:35.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.170+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f44d40d4320 con 0x7f44d40d0e00 2024-04-25T15:26:35.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.171+0000 7f44cb7fe640 1 -- 172.21.15.67:0/1566077435 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) ==== 104+0+11 (secure 0 0 0) 0x7f44c405f660 con 0x7f44d40d0e00 2024-04-25T15:26:35.173 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44b40744b0 msgr2=0x7f44b40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44b40744b0 0x7f44b40769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f44bc006fd0 tx=0x7f44bc008040 comp rx=0 tx=0).stop 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d0e00 msgr2=0x7f44d40b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d0e00 0x7f44d40b7120 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f44c400c430 tx=0x7f44c400c8f0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 shutdown_connections 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44b40744b0 0x7f44b40769c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44d415afd0 0x7f44d415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44d40d1210 0x7f44d415bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 --2- 172.21.15.67:0/1566077435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44d40d0e00 0x7f44d40b7120 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 >> 172.21.15.67:0/1566077435 conn(0x7f44d40d5d90 msgr2=0x7f44d40684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 shutdown_connections 2024-04-25T15:26:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.173+0000 7f44d95b2640 1 -- 172.21.15.67:0/1566077435 wait complete. 2024-04-25T15:26:35.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.299+0000 7f4802a42640 1 Processor -- start 2024-04-25T15:26:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.299+0000 7f4802a42640 1 -- start start 2024-04-25T15:26:35.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4802a42640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47fc169200 0x7f47fc16b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4802a42640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47fc0d0a00 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4802a42640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47fc0d0ce0 con 0x7f47fc0d0220 2024-04-25T15:26:35.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4802a42640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f47fc160c20 con 0x7f47fc169200 2024-04-25T15:26:35.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4800fdf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47fc169200 0x7f47fc16b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f47fb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f47fc0ccbe0 0x7f47fc0ccff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53838/0 (socket says 172.21.15.67:53838) 2024-04-25T15:26:35.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f47fb7fe640 1 -- 172.21.15.67:0/2422223067 learned_addr learned my addr 172.21.15.67:0/2422223067 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4800fdf640 1 -- 172.21.15.67:0/2422223067 >> v1:172.21.15.176:6789/0 conn(0x7f47fc0d0220 legacy=0x7f47fc160510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4800fdf640 1 -- 172.21.15.67:0/2422223067 >> v1:172.21.15.67:6789/0 conn(0x7f47fc0ccbe0 legacy=0x7f47fc0ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f4800fdf640 1 -- 172.21.15.67:0/2422223067 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc0d0fc0 con 0x7f47fc169200 2024-04-25T15:26:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.300+0000 7f47faffd640 1 -- 172.21.15.67:0/2422223067 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728982601 0 0) 0x7f47fc0d0a00 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.301+0000 7f4800fdf640 1 --2- 172.21.15.67:0/2422223067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc169200 0x7f47fc16b620 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009d90 tx=0x7f47f402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e458dd357ae9a57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.301+0000 7f47faffd640 1 -- 172.21.15.67:0/2422223067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4007740 con 0x7f47fc169200 2024-04-25T15:26:35.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.301+0000 7f47faffd640 1 -- 172.21.15.67:0/2422223067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f40078e0 con 0x7f47fc169200 2024-04-25T15:26:35.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.301+0000 7f47faffd640 1 -- 172.21.15.67:0/2422223067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4007cb0 con 0x7f47fc169200 2024-04-25T15:26:35.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.302+0000 7f4802a42640 1 -- 172.21.15.67:0/2422223067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc169200 msgr2=0x7f47fc16b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.302+0000 7f4802a42640 1 --2- 172.21.15.67:0/2422223067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc169200 0x7f47fc16b620 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009d90 tx=0x7f47f402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.302+0000 7f4802a42640 1 -- 172.21.15.67:0/2422223067 shutdown_connections 2024-04-25T15:26:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.302+0000 7f4802a42640 1 --2- 172.21.15.67:0/2422223067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc169200 0x7f47fc16b620 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.302+0000 7f4802a42640 1 -- 172.21.15.67:0/2422223067 >> 172.21.15.67:0/2422223067 conn(0x7f47fc0ce790 msgr2=0x7f47fc0ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.302+0000 7f4802a42640 1 -- 172.21.15.67:0/2422223067 shutdown_connections 2024-04-25T15:26:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.302+0000 7f4802a42640 1 -- 172.21.15.67:0/2422223067 wait complete. 2024-04-25T15:26:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.304+0000 7f4802a42640 1 Processor -- start 2024-04-25T15:26:35.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.304+0000 7f4802a42640 1 -- start start 2024-04-25T15:26:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f4802a42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0ccbe0 0x7f47fc15de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f4802a42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47fc0d0220 0x7f47fc0b6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f4802a42640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47fc169200 0x7f47fc0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f4802a42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc06d5d0 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f4802a42640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc0d1580 con 0x7f47fc169200 2024-04-25T15:26:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f4802a42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc0d1aa0 con 0x7f47fc0d0220 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f47fbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0ccbe0 0x7f47fc15de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f47fbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0ccbe0 0x7f47fc15de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59438/0 (socket says 172.21.15.67:59438) 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.305+0000 7f47fbfff640 1 -- 172.21.15.67:0/3033800272 learned_addr learned my addr 172.21.15.67:0/3033800272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f4800fdf640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47fc169200 0x7f47fc0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47fb7fe640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47fc0d0220 0x7f47fc0b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47fbfff640 1 -- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47fc0d0220 msgr2=0x7f47fc0b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47fbfff640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47fc0d0220 0x7f47fc0b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47fbfff640 1 -- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47fc169200 msgr2=0x7f47fc0b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47fbfff640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47fc169200 0x7f47fc0b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47fbfff640 1 -- 172.21.15.67:0/3033800272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc15e340 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47fbfff640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0ccbe0 0x7f47fc15de00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00b420 tx=0x7f47ec00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47f8ff9640 1 -- 172.21.15.67:0/3033800272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec011830 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47f8ff9640 1 -- 172.21.15.67:0/3033800272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47ec016070 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47fc0b75c0 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f47f8ff9640 1 -- 172.21.15.67:0/3033800272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec011c10 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.306+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47fc0a0360 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.307+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47fc050740 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.308+0000 7f47f8ff9640 1 -- 172.21.15.67:0/3033800272 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f47ec01b460 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.308+0000 7f47f8ff9640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47dc0744e0 0x7f47dc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.309+0000 7f47f8ff9640 1 -- 172.21.15.67:0/3033800272 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f47ec094d10 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.309+0000 7f47fb7fe640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47dc0744e0 0x7f47dc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.309+0000 7f47fb7fe640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47dc0744e0 0x7f47dc0769f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0689b0 tx=0x7f47e4007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:35.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.313+0000 7f47f8ff9640 1 -- 172.21.15.67:0/3033800272 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47ec020020 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.497+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f47fc0d0000 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.498+0000 7f47f8ff9640 1 -- 172.21.15.67:0/3033800272 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) ==== 100+0+11 (secure 0 0 0) 0x7f47ec05f4a0 con 0x7f47fc0ccbe0 2024-04-25T15:26:35.500 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47dc0744e0 msgr2=0x7f47dc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47dc0744e0 0x7f47dc0769f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f47fc0689b0 tx=0x7f47e4007c00 comp rx=0 tx=0).stop 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0ccbe0 msgr2=0x7f47fc15de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0ccbe0 0x7f47fc15de00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00b420 tx=0x7f47ec00b8e0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 shutdown_connections 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47dc0744e0 0x7f47dc0769f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47fc169200 0x7f47fc0b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47fc0d0220 0x7f47fc0b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 --2- 172.21.15.67:0/3033800272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc0ccbe0 0x7f47fc15de00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 >> 172.21.15.67:0/3033800272 conn(0x7f47fc0ce790 msgr2=0x7f47fc162d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:35.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 shutdown_connections 2024-04-25T15:26:35.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.500+0000 7f4802a42640 1 -- 172.21.15.67:0/3033800272 wait complete. 2024-04-25T15:26:35.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:35.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.623+0000 7fc609827640 1 Processor -- start 2024-04-25T15:26:35.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.623+0000 7fc609827640 1 -- start start 2024-04-25T15:26:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.624+0000 7fc609827640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc60415d400 0x7fc6040b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.624+0000 7fc609827640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6040b80b0 con 0x7fc6040b7ca0 2024-04-25T15:26:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.624+0000 7fc609827640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6040d0010 con 0x7fc6040cd050 2024-04-25T15:26:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.624+0000 7fc609827640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc6040d02f0 con 0x7fc60415d400 2024-04-25T15:26:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.624+0000 7fc6037fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc6040b7ca0 0x7fc60406e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53854/0 (socket says 172.21.15.67:53854) 2024-04-25T15:26:35.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.624+0000 7fc6037fe640 1 -- 172.21.15.67:0/1295359377 learned_addr learned my addr 172.21.15.67:0/1295359377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc5fbfff640 1 --2- 172.21.15.67:0/1295359377 >> v2:172.21.15.176:3300/0 conn(0x7fc60415d400 0x7fc6040b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735077095 0 0) 0x7fc5fc001490 con 0x7fc6040b7ca0 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f0005730 con 0x7fc6040b7ca0 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4239563249 0 0) 0x7fc5f0005730 con 0x7fc6040b7ca0 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 >> v2:172.21.15.176:3300/0 conn(0x7fc60415d400 msgr2=0x7fc6040b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 --2- 172.21.15.67:0/1295359377 >> v2:172.21.15.176:3300/0 conn(0x7fc60415d400 0x7fc6040b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 >> v1:172.21.15.176:6789/0 conn(0x7fc6040cd050 legacy=0x7fc6040b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60406d770 con 0x7fc6040b7ca0 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fc5fc001bc0 con 0x7fc6040b7ca0 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 >> v1:172.21.15.67:6789/0 conn(0x7fc6040b7ca0 legacy=0x7fc60406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc5f0006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5f0006a10 0x7fc5f0006e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5f00073d0 0x7fc5f000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc6040d02f0 con 0x7fc5f0005bc0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc5f000e370 con 0x7fc5f00073d0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc5f000e570 con 0x7fc5f0006a10 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc602ffd640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc5f0006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc5fbfff640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5f0006a10 0x7fc5f0006e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.625+0000 7fc6037fe640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5f00073d0 0x7fc5f000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc602ffd640 1 -- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5f0006a10 msgr2=0x7fc5f0006e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc602ffd640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5f0006a10 0x7fc5f0006e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc602ffd640 1 -- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5f00073d0 msgr2=0x7fc5f000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc602ffd640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5f00073d0 0x7fc5f000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc602ffd640 1 -- 172.21.15.67:0/1295359377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f0005730 con 0x7fc5f0005bc0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc6037fe640 1 -- 172.21.15.67:0/1295359377 reap_dead start 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc602ffd640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc5f0006030 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4001720 tx=0x7fc5f0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9232e6064fbfad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4001370 con 0x7fc5f0005bc0 2024-04-25T15:26:35.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f40052d0 con 0x7fc5f0005bc0 2024-04-25T15:26:35.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.626+0000 7fc6027fc640 1 -- 172.21.15.67:0/1295359377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f402aaf0 con 0x7fc5f0005bc0 2024-04-25T15:26:35.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.627+0000 7fc609827640 1 -- 172.21.15.67:0/1295359377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 msgr2=0x7fc5f0006030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.627+0000 7fc609827640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc5f0006030 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4001720 tx=0x7fc5f0007a40 comp rx=0 tx=0).stop 2024-04-25T15:26:35.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.627+0000 7fc609827640 1 -- 172.21.15.67:0/1295359377 shutdown_connections 2024-04-25T15:26:35.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.627+0000 7fc609827640 1 --2- 172.21.15.67:0/1295359377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc5f0006030 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.627+0000 7fc609827640 1 -- 172.21.15.67:0/1295359377 >> 172.21.15.67:0/1295359377 conn(0x7fc60409b7d0 msgr2=0x7fc60409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.627+0000 7fc609827640 1 -- 172.21.15.67:0/1295359377 shutdown_connections 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.627+0000 7fc609827640 1 -- 172.21.15.67:0/1295359377 wait complete. 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.629+0000 7fc609827640 1 Processor -- start 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.629+0000 7fc609827640 1 -- start start 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.630+0000 7fc609827640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc6040d3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.630+0000 7fc609827640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc60415fb80 0x7fc60414f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.630+0000 7fc609827640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc60414cf40 0x7fc60414d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.630+0000 7fc609827640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc6040cfc90 con 0x7fc5f0005bc0 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.630+0000 7fc609827640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc60406dd30 con 0x7fc60414cf40 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.630+0000 7fc609827640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc6040b40d0 con 0x7fc60415fb80 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.630+0000 7fc602ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc6040d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc6040d3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59464/0 (socket says 172.21.15.67:59464) 2024-04-25T15:26:35.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 -- 172.21.15.67:0/4043304947 learned_addr learned my addr 172.21.15.67:0/4043304947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc5fbfff640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc60415fb80 0x7fc60414f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc6037fe640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc60414cf40 0x7fc60414d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 -- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc60415fb80 msgr2=0x7fc60414f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc60415fb80 0x7fc60414f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 -- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc60414cf40 msgr2=0x7fc60414d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc60414cf40 0x7fc60414d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 -- 172.21.15.67:0/4043304947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6040b17f0 con 0x7fc5f0005bc0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc5fbfff640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc60415fb80 0x7fc60414f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc6037fe640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc60414cf40 0x7fc60414d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc602ffd640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc6040d3330 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4001780 tx=0x7fc5f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc600ff9640 1 -- 172.21.15.67:0/4043304947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f403f030 con 0x7fc5f0005bc0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc600ff9640 1 -- 172.21.15.67:0/4043304947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5f4005480 con 0x7fc5f0005bc0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6040d1040 con 0x7fc5f0005bc0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc600ff9640 1 -- 172.21.15.67:0/4043304947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4037040 con 0x7fc5f0005bc0 2024-04-25T15:26:35.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.631+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6040b1510 con 0x7fc5f0005bc0 2024-04-25T15:26:35.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.632+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc604050740 con 0x7fc5f0005bc0 2024-04-25T15:26:35.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.633+0000 7fc600ff9640 1 -- 172.21.15.67:0/4043304947 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc5f4042070 con 0x7fc5f0005bc0 2024-04-25T15:26:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.633+0000 7fc600ff9640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc5e40744e0 0x7fc5e40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.634+0000 7fc600ff9640 1 -- 172.21.15.67:0/4043304947 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc5f40b4b30 con 0x7fc5f0005bc0 2024-04-25T15:26:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.634+0000 7fc5fbfff640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc5e40744e0 0x7fc5e40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.634+0000 7fc5fbfff640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc5e40744e0 0x7fc5e40769f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc604159f70 tx=0x7fc5ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.638+0000 7fc600ff9640 1 -- 172.21.15.67:0/4043304947 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5f4049070 con 0x7fc5f0005bc0 2024-04-25T15:26:35.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.867+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc60415b300 con 0x7fc5f0005bc0 2024-04-25T15:26:35.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.867+0000 7fc600ff9640 1 -- 172.21.15.67:0/4043304947 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7fc5f407ed00 con 0x7fc5f0005bc0 2024-04-25T15:26:35.869 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:35.869 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/4217070487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4217070487},{"type":"v1","addr":"172.21.15.176:6835","nonce":4217070487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4207485404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4207485404},{"type":"v1","addr":"172.21.15.176:6837","nonce":4207485404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5067,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2355578883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2355578883},{"type":"v1","addr":"172.21.15.67:6834","nonce":2355578883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:20.974268+0000","modified":"2024-04-25T15:26:23.988098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2490033101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2490033101},{"type":"v1","addr":"172.21.15.67:6835","nonce":2490033101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-04-25T15:26:35.869 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-04-25T15:26:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc5e40744e0 msgr2=0x7fc5e40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc5e40744e0 0x7fc5e40769f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc604159f70 tx=0x7fc5ec008040 comp rx=0 tx=0).stop 2024-04-25T15:26:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 msgr2=0x7fc6040d3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc6040d3330 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4001780 tx=0x7fc5f4001120 comp rx=0 tx=0).stop 2024-04-25T15:26:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 shutdown_connections 2024-04-25T15:26:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc5e40744e0 0x7fc5e40769f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc60414cf40 0x7fc60414d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc60415fb80 0x7fc60414f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 --2- 172.21.15.67:0/4043304947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5f0005bc0 0x7fc6040d3330 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 >> 172.21.15.67:0/4043304947 conn(0x7fc60409b7d0 msgr2=0x7fc60415bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 shutdown_connections 2024-04-25T15:26:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.869+0000 7fc609827640 1 -- 172.21.15.67:0/4043304947 wait complete. 2024-04-25T15:26:35.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:35.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.992+0000 7f02b48d1640 1 Processor -- start 2024-04-25T15:26:35.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.992+0000 7f02b48d1640 1 -- start start 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02b48d1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02b00b6a00 0x7f02b015d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02b48d1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b006ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02b48d1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b01611e0 con 0x7f02b006d390 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02b48d1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f02b0160d50 con 0x7f02b00b6a00 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02b48d1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f02b015a700 con 0x7f02b015a2f0 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02ae59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02b00b6a00 0x7f02b015d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02aed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f02b006d390 0x7f02b006f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53856/0 (socket says 172.21.15.67:53856) 2024-04-25T15:26:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02aed9d640 1 -- 172.21.15.67:0/991916646 learned_addr learned my addr 172.21.15.67:0/991916646 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02add9b640 1 --2- 172.21.15.67:0/991916646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b006ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02ae59c640 1 -- 172.21.15.67:0/991916646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 msgr2=0x7f02b006ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02ae59c640 1 --2- 172.21.15.67:0/991916646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b006ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02ae59c640 1 -- 172.21.15.67:0/991916646 >> v1:172.21.15.67:6789/0 conn(0x7f02b006d390 legacy=0x7f02b006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02ae59c640 1 -- 172.21.15.67:0/991916646 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b006d7a0 con 0x7f02b00b6a00 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02ad59a640 1 -- 172.21.15.67:0/991916646 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4091612476 0 0) 0x7f02b01611e0 con 0x7f02b006d390 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.994+0000 7f02add9b640 1 --2- 172.21.15.67:0/991916646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b006ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.995+0000 7f02ae59c640 1 --2- 172.21.15.67:0/991916646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b015d6c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f02a000bfa0 tx=0x7f02a002c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73af990a4c145b72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.995+0000 7f02ad59a640 1 -- 172.21.15.67:0/991916646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0039070 con 0x7f02b00b6a00 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.995+0000 7f02ad59a640 1 -- 172.21.15.67:0/991916646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a0009540 con 0x7f02b00b6a00 2024-04-25T15:26:35.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.995+0000 7f02ad59a640 1 -- 172.21.15.67:0/991916646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0009910 con 0x7f02b00b6a00 2024-04-25T15:26:35.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 -- 172.21.15.67:0/991916646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 msgr2=0x7f02b015d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:35.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 --2- 172.21.15.67:0/991916646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b015d6c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f02a000bfa0 tx=0x7f02a002c860 comp rx=0 tx=0).stop 2024-04-25T15:26:35.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 -- 172.21.15.67:0/991916646 shutdown_connections 2024-04-25T15:26:35.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 --2- 172.21.15.67:0/991916646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b006ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:35.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 --2- 172.21.15.67:0/991916646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b015d6c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 -- 172.21.15.67:0/991916646 >> 172.21.15.67:0/991916646 conn(0x7f02b00d1ae0 msgr2=0x7f02b00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:36.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 -- 172.21.15.67:0/991916646 shutdown_connections 2024-04-25T15:26:36.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.996+0000 7f02b48d1640 1 -- 172.21.15.67:0/991916646 wait complete. 2024-04-25T15:26:36.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.998+0000 7f02b48d1640 1 Processor -- start 2024-04-25T15:26:36.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.998+0000 7f02b48d1640 1 -- start start 2024-04-25T15:26:36.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.999+0000 7f02b48d1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b006d390 0x7f02b014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.999+0000 7f02b48d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b01f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.999+0000 7f02b48d1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b01f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.999+0000 7f02b48d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01609e0 con 0x7f02b00b6a00 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.999+0000 7f02b48d1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02b0160700 con 0x7f02b006d390 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:35.999+0000 7f02b48d1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f02b00b5980 con 0x7f02b015a2f0 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b01f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b01f5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59484/0 (socket says 172.21.15.67:59484) 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 -- 172.21.15.67:0/195418485 learned_addr learned my addr 172.21.15.67:0/195418485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02aed9d640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b01f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02ae59c640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b006d390 0x7f02b014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 -- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 msgr2=0x7f02b01f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b01f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 -- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b006d390 msgr2=0x7f02b014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b006d390 0x7f02b014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 -- 172.21.15.67:0/195418485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b0155920 con 0x7f02b00b6a00 2024-04-25T15:26:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02add9b640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b01f5c50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f029800d800 tx=0x7f029800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02a6ffd640 1 -- 172.21.15.67:0/195418485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298004030 con 0x7f02b00b6a00 2024-04-25T15:26:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02a6ffd640 1 -- 172.21.15.67:0/195418485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02980041d0 con 0x7f02b00b6a00 2024-04-25T15:26:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b0155630 con 0x7f02b00b6a00 2024-04-25T15:26:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02a6ffd640 1 -- 172.21.15.67:0/195418485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02980096a0 con 0x7f02b00b6a00 2024-04-25T15:26:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.000+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b014f750 con 0x7f02b00b6a00 2024-04-25T15:26:36.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.001+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b0050740 con 0x7f02b00b6a00 2024-04-25T15:26:36.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.002+0000 7f02a6ffd640 1 -- 172.21.15.67:0/195418485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0298009840 con 0x7f02b00b6a00 2024-04-25T15:26:36.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.002+0000 7f02a6ffd640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f02900744e0 0x7f02900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.003+0000 7f02a6ffd640 1 -- 172.21.15.67:0/195418485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0298095390 con 0x7f02b00b6a00 2024-04-25T15:26:36.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.003+0000 7f02ae59c640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f02900744e0 0x7f02900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.003+0000 7f02ae59c640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f02900744e0 0x7f02900769f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f02a0036000 tx=0x7f02a000b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.007+0000 7f02a6ffd640 1 -- 172.21.15.67:0/195418485 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f029805f4d0 con 0x7f02b00b6a00 2024-04-25T15:26:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.194+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02b00d09c0 con 0x7f02b00b6a00 2024-04-25T15:26:36.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.195+0000 7f02a6ffd640 1 -- 172.21.15.67:0/195418485 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16633 (secure 0 0 0) 0x7f02980637d0 con 0x7f02b00b6a00 2024-04-25T15:26:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:36.197 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:22.596544+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:36.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f02900744e0 msgr2=0x7f02900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f02900744e0 0x7f02900769f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f02a0036000 tx=0x7f02a000b000 comp rx=0 tx=0).stop 2024-04-25T15:26:36.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 msgr2=0x7f02b01f5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b01f5c50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f029800d800 tx=0x7f029800dcc0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 shutdown_connections 2024-04-25T15:26:36.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f02900744e0 0x7f02900769f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02b015a2f0 0x7f02b01f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b00b6a00 0x7f02b01f5c50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 --2- 172.21.15.67:0/195418485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b006d390 0x7f02b014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 >> 172.21.15.67:0/195418485 conn(0x7f02b00d1ae0 msgr2=0x7f02b0151660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:36.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 shutdown_connections 2024-04-25T15:26:36.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.197+0000 7f02b48d1640 1 -- 172.21.15.67:0/195418485 wait complete. 2024-04-25T15:26:36.211 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:26:36.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.321+0000 7f1ba4972640 1 Processor -- start 2024-04-25T15:26:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.321+0000 7f1ba4972640 1 -- start start 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1ba4972640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ba0164b00 0x7f1ba00b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1ba4972640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1ba0160fa0 0x7f1ba00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1ba4972640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ba01615f0 con 0x7f1ba015df80 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1ba4972640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1ba01613b0 con 0x7f1ba0164b00 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1ba4972640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1ba0164670 con 0x7f1ba0160fa0 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ba0164b00 0x7f1ba00b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ba0164b00 0x7f1ba00b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59494/0 (socket says 172.21.15.67:59494) 2024-04-25T15:26:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9e59c640 1 -- 172.21.15.67:0/2637780782 learned_addr learned my addr 172.21.15.67:0/2637780782 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9ed9d640 1 --2- 172.21.15.67:0/2637780782 >> v2:172.21.15.176:3300/0 conn(0x7f1ba0160fa0 0x7f1ba00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9e59c640 1 -- 172.21.15.67:0/2637780782 >> v2:172.21.15.176:3300/0 conn(0x7f1ba0160fa0 msgr2=0x7f1ba00b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9e59c640 1 --2- 172.21.15.67:0/2637780782 >> v2:172.21.15.176:3300/0 conn(0x7f1ba0160fa0 0x7f1ba00b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9e59c640 1 -- 172.21.15.67:0/2637780782 >> v1:172.21.15.176:6789/0 conn(0x7f1ba015df80 legacy=0x7f1ba0160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:36.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.323+0000 7f1b9e59c640 1 -- 172.21.15.67:0/2637780782 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba0164020 con 0x7f1ba0164b00 2024-04-25T15:26:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.324+0000 7f1b9e59c640 1 --2- 172.21.15.67:0/2637780782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b7b80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00d3e80 tx=0x7f1b9802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654f70e4efbd815c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.324+0000 7f1b9d59a640 1 -- 172.21.15.67:0/2637780782 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98037070 con 0x7f1ba0164b00 2024-04-25T15:26:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.324+0000 7f1b9d59a640 1 -- 172.21.15.67:0/2637780782 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b98032440 con 0x7f1ba0164b00 2024-04-25T15:26:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.324+0000 7f1b9d59a640 1 -- 172.21.15.67:0/2637780782 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98032810 con 0x7f1ba0164b00 2024-04-25T15:26:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 -- 172.21.15.67:0/2637780782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 msgr2=0x7f1ba00b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2637780782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b7b80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00d3e80 tx=0x7f1b9802a230 comp rx=0 tx=0).stop 2024-04-25T15:26:36.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 -- 172.21.15.67:0/2637780782 shutdown_connections 2024-04-25T15:26:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2637780782 >> v2:172.21.15.176:3300/0 conn(0x7f1ba0160fa0 0x7f1ba00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2637780782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b7b80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 -- 172.21.15.67:0/2637780782 >> 172.21.15.67:0/2637780782 conn(0x7f1ba00d2bc0 msgr2=0x7f1ba00d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 -- 172.21.15.67:0/2637780782 shutdown_connections 2024-04-25T15:26:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.325+0000 7f1ba4972640 1 -- 172.21.15.67:0/2637780782 wait complete. 2024-04-25T15:26:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.327+0000 7f1ba4972640 1 Processor -- start 2024-04-25T15:26:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.327+0000 7f1ba4972640 1 -- start start 2024-04-25T15:26:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1ba4972640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba015df80 0x7f1ba014b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1ba4972640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba0160fa0 0x7f1ba01632d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1ba4972640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1ba4972640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba00b4a00 con 0x7f1ba0164b00 2024-04-25T15:26:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1ba4972640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba00b4880 con 0x7f1ba015df80 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1ba4972640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba00b4b80 con 0x7f1ba0160fa0 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1b9ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1b9ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59510/0 (socket says 172.21.15.67:59510) 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1b9ed9d640 1 -- 172.21.15.67:0/2473398778 learned_addr learned my addr 172.21.15.67:0/2473398778 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1b9e59c640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba015df80 0x7f1ba014b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.328+0000 7f1b9dd9b640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba0160fa0 0x7f1ba01632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b9ed9d640 1 -- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba0160fa0 msgr2=0x7f1ba01632d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b9ed9d640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba0160fa0 0x7f1ba01632d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b9ed9d640 1 -- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba015df80 msgr2=0x7f1ba014b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b9ed9d640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba015df80 0x7f1ba014b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b9ed9d640 1 -- 172.21.15.67:0/2473398778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba0156a80 con 0x7f1ba0164b00 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b9e59c640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba015df80 0x7f1ba014b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b9ed9d640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b60e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1b90007d80 tx=0x7f1b9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b96ffd640 1 -- 172.21.15.67:0/2473398778 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b90014420 con 0x7f1ba0164b00 2024-04-25T15:26:36.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b96ffd640 1 -- 172.21.15.67:0/2473398778 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b90019070 con 0x7f1ba0164b00 2024-04-25T15:26:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba0156d60 con 0x7f1ba0164b00 2024-04-25T15:26:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1b96ffd640 1 -- 172.21.15.67:0/2473398778 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b90014800 con 0x7f1ba0164b00 2024-04-25T15:26:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.329+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba0157320 con 0x7f1ba0164b00 2024-04-25T15:26:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.330+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ba0068c90 con 0x7f1ba0164b00 2024-04-25T15:26:36.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.331+0000 7f1b96ffd640 1 -- 172.21.15.67:0/2473398778 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1b900149a0 con 0x7f1ba0164b00 2024-04-25T15:26:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.331+0000 7f1b96ffd640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1b800744e0 0x7f1b800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.332+0000 7f1b96ffd640 1 -- 172.21.15.67:0/2473398778 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f1b90095a90 con 0x7f1ba0164b00 2024-04-25T15:26:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.332+0000 7f1b9e59c640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1b800744e0 0x7f1b800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.332+0000 7f1b9e59c640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1b800744e0 0x7f1b800769f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1b98034000 tx=0x7f1b9802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.336+0000 7f1b96ffd640 1 -- 172.21.15.67:0/2473398778 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ba0068c90 con 0x7f1ba0164b00 2024-04-25T15:26:36.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.558+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ba0050740 con 0x7f1ba0164b00 2024-04-25T15:26:36.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.559+0000 7f1b96ffd640 1 -- 172.21.15.67:0/2473398778 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7f1b900601a0 con 0x7f1ba0164b00 2024-04-25T15:26:36.561 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:36.561 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/4217070487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4217070487},{"type":"v1","addr":"172.21.15.176:6835","nonce":4217070487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4207485404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4207485404},{"type":"v1","addr":"172.21.15.176:6837","nonce":4207485404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5067,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2355578883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2355578883},{"type":"v1","addr":"172.21.15.67:6834","nonce":2355578883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:20.974268+0000","modified":"2024-04-25T15:26:23.988098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2490033101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2490033101},{"type":"v1","addr":"172.21.15.67:6835","nonce":2490033101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-04-25T15:26:36.561 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1b800744e0 msgr2=0x7f1b800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1b800744e0 0x7f1b800769f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1b98034000 tx=0x7f1b9802e000 comp rx=0 tx=0).stop 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 msgr2=0x7f1ba00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b60e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1b90007d80 tx=0x7f1b9000e4b0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 shutdown_connections 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1b800744e0 0x7f1b800769f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba0164b00 0x7f1ba00b60e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba0160fa0 0x7f1ba01632d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 --2- 172.21.15.67:0/2473398778 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba015df80 0x7f1ba014b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 >> 172.21.15.67:0/2473398778 conn(0x7f1ba00d2bc0 msgr2=0x7f1ba00d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 shutdown_connections 2024-04-25T15:26:36.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.561+0000 7f1ba4972640 1 -- 172.21.15.67:0/2473398778 wait complete. 2024-04-25T15:26:36.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.686+0000 7f67e7a4c640 1 Processor -- start 2024-04-25T15:26:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.686+0000 7f67e7a4c640 1 -- start start 2024-04-25T15:26:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.687+0000 7f67e7a4c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.687+0000 7f67e7a4c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f67e006c5c0 0x7f67e006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.687+0000 7f67e7a4c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e006cf10 con 0x7f67e00b7810 2024-04-25T15:26:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.687+0000 7f67e7a4c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f67e00d5a60 con 0x7f67e006c5c0 2024-04-25T15:26:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.687+0000 7f67e7a4c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f67e00d1850 con 0x7f67e00d0e00 2024-04-25T15:26:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e57e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e4fe7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f67e006c5c0 0x7f67e006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e57e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:35716/0 (socket says 172.21.15.67:35716) 2024-04-25T15:26:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e5fe9640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f67e00b7810 0x7f67e00b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:41276/0 (socket says 172.21.15.67:41276) 2024-04-25T15:26:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e57e8640 1 -- 172.21.15.67:0/207178936 learned_addr learned my addr 172.21.15.67:0/207178936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67d7fff640 1 -- 172.21.15.67:0/207178936 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2427729459 0 0) 0x7f67d8001490 con 0x7f67e00b7810 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67d7fff640 1 -- 172.21.15.67:0/207178936 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67cc003520 con 0x7f67e00b7810 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e57e8640 1 -- 172.21.15.67:0/207178936 >> v2:172.21.15.176:3300/0 conn(0x7f67e006c5c0 msgr2=0x7f67e006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e57e8640 1 --2- 172.21.15.67:0/207178936 >> v2:172.21.15.176:3300/0 conn(0x7f67e006c5c0 0x7f67e006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e57e8640 1 -- 172.21.15.67:0/207178936 >> v1:172.21.15.176:6789/0 conn(0x7f67e00b7810 legacy=0x7f67e00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e57e8640 1 -- 172.21.15.67:0/207178936 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e0160de0 con 0x7f67e00d0e00 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.688+0000 7f67e4fe7640 1 --2- 172.21.15.67:0/207178936 >> v2:172.21.15.176:3300/0 conn(0x7f67e006c5c0 0x7f67e006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.689+0000 7f67e57e8640 1 --2- 172.21.15.67:0/207178936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00d1210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009870 tx=0x7f67dc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9b7e6b0aa38c264 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.689+0000 7f67d7fff640 1 -- 172.21.15.67:0/207178936 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc037070 con 0x7f67e00d0e00 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.689+0000 7f67d7fff640 1 -- 172.21.15.67:0/207178936 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc032440 con 0x7f67e00d0e00 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.689+0000 7f67d7fff640 1 -- 172.21.15.67:0/207178936 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc032810 con 0x7f67e00d0e00 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 -- 172.21.15.67:0/207178936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 msgr2=0x7f67e00d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/207178936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00d1210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009870 tx=0x7f67dc02a270 comp rx=0 tx=0).stop 2024-04-25T15:26:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 -- 172.21.15.67:0/207178936 shutdown_connections 2024-04-25T15:26:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/207178936 >> v2:172.21.15.176:3300/0 conn(0x7f67e006c5c0 0x7f67e006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/207178936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00d1210 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 -- 172.21.15.67:0/207178936 >> 172.21.15.67:0/207178936 conn(0x7f67e00d5d90 msgr2=0x7f67e00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 -- 172.21.15.67:0/207178936 shutdown_connections 2024-04-25T15:26:36.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.690+0000 7f67e7a4c640 1 -- 172.21.15.67:0/207178936 wait complete. 2024-04-25T15:26:36.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.692+0000 7f67e7a4c640 1 Processor -- start 2024-04-25T15:26:36.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.692+0000 7f67e7a4c640 1 -- start start 2024-04-25T15:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e7a4c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e006c5c0 0x7f67e00cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e7a4c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67e00b7810 0x7f67e00c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e7a4c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e7a4c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f67e0160ad0 con 0x7f67e006c5c0 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e7a4c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f67e01613a0 con 0x7f67e00b7810 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e7a4c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f67e00b5320 con 0x7f67e00d0e00 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e006c5c0 0x7f67e00cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e006c5c0 0x7f67e00cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59526/0 (socket says 172.21.15.67:59526) 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 -- 172.21.15.67:0/1858507469 learned_addr learned my addr 172.21.15.67:0/1858507469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e5fe9640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e4fe7640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67e00b7810 0x7f67e00c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 -- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 msgr2=0x7f67e00ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 -- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67e00b7810 msgr2=0x7f67e00c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67e00b7810 0x7f67e00c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e57e8640 1 -- 172.21.15.67:0/1858507469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e0147fd0 con 0x7f67e006c5c0 2024-04-25T15:26:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.693+0000 7f67e4fe7640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67e00b7810 0x7f67e00c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.694+0000 7f67e57e8640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e006c5c0 0x7f67e00cc6e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0090b0 tx=0x7f67dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.694+0000 7f67d5ffb640 1 -- 172.21.15.67:0/1858507469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc041020 con 0x7f67e006c5c0 2024-04-25T15:26:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.694+0000 7f67d5ffb640 1 -- 172.21.15.67:0/1858507469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc03d640 con 0x7f67e006c5c0 2024-04-25T15:26:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.694+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e01482b0 con 0x7f67e006c5c0 2024-04-25T15:26:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.694+0000 7f67d5ffb640 1 -- 172.21.15.67:0/1858507469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc037040 con 0x7f67e006c5c0 2024-04-25T15:26:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.694+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e0148870 con 0x7f67e006c5c0 2024-04-25T15:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.695+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e0050740 con 0x7f67e006c5c0 2024-04-25T15:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.696+0000 7f67d5ffb640 1 -- 172.21.15.67:0/1858507469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f67dc044020 con 0x7f67e006c5c0 2024-04-25T15:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.696+0000 7f67d5ffb640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67c40744e0 0x7f67c40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.696+0000 7f67d5ffb640 1 -- 172.21.15.67:0/1858507469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f67dc0b5860 con 0x7f67e006c5c0 2024-04-25T15:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.697+0000 7f67e4fe7640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67c40744e0 0x7f67c40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:36.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.697+0000 7f67e4fe7640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67c40744e0 0x7f67c40769f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f67e006cc40 tx=0x7f67d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.701+0000 7f67d5ffb640 1 -- 172.21.15.67:0/1858507469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67dc04b030 con 0x7f67e006c5c0 2024-04-25T15:26:36.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.928+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67e00d4320 con 0x7f67e006c5c0 2024-04-25T15:26:36.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.929+0000 7f67d5ffb640 1 -- 172.21.15.67:0/1858507469 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5187 (secure 0 0 0) 0x7f67dc080080 con 0x7f67e006c5c0 2024-04-25T15:26:36.931 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:36.932 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/4217070487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4217070487},{"type":"v1","addr":"172.21.15.176:6835","nonce":4217070487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4207485404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4207485404},{"type":"v1","addr":"172.21.15.176:6837","nonce":4207485404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5067,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2355578883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2355578883},{"type":"v1","addr":"172.21.15.67:6834","nonce":2355578883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:20.974268+0000","modified":"2024-04-25T15:26:23.988098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5070},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5070":{"gid":5070,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/2490033101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2490033101},{"type":"v1","addr":"172.21.15.67:6835","nonce":2490033101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5070,"qdb_cluster":[5070]},"id":2}]} 2024-04-25T15:26:36.932 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-04-25T15:26:36.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67c40744e0 msgr2=0x7f67c40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67c40744e0 0x7f67c40769f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f67e006cc40 tx=0x7f67d0008040 comp rx=0 tx=0).stop 2024-04-25T15:26:36.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e006c5c0 msgr2=0x7f67e00cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:36.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e006c5c0 0x7f67e00cc6e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f67dc0090b0 tx=0x7f67dc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 shutdown_connections 2024-04-25T15:26:36.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f67c40744e0 0x7f67c40769f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67e00d0e00 0x7f67e00ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67e00b7810 0x7f67e00c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 --2- 172.21.15.67:0/1858507469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67e006c5c0 0x7f67e00cc6e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 >> 172.21.15.67:0/1858507469 conn(0x7f67e00d5d90 msgr2=0x7f67e00d27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 shutdown_connections 2024-04-25T15:26:36.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:36.931+0000 7f67e7a4c640 1 -- 172.21.15.67:0/1858507469 wait complete. 2024-04-25T15:26:36.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.056+0000 7f7bf5b46640 1 Processor -- start 2024-04-25T15:26:37.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.056+0000 7f7bf5b46640 1 -- start start 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7bf5b46640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bf00d4130 0x7f7bf00d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7bf5b46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006dcf0 0x7f7bf00d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7bf5b46640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bf00b3c10 con 0x7f7bf006d8e0 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7bf5b46640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf00b9520 con 0x7f7bf00d4130 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7bf5b46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf00b4820 con 0x7f7bf006dcf0 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bf00d4130 0x7f7bf00d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7bf00d4130 0x7f7bf00d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59534/0 (socket says 172.21.15.67:59534) 2024-04-25T15:26:37.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beeffd640 1 -- 172.21.15.67:0/3197132296 learned_addr learned my addr 172.21.15.67:0/3197132296 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beffff640 1 --2- 172.21.15.67:0/3197132296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006dcf0 0x7f7bf00d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beeffd640 1 -- 172.21.15.67:0/3197132296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006dcf0 msgr2=0x7f7bf00d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beeffd640 1 --2- 172.21.15.67:0/3197132296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006dcf0 0x7f7bf00d22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beeffd640 1 -- 172.21.15.67:0/3197132296 >> v1:172.21.15.176:6789/0 conn(0x7f7bf006d8e0 legacy=0x7f7bf00d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:37.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.058+0000 7f7beeffd640 1 -- 172.21.15.67:0/3197132296 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf00b3ef0 con 0x7f7bf00d4130 2024-04-25T15:26:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.059+0000 7f7beeffd640 1 --2- 172.21.15.67:0/3197132296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00d4130 0x7f7bf00d1d60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e7af004386e44d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.059+0000 7f7bee7fc640 1 -- 172.21.15.67:0/3197132296 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4037070 con 0x7f7bf00d4130 2024-04-25T15:26:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.059+0000 7f7bee7fc640 1 -- 172.21.15.67:0/3197132296 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be4032440 con 0x7f7bf00d4130 2024-04-25T15:26:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.059+0000 7f7bee7fc640 1 -- 172.21.15.67:0/3197132296 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4032810 con 0x7f7bf00d4130 2024-04-25T15:26:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 -- 172.21.15.67:0/3197132296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00d4130 msgr2=0x7f7bf00d1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/3197132296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00d4130 0x7f7bf00d1d60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402a180 comp rx=0 tx=0).stop 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 -- 172.21.15.67:0/3197132296 shutdown_connections 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/3197132296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006dcf0 0x7f7bf00d22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/3197132296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00d4130 0x7f7bf00d1d60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 -- 172.21.15.67:0/3197132296 >> 172.21.15.67:0/3197132296 conn(0x7f7bf008d110 msgr2=0x7f7bf008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 -- 172.21.15.67:0/3197132296 shutdown_connections 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.060+0000 7f7bf5b46640 1 -- 172.21.15.67:0/3197132296 wait complete. 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.062+0000 7f7bf5b46640 1 Processor -- start 2024-04-25T15:26:37.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.062+0000 7f7bf5b46640 1 -- start start 2024-04-25T15:26:37.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7bf5b46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006d8e0 0x7f7bf0159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7bf5b46640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bf006dcf0 0x7f7bf00c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7bf5b46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00c9c70 0x7f7bf00ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7bf5b46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf00b38e0 con 0x7f7bf00c9c70 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7bf5b46640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf00b44b0 con 0x7f7bf006dcf0 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7bf5b46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf00d27e0 con 0x7f7bf006d8e0 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7beffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00c9c70 0x7f7bf00ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7beffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00c9c70 0x7f7bf00ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59544/0 (socket says 172.21.15.67:59544) 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7beffff640 1 -- 172.21.15.67:0/660496817 learned_addr learned my addr 172.21.15.67:0/660496817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7beeffd640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bf006dcf0 0x7f7bf00c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.063+0000 7f7bef7fe640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006d8e0 0x7f7bf0159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7beffff640 1 -- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006d8e0 msgr2=0x7f7bf0159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7beffff640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006d8e0 0x7f7bf0159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7beffff640 1 -- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bf006dcf0 msgr2=0x7f7bf00c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7beffff640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bf006dcf0 0x7f7bf00c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7beffff640 1 -- 172.21.15.67:0/660496817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf00ca5c0 con 0x7f7bf00c9c70 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7bef7fe640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006d8e0 0x7f7bf0159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7beffff640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00c9c70 0x7f7bf00ca080 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7be8007e60 tx=0x7f7be800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7bd7fff640 1 -- 172.21.15.67:0/660496817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be8014420 con 0x7f7bf00c9c70 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7bd7fff640 1 -- 172.21.15.67:0/660496817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be8019070 con 0x7f7bf00c9c70 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf009f830 con 0x7f7bf00c9c70 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7bd7fff640 1 -- 172.21.15.67:0/660496817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be8014820 con 0x7f7bf00c9c70 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.064+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf009fdf0 con 0x7f7bf00c9c70 2024-04-25T15:26:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.065+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf0050740 con 0x7f7bf00c9c70 2024-04-25T15:26:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.066+0000 7f7bd7fff640 1 -- 172.21.15.67:0/660496817 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7be80149c0 con 0x7f7bf00c9c70 2024-04-25T15:26:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.066+0000 7f7bd7fff640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7bdc0744b0 0x7f7bdc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.066+0000 7f7bd7fff640 1 -- 172.21.15.67:0/660496817 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7be8095ac0 con 0x7f7bf00c9c70 2024-04-25T15:26:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.066+0000 7f7bef7fe640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7bdc0744b0 0x7f7bdc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.067+0000 7f7bef7fe640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7bdc0744b0 0x7f7bdc0769c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00744c0 tx=0x7f7be0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.071+0000 7f7bd7fff640 1 -- 172.21.15.67:0/660496817 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7bf00ca5c0 con 0x7f7bf00c9c70 2024-04-25T15:26:37.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.262+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7bf0072eb0 con 0x7f7bf00c9c70 2024-04-25T15:26:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.263+0000 7f7bd7fff640 1 -- 172.21.15.67:0/660496817 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16633 (secure 0 0 0) 0x7f7be805fc00 con 0x7f7bf00c9c70 2024-04-25T15:26:37.265 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:22.596544+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:18.702421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:19.919128+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7bdc0744b0 msgr2=0x7f7bdc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7bdc0744b0 0x7f7bdc0769c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00744c0 tx=0x7f7be0008040 comp rx=0 tx=0).stop 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00c9c70 msgr2=0x7f7bf00ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00c9c70 0x7f7bf00ca080 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7be8007e60 tx=0x7f7be800e4b0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 shutdown_connections 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7bdc0744b0 0x7f7bdc0769c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf00c9c70 0x7f7bf00ca080 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bf006dcf0 0x7f7bf00c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 --2- 172.21.15.67:0/660496817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7bf006d8e0 0x7f7bf0159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 >> 172.21.15.67:0/660496817 conn(0x7f7bf008d110 msgr2=0x7f7bf00d3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 shutdown_connections 2024-04-25T15:26:37.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.266+0000 7f7bf5b46640 1 -- 172.21.15.67:0/660496817 wait complete. 2024-04-25T15:26:37.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:26:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.390+0000 7fe9e1df3640 1 Processor -- start 2024-04-25T15:26:37.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.390+0000 7fe9e1df3640 1 -- start start 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9e1df3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9dc0d4570 0x7fe9dc0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9e1df3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe9dc06e8f0 0x7fe9dc0d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9e1df3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9e1df3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe9dc06e460 con 0x7fe9dc0d4570 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9e1df3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe9dc06d870 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9e1df3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe9dc06db10 con 0x7fe9dc0d09f0 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9db7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9dc0d4570 0x7fe9dc0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9db7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9dc0d4570 0x7fe9dc0b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59552/0 (socket says 172.21.15.67:59552) 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.391+0000 7fe9db7fe640 1 -- 172.21.15.67:0/570252338 learned_addr learned my addr 172.21.15.67:0/570252338 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9daffd640 1 --2- 172.21.15.67:0/570252338 >> v2:172.21.15.176:3300/0 conn(0x7fe9dc06e8f0 0x7fe9dc0d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9dbfff640 1 --2- 172.21.15.67:0/570252338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9db7fe640 1 -- 172.21.15.67:0/570252338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 msgr2=0x7fe9dc0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9db7fe640 1 --2- 172.21.15.67:0/570252338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9db7fe640 1 -- 172.21.15.67:0/570252338 >> v2:172.21.15.176:3300/0 conn(0x7fe9dc06e8f0 msgr2=0x7fe9dc0d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9db7fe640 1 --2- 172.21.15.67:0/570252338 >> v2:172.21.15.176:3300/0 conn(0x7fe9dc06e8f0 0x7fe9dc0d04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9db7fe640 1 -- 172.21.15.67:0/570252338 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9dc06e0f0 con 0x7fe9dc0d4570 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9daffd640 1 --2- 172.21.15.67:0/570252338 >> v2:172.21.15.176:3300/0 conn(0x7fe9dc06e8f0 0x7fe9dc0d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9dbfff640 1 --2- 172.21.15.67:0/570252338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9db7fe640 1 --2- 172.21.15.67:0/570252338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc0d4570 0x7fe9dc0b4830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc070c30 tx=0x7fe9cc02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6464f1f4ae58e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9da7fc640 1 -- 172.21.15.67:0/570252338 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc004030 con 0x7fe9dc0d4570 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.392+0000 7fe9da7fc640 1 -- 172.21.15.67:0/570252338 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9cc0041d0 con 0x7fe9dc0d4570 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9da7fc640 1 -- 172.21.15.67:0/570252338 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc003610 con 0x7fe9dc0d4570 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 -- 172.21.15.67:0/570252338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc0d4570 msgr2=0x7fe9dc0b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/570252338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc0d4570 0x7fe9dc0b4830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc070c30 tx=0x7fe9cc02a130 comp rx=0 tx=0).stop 2024-04-25T15:26:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 -- 172.21.15.67:0/570252338 shutdown_connections 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/570252338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/570252338 >> v2:172.21.15.176:3300/0 conn(0x7fe9dc06e8f0 0x7fe9dc0d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/570252338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc0d4570 0x7fe9dc0b4830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 -- 172.21.15.67:0/570252338 >> 172.21.15.67:0/570252338 conn(0x7fe9dc0d3a50 msgr2=0x7fe9dc0d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 -- 172.21.15.67:0/570252338 shutdown_connections 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.393+0000 7fe9e1df3640 1 -- 172.21.15.67:0/570252338 wait complete. 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.395+0000 7fe9e1df3640 1 Processor -- start 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.395+0000 7fe9e1df3640 1 -- start start 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.396+0000 7fe9e1df3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc06e8f0 0x7fe9dc1f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.396+0000 7fe9e1df3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.396+0000 7fe9e1df3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9dc0d4570 0x7fe9dc1f98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.396+0000 7fe9e1df3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9dc0b3300 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.396+0000 7fe9e1df3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe9dc06ed00 con 0x7fe9dc0d4570 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.396+0000 7fe9e1df3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe9dc0b3480 con 0x7fe9dc0d09f0 2024-04-25T15:26:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc06e8f0 0x7fe9dc1f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc06e8f0 0x7fe9dc1f5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59566/0 (socket says 172.21.15.67:59566) 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 -- 172.21.15.67:0/2310156286 learned_addr learned my addr 172.21.15.67:0/2310156286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 -- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 msgr2=0x7fe9dc0b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9daffd640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9dbfff640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9dc0d4570 0x7fe9dc1f98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 -- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9dc0d4570 msgr2=0x7fe9dc1f98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9dc0d4570 0x7fe9dc1f98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 -- 172.21.15.67:0/2310156286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9dc0b7840 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9db7fe640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc06e8f0 0x7fe9dc1f5d00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc02a170 tx=0x7fe9cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9c3fff640 1 -- 172.21.15.67:0/2310156286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc041020 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9c3fff640 1 -- 172.21.15.67:0/2310156286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9cc003c50 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9dc0b7b20 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9c3fff640 1 -- 172.21.15.67:0/2310156286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc006040 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.397+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9dc0b80e0 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.398+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9dc050740 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.399+0000 7fe9c3fff640 1 -- 172.21.15.67:0/2310156286 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe9cc044020 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.399+0000 7fe9c3fff640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe9c80744b0 0x7fe9c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.400+0000 7fe9c3fff640 1 -- 172.21.15.67:0/2310156286 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe9cc0b5eb0 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.400+0000 7fe9daffd640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe9c80744b0 0x7fe9c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.400+0000 7fe9daffd640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe9c80744b0 0x7fe9c80769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc1fa930 tx=0x7fe9d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.404+0000 7fe9c3fff640 1 -- 172.21.15.67:0/2310156286 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9cc04b030 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.634+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe9dc0d1220 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.636 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:26:37.635+0000 7f3e02b50640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:26:37.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.636+0000 7fe9c3fff640 1 -- 172.21.15.67:0/2310156286 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7fe9cc0806d0 con 0x7fe9dc06e8f0 2024-04-25T15:26:37.638 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:26:37.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe9c80744b0 msgr2=0x7fe9c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe9c80744b0 0x7fe9c80769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc1fa930 tx=0x7fe9d0008040 comp rx=0 tx=0).stop 2024-04-25T15:26:37.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc06e8f0 msgr2=0x7fe9dc1f5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc06e8f0 0x7fe9dc1f5d00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc02a170 tx=0x7fe9cc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 shutdown_connections 2024-04-25T15:26:37.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe9c80744b0 0x7fe9c80769c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9dc0d4570 0x7fe9dc1f98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9dc0d09f0 0x7fe9dc0b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 --2- 172.21.15.67:0/2310156286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9dc06e8f0 0x7fe9dc1f5d00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 >> 172.21.15.67:0/2310156286 conn(0x7fe9dc0d3a50 msgr2=0x7fe9dc163ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:37.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.638+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 shutdown_connections 2024-04-25T15:26:37.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.639+0000 7fe9e1df3640 1 -- 172.21.15.67:0/2310156286 wait complete. 2024-04-25T15:26:37.643 INFO:tasks.ceph.mds.a.smithi067.stderr: -3199> 2024-04-25T15:26:25.551+0000 7fbfacaf6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:26:37.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:26:37.702 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:26:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.770+0000 7fdfd8413640 1 Processor -- start 2024-04-25T15:26:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.770+0000 7fdfd8413640 1 -- start start 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd8413640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfd0164ad0 0x7fdfd00b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd8413640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdfd00b0a70 0x7fdfd00b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd8413640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd00d3170 con 0x7fdfd0163f30 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd8413640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd00d2f40 con 0x7fdfd0164ad0 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd8413640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd0164ee0 con 0x7fdfd00b0a70 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd61af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfd0164ad0 0x7fdfd00b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd61af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfd0164ad0 0x7fdfd00b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49484/0 (socket says 172.21.15.67:49484) 2024-04-25T15:26:37.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd61af640 1 -- 172.21.15.67:0/221257914 learned_addr learned my addr 172.21.15.67:0/221257914 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd69b0640 1 --2- 172.21.15.67:0/221257914 >> v2:172.21.15.176:3300/0 conn(0x7fdfd00b0a70 0x7fdfd00b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd61af640 1 -- 172.21.15.67:0/221257914 >> v2:172.21.15.176:3300/0 conn(0x7fdfd00b0a70 msgr2=0x7fdfd00b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd61af640 1 --2- 172.21.15.67:0/221257914 >> v2:172.21.15.176:3300/0 conn(0x7fdfd00b0a70 0x7fdfd00b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd61af640 1 -- 172.21.15.67:0/221257914 >> v1:172.21.15.176:6789/0 conn(0x7fdfd0163f30 legacy=0x7fdfd0164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.772+0000 7fdfd61af640 1 -- 172.21.15.67:0/221257914 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd00d3450 con 0x7fdfd0164ad0 2024-04-25T15:26:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.773+0000 7fdfd61af640 1 --2- 172.21.15.67:0/221257914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0164ad0 0x7fdfd00b2ca0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdfd006dc90 tx=0x7fdfcc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f51ac6a58d98a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.773+0000 7fdfd51ad640 1 -- 172.21.15.67:0/221257914 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc037070 con 0x7fdfd0164ad0 2024-04-25T15:26:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.773+0000 7fdfd51ad640 1 -- 172.21.15.67:0/221257914 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfcc032440 con 0x7fdfd0164ad0 2024-04-25T15:26:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.773+0000 7fdfd51ad640 1 -- 172.21.15.67:0/221257914 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc032810 con 0x7fdfd0164ad0 2024-04-25T15:26:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 -- 172.21.15.67:0/221257914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0164ad0 msgr2=0x7fdfd00b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 --2- 172.21.15.67:0/221257914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0164ad0 0x7fdfd00b2ca0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdfd006dc90 tx=0x7fdfcc02a270 comp rx=0 tx=0).stop 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 -- 172.21.15.67:0/221257914 shutdown_connections 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 --2- 172.21.15.67:0/221257914 >> v2:172.21.15.176:3300/0 conn(0x7fdfd00b0a70 0x7fdfd00b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 --2- 172.21.15.67:0/221257914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0164ad0 0x7fdfd00b2ca0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 -- 172.21.15.67:0/221257914 >> 172.21.15.67:0/221257914 conn(0x7fdfd00d3aa0 msgr2=0x7fdfd00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 -- 172.21.15.67:0/221257914 shutdown_connections 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.774+0000 7fdfd8413640 1 -- 172.21.15.67:0/221257914 wait complete. 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.776+0000 7fdfd8413640 1 Processor -- start 2024-04-25T15:26:37.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.776+0000 7fdfd8413640 1 -- start start 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd8413640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfd00b0a70 0x7fdfd015a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd8413640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0163f30 0x7fdfd00cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd8413640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfd0164ad0 0x7fdfd0162c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd8413640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd006d9b0 con 0x7fdfd0163f30 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd8413640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd00b31e0 con 0x7fdfd00b0a70 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd8413640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd00b9430 con 0x7fdfd0164ad0 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0163f30 0x7fdfd00cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0163f30 0x7fdfd00cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49500/0 (socket says 172.21.15.67:49500) 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 -- 172.21.15.67:0/1327454877 learned_addr learned my addr 172.21.15.67:0/1327454877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd69b0640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfd0164ad0 0x7fdfd0162c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd61af640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfd00b0a70 0x7fdfd015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 -- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfd0164ad0 msgr2=0x7fdfd0162c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfd0164ad0 0x7fdfd0162c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 -- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfd00b0a70 msgr2=0x7fdfd015a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfd00b0a70 0x7fdfd015a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.777+0000 7fdfd59ae640 1 -- 172.21.15.67:0/1327454877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd01f6bf0 con 0x7fdfd0163f30 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.778+0000 7fdfd59ae640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0163f30 0x7fdfd00cc250 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000c3f0 tx=0x7fdfc000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.778+0000 7fdfc6ffd640 1 -- 172.21.15.67:0/1327454877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00052d0 con 0x7fdfd0163f30 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.778+0000 7fdfc6ffd640 1 -- 172.21.15.67:0/1327454877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc0005490 con 0x7fdfd0163f30 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.778+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd01f6ed0 con 0x7fdfd0163f30 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.778+0000 7fdfc6ffd640 1 -- 172.21.15.67:0/1327454877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0007260 con 0x7fdfd0163f30 2024-04-25T15:26:37.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.778+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd01f7490 con 0x7fdfd0163f30 2024-04-25T15:26:37.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.779+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd00d4770 con 0x7fdfd0163f30 2024-04-25T15:26:37.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.780+0000 7fdfc6ffd640 1 -- 172.21.15.67:0/1327454877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdfc0007400 con 0x7fdfd0163f30 2024-04-25T15:26:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.780+0000 7fdfc6ffd640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfb40744b0 0x7fdfb40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.780+0000 7fdfc6ffd640 1 -- 172.21.15.67:0/1327454877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7fdfc00946c0 con 0x7fdfd0163f30 2024-04-25T15:26:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.780+0000 7fdfd61af640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfb40744b0 0x7fdfb40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.781+0000 7fdfd61af640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfb40744b0 0x7fdfb40769c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc034000 tx=0x7fdfcc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:37.785+0000 7fdfc6ffd640 1 -- 172.21.15.67:0/1327454877 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdfc005ee80 con 0x7fdfd0163f30 2024-04-25T15:26:38.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.011+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fdfd00717f0 con 0x7fdfd0163f30 2024-04-25T15:26:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.663+0000 7fdfc6ffd640 1 -- 172.21.15.67:0/1327454877 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7fdfc0065e20 con 0x7fdfd0163f30 2024-04-25T15:26:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfb40744b0 msgr2=0x7fdfb40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfb40744b0 0x7fdfb40769c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc034000 tx=0x7fdfcc02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0163f30 msgr2=0x7fdfd00cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0163f30 0x7fdfd00cc250 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000c3f0 tx=0x7fdfc000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 shutdown_connections 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfb40744b0 0x7fdfb40769c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfd0164ad0 0x7fdfd0162c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfd0163f30 0x7fdfd00cc250 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 --2- 172.21.15.67:0/1327454877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfd00b0a70 0x7fdfd015a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.666+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 >> 172.21.15.67:0/1327454877 conn(0x7fdfd00d3aa0 msgr2=0x7fdfd0071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.667+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 shutdown_connections 2024-04-25T15:26:38.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.667+0000 7fdfd8413640 1 -- 172.21.15.67:0/1327454877 wait complete. 2024-04-25T15:26:38.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:26:38.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.804+0000 7f0542084640 1 Processor -- start 2024-04-25T15:26:38.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.804+0000 7f0542084640 1 -- start start 2024-04-25T15:26:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f0542084640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f053c161710 0x7f053c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f0542084640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f053c06c5c0 0x7f053c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f0542084640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f053c1609b0 con 0x7f053c163fa0 2024-04-25T15:26:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f0542084640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f053c06cf10 con 0x7f053c161710 2024-04-25T15:26:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f0542084640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f053c164cd0 con 0x7f053c06c5c0 2024-04-25T15:26:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f053b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f053c161710 0x7f053c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:38.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f053b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f053c161710 0x7f053c15e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49506/0 (socket says 172.21.15.67:49506) 2024-04-25T15:26:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.805+0000 7f053b7fe640 1 -- 172.21.15.67:0/3625825600 learned_addr learned my addr 172.21.15.67:0/3625825600 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.806+0000 7f053affd640 1 --2- 172.21.15.67:0/3625825600 >> v2:172.21.15.176:3300/0 conn(0x7f053c06c5c0 0x7f053c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.806+0000 7f053b7fe640 1 -- 172.21.15.67:0/3625825600 >> v2:172.21.15.176:3300/0 conn(0x7f053c06c5c0 msgr2=0x7f053c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.806+0000 7f053b7fe640 1 --2- 172.21.15.67:0/3625825600 >> v2:172.21.15.176:3300/0 conn(0x7f053c06c5c0 0x7f053c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.806+0000 7f053b7fe640 1 -- 172.21.15.67:0/3625825600 >> v1:172.21.15.67:6789/0 conn(0x7f053c163fa0 legacy=0x7f053c1643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:38.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.806+0000 7f053b7fe640 1 -- 172.21.15.67:0/3625825600 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c160c30 con 0x7f053c161710 2024-04-25T15:26:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.806+0000 7f053a7fc640 1 -- 172.21.15.67:0/3625825600 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1907549548 0 0) 0x7f0534001490 con 0x7f053c163fa0 2024-04-25T15:26:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.806+0000 7f053affd640 1 --2- 172.21.15.67:0/3625825600 >> v2:172.21.15.176:3300/0 conn(0x7f053c06c5c0 0x7f053c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f053b7fe640 1 --2- 172.21.15.67:0/3625825600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c15e080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f053c0d2770 tx=0x7f052c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf168efcc7b39d55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f053a7fc640 1 -- 172.21.15.67:0/3625825600 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c037070 con 0x7f053c161710 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f053a7fc640 1 -- 172.21.15.67:0/3625825600 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f052c032440 con 0x7f053c161710 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f053a7fc640 1 -- 172.21.15.67:0/3625825600 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c032810 con 0x7f053c161710 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 -- 172.21.15.67:0/3625825600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 msgr2=0x7f053c15e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 --2- 172.21.15.67:0/3625825600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c15e080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f053c0d2770 tx=0x7f052c02a200 comp rx=0 tx=0).stop 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 -- 172.21.15.67:0/3625825600 shutdown_connections 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 --2- 172.21.15.67:0/3625825600 >> v2:172.21.15.176:3300/0 conn(0x7f053c06c5c0 0x7f053c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 --2- 172.21.15.67:0/3625825600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c15e080 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 -- 172.21.15.67:0/3625825600 >> 172.21.15.67:0/3625825600 conn(0x7f053c0d5d90 msgr2=0x7f053c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 -- 172.21.15.67:0/3625825600 shutdown_connections 2024-04-25T15:26:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.807+0000 7f0542084640 1 -- 172.21.15.67:0/3625825600 wait complete. 2024-04-25T15:26:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.809+0000 7f0542084640 1 Processor -- start 2024-04-25T15:26:38.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.809+0000 7f0542084640 1 -- start start 2024-04-25T15:26:38.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f0542084640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f053c06c5c0 0x7f053c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:38.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f0542084640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f0542084640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f053c163fa0 0x7f053c0ba430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f0542084640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f053c0d49f0 con 0x7f053c161710 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f0542084640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f053c1611f0 con 0x7f053c06c5c0 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f0542084640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f053c0d4b70 con 0x7f053c163fa0 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49512/0 (socket says 172.21.15.67:49512) 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 -- 172.21.15.67:0/2984747377 learned_addr learned my addr 172.21.15.67:0/2984747377 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053b7fe640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f053c06c5c0 0x7f053c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053bfff640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f053c163fa0 0x7f053c0ba430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 -- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f053c163fa0 msgr2=0x7f053c0ba430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f053c163fa0 0x7f053c0ba430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 -- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f053c06c5c0 msgr2=0x7f053c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f053c06c5c0 0x7f053c0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 -- 172.21.15.67:0/2984747377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c155f50 con 0x7f053c161710 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.811+0000 7f053affd640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c155570 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f053000d7d0 tx=0x7f053000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.812+0000 7f0523fff640 1 -- 172.21.15.67:0/2984747377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530004030 con 0x7f053c161710 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.812+0000 7f0523fff640 1 -- 172.21.15.67:0/2984747377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05300041d0 con 0x7f053c161710 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.812+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f053c1f6bc0 con 0x7f053c161710 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.812+0000 7f0523fff640 1 -- 172.21.15.67:0/2984747377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05300096a0 con 0x7f053c161710 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.813+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f053c1f7180 con 0x7f053c161710 2024-04-25T15:26:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.813+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053c155cc0 con 0x7f053c161710 2024-04-25T15:26:38.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.814+0000 7f0523fff640 1 -- 172.21.15.67:0/2984747377 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0530009840 con 0x7f053c161710 2024-04-25T15:26:38.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.815+0000 7f0523fff640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05280744b0 0x7f05280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:38.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.815+0000 7f0523fff640 1 -- 172.21.15.67:0/2984747377 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7f0530095460 con 0x7f053c161710 2024-04-25T15:26:38.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.815+0000 7f053b7fe640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05280744b0 0x7f05280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:38.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.816+0000 7f053b7fe640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05280744b0 0x7f05280769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f052c034000 tx=0x7f052c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:38.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:38.819+0000 7f0523fff640 1 -- 172.21.15.67:0/2984747377 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f053005f540 con 0x7f053c161710 2024-04-25T15:26:39.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.004+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f053c050740 con 0x7f053c161710 2024-04-25T15:26:39.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.675+0000 7f0523fff640 1 -- 172.21.15.67:0/2984747377 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) ==== 181+0+0 (secure 0 0 0) 0x7f0530063840 con 0x7f053c161710 2024-04-25T15:26:39.677 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:26:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.680+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05280744b0 msgr2=0x7f05280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.680+0000 7f0542084640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05280744b0 0x7f05280769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f052c034000 tx=0x7f052c02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.680+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 msgr2=0x7f053c155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:39.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.680+0000 7f0542084640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c155570 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f053000d7d0 tx=0x7f053000dc90 comp rx=0 tx=0).stop 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 shutdown_connections 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05280744b0 0x7f05280769c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f053c163fa0 0x7f053c0ba430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f053c161710 0x7f053c155570 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 --2- 172.21.15.67:0/2984747377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f053c06c5c0 0x7f053c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 >> 172.21.15.67:0/2984747377 conn(0x7f053c0d5d90 msgr2=0x7f053c15e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 shutdown_connections 2024-04-25T15:26:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.682+0000 7f0542084640 1 -- 172.21.15.67:0/2984747377 wait complete. 2024-04-25T15:26:39.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:26:39.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.819+0000 7fdf25451640 1 Processor -- start 2024-04-25T15:26:39.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.819+0000 7fdf25451640 1 -- start start 2024-04-25T15:26:39.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf25451640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf200d0e00 0x7fdf200d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:39.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf25451640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf2006c5c0 0x7fdf2006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:39.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf25451640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdf200b7810 0x7fdf200b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:39.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf25451640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdf2006cf10 con 0x7fdf200d0e00 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf25451640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdf200d1850 con 0x7fdf200b7810 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf25451640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdf20160de0 con 0x7fdf2006c5c0 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf200d0e00 0x7fdf200d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf200d0e00 0x7fdf200d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49528/0 (socket says 172.21.15.67:49528) 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 -- 172.21.15.67:0/1787789060 learned_addr learned my addr 172.21.15.67:0/1787789060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1e7fc640 1 --2- 172.21.15.67:0/1787789060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf2006c5c0 0x7fdf2006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1f7fe640 1 --2- 172.21.15.67:0/1787789060 >> v2:172.21.15.176:3300/0 conn(0x7fdf200b7810 0x7fdf200b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 -- 172.21.15.67:0/1787789060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf2006c5c0 msgr2=0x7fdf2006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 --2- 172.21.15.67:0/1787789060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf2006c5c0 0x7fdf2006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 -- 172.21.15.67:0/1787789060 >> v2:172.21.15.176:3300/0 conn(0x7fdf200b7810 msgr2=0x7fdf200b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 --2- 172.21.15.67:0/1787789060 >> v2:172.21.15.176:3300/0 conn(0x7fdf200b7810 0x7fdf200b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.821+0000 7fdf1effd640 1 -- 172.21.15.67:0/1787789060 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf201610c0 con 0x7fdf200d0e00 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.822+0000 7fdf1effd640 1 --2- 172.21.15.67:0/1787789060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf200d0e00 0x7fdf200d1210 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdf2006d1e0 tx=0x7fdf1002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb2a4f06a0e8600 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.822+0000 7fdf1dffb640 1 -- 172.21.15.67:0/1787789060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10037070 con 0x7fdf200d0e00 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.822+0000 7fdf1dffb640 1 -- 172.21.15.67:0/1787789060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf10032440 con 0x7fdf200d0e00 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.822+0000 7fdf1dffb640 1 -- 172.21.15.67:0/1787789060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10032810 con 0x7fdf200d0e00 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 -- 172.21.15.67:0/1787789060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf200d0e00 msgr2=0x7fdf200d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 --2- 172.21.15.67:0/1787789060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf200d0e00 0x7fdf200d1210 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fdf2006d1e0 tx=0x7fdf1002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 -- 172.21.15.67:0/1787789060 shutdown_connections 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 --2- 172.21.15.67:0/1787789060 >> v2:172.21.15.176:3300/0 conn(0x7fdf200b7810 0x7fdf200b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 --2- 172.21.15.67:0/1787789060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf2006c5c0 0x7fdf2006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 --2- 172.21.15.67:0/1787789060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf200d0e00 0x7fdf200d1210 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 -- 172.21.15.67:0/1787789060 >> 172.21.15.67:0/1787789060 conn(0x7fdf200d5d90 msgr2=0x7fdf200d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 -- 172.21.15.67:0/1787789060 shutdown_connections 2024-04-25T15:26:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.823+0000 7fdf25451640 1 -- 172.21.15.67:0/1787789060 wait complete. 2024-04-25T15:26:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.825+0000 7fdf25451640 1 Processor -- start 2024-04-25T15:26:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.825+0000 7fdf25451640 1 -- start start 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf25451640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf2006c5c0 0x7fdf20150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf25451640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf200b7810 0x7fdf200ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf25451640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf200d0e00 0x7fdf200ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf25451640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf200b5320 con 0x7fdf2006c5c0 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf25451640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdf20160ad0 con 0x7fdf200b7810 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf25451640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdf20164e00 con 0x7fdf200d0e00 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf1effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf2006c5c0 0x7fdf20150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf1effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf2006c5c0 0x7fdf20150240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49542/0 (socket says 172.21.15.67:49542) 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.826+0000 7fdf1effd640 1 -- 172.21.15.67:0/2404282073 learned_addr learned my addr 172.21.15.67:0/2404282073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1e7fc640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf200b7810 0x7fdf200ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1f7fe640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf200d0e00 0x7fdf200ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1effd640 1 -- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf200d0e00 msgr2=0x7fdf200ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1effd640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf200d0e00 0x7fdf200ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1effd640 1 -- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf200b7810 msgr2=0x7fdf200ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1effd640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf200b7810 0x7fdf200ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1effd640 1 -- 172.21.15.67:0/2404282073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf201ea740 con 0x7fdf2006c5c0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf1effd640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf2006c5c0 0x7fdf20150240 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdf1002a220 tx=0x7fdf100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf077fe640 1 -- 172.21.15.67:0/2404282073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10041020 con 0x7fdf2006c5c0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf077fe640 1 -- 172.21.15.67:0/2404282073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf1003d620 con 0x7fdf2006c5c0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf201ea460 con 0x7fdf2006c5c0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf077fe640 1 -- 172.21.15.67:0/2404282073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10037040 con 0x7fdf2006c5c0 2024-04-25T15:26:39.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.827+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf201eae00 con 0x7fdf2006c5c0 2024-04-25T15:26:39.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.828+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf20050740 con 0x7fdf2006c5c0 2024-04-25T15:26:39.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.829+0000 7fdf077fe640 1 -- 172.21.15.67:0/2404282073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdf10044020 con 0x7fdf2006c5c0 2024-04-25T15:26:39.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.829+0000 7fdf077fe640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdf0c0744b0 0x7fdf0c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:39.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.829+0000 7fdf077fe640 1 -- 172.21.15.67:0/2404282073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5357+0+0 (secure 0 0 0) 0x7fdf100b56a0 con 0x7fdf2006c5c0 2024-04-25T15:26:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.830+0000 7fdf1e7fc640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdf0c0744b0 0x7fdf0c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.830+0000 7fdf1e7fc640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdf0c0744b0 0x7fdf0c0769c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdf200cbdb0 tx=0x7fdf14009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:39.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:39.834+0000 7fdf077fe640 1 -- 172.21.15.67:0/2404282073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdf1004b030 con 0x7fdf2006c5c0 2024-04-25T15:26:40.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.025+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdf200b5b40 con 0x7fdf2006c5c0 2024-04-25T15:26:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.677+0000 7fdf077fe640 1 -- 172.21.15.67:0/2404282073 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) ==== 169+0+0 (secure 0 0 0) 0x7fdf1007f940 con 0x7fdf2006c5c0 2024-04-25T15:26:40.678 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:26:40.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.679+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdf0c0744b0 msgr2=0x7fdf0c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.679+0000 7fdf25451640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdf0c0744b0 0x7fdf0c0769c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdf200cbdb0 tx=0x7fdf14009190 comp rx=0 tx=0).stop 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf2006c5c0 msgr2=0x7fdf20150240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf2006c5c0 0x7fdf20150240 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdf1002a220 tx=0x7fdf100046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 shutdown_connections 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdf0c0744b0 0x7fdf0c0769c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf200d0e00 0x7fdf200ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf200b7810 0x7fdf200ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 --2- 172.21.15.67:0/2404282073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf2006c5c0 0x7fdf20150240 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.680+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 >> 172.21.15.67:0/2404282073 conn(0x7fdf200d5d90 msgr2=0x7fdf200b6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:40.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.681+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 shutdown_connections 2024-04-25T15:26:40.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.681+0000 7fdf25451640 1 -- 172.21.15.67:0/2404282073 wait complete. 2024-04-25T15:26:40.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:26:40.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.817+0000 7fbfb69a8640 1 Processor -- start 2024-04-25T15:26:40.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.817+0000 7fbfb69a8640 1 -- start start 2024-04-25T15:26:40.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfb69a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0164b00 0x7fbfb00b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:40.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfb69a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbfb015df80 0x7fbfb00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:40.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfb69a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfb01615f0 con 0x7fbfb0160890 2024-04-25T15:26:40.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfb69a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbfb01613b0 con 0x7fbfb015df80 2024-04-25T15:26:40.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfb69a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb0164670 con 0x7fbfb0164b00 2024-04-25T15:26:40.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfaf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbfb015df80 0x7fbfb00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:40.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfb4f45640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbfb0160890 0x7fbfb0160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36242/0 (socket says 172.21.15.67:36242) 2024-04-25T15:26:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfb4f45640 1 -- 172.21.15.67:0/911049480 learned_addr learned my addr 172.21.15.67:0/911049480 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfaffff640 1 --2- 172.21.15.67:0/911049480 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0164b00 0x7fbfb00b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfaf7fe640 1 -- 172.21.15.67:0/911049480 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0164b00 msgr2=0x7fbfb00b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfaf7fe640 1 --2- 172.21.15.67:0/911049480 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0164b00 0x7fbfb00b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfaf7fe640 1 -- 172.21.15.67:0/911049480 >> v1:172.21.15.67:6789/0 conn(0x7fbfb0160890 legacy=0x7fbfb0160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfaeffd640 1 -- 172.21.15.67:0/911049480 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1512914536 0 0) 0x7fbfb01615f0 con 0x7fbfb0160890 2024-04-25T15:26:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.819+0000 7fbfaf7fe640 1 -- 172.21.15.67:0/911049480 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb0164020 con 0x7fbfb015df80 2024-04-25T15:26:40.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.820+0000 7fbfaf7fe640 1 --2- 172.21.15.67:0/911049480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb00b80c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b79e972b3e72d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:40.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.820+0000 7fbfaeffd640 1 -- 172.21.15.67:0/911049480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0037070 con 0x7fbfb015df80 2024-04-25T15:26:40.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.820+0000 7fbfaeffd640 1 -- 172.21.15.67:0/911049480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa0032440 con 0x7fbfb015df80 2024-04-25T15:26:40.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.820+0000 7fbfaeffd640 1 -- 172.21.15.67:0/911049480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0032810 con 0x7fbfb015df80 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 -- 172.21.15.67:0/911049480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 msgr2=0x7fbfb00b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/911049480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb00b80c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa002a180 comp rx=0 tx=0).stop 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 -- 172.21.15.67:0/911049480 shutdown_connections 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/911049480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb00b80c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/911049480 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0164b00 0x7fbfb00b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 -- 172.21.15.67:0/911049480 >> 172.21.15.67:0/911049480 conn(0x7fbfb00d2bc0 msgr2=0x7fbfb00d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 -- 172.21.15.67:0/911049480 shutdown_connections 2024-04-25T15:26:40.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.821+0000 7fbfb69a8640 1 -- 172.21.15.67:0/911049480 wait complete. 2024-04-25T15:26:40.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.823+0000 7fbfb69a8640 1 Processor -- start 2024-04-25T15:26:40.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.823+0000 7fbfb69a8640 1 -- start start 2024-04-25T15:26:40.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfb69a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb0154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:40.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfb69a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0160890 0x7fbfb0152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:40.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfb69a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfb0164b00 0x7fbfb0156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:40.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfb69a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb00b4a00 con 0x7fbfb015df80 2024-04-25T15:26:40.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfb69a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb00b4880 con 0x7fbfb0164b00 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfb69a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb00b4b80 con 0x7fbfb0160890 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfaffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb0154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfaffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb0154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49572/0 (socket says 172.21.15.67:49572) 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfaffff640 1 -- 172.21.15.67:0/2920162950 learned_addr learned my addr 172.21.15.67:0/2920162950 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfaf7fe640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0160890 0x7fbfb0152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.824+0000 7fbfb4f45640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfb0164b00 0x7fbfb0156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfaffff640 1 -- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0160890 msgr2=0x7fbfb0152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfaffff640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0160890 0x7fbfb0152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfaffff640 1 -- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfb0164b00 msgr2=0x7fbfb0156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfaffff640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfb0164b00 0x7fbfb0156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfaffff640 1 -- 172.21.15.67:0/2920162950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb0155380 con 0x7fbfb015df80 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfaffff640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb0154bc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400d620 tx=0x7fbfa400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfacff9640 1 -- 172.21.15.67:0/2920162950 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa400b620 con 0x7fbfb015df80 2024-04-25T15:26:40.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfacff9640 1 -- 172.21.15.67:0/2920162950 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa4019070 con 0x7fbfb015df80 2024-04-25T15:26:40.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb01618f0 con 0x7fbfb015df80 2024-04-25T15:26:40.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfacff9640 1 -- 172.21.15.67:0/2920162950 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa400ba00 con 0x7fbfb015df80 2024-04-25T15:26:40.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.825+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb0161eb0 con 0x7fbfb015df80 2024-04-25T15:26:40.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.826+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb0050740 con 0x7fbfb015df80 2024-04-25T15:26:40.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.827+0000 7fbfacff9640 1 -- 172.21.15.67:0/2920162950 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbfa400bba0 con 0x7fbfb015df80 2024-04-25T15:26:40.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.827+0000 7fbfacff9640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbf9c0744b0 0x7fbf9c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:40.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.827+0000 7fbfacff9640 1 -- 172.21.15.67:0/2920162950 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fbfa4095210 con 0x7fbfb015df80 2024-04-25T15:26:40.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.827+0000 7fbfaf7fe640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbf9c0744b0 0x7fbf9c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:40.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.828+0000 7fbfaf7fe640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbf9c0744b0 0x7fbf9c0769c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0034000 tx=0x7fbfa002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:40.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:40.832+0000 7fbfacff9640 1 -- 172.21.15.67:0/2920162950 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfa405f650 con 0x7fbfb015df80 2024-04-25T15:26:41.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.017+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fbfb00b8c00 con 0x7fbfb015df80 2024-04-25T15:26:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.021+0000 7fbfacff9640 1 -- 172.21.15.67:0/2920162950 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fbfa4063950 con 0x7fbfb015df80 2024-04-25T15:26:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbf9c0744b0 msgr2=0x7fbf9c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbf9c0744b0 0x7fbf9c0769c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0034000 tx=0x7fbfa002e000 comp rx=0 tx=0).stop 2024-04-25T15:26:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 msgr2=0x7fbfb0154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb0154bc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbfa400d620 tx=0x7fbfa400dae0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 shutdown_connections 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbf9c0744b0 0x7fbf9c0769c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfb0164b00 0x7fbfb0156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfb0160890 0x7fbfb0152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 --2- 172.21.15.67:0/2920162950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfb015df80 0x7fbfb0154bc0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 >> 172.21.15.67:0/2920162950 conn(0x7fbfb00d2bc0 msgr2=0x7fbfb015bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 shutdown_connections 2024-04-25T15:26:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.023+0000 7fbfb69a8640 1 -- 172.21.15.67:0/2920162950 wait complete. 2024-04-25T15:26:41.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values' 2024-04-25T15:26:41.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.145+0000 7fd28910d640 1 Processor -- start 2024-04-25T15:26:41.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.145+0000 7fd28910d640 1 -- start start 2024-04-25T15:26:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd28910d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2840d0e00 0x7fd2840d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd28910d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd2840d4a50 0x7fd2840d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd28910d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2840d1850 con 0x7fd28406c5c0 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd28910d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd284160de0 con 0x7fd2840d4a50 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd28910d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2841610c0 con 0x7fd2840d0e00 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd282d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2840d0e00 0x7fd2840d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd28359e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd2840d4a50 0x7fd2840d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd282d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2840d0e00 0x7fd2840d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:34294/0 (socket says 172.21.15.67:34294) 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd282d9d640 1 -- 172.21.15.67:0/3156197708 learned_addr learned my addr 172.21.15.67:0/3156197708 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.147+0000 7fd282d9d640 1 -- 172.21.15.67:0/3156197708 >> v2:172.21.15.176:3300/0 conn(0x7fd2840d4a50 msgr2=0x7fd2840d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.148+0000 7fd282d9d640 1 --2- 172.21.15.67:0/3156197708 >> v2:172.21.15.176:3300/0 conn(0x7fd2840d4a50 0x7fd2840d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.148+0000 7fd282d9d640 1 -- 172.21.15.67:0/3156197708 >> v1:172.21.15.176:6789/0 conn(0x7fd28406c5c0 legacy=0x7fd28406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.148+0000 7fd282d9d640 1 -- 172.21.15.67:0/3156197708 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2841613a0 con 0x7fd2840d0e00 2024-04-25T15:26:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.148+0000 7fd282d9d640 1 --2- 172.21.15.67:0/3156197708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2840d0e00 0x7fd2840d1210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009870 tx=0x7fd27c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a3141bfe6f838f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd281d9b640 1 -- 172.21.15.67:0/3156197708 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c037070 con 0x7fd2840d0e00 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd281d9b640 1 -- 172.21.15.67:0/3156197708 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27c032440 con 0x7fd2840d0e00 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd281d9b640 1 -- 172.21.15.67:0/3156197708 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c032810 con 0x7fd2840d0e00 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 -- 172.21.15.67:0/3156197708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2840d0e00 msgr2=0x7fd2840d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 --2- 172.21.15.67:0/3156197708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2840d0e00 0x7fd2840d1210 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009870 tx=0x7fd27c02a270 comp rx=0 tx=0).stop 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 -- 172.21.15.67:0/3156197708 shutdown_connections 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 --2- 172.21.15.67:0/3156197708 >> v2:172.21.15.176:3300/0 conn(0x7fd2840d4a50 0x7fd2840d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 --2- 172.21.15.67:0/3156197708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2840d0e00 0x7fd2840d1210 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 -- 172.21.15.67:0/3156197708 >> 172.21.15.67:0/3156197708 conn(0x7fd2840d5d90 msgr2=0x7fd2840d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 -- 172.21.15.67:0/3156197708 shutdown_connections 2024-04-25T15:26:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.149+0000 7fd28910d640 1 -- 172.21.15.67:0/3156197708 wait complete. 2024-04-25T15:26:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.151+0000 7fd28910d640 1 Processor -- start 2024-04-25T15:26:41.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.151+0000 7fd28910d640 1 -- start start 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28910d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd28406c5c0 0x7fd284157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28910d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2840d0e00 0x7fd2840cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28910d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2840d4a50 0x7fd28415bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28910d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd284164e00 con 0x7fd2840d0e00 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28910d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2840b5320 con 0x7fd2840d4a50 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28910d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd284164f80 con 0x7fd28406c5c0 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2840d0e00 0x7fd2840cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2840d0e00 0x7fd2840cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49582/0 (socket says 172.21.15.67:49582) 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 -- 172.21.15.67:0/1188320297 learned_addr learned my addr 172.21.15.67:0/1188320297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd282d9d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd28406c5c0 0x7fd284157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28359e640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2840d4a50 0x7fd28415bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 -- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd28406c5c0 msgr2=0x7fd284157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd28406c5c0 0x7fd284157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 -- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2840d4a50 msgr2=0x7fd28415bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2840d4a50 0x7fd28415bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 -- 172.21.15.67:0/1188320297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2840cead0 con 0x7fd2840d0e00 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd282d9d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd28406c5c0 0x7fd284157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.153+0000 7fd28259c640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2840d0e00 0x7fd2840cf890 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00c430 tx=0x7fd26c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.154+0000 7fd27b7fe640 1 -- 172.21.15.67:0/1188320297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c017070 con 0x7fd2840d0e00 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.154+0000 7fd27b7fe640 1 -- 172.21.15.67:0/1188320297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd26c0052d0 con 0x7fd2840d0e00 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.154+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2840cedb0 con 0x7fd2840d0e00 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.154+0000 7fd27b7fe640 1 -- 172.21.15.67:0/1188320297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c003830 con 0x7fd2840d0e00 2024-04-25T15:26:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.154+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2840ce360 con 0x7fd2840d0e00 2024-04-25T15:26:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.155+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd284050740 con 0x7fd2840d0e00 2024-04-25T15:26:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.156+0000 7fd27b7fe640 1 -- 172.21.15.67:0/1188320297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd26c01a070 con 0x7fd2840d0e00 2024-04-25T15:26:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.156+0000 7fd27b7fe640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2640744b0 0x7fd2640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.156+0000 7fd27b7fe640 1 -- 172.21.15.67:0/1188320297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fd26c0941a0 con 0x7fd2840d0e00 2024-04-25T15:26:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.156+0000 7fd282d9d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2640744b0 0x7fd2640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.157+0000 7fd282d9d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2640744b0 0x7fd2640769c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd27c034000 tx=0x7fd27c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:41.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.161+0000 7fd27b7fe640 1 -- 172.21.15.67:0/1188320297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd26c05e670 con 0x7fd2840d0e00 2024-04-25T15:26:41.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.416+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]} v 0) -- 0x7fd2840d4320 con 0x7fd2840d0e00 2024-04-25T15:26:41.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.693+0000 7fd27b7fe640 1 -- 172.21.15.67:0/1188320297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]}]=0 v91) ==== 160+0+0 (secure 0 0 0) 0x7fd26c062970 con 0x7fd2840d0e00 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2640744b0 msgr2=0x7fd2640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2640744b0 0x7fd2640769c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd27c034000 tx=0x7fd27c02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2840d0e00 msgr2=0x7fd2840cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2840d0e00 0x7fd2840cf890 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00c430 tx=0x7fd26c00c8f0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 shutdown_connections 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2640744b0 0x7fd2640769c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2840d4a50 0x7fd28415bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2840d0e00 0x7fd2840cf890 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 --2- 172.21.15.67:0/1188320297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd28406c5c0 0x7fd284157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 >> 172.21.15.67:0/1188320297 conn(0x7fd2840d5d90 msgr2=0x7fd2840b5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:41.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 shutdown_connections 2024-04-25T15:26:41.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.695+0000 7fd28910d640 1 -- 172.21.15.67:0/1188320297 wait complete. 2024-04-25T15:26:41.711 INFO:tasks.cephfs_test_runner:test: ... ok 2024-04-25T15:26:41.711 INFO:tasks.cephfs_test_runner:Starting test: test_human_readable_quota_invalid_values (tasks.cephfs.test_quota.TestQuota) test invalid values for ceph.quota.max_bytes 2024-04-25T15:26:41.711 INFO:tasks.cephfs_test_runner:test_human_readable_quota_invalid_values (tasks.cephfs.test_quota.TestQuota) 2024-04-25T15:26:41.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:41.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.828+0000 7fb6fe584640 1 Processor -- start 2024-04-25T15:26:41.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.828+0000 7fb6fe584640 1 -- start start 2024-04-25T15:26:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fe584640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d0be0 0x7fb6f80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fe584640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6f80d45e0 0x7fb6f80ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fe584640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f80d2330 con 0x7fb6f80d33d0 2024-04-25T15:26:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fe584640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb6f80d2610 con 0x7fb6f80d45e0 2024-04-25T15:26:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fe584640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb6f80d28f0 con 0x7fb6f80d0be0 2024-04-25T15:26:41.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fcb21640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6f80d45e0 0x7fb6f80ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6f7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb6f80d33d0 0x7fb6f80d04d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36254/0 (socket says 172.21.15.67:36254) 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6f7fff640 1 -- 172.21.15.67:0/819027591 learned_addr learned my addr 172.21.15.67:0/819027591 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6f77fe640 1 --2- 172.21.15.67:0/819027591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d0be0 0x7fb6f80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fcb21640 1 -- 172.21.15.67:0/819027591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d0be0 msgr2=0x7fb6f80b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fcb21640 1 --2- 172.21.15.67:0/819027591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d0be0 0x7fb6f80b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fcb21640 1 -- 172.21.15.67:0/819027591 >> v1:172.21.15.67:6789/0 conn(0x7fb6f80d33d0 legacy=0x7fb6f80d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6f6ffd640 1 -- 172.21.15.67:0/819027591 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915507388 0 0) 0x7fb6f80d2330 con 0x7fb6f80d33d0 2024-04-25T15:26:41.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.830+0000 7fb6fcb21640 1 -- 172.21.15.67:0/819027591 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f80d2bd0 con 0x7fb6f80d45e0 2024-04-25T15:26:41.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.831+0000 7fb6fcb21640 1 --2- 172.21.15.67:0/819027591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f80d45e0 0x7fb6f80ce110 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009d90 tx=0x7fb6f002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffd86333af6f0eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:41.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.831+0000 7fb6f6ffd640 1 -- 172.21.15.67:0/819027591 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0007740 con 0x7fb6f80d45e0 2024-04-25T15:26:41.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.831+0000 7fb6f6ffd640 1 -- 172.21.15.67:0/819027591 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f00078e0 con 0x7fb6f80d45e0 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.831+0000 7fb6f6ffd640 1 -- 172.21.15.67:0/819027591 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0007cb0 con 0x7fb6f80d45e0 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 -- 172.21.15.67:0/819027591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f80d45e0 msgr2=0x7fb6f80ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 --2- 172.21.15.67:0/819027591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f80d45e0 0x7fb6f80ce110 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009d90 tx=0x7fb6f002a660 comp rx=0 tx=0).stop 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 -- 172.21.15.67:0/819027591 shutdown_connections 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 --2- 172.21.15.67:0/819027591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f80d45e0 0x7fb6f80ce110 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 --2- 172.21.15.67:0/819027591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d0be0 0x7fb6f80b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 -- 172.21.15.67:0/819027591 >> 172.21.15.67:0/819027591 conn(0x7fb6f80d3ac0 msgr2=0x7fb6f80d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 -- 172.21.15.67:0/819027591 shutdown_connections 2024-04-25T15:26:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.832+0000 7fb6fe584640 1 -- 172.21.15.67:0/819027591 wait complete. 2024-04-25T15:26:41.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.834+0000 7fb6fe584640 1 Processor -- start 2024-04-25T15:26:41.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.834+0000 7fb6fe584640 1 -- start start 2024-04-25T15:26:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fe584640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6f80d33d0 0x7fb6f81f9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fe584640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d45e0 0x7fb6f80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fe584640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f8153360 0x7fb6f80ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fe584640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f80d2040 con 0x7fb6f8153360 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fe584640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f80d1ec0 con 0x7fb6f80d33d0 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fe584640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb6f806dc70 con 0x7fb6f80d45e0 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fcb21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f8153360 0x7fb6f80ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fcb21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f8153360 0x7fb6f80ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49598/0 (socket says 172.21.15.67:49598) 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.835+0000 7fb6fcb21640 1 -- 172.21.15.67:0/2352439381 learned_addr learned my addr 172.21.15.67:0/2352439381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6f77fe640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d45e0 0x7fb6f80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6f7fff640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6f80d33d0 0x7fb6f81f9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fcb21640 1 -- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d45e0 msgr2=0x7fb6f80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fcb21640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d45e0 0x7fb6f80ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fcb21640 1 -- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6f80d33d0 msgr2=0x7fb6f81f9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:41.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fcb21640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6f80d33d0 0x7fb6f81f9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fcb21640 1 -- 172.21.15.67:0/2352439381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f8159d40 con 0x7fb6f8153360 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6f77fe640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d45e0 0x7fb6f80ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6f7fff640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6f80d33d0 0x7fb6f81f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fcb21640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f8153360 0x7fb6f80ce630 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009d90 tx=0x7fb6f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6f4ff9640 1 -- 172.21.15.67:0/2352439381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f003f070 con 0x7fb6f8153360 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6f4ff9640 1 -- 172.21.15.67:0/2352439381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f0031bc0 con 0x7fb6f8153360 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f815ccd0 con 0x7fb6f8153360 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6f4ff9640 1 -- 172.21.15.67:0/2352439381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0009450 con 0x7fb6f8153360 2024-04-25T15:26:41.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.836+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f81fa1b0 con 0x7fb6f8153360 2024-04-25T15:26:41.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.837+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6f81f9c40 con 0x7fb6f8153360 2024-04-25T15:26:41.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.838+0000 7fb6f4ff9640 1 -- 172.21.15.67:0/2352439381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb6f003b410 con 0x7fb6f8153360 2024-04-25T15:26:41.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.838+0000 7fb6f4ff9640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6d80744e0 0x7fb6d80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:41.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.838+0000 7fb6f4ff9640 1 -- 172.21.15.67:0/2352439381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fb6f00b6360 con 0x7fb6f8153360 2024-04-25T15:26:41.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.838+0000 7fb6f7fff640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6d80744e0 0x7fb6d80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:41.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.839+0000 7fb6f7fff640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6d80744e0 0x7fb6d80769f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb6f806f8b0 tx=0x7fb6e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:41.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:41.843+0000 7fb6f4ff9640 1 -- 172.21.15.67:0/2352439381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6f00807a0 con 0x7fb6f8153360 2024-04-25T15:26:42.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.031+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6f8050740 con 0x7fb6f8153360 2024-04-25T15:26:42.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.031+0000 7fb6f4ff9640 1 -- 172.21.15.67:0/2352439381 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+12984 (secure 0 0 0) 0x7fb6f0087e20 con 0x7fb6f8153360 2024-04-25T15:26:42.032 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:42.032 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":36,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:40.676616+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/2490033101":"2024-04-26T15:26:37.636539+0000","172.21.15.67:6833/2490033101":"2024-04-26T15:26:37.636539+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.033+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6d80744e0 msgr2=0x7fb6d80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.033+0000 7fb6fe584640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6d80744e0 0x7fb6d80769f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb6f806f8b0 tx=0x7fb6e0009040 comp rx=0 tx=0).stop 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.033+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f8153360 msgr2=0x7fb6f80ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.033+0000 7fb6fe584640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f8153360 0x7fb6f80ce630 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009d90 tx=0x7fb6f0004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 shutdown_connections 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6d80744e0 0x7fb6d80769f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6f8153360 0x7fb6f80ce630 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6f80d45e0 0x7fb6f80ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 --2- 172.21.15.67:0/2352439381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6f80d33d0 0x7fb6f81f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 >> 172.21.15.67:0/2352439381 conn(0x7fb6f80d3ac0 msgr2=0x7fb6f8072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 shutdown_connections 2024-04-25T15:26:42.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.034+0000 7fb6fe584640 1 -- 172.21.15.67:0/2352439381 wait complete. 2024-04-25T15:26:42.046 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:42.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.160+0000 7f1a7800c640 1 Processor -- start 2024-04-25T15:26:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.160+0000 7f1a7800c640 1 -- start start 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.161+0000 7f1a7800c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a7006ccd0 0x7f1a700d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a7800c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1a700d4a50 0x7f1a700d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a7800c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a700d1850 con 0x7f1a700d0e00 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a7800c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1a70160de0 con 0x7f1a7006ccd0 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a7800c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1a701610c0 con 0x7f1a700d4a50 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a755a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a7006ccd0 0x7f1a700d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a75da8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1a700d0e00 0x7f1a7006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36256/0 (socket says 172.21.15.67:36256) 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a75da8640 1 -- 172.21.15.67:0/32149911 learned_addr learned my addr 172.21.15.67:0/32149911 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a765a9640 1 --2- 172.21.15.67:0/32149911 >> v2:172.21.15.176:3300/0 conn(0x7f1a700d4a50 0x7f1a700d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a755a7640 1 -- 172.21.15.67:0/32149911 >> v2:172.21.15.176:3300/0 conn(0x7f1a700d4a50 msgr2=0x7f1a700d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a755a7640 1 --2- 172.21.15.67:0/32149911 >> v2:172.21.15.176:3300/0 conn(0x7f1a700d4a50 0x7f1a700d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a755a7640 1 -- 172.21.15.67:0/32149911 >> v1:172.21.15.67:6789/0 conn(0x7f1a700d0e00 legacy=0x7f1a7006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a755a7640 1 -- 172.21.15.67:0/32149911 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a701613a0 con 0x7f1a7006ccd0 2024-04-25T15:26:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a74da6640 1 -- 172.21.15.67:0/32149911 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515310945 0 0) 0x7f1a700d1850 con 0x7f1a700d0e00 2024-04-25T15:26:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.162+0000 7f1a765a9640 1 --2- 172.21.15.67:0/32149911 >> v2:172.21.15.176:3300/0 conn(0x7f1a700d4a50 0x7f1a700d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:42.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.163+0000 7f1a755a7640 1 --2- 172.21.15.67:0/32149911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a7006ccd0 0x7f1a700d1210 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1a700d1750 tx=0x7f1a6402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a7fe20419b2766 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:42.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.163+0000 7f1a74da6640 1 -- 172.21.15.67:0/32149911 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64037070 con 0x7f1a7006ccd0 2024-04-25T15:26:42.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.163+0000 7f1a74da6640 1 -- 172.21.15.67:0/32149911 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a64032440 con 0x7f1a7006ccd0 2024-04-25T15:26:42.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.163+0000 7f1a74da6640 1 -- 172.21.15.67:0/32149911 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6402ae60 con 0x7f1a7006ccd0 2024-04-25T15:26:42.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 -- 172.21.15.67:0/32149911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a7006ccd0 msgr2=0x7f1a700d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 --2- 172.21.15.67:0/32149911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a7006ccd0 0x7f1a700d1210 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1a700d1750 tx=0x7f1a6402a110 comp rx=0 tx=0).stop 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 -- 172.21.15.67:0/32149911 shutdown_connections 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 --2- 172.21.15.67:0/32149911 >> v2:172.21.15.176:3300/0 conn(0x7f1a700d4a50 0x7f1a700d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 --2- 172.21.15.67:0/32149911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a7006ccd0 0x7f1a700d1210 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 -- 172.21.15.67:0/32149911 >> 172.21.15.67:0/32149911 conn(0x7f1a700d5d90 msgr2=0x7f1a700d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 -- 172.21.15.67:0/32149911 shutdown_connections 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.164+0000 7f1a7800c640 1 -- 172.21.15.67:0/32149911 wait complete. 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.165+0000 7f1a7800c640 1 Processor -- start 2024-04-25T15:26:42.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.166+0000 7f1a7800c640 1 -- start start 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a7800c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a700d0e00 0x7f1a700ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a7800c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a700d4a50 0x7f1a700b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a7800c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a700b5c30 0x7f1a700b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a7800c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a70164e00 con 0x7f1a700d0e00 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a7800c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700b5320 con 0x7f1a700b5c30 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a7800c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a70164f80 con 0x7f1a700d4a50 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a700d0e00 0x7f1a700ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a700d0e00 0x7f1a700ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49628/0 (socket says 172.21.15.67:49628) 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 -- 172.21.15.67:0/2751403593 learned_addr learned my addr 172.21.15.67:0/2751403593 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a755a7640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a700d4a50 0x7f1a700b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a765a9640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a700b5c30 0x7f1a700b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 -- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a700d4a50 msgr2=0x7f1a700b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a700d4a50 0x7f1a700b56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 -- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a700b5c30 msgr2=0x7f1a700b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a700b5c30 0x7f1a700b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 -- 172.21.15.67:0/2751403593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a700b83a0 con 0x7f1a700d0e00 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a765a9640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a700b5c30 0x7f1a700b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.167+0000 7f1a75da8640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a700d0e00 0x7f1a700ccdd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000c7f0 tx=0x7f1a6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.168+0000 7f1a6e7fc640 1 -- 172.21.15.67:0/2751403593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a60018070 con 0x7f1a700d0e00 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.168+0000 7f1a6e7fc640 1 -- 172.21.15.67:0/2751403593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6000a640 con 0x7f1a700d0e00 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.168+0000 7f1a6e7fc640 1 -- 172.21.15.67:0/2751403593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000aa30 con 0x7f1a700d0e00 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.168+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a700cd9b0 con 0x7f1a700d0e00 2024-04-25T15:26:42.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.168+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a700cdf70 con 0x7f1a700d0e00 2024-04-25T15:26:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.169+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a70050740 con 0x7f1a700d0e00 2024-04-25T15:26:42.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.169+0000 7f1a6e7fc640 1 -- 172.21.15.67:0/2751403593 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1a6000abd0 con 0x7f1a700d0e00 2024-04-25T15:26:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.170+0000 7f1a6e7fc640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a5c0744e0 0x7f1a5c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.170+0000 7f1a6e7fc640 1 -- 172.21.15.67:0/2751403593 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f1a600942a0 con 0x7f1a700d0e00 2024-04-25T15:26:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.170+0000 7f1a755a7640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a5c0744e0 0x7f1a5c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.171+0000 7f1a755a7640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a5c0744e0 0x7f1a5c0769f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1a64034000 tx=0x7f1a6402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:42.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.175+0000 7f1a6e7fc640 1 -- 172.21.15.67:0/2751403593 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a6005e800 con 0x7f1a700d0e00 2024-04-25T15:26:42.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.361+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1a700d4320 con 0x7f1a700d0e00 2024-04-25T15:26:42.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.361+0000 7f1a6e7fc640 1 -- 172.21.15.67:0/2751403593 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f1a60062b00 con 0x7f1a700d0e00 2024-04-25T15:26:42.362 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a5c0744e0 msgr2=0x7f1a5c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a5c0744e0 0x7f1a5c0769f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1a64034000 tx=0x7f1a6402e000 comp rx=0 tx=0).stop 2024-04-25T15:26:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a700d0e00 msgr2=0x7f1a700ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a700d0e00 0x7f1a700ccdd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000c7f0 tx=0x7f1a6000ccb0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 shutdown_connections 2024-04-25T15:26:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a5c0744e0 0x7f1a5c0769f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a700b5c30 0x7f1a700b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a700d4a50 0x7f1a700b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 --2- 172.21.15.67:0/2751403593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a700d0e00 0x7f1a700ccdd0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 >> 172.21.15.67:0/2751403593 conn(0x7f1a700d5d90 msgr2=0x7f1a70160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 shutdown_connections 2024-04-25T15:26:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.363+0000 7f1a7800c640 1 -- 172.21.15.67:0/2751403593 wait complete. 2024-04-25T15:26:42.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values' 2024-04-25T15:26:42.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.486+0000 7fd557c27640 1 Processor -- start 2024-04-25T15:26:42.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.486+0000 7fd557c27640 1 -- start start 2024-04-25T15:26:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.487+0000 7fd557c27640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd5500d4130 0x7fd5500d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.487+0000 7fd557c27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd55006dcf0 0x7fd5500d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.487+0000 7fd557c27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5500b3c10 con 0x7fd55006d8e0 2024-04-25T15:26:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.487+0000 7fd557c27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd5500b9520 con 0x7fd55006dcf0 2024-04-25T15:26:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.487+0000 7fd557c27640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd5500b4820 con 0x7fd5500d4130 2024-04-25T15:26:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5561c4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd55006dcf0 0x7fd5500d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5559c3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd55006d8e0 0x7fd5500d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36262/0 (socket says 172.21.15.67:36262) 2024-04-25T15:26:42.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5559c3640 1 -- 172.21.15.67:0/2873083509 learned_addr learned my addr 172.21.15.67:0/2873083509 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:42.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5551c2640 1 --2- 172.21.15.67:0/2873083509 >> v2:172.21.15.176:3300/0 conn(0x7fd5500d4130 0x7fd5500d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5561c4640 1 -- 172.21.15.67:0/2873083509 >> v2:172.21.15.176:3300/0 conn(0x7fd5500d4130 msgr2=0x7fd5500d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:42.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5561c4640 1 --2- 172.21.15.67:0/2873083509 >> v2:172.21.15.176:3300/0 conn(0x7fd5500d4130 0x7fd5500d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5561c4640 1 -- 172.21.15.67:0/2873083509 >> v1:172.21.15.67:6789/0 conn(0x7fd55006d8e0 legacy=0x7fd5500d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.488+0000 7fd5561c4640 1 -- 172.21.15.67:0/2873083509 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5500b3ef0 con 0x7fd55006dcf0 2024-04-25T15:26:42.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.489+0000 7fd5561c4640 1 --2- 172.21.15.67:0/2873083509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd55006dcf0 0x7fd5500d22a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd55006f790 tx=0x7fd54802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d8997a4f276fab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:42.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.489+0000 7fd5549c1640 1 -- 172.21.15.67:0/2873083509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548007740 con 0x7fd55006dcf0 2024-04-25T15:26:42.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.489+0000 7fd5549c1640 1 -- 172.21.15.67:0/2873083509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5480078e0 con 0x7fd55006dcf0 2024-04-25T15:26:42.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.489+0000 7fd5549c1640 1 -- 172.21.15.67:0/2873083509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548007cb0 con 0x7fd55006dcf0 2024-04-25T15:26:42.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.489+0000 7fd557c27640 1 -- 172.21.15.67:0/2873083509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd55006dcf0 msgr2=0x7fd5500d22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.489+0000 7fd557c27640 1 --2- 172.21.15.67:0/2873083509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd55006dcf0 0x7fd5500d22a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd55006f790 tx=0x7fd54802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.490+0000 7fd557c27640 1 -- 172.21.15.67:0/2873083509 shutdown_connections 2024-04-25T15:26:42.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.490+0000 7fd557c27640 1 --2- 172.21.15.67:0/2873083509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd55006dcf0 0x7fd5500d22a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.490+0000 7fd557c27640 1 --2- 172.21.15.67:0/2873083509 >> v2:172.21.15.176:3300/0 conn(0x7fd5500d4130 0x7fd5500d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.490+0000 7fd557c27640 1 -- 172.21.15.67:0/2873083509 >> 172.21.15.67:0/2873083509 conn(0x7fd55008d110 msgr2=0x7fd55008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:42.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.490+0000 7fd557c27640 1 -- 172.21.15.67:0/2873083509 shutdown_connections 2024-04-25T15:26:42.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.490+0000 7fd557c27640 1 -- 172.21.15.67:0/2873083509 wait complete. 2024-04-25T15:26:42.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.491+0000 7fd557c27640 1 Processor -- start 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.491+0000 7fd557c27640 1 -- start start 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd557c27640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd55006d8e0 0x7fd5501e84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd557c27640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd55006dcf0 0x7fd550152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd557c27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550152940 0x7fd550152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd557c27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5500b38e0 con 0x7fd550152940 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd557c27640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd5500b44b0 con 0x7fd55006dcf0 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd557c27640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd5500d27e0 con 0x7fd55006d8e0 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550152940 0x7fd550152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550152940 0x7fd550152d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49650/0 (socket says 172.21.15.67:49650) 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 -- 172.21.15.67:0/2002973332 learned_addr learned my addr 172.21.15.67:0/2002973332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5559c3640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd55006d8e0 0x7fd5501e84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5551c2640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd55006dcf0 0x7fd550152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 -- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd55006d8e0 msgr2=0x7fd5501e84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd55006d8e0 0x7fd5501e84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 -- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd55006dcf0 msgr2=0x7fd550152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd55006dcf0 0x7fd550152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 -- 172.21.15.67:0/2002973332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd550152640 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5561c4640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550152940 0x7fd550152d50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd54802abe0 tx=0x7fd548004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.493+0000 7fd5467fc640 1 -- 172.21.15.67:0/2002973332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54803f070 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.494+0000 7fd5467fc640 1 -- 172.21.15.67:0/2002973332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd548031c60 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.494+0000 7fd5467fc640 1 -- 172.21.15.67:0/2002973332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548043650 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.494+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5501e7420 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.494+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5501e79e0 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.495+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5501e7170 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.495+0000 7fd5467fc640 1 -- 172.21.15.67:0/2002973332 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd54803b410 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.496+0000 7fd5467fc640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5340744e0 0x7fd5340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.496+0000 7fd5467fc640 1 -- 172.21.15.67:0/2002973332 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fd5480b6490 con 0x7fd550152940 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.496+0000 7fd5559c3640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5340744e0 0x7fd5340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:42.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.497+0000 7fd5559c3640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5340744e0 0x7fd5340769f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd54c000f60 tx=0x7fd54c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:42.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.500+0000 7fd5467fc640 1 -- 172.21.15.67:0/2002973332 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd548080f20 con 0x7fd550152940 2024-04-25T15:26:42.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:42.766+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]} v 0) -- 0x7fd550050740 con 0x7fd550152940 2024-04-25T15:26:43.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.701+0000 7fd5467fc640 1 -- 172.21.15.67:0/2002973332 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]}]=0 v93) ==== 156+0+0 (secure 0 0 0) 0x7fd548080820 con 0x7fd550152940 2024-04-25T15:26:43.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5340744e0 msgr2=0x7fd5340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:43.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5340744e0 0x7fd5340769f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd54c000f60 tx=0x7fd54c009040 comp rx=0 tx=0).stop 2024-04-25T15:26:43.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550152940 msgr2=0x7fd550152d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550152940 0x7fd550152d50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd54802abe0 tx=0x7fd548004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 shutdown_connections 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5340744e0 0x7fd5340769f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd550152940 0x7fd550152d50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd55006dcf0 0x7fd550152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 --2- 172.21.15.67:0/2002973332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd55006d8e0 0x7fd5501e84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 >> 172.21.15.67:0/2002973332 conn(0x7fd55008d110 msgr2=0x7fd5500b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 shutdown_connections 2024-04-25T15:26:43.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.704+0000 7fd557c27640 1 -- 172.21.15.67:0/2002973332 wait complete. 2024-04-25T15:26:43.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:26:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.840+0000 7ffaa3842640 1 Processor -- start 2024-04-25T15:26:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.840+0000 7ffaa3842640 1 -- start start 2024-04-25T15:26:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.841+0000 7ffaa3842640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 0x7ffa9c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.841+0000 7ffaa3842640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa9c0d3170 con 0x7ffa9c164ad0 2024-04-25T15:26:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.841+0000 7ffaa3842640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa9c0d3450 con 0x7ffa9c164640 2024-04-25T15:26:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.841+0000 7ffaa3842640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa9c0d2f40 con 0x7ffa9c0b2ca0 2024-04-25T15:26:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.841+0000 7ffaa15de640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffa9c164ad0 0x7ffa9c163f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36274/0 (socket says 172.21.15.67:36274) 2024-04-25T15:26:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.841+0000 7ffaa15de640 1 -- 172.21.15.67:0/181935911 learned_addr learned my addr 172.21.15.67:0/181935911 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:43.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffaa1ddf640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 0x7ffa9c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 39669892 0 0) 0x7ffa9c0d3170 con 0x7ffa9c164ad0 2024-04-25T15:26:43.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa88003520 con 0x7ffa9c164ad0 2024-04-25T15:26:43.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3179015841 0 0) 0x7ffa88003520 con 0x7ffa9c164ad0 2024-04-25T15:26:43.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 msgr2=0x7ffa9c0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 0x7ffa9c0b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 >> v1:172.21.15.176:6789/0 conn(0x7ffa9c164640 legacy=0x7ffa9c0b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9c06d6d0 con 0x7ffa9c164ad0 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7ffa98001670 con 0x7ffa9c164ad0 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 >> v1:172.21.15.67:6789/0 conn(0x7ffa9c164ad0 legacy=0x7ffa9c163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 0x7ffa880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa88004d40 0x7ffa88005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8800baa0 con 0x7ffa88003f40 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8800bc50 con 0x7ffa88004d40 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa8800be50 con 0x7ffa9c0b2ca0 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffaa0ddd640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffaa15de640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 0x7ffa880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.842+0000 7ffaa1ddf640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa88004d40 0x7ffa88005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.843+0000 7ffaa0ddd640 1 -- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 msgr2=0x7ffa880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.843+0000 7ffaa0ddd640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b2ca0 0x7ffa880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.843+0000 7ffaa0ddd640 1 -- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa88004d40 msgr2=0x7ffa88005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.843+0000 7ffaa0ddd640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa88004d40 0x7ffa88005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.843+0000 7ffaa0ddd640 1 -- 172.21.15.67:0/181935911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa88003520 con 0x7ffa88003f40 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.843+0000 7ffaa0ddd640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa88004800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c001860 tx=0x7ffa88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961f2486d4b4098e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c037070 con 0x7ffa88003f40 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa8c02abf0 con 0x7ffa88003f40 2024-04-25T15:26:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffa93fff640 1 -- 172.21.15.67:0/181935911 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c001430 con 0x7ffa88003f40 2024-04-25T15:26:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffaa3842640 1 -- 172.21.15.67:0/181935911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 msgr2=0x7ffa88004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffaa3842640 1 --2- 172.21.15.67:0/181935911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa88004800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c001860 tx=0x7ffa88005830 comp rx=0 tx=0).stop 2024-04-25T15:26:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffaa1ddf640 1 -- 172.21.15.67:0/181935911 reap_dead start 2024-04-25T15:26:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffaa3842640 1 -- 172.21.15.67:0/181935911 shutdown_connections 2024-04-25T15:26:43.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffaa3842640 1 -- 172.21.15.67:0/181935911 >> 172.21.15.67:0/181935911 conn(0x7ffa9c0d3aa0 msgr2=0x7ffa9c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffaa3842640 1 -- 172.21.15.67:0/181935911 shutdown_connections 2024-04-25T15:26:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.844+0000 7ffaa3842640 1 -- 172.21.15.67:0/181935911 wait complete. 2024-04-25T15:26:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.846+0000 7ffaa3842640 1 Processor -- start 2024-04-25T15:26:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.846+0000 7ffaa3842640 1 -- start start 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa3842640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa9c0b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa3842640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa9c0c9ec0 0x7ffa9c0c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa3842640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b5780 0x7ffa9c0c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa3842640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c0b30b0 con 0x7ffa88003f40 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa3842640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c164ee0 con 0x7ffa9c0c9ec0 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa3842640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa9c0b9430 con 0x7ffa9c0b5780 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa9c0b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa9c0b3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49662/0 (socket says 172.21.15.67:49662) 2024-04-25T15:26:43.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 -- 172.21.15.67:0/3971886959 learned_addr learned my addr 172.21.15.67:0/3971886959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa1ddf640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b5780 0x7ffa9c0c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa0ddd640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa9c0c9ec0 0x7ffa9c0c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 -- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b5780 msgr2=0x7ffa9c0c80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b5780 0x7ffa9c0c80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 -- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa9c0c9ec0 msgr2=0x7ffa9c0c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa9c0c9ec0 0x7ffa9c0c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 -- 172.21.15.67:0/3971886959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9c0b3d70 con 0x7ffa88003f40 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa15de640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa9c0b3400 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa98007310 tx=0x7ffa980077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffa91ffb640 1 -- 172.21.15.67:0/3971886959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800f940 con 0x7ffa88003f40 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffa91ffb640 1 -- 172.21.15.67:0/3971886959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa98018070 con 0x7ffa88003f40 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9c0b4050 con 0x7ffa88003f40 2024-04-25T15:26:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.848+0000 7ffa91ffb640 1 -- 172.21.15.67:0/3971886959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800fd60 con 0x7ffa88003f40 2024-04-25T15:26:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.849+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa9c0c7670 con 0x7ffa88003f40 2024-04-25T15:26:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.850+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa9c0d4770 con 0x7ffa88003f40 2024-04-25T15:26:43.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.850+0000 7ffa91ffb640 1 -- 172.21.15.67:0/3971886959 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffa9801d460 con 0x7ffa88003f40 2024-04-25T15:26:43.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.850+0000 7ffa91ffb640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa800744e0 0x7ffa800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:43.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.851+0000 7ffa91ffb640 1 -- 172.21.15.67:0/3971886959 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7ffa98094f60 con 0x7ffa88003f40 2024-04-25T15:26:43.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.851+0000 7ffaa0ddd640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa800744e0 0x7ffa800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:43.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.851+0000 7ffaa0ddd640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa800744e0 0x7ffa800769f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c034000 tx=0x7ffa8c009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:43.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:43.855+0000 7ffa91ffb640 1 -- 172.21.15.67:0/3971886959 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa9805fa80 con 0x7ffa88003f40 2024-04-25T15:26:44.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.041+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ffa9c068860 con 0x7ffa88003f40 2024-04-25T15:26:44.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.047+0000 7ffa91ffb640 1 -- 172.21.15.67:0/3971886959 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7ffa9805f380 con 0x7ffa88003f40 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa800744e0 msgr2=0x7ffa800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa800744e0 0x7ffa800769f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c034000 tx=0x7ffa8c009030 comp rx=0 tx=0).stop 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 msgr2=0x7ffa9c0b3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa9c0b3400 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa98007310 tx=0x7ffa980077d0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 shutdown_connections 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa800744e0 0x7ffa800769f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa9c0b5780 0x7ffa9c0c80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa9c0c9ec0 0x7ffa9c0c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 --2- 172.21.15.67:0/3971886959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa88003f40 0x7ffa9c0b3400 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 >> 172.21.15.67:0/3971886959 conn(0x7ffa9c0d3aa0 msgr2=0x7ffa9c0cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 shutdown_connections 2024-04-25T15:26:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.049+0000 7ffaa3842640 1 -- 172.21.15.67:0/3971886959 wait complete. 2024-04-25T15:26:44.060 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:26:44.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:26:44.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:26:44.142 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:26:44.142 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:44.142 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:44.157 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:26:44.158 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:44.158 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:44.199 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:26:44.199 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:44.199 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:44.215 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:26:44.215 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:44.215 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:44.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:26:44.257 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:26:44.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:44.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.367+0000 7fce63b64640 1 Processor -- start 2024-04-25T15:26:44.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.367+0000 7fce63b64640 1 -- start start 2024-04-25T15:26:44.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce63b64640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fce5c06e8f0 0x7fce5c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce63b64640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce5c06d870 con 0x7fce5c0b4830 2024-04-25T15:26:44.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce63b64640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce5c06db10 con 0x7fce5c0d4570 2024-04-25T15:26:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce63b64640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fce5c0b4c40 con 0x7fce5c06e8f0 2024-04-25T15:26:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce610ff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fce5c0b4830 0x7fce5c0b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36276/0 (socket says 172.21.15.67:36276) 2024-04-25T15:26:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce610ff640 1 -- 172.21.15.67:0/639630423 learned_addr learned my addr 172.21.15.67:0/639630423 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:44.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543905159 0 0) 0x7fce5c06d870 con 0x7fce5c0b4830 2024-04-25T15:26:44.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce62101640 1 --2- 172.21.15.67:0/639630423 >> v2:172.21.15.176:3300/0 conn(0x7fce5c06e8f0 0x7fce5c0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce44003520 con 0x7fce5c0b4830 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2344502401 0 0) 0x7fce44003520 con 0x7fce5c0b4830 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 >> v2:172.21.15.176:3300/0 conn(0x7fce5c06e8f0 msgr2=0x7fce5c0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 --2- 172.21.15.67:0/639630423 >> v2:172.21.15.176:3300/0 conn(0x7fce5c06e8f0 0x7fce5c0b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 >> v1:172.21.15.176:6789/0 conn(0x7fce5c0d4570 legacy=0x7fce5c0d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c06e0f0 con 0x7fce5c0b4830 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fce50001670 con 0x7fce5c0b4830 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 >> v1:172.21.15.67:6789/0 conn(0x7fce5c0b4830 legacy=0x7fce5c0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce440039b0 0x7fce44003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce44004800 0x7fce44004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44005250 0x7fce4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400c1b0 con 0x7fce44004800 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fce4400c360 con 0x7fce44005250 2024-04-25T15:26:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.369+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fce4400c560 con 0x7fce440039b0 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce610ff640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce44004800 0x7fce44004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce62101640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44005250 0x7fce4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce61900640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce440039b0 0x7fce44003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce610ff640 1 -- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce440039b0 msgr2=0x7fce44003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce610ff640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce440039b0 0x7fce44003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce610ff640 1 -- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44005250 msgr2=0x7fce4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce610ff640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44005250 0x7fce4400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce610ff640 1 -- 172.21.15.67:0/639630423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce44003520 con 0x7fce44004800 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.370+0000 7fce62101640 1 -- 172.21.15.67:0/639630423 reap_dead start 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce610ff640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce44004800 0x7fce44004c10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fce5c074cf0 tx=0x7fce44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de8e6361321320f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50007d10 con 0x7fce44004800 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce50005ad0 con 0x7fce44004800 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce608fe640 1 -- 172.21.15.67:0/639630423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50007070 con 0x7fce44004800 2024-04-25T15:26:44.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce63b64640 1 -- 172.21.15.67:0/639630423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce44004800 msgr2=0x7fce44004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce63b64640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce44004800 0x7fce44004c10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fce5c074cf0 tx=0x7fce44005830 comp rx=0 tx=0).stop 2024-04-25T15:26:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce63b64640 1 -- 172.21.15.67:0/639630423 shutdown_connections 2024-04-25T15:26:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce63b64640 1 --2- 172.21.15.67:0/639630423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce44004800 0x7fce44004c10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.371+0000 7fce63b64640 1 -- 172.21.15.67:0/639630423 >> 172.21.15.67:0/639630423 conn(0x7fce5c0d3a50 msgr2=0x7fce5c0d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.372+0000 7fce63b64640 1 -- 172.21.15.67:0/639630423 shutdown_connections 2024-04-25T15:26:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.372+0000 7fce63b64640 1 -- 172.21.15.67:0/639630423 wait complete. 2024-04-25T15:26:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.373+0000 7fce63b64640 1 Processor -- start 2024-04-25T15:26:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.373+0000 7fce63b64640 1 -- start start 2024-04-25T15:26:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44004800 0x7fce5c15aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce5c14bf70 0x7fce5c15e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5c15ee60 0x7fce5c15f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c06ed00 con 0x7fce5c15ee60 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c06e460 con 0x7fce44004800 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fce5c0d04b0 con 0x7fce5c14bf70 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5c15ee60 0x7fce5c15f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5c15ee60 0x7fce5c15f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49672/0 (socket says 172.21.15.67:49672) 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 -- 172.21.15.67:0/2364497770 learned_addr learned my addr 172.21.15.67:0/2364497770 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce61900640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44004800 0x7fce5c15aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce610ff640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce5c14bf70 0x7fce5c15e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 -- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce5c14bf70 msgr2=0x7fce5c15e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce5c14bf70 0x7fce5c15e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 -- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44004800 msgr2=0x7fce5c15aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44004800 0x7fce5c15aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 -- 172.21.15.67:0/2364497770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c15f7b0 con 0x7fce5c15ee60 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce62101640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5c15ee60 0x7fce5c15f270 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fce5c06e410 tx=0x7fce4c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce5a7fc640 1 -- 172.21.15.67:0/2364497770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00fe10 con 0x7fce5c15ee60 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce5a7fc640 1 -- 172.21.15.67:0/2364497770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4c01d070 con 0x7fce5c15ee60 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5c15b400 con 0x7fce5c15ee60 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce5a7fc640 1 -- 172.21.15.67:0/2364497770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c015670 con 0x7fce5c15ee60 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.375+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5c15b9c0 con 0x7fce5c15ee60 2024-04-25T15:26:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.376+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce5c050740 con 0x7fce5c15ee60 2024-04-25T15:26:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.377+0000 7fce5a7fc640 1 -- 172.21.15.67:0/2364497770 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fce4c015810 con 0x7fce5c15ee60 2024-04-25T15:26:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.378+0000 7fce5a7fc640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fce480744b0 0x7fce480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.378+0000 7fce5a7fc640 1 -- 172.21.15.67:0/2364497770 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fce4c095880 con 0x7fce5c15ee60 2024-04-25T15:26:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.378+0000 7fce61900640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fce480744b0 0x7fce480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.379+0000 7fce61900640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fce480744b0 0x7fce480769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:44.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.382+0000 7fce5a7fc640 1 -- 172.21.15.67:0/2364497770 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce4c010040 con 0x7fce5c15ee60 2024-04-25T15:26:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.604+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce5c0d1220 con 0x7fce5c15ee60 2024-04-25T15:26:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.605+0000 7fce5a7fc640 1 -- 172.21.15.67:0/2364497770 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3875 (secure 0 0 0) 0x7fce4c05fcc0 con 0x7fce5c15ee60 2024-04-25T15:26:44.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:44.606 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/4217070487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4217070487},{"type":"v1","addr":"172.21.15.176:6835","nonce":4217070487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4781,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4207485404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":4207485404},{"type":"v1","addr":"172.21.15.176:6837","nonce":4207485404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5067,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2355578883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2355578883},{"type":"v1","addr":"172.21.15.67:6834","nonce":2355578883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5409,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1432569266","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1432569266},{"type":"v1","addr":"172.21.15.67:6835","nonce":1432569266}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-04-25T15:26:44.606 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 18 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fce480744b0 msgr2=0x7fce480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fce480744b0 0x7fce480769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce54009040 comp rx=0 tx=0).stop 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5c15ee60 msgr2=0x7fce5c15f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5c15ee60 0x7fce5c15f270 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fce5c06e410 tx=0x7fce4c00f580 comp rx=0 tx=0).stop 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 shutdown_connections 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fce480744b0 0x7fce480769c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce5c15ee60 0x7fce5c15f270 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce5c14bf70 0x7fce5c15e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 --2- 172.21.15.67:0/2364497770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce44004800 0x7fce5c15aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 >> 172.21.15.67:0/2364497770 conn(0x7fce5c0d3a50 msgr2=0x7fce5c0ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 shutdown_connections 2024-04-25T15:26:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.607+0000 7fce63b64640 1 -- 172.21.15.67:0/2364497770 wait complete. 2024-04-25T15:26:44.619 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:26:44.619 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:26:44.619 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:26:44.620 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:44.620 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:26:44.620 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:26:44.620 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:26:44.620 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:44.620 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:26:44.620 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:26:44.620 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:26:44.621 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:44.621 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:26:44.621 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:26:44.621 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:26:44.621 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:26:44.632 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:26:44.632 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:26:44.634 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:26:44.635 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:26:44.635 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:26:44.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:26:44.636 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:26:44.636 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:26:44.677 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:26:44.677 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:44.678 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:26:44.678 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:44.679 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:26:44.679 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:26:44.679 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:44.679 DEBUG:teuthology.parallel:result is None 2024-04-25T15:26:44.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:26:44.765 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:26:44.767 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:26:44.771 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:26:44.771 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:26:44.771 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-04-25T15:26:44.771+0000 7f7baa5f1640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-25T15:26:44.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.799+0000 7f70a6020640 1 Processor -- start 2024-04-25T15:26:44.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.799+0000 7f70a6020640 1 -- start start 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f70a6020640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a0162580 0x7f70a00d4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f70a6020640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70a01657f0 0x7f70a00d56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f70a6020640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f70a006c5c0 0x7f70a006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f70a6020640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f70a00d5be0 con 0x7f70a01657f0 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f70a6020640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f70a006cf10 con 0x7f70a006c5c0 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f70a6020640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70a00d4170 con 0x7f70a0162580 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70a01657f0 0x7f70a00d56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70a01657f0 0x7f70a00d56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49708/0 (socket says 172.21.15.67:49708) 2024-04-25T15:26:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 -- 172.21.15.67:0/2945779798 learned_addr learned my addr 172.21.15.67:0/2945779798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709ffff640 1 --2- 172.21.15.67:0/2945779798 >> v2:172.21.15.176:3300/0 conn(0x7f70a006c5c0 0x7f70a006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709f7fe640 1 --2- 172.21.15.67:0/2945779798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a0162580 0x7f70a00d4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 -- 172.21.15.67:0/2945779798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a0162580 msgr2=0x7f70a00d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 --2- 172.21.15.67:0/2945779798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a0162580 0x7f70a00d4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 -- 172.21.15.67:0/2945779798 >> v2:172.21.15.176:3300/0 conn(0x7f70a006c5c0 msgr2=0x7f70a006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 --2- 172.21.15.67:0/2945779798 >> v2:172.21.15.176:3300/0 conn(0x7f70a006c5c0 0x7f70a006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.801+0000 7f709effd640 1 -- 172.21.15.67:0/2945779798 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a00d4450 con 0x7f70a01657f0 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.802+0000 7f709effd640 1 --2- 172.21.15.67:0/2945779798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00d56a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f70a00b88f0 tx=0x7f709002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ae92440f4fccfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.802+0000 7f709e7fc640 1 -- 172.21.15.67:0/2945779798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090037070 con 0x7f70a01657f0 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.802+0000 7f709e7fc640 1 -- 172.21.15.67:0/2945779798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7090032440 con 0x7f70a01657f0 2024-04-25T15:26:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.802+0000 7f709e7fc640 1 -- 172.21.15.67:0/2945779798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090032810 con 0x7f70a01657f0 2024-04-25T15:26:44.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 -- 172.21.15.67:0/2945779798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 msgr2=0x7f70a00d56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 --2- 172.21.15.67:0/2945779798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00d56a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f70a00b88f0 tx=0x7f709002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 -- 172.21.15.67:0/2945779798 shutdown_connections 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 --2- 172.21.15.67:0/2945779798 >> v2:172.21.15.176:3300/0 conn(0x7f70a006c5c0 0x7f70a006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 --2- 172.21.15.67:0/2945779798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00d56a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 --2- 172.21.15.67:0/2945779798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a0162580 0x7f70a00d4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 -- 172.21.15.67:0/2945779798 >> 172.21.15.67:0/2945779798 conn(0x7f70a00d5d90 msgr2=0x7f70a00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 -- 172.21.15.67:0/2945779798 shutdown_connections 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.803+0000 7f70a6020640 1 -- 172.21.15.67:0/2945779798 wait complete. 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.805+0000 7f70a6020640 1 Processor -- start 2024-04-25T15:26:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.805+0000 7f70a6020640 1 -- start start 2024-04-25T15:26:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f70a6020640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a006c5c0 0x7f70a00ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f70a6020640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70a0162580 0x7f70a01efae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f70a6020640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f70a6020640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70a0165f30 con 0x7f70a01657f0 2024-04-25T15:26:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f70a6020640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f70a00d4a10 con 0x7f70a0162580 2024-04-25T15:26:44.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f70a6020640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70a01660b0 con 0x7f70a006c5c0 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f709ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f709f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a006c5c0 0x7f70a00ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f709ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49720/0 (socket says 172.21.15.67:49720) 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f709f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a006c5c0 0x7f70a00ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:34398/0 (socket says 172.21.15.67:34398) 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f709ffff640 1 -- 172.21.15.67:0/3996720474 learned_addr learned my addr 172.21.15.67:0/3996720474 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.806+0000 7f709effd640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70a0162580 0x7f70a01efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f709ffff640 1 -- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a006c5c0 msgr2=0x7f70a00ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f709ffff640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a006c5c0 0x7f70a00ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f709ffff640 1 -- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70a0162580 msgr2=0x7f70a01efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f709ffff640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70a0162580 0x7f70a01efae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f709ffff640 1 -- 172.21.15.67:0/3996720474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a01f0020 con 0x7f70a01657f0 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f709f7fe640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a006c5c0 0x7f70a00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:44.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f709ffff640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00cc2c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7098007d80 tx=0x7f709800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f7087fff640 1 -- 172.21.15.67:0/3996720474 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098014420 con 0x7f70a01657f0 2024-04-25T15:26:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f7087fff640 1 -- 172.21.15.67:0/3996720474 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7098019070 con 0x7f70a01657f0 2024-04-25T15:26:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a00cd260 con 0x7f70a01657f0 2024-04-25T15:26:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f7087fff640 1 -- 172.21.15.67:0/3996720474 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098014800 con 0x7f70a01657f0 2024-04-25T15:26:44.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.807+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a00cd820 con 0x7f70a01657f0 2024-04-25T15:26:44.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.808+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a00ccfd0 con 0x7f70a01657f0 2024-04-25T15:26:44.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.809+0000 7f7087fff640 1 -- 172.21.15.67:0/3996720474 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f70980149a0 con 0x7f70a01657f0 2024-04-25T15:26:44.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.809+0000 7f7087fff640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f708c0744b0 0x7f708c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:44.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.809+0000 7f7087fff640 1 -- 172.21.15.67:0/3996720474 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f70980960b0 con 0x7f70a01657f0 2024-04-25T15:26:44.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.810+0000 7f709f7fe640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f708c0744b0 0x7f708c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:44.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.810+0000 7f709f7fe640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f708c0744b0 0x7f708c0769c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f70a00d2130 tx=0x7f7094008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:44.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:44.813+0000 7f7087fff640 1 -- 172.21.15.67:0/3996720474 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7098060560 con 0x7f70a01657f0 2024-04-25T15:26:45.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.010+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f70a0050740 con 0x7f70a01657f0 2024-04-25T15:26:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.714+0000 7f7087fff640 1 -- 172.21.15.67:0/3996720474 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) ==== 97+0+0 (secure 0 0 0) 0x7f7098064860 con 0x7f70a01657f0 2024-04-25T15:26:45.715 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f708c0744b0 msgr2=0x7f708c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f708c0744b0 0x7f708c0769c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f70a00d2130 tx=0x7f7094008040 comp rx=0 tx=0).stop 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 msgr2=0x7f70a00cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00cc2c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7098007d80 tx=0x7f709800e4b0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 shutdown_connections 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f708c0744b0 0x7f708c0769c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a01657f0 0x7f70a00cc2c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70a0162580 0x7f70a01efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 --2- 172.21.15.67:0/3996720474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70a006c5c0 0x7f70a00ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 >> 172.21.15.67:0/3996720474 conn(0x7f70a00d5d90 msgr2=0x7f70a00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 shutdown_connections 2024-04-25T15:26:45.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.718+0000 7f70a6020640 1 -- 172.21.15.67:0/3996720474 wait complete. 2024-04-25T15:26:45.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:26:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.850+0000 7fcb18bce640 1 Processor -- start 2024-04-25T15:26:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.850+0000 7fcb18bce640 1 -- start start 2024-04-25T15:26:45.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb18bce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14070480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:45.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb18bce640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb141612f0 0x7fcb140d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:45.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb18bce640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb140709c0 con 0x7fcb140d0ed0 2024-04-25T15:26:45.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb18bce640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcb141438e0 con 0x7fcb141612f0 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb18bce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1406f3e0 con 0x7fcb141434d0 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb12d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb141612f0 0x7fcb140d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb12d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb141612f0 0x7fcb140d12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49734/0 (socket says 172.21.15.67:49734) 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb12d9d640 1 -- 172.21.15.67:0/671682042 learned_addr learned my addr 172.21.15.67:0/671682042 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb1259c640 1 --2- 172.21.15.67:0/671682042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb12d9d640 1 -- 172.21.15.67:0/671682042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 msgr2=0x7fcb14070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb12d9d640 1 --2- 172.21.15.67:0/671682042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14070480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb12d9d640 1 -- 172.21.15.67:0/671682042 >> v1:172.21.15.176:6789/0 conn(0x7fcb140d0ed0 legacy=0x7fcb14160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb12d9d640 1 -- 172.21.15.67:0/671682042 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1406f6c0 con 0x7fcb141612f0 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.852+0000 7fcb1259c640 1 --2- 172.21.15.67:0/671682042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14070480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.853+0000 7fcb12d9d640 1 --2- 172.21.15.67:0/671682042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb141612f0 0x7fcb140d12e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400d150 tx=0x7fcb0402c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05e2ff6adc8c972 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:45.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.853+0000 7fcb1159a640 1 -- 172.21.15.67:0/671682042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400b2e0 con 0x7fcb141612f0 2024-04-25T15:26:45.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.853+0000 7fcb1159a640 1 -- 172.21.15.67:0/671682042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb04009600 con 0x7fcb141612f0 2024-04-25T15:26:45.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.853+0000 7fcb1159a640 1 -- 172.21.15.67:0/671682042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400b480 con 0x7fcb141612f0 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 -- 172.21.15.67:0/671682042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb141612f0 msgr2=0x7fcb140d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 --2- 172.21.15.67:0/671682042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb141612f0 0x7fcb140d12e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400d150 tx=0x7fcb0402c8e0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 -- 172.21.15.67:0/671682042 shutdown_connections 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 --2- 172.21.15.67:0/671682042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb141612f0 0x7fcb140d12e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 --2- 172.21.15.67:0/671682042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14070480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 -- 172.21.15.67:0/671682042 >> 172.21.15.67:0/671682042 conn(0x7fcb140d1ae0 msgr2=0x7fcb140d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 -- 172.21.15.67:0/671682042 shutdown_connections 2024-04-25T15:26:45.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.854+0000 7fcb18bce640 1 -- 172.21.15.67:0/671682042 wait complete. 2024-04-25T15:26:45.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.856+0000 7fcb18bce640 1 Processor -- start 2024-04-25T15:26:45.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.856+0000 7fcb18bce640 1 -- start start 2024-04-25T15:26:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb18bce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb140d0ed0 0x7fcb14144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb18bce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb18bce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1415a4a0 0x7fcb1415d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb18bce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1406f090 con 0x7fcb1415a4a0 2024-04-25T15:26:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb18bce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1406fc80 con 0x7fcb140d0ed0 2024-04-25T15:26:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb18bce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb140b51a0 con 0x7fcb141434d0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb12d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1415a4a0 0x7fcb1415d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb12d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1415a4a0 0x7fcb1415d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49738/0 (socket says 172.21.15.67:49738) 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb12d9d640 1 -- 172.21.15.67:0/1305739305 learned_addr learned my addr 172.21.15.67:0/1305739305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb1259c640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb140d0ed0 0x7fcb14144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.857+0000 7fcb11d9b640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb12d9d640 1 -- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 msgr2=0x7fcb14159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb12d9d640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb12d9d640 1 -- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb140d0ed0 msgr2=0x7fcb14144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb12d9d640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb140d0ed0 0x7fcb14144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb12d9d640 1 -- 172.21.15.67:0/1305739305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1415ec40 con 0x7fcb1415a4a0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb1259c640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb140d0ed0 0x7fcb14144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb12d9d640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1415a4a0 0x7fcb1415d460 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcb0402ced0 tx=0x7fcb04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb0affd640 1 -- 172.21.15.67:0/1305739305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04041070 con 0x7fcb1415a4a0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb0affd640 1 -- 172.21.15.67:0/1305739305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0403d820 con 0x7fcb1415a4a0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1415ef20 con 0x7fcb1415a4a0 2024-04-25T15:26:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb0affd640 1 -- 172.21.15.67:0/1305739305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0403dbd0 con 0x7fcb1415a4a0 2024-04-25T15:26:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.858+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb141f1bf0 con 0x7fcb1415a4a0 2024-04-25T15:26:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.859+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb14050740 con 0x7fcb1415a4a0 2024-04-25T15:26:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.860+0000 7fcb0affd640 1 -- 172.21.15.67:0/1305739305 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcb04046460 con 0x7fcb1415a4a0 2024-04-25T15:26:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.860+0000 7fcb0affd640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf40744e0 0x7fcaf40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.860+0000 7fcb0affd640 1 -- 172.21.15.67:0/1305739305 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7fcb04030030 con 0x7fcb1415a4a0 2024-04-25T15:26:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.860+0000 7fcb1259c640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf40744e0 0x7fcaf40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.861+0000 7fcb1259c640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf40744e0 0x7fcaf40769f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00a990 tx=0x7fcb0c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:45.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:45.865+0000 7fcb0affd640 1 -- 172.21.15.67:0/1305739305 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb0404b060 con 0x7fcb1415a4a0 2024-04-25T15:26:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.119+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fcb140d14f0 con 0x7fcb1415a4a0 2024-04-25T15:26:46.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.119+0000 7fcb0affd640 1 -- 172.21.15.67:0/1305739305 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7fcb04083470 con 0x7fcb1415a4a0 2024-04-25T15:26:46.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg==", 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.121 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA==", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q==", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.122 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw==", 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg==", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw==", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA==", 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g==", 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.125 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA==", 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.126 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA==", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ==", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA==", 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q==", 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.128 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ==", 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg==", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA==", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:26:46.130 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmVIPuGRAAswFIakCxyA7OvvoJIBYUiQ==", 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg==", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag==", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:26:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:46.133 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:26:46.133 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:26:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf40744e0 msgr2=0x7fcaf40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf40744e0 0x7fcaf40769f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00a990 tx=0x7fcb0c008040 comp rx=0 tx=0).stop 2024-04-25T15:26:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1415a4a0 msgr2=0x7fcb1415d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1415a4a0 0x7fcb1415d460 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcb0402ced0 tx=0x7fcb04004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:46.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 shutdown_connections 2024-04-25T15:26:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcaf40744e0 0x7fcaf40769f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb1415a4a0 0x7fcb1415d460 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb141434d0 0x7fcb14159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 --2- 172.21.15.67:0/1305739305 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb140d0ed0 0x7fcb14144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 >> 172.21.15.67:0/1305739305 conn(0x7fcb140d1ae0 msgr2=0x7fcb1413fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 shutdown_connections 2024-04-25T15:26:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.121+0000 7fcb18bce640 1 -- 172.21.15.67:0/1305739305 wait complete. 2024-04-25T15:26:46.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:46.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.246+0000 7f854169f640 1 Processor -- start 2024-04-25T15:26:46.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.246+0000 7f854169f640 1 -- start start 2024-04-25T15:26:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f854169f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f854169f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f853c06c5c0 0x7f853c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f854169f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f853c06cf10 con 0x7f853c0b7810 2024-04-25T15:26:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f854169f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f853c0d5a60 con 0x7f853c06c5c0 2024-04-25T15:26:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f854169f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f853c0d1850 con 0x7f853c0d0e00 2024-04-25T15:26:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f853c06c5c0 0x7f853c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853a7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f853c06c5c0 0x7f853c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49744/0 (socket says 172.21.15.67:49744) 2024-04-25T15:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853a7fc640 1 -- 172.21.15.67:0/1244664836 learned_addr learned my addr 172.21.15.67:0/1244664836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853affd640 1 --2- 172.21.15.67:0/1244664836 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853a7fc640 1 -- 172.21.15.67:0/1244664836 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 msgr2=0x7f853c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853a7fc640 1 --2- 172.21.15.67:0/1244664836 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853a7fc640 1 -- 172.21.15.67:0/1244664836 >> v1:172.21.15.176:6789/0 conn(0x7f853c0b7810 legacy=0x7f853c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.248+0000 7f853a7fc640 1 -- 172.21.15.67:0/1244664836 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c160de0 con 0x7f853c06c5c0 2024-04-25T15:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.249+0000 7f853a7fc640 1 --2- 172.21.15.67:0/1244664836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c06c5c0 0x7f853c06c9d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8524009870 tx=0x7f852402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed88fbe5c28306a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.249+0000 7f8539ffb640 1 -- 172.21.15.67:0/1244664836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524037070 con 0x7f853c06c5c0 2024-04-25T15:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.249+0000 7f8539ffb640 1 -- 172.21.15.67:0/1244664836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8524032440 con 0x7f853c06c5c0 2024-04-25T15:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.249+0000 7f8539ffb640 1 -- 172.21.15.67:0/1244664836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524032810 con 0x7f853c06c5c0 2024-04-25T15:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 -- 172.21.15.67:0/1244664836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c06c5c0 msgr2=0x7f853c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 --2- 172.21.15.67:0/1244664836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c06c5c0 0x7f853c06c9d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8524009870 tx=0x7f852402a210 comp rx=0 tx=0).stop 2024-04-25T15:26:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 -- 172.21.15.67:0/1244664836 shutdown_connections 2024-04-25T15:26:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 --2- 172.21.15.67:0/1244664836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c06c5c0 0x7f853c06c9d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 --2- 172.21.15.67:0/1244664836 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 -- 172.21.15.67:0/1244664836 >> 172.21.15.67:0/1244664836 conn(0x7f853c0d5d90 msgr2=0x7f853c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 -- 172.21.15.67:0/1244664836 shutdown_connections 2024-04-25T15:26:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.250+0000 7f854169f640 1 -- 172.21.15.67:0/1244664836 wait complete. 2024-04-25T15:26:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.252+0000 7f854169f640 1 Processor -- start 2024-04-25T15:26:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.252+0000 7f854169f640 1 -- start start 2024-04-25T15:26:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f854169f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f853c06c5c0 0x7f853c1e9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f854169f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c0b7810 0x7f853c150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f854169f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f854169f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f853c160ad0 con 0x7f853c0b7810 2024-04-25T15:26:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f854169f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f853c1613a0 con 0x7f853c06c5c0 2024-04-25T15:26:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f854169f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f853c0b5320 con 0x7f853c0d0e00 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c0b7810 0x7f853c150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c0b7810 0x7f853c150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49748/0 (socket says 172.21.15.67:49748) 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853a7fc640 1 -- 172.21.15.67:0/1248738028 learned_addr learned my addr 172.21.15.67:0/1248738028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853affd640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f853c06c5c0 0x7f853c1e9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853b7fe640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853a7fc640 1 -- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 msgr2=0x7f853c0ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853a7fc640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853a7fc640 1 -- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f853c06c5c0 msgr2=0x7f853c1e9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.253+0000 7f853a7fc640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f853c06c5c0 0x7f853c1e9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f853a7fc640 1 -- 172.21.15.67:0/1248738028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853c1eb7a0 con 0x7f853c0b7810 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f853affd640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f853c06c5c0 0x7f853c1e9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f853b7fe640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f853a7fc640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c0b7810 0x7f853c150140 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8524000c00 tx=0x7f85240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f85337fe640 1 -- 172.21.15.67:0/1248738028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524037040 con 0x7f853c0b7810 2024-04-25T15:26:46.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f85337fe640 1 -- 172.21.15.67:0/1248738028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8524042070 con 0x7f853c0b7810 2024-04-25T15:26:46.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f853c1ea700 con 0x7f853c0b7810 2024-04-25T15:26:46.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f85337fe640 1 -- 172.21.15.67:0/1248738028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524030bb0 con 0x7f853c0b7810 2024-04-25T15:26:46.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.254+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f853c1eacc0 con 0x7f853c0b7810 2024-04-25T15:26:46.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.255+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f853c1ea470 con 0x7f853c0b7810 2024-04-25T15:26:46.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.256+0000 7f85337fe640 1 -- 172.21.15.67:0/1248738028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8524047460 con 0x7f853c0b7810 2024-04-25T15:26:46.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.256+0000 7f85337fe640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f851c0744e0 0x7f851c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.256+0000 7f85337fe640 1 -- 172.21.15.67:0/1248738028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f85240b5c40 con 0x7f853c0b7810 2024-04-25T15:26:46.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.256+0000 7f853affd640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f851c0744e0 0x7f851c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.257+0000 7f853affd640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f851c0744e0 0x7f851c0769f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f853400b820 tx=0x7f8534008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.261+0000 7f85337fe640 1 -- 172.21.15.67:0/1248738028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f853c06cf10 con 0x7f853c0b7810 2024-04-25T15:26:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.448+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f853c050740 con 0x7f853c0b7810 2024-04-25T15:26:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.449+0000 7f85337fe640 1 -- 172.21.15.67:0/1248738028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+12855 (secure 0 0 0) 0x7f8524080030 con 0x7f853c0b7810 2024-04-25T15:26:46.449 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:46.450 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:45.724992+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f851c0744e0 msgr2=0x7f851c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f851c0744e0 0x7f851c0769f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f853400b820 tx=0x7f8534008040 comp rx=0 tx=0).stop 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c0b7810 msgr2=0x7f853c150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c0b7810 0x7f853c150140 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8524000c00 tx=0x7f85240046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 shutdown_connections 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f851c0744e0 0x7f851c0769f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f853c0d0e00 0x7f853c0ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f853c0b7810 0x7f853c150140 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 --2- 172.21.15.67:0/1248738028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f853c06c5c0 0x7f853c1e9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 >> 172.21.15.67:0/1248738028 conn(0x7f853c0d5d90 msgr2=0x7f853c0b61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 shutdown_connections 2024-04-25T15:26:46.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.451+0000 7f854169f640 1 -- 172.21.15.67:0/1248738028 wait complete. 2024-04-25T15:26:46.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:46.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.578+0000 7f0f05abd640 1 Processor -- start 2024-04-25T15:26:46.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.578+0000 7f0f05abd640 1 -- start start 2024-04-25T15:26:46.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0f05abd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f0015e3b0 0x7f0f000cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0f05abd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f000cfe60 con 0x7f0f000d28f0 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0f05abd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f000d0140 con 0x7f0f0006c5c0 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0f05abd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0f000b3bf0 con 0x7f0f0015e3b0 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0efeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f0015e3b0 0x7f0f000cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0efeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f0015e3b0 0x7f0f000cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49764/0 (socket says 172.21.15.67:49764) 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0efffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0f000d28f0 0x7f0f000d2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36288/0 (socket says 172.21.15.67:36288) 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.579+0000 7f0efeffd640 1 -- 172.21.15.67:0/1313028583 learned_addr learned my addr 172.21.15.67:0/1313028583 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.580+0000 7f0efeffd640 1 -- 172.21.15.67:0/1313028583 >> v1:172.21.15.176:6789/0 conn(0x7f0f0006c5c0 legacy=0x7f0f000cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.580+0000 7f0efeffd640 1 -- 172.21.15.67:0/1313028583 >> v1:172.21.15.67:6789/0 conn(0x7f0f000d28f0 legacy=0x7f0f000d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.580+0000 7f0efeffd640 1 -- 172.21.15.67:0/1313028583 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f000b3ed0 con 0x7f0f0015e3b0 2024-04-25T15:26:46.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.580+0000 7f0efe7fc640 1 -- 172.21.15.67:0/1313028583 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2654799544 0 0) 0x7f0f000cfe60 con 0x7f0f000d28f0 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0efeffd640 1 --2- 172.21.15.67:0/1313028583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0015e3b0 0x7f0f000cef00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009870 tx=0x7f0ef402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf6f6931592367b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0efe7fc640 1 -- 172.21.15.67:0/1313028583 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4037070 con 0x7f0f0015e3b0 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0efe7fc640 1 -- 172.21.15.67:0/1313028583 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef402ae50 con 0x7f0f0015e3b0 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0efe7fc640 1 -- 172.21.15.67:0/1313028583 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4032860 con 0x7f0f0015e3b0 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0f05abd640 1 -- 172.21.15.67:0/1313028583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0015e3b0 msgr2=0x7f0f000cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0f05abd640 1 --2- 172.21.15.67:0/1313028583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0015e3b0 0x7f0f000cef00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009870 tx=0x7f0ef402a300 comp rx=0 tx=0).stop 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0f05abd640 1 -- 172.21.15.67:0/1313028583 shutdown_connections 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0f05abd640 1 --2- 172.21.15.67:0/1313028583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f0015e3b0 0x7f0f000cef00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0f05abd640 1 -- 172.21.15.67:0/1313028583 >> 172.21.15.67:0/1313028583 conn(0x7f0f0009b7d0 msgr2=0x7f0f0009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0f05abd640 1 -- 172.21.15.67:0/1313028583 shutdown_connections 2024-04-25T15:26:46.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.581+0000 7f0f05abd640 1 -- 172.21.15.67:0/1313028583 wait complete. 2024-04-25T15:26:46.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.583+0000 7f0f05abd640 1 Processor -- start 2024-04-25T15:26:46.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.583+0000 7f0f05abd640 1 -- start start 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0f05abd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f0006c5c0 0x7f0f0015a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0f05abd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f000d28f0 0x7f0f00160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0f05abd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f0015e3b0 0x7f0f000c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0f05abd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00169490 con 0x7f0f000d28f0 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0f05abd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00169310 con 0x7f0f0015e3b0 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0f05abd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0f00169610 con 0x7f0f0006c5c0 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f000d28f0 0x7f0f00160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f000d28f0 0x7f0f00160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49768/0 (socket says 172.21.15.67:49768) 2024-04-25T15:26:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 -- 172.21.15.67:0/750296031 learned_addr learned my addr 172.21.15.67:0/750296031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efffff640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f0015e3b0 0x7f0f000c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0eff7fe640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f0006c5c0 0x7f0f0015a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 -- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f0006c5c0 msgr2=0x7f0f0015a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f0006c5c0 0x7f0f0015a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 -- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f0015e3b0 msgr2=0x7f0f000c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f0015e3b0 0x7f0f000c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 -- 172.21.15.67:0/750296031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f00153700 con 0x7f0f000d28f0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0eff7fe640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f0006c5c0 0x7f0f0015a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0efeffd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f000d28f0 0x7f0f00160340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009870 tx=0x7f0ef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.585+0000 7f0ee7fff640 1 -- 172.21.15.67:0/750296031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4042070 con 0x7f0f000d28f0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.586+0000 7f0ee7fff640 1 -- 172.21.15.67:0/750296031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef403e870 con 0x7f0f000d28f0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.586+0000 7f0ee7fff640 1 -- 172.21.15.67:0/750296031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4037040 con 0x7f0f000d28f0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.586+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f00153970 con 0x7f0f000d28f0 2024-04-25T15:26:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.586+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0015a760 con 0x7f0f000d28f0 2024-04-25T15:26:46.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.587+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f00071f70 con 0x7f0f000d28f0 2024-04-25T15:26:46.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.587+0000 7f0ee7fff640 1 -- 172.21.15.67:0/750296031 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0ef4045070 con 0x7f0f000d28f0 2024-04-25T15:26:46.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.588+0000 7f0ee7fff640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0eec0744b0 0x7f0eec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.588+0000 7f0ee7fff640 1 -- 172.21.15.67:0/750296031 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f0ef40b58f0 con 0x7f0f000d28f0 2024-04-25T15:26:46.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.588+0000 7f0eff7fe640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0eec0744b0 0x7f0eec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.589+0000 7f0eff7fe640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0eec0744b0 0x7f0eec0769c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000fb0 tx=0x7f0ef0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.593+0000 7f0ee7fff640 1 -- 172.21.15.67:0/750296031 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f00071f70 con 0x7f0f000d28f0 2024-04-25T15:26:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.778+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0f00155790 con 0x7f0f000d28f0 2024-04-25T15:26:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.778+0000 7f0ee7fff640 1 -- 172.21.15.67:0/750296031 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) ==== 93+0+10 (secure 0 0 0) 0x7f0ef407fd80 con 0x7f0f000d28f0 2024-04-25T15:26:46.779 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0eec0744b0 msgr2=0x7f0eec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0eec0744b0 0x7f0eec0769c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0000fb0 tx=0x7f0ef0009040 comp rx=0 tx=0).stop 2024-04-25T15:26:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f000d28f0 msgr2=0x7f0f00160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f000d28f0 0x7f0f00160340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009870 tx=0x7f0ef40046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 shutdown_connections 2024-04-25T15:26:46.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0eec0744b0 0x7f0eec0769c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f0015e3b0 0x7f0f000c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f000d28f0 0x7f0f00160340 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 --2- 172.21.15.67:0/750296031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f0006c5c0 0x7f0f0015a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 >> 172.21.15.67:0/750296031 conn(0x7f0f0009b7d0 msgr2=0x7f0f000d2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:46.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 shutdown_connections 2024-04-25T15:26:46.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.780+0000 7f0f05abd640 1 -- 172.21.15.67:0/750296031 wait complete. 2024-04-25T15:26:46.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:26:46.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.904+0000 7f88879d7640 1 Processor -- start 2024-04-25T15:26:46.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.904+0000 7f88879d7640 1 -- start start 2024-04-25T15:26:46.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f88879d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 0x7f88800b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f88879d7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88801615f0 con 0x7f888015df80 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f88879d7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8880164020 con 0x7f8880160fa0 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f88879d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f88801613b0 con 0x7f8880164b00 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f8884f72640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f888015df80 0x7f8880160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36296/0 (socket says 172.21.15.67:36296) 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f8884f72640 1 -- 172.21.15.67:0/3825404882 learned_addr learned my addr 172.21.15.67:0/3825404882 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f8885773640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 0x7f88800b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 375289168 0 0) 0x7f88801615f0 con 0x7f888015df80 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8868003520 con 0x7f888015df80 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1389784038 0 0) 0x7f8880164020 con 0x7f8880160fa0 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88801615f0 con 0x7f8880160fa0 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2831550792 0 0) 0x7f8868003520 con 0x7f888015df80 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 msgr2=0x7f88800b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 0x7f88800b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.906+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 >> v1:172.21.15.176:6789/0 conn(0x7f8880160fa0 legacy=0x7f88800b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8880164300 con 0x7f888015df80 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8874001670 con 0x7f888015df80 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 >> v1:172.21.15.67:6789/0 conn(0x7f888015df80 legacy=0x7f8880160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8868003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 0x7f8868004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8868004d40 0x7f8868005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88801613b0 con 0x7f8868003b70 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f886800ba70 con 0x7f8868004d40 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f886800bc70 con 0x7f8880164b00 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8885773640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8868003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8884f72640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 0x7f8868004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8885f74640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8868004d40 0x7f8868005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8885773640 1 -- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 msgr2=0x7f8868004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8885773640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880164b00 0x7f8868004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8885773640 1 -- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8868004d40 msgr2=0x7f8868005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8885773640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8868004d40 0x7f8868005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.907+0000 7f8885773640 1 -- 172.21.15.67:0/3825404882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8868003520 con 0x7f8868003b70 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.908+0000 7f8885773640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8868003fa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8878009ac0 tx=0x7f8868005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edbd9e4cc0e82af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.908+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878036070 con 0x7f8868003b70 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.908+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8878031400 con 0x7f8868003b70 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.908+0000 7f887ffff640 1 -- 172.21.15.67:0/3825404882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878009690 con 0x7f8868003b70 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.909+0000 7f88879d7640 1 -- 172.21.15.67:0/3825404882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 msgr2=0x7f8868003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.909+0000 7f88879d7640 1 --2- 172.21.15.67:0/3825404882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8868003fa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8878009ac0 tx=0x7f8868005830 comp rx=0 tx=0).stop 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.909+0000 7f8885f74640 1 -- 172.21.15.67:0/3825404882 reap_dead start 2024-04-25T15:26:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.909+0000 7f88879d7640 1 -- 172.21.15.67:0/3825404882 shutdown_connections 2024-04-25T15:26:46.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.909+0000 7f88879d7640 1 -- 172.21.15.67:0/3825404882 >> 172.21.15.67:0/3825404882 conn(0x7f88800d2bc0 msgr2=0x7f88800d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:46.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.909+0000 7f88879d7640 1 -- 172.21.15.67:0/3825404882 shutdown_connections 2024-04-25T15:26:46.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.909+0000 7f88879d7640 1 -- 172.21.15.67:0/3825404882 wait complete. 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.911+0000 7f88879d7640 1 Processor -- start 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.911+0000 7f88879d7640 1 -- start start 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f88879d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8880161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f88879d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880162010 0x7f88801549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f88879d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8880154f20 0x7f8880155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f88879d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88800b53a0 con 0x7f8868003b70 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f88879d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f88800b5220 con 0x7f8880154f20 2024-04-25T15:26:46.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f88879d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f88800b5520 con 0x7f8880162010 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8880161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8880161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49776/0 (socket says 172.21.15.67:49776) 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 -- 172.21.15.67:0/3518636522 learned_addr learned my addr 172.21.15.67:0/3518636522 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885f74640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8880154f20 0x7f8880155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8884f72640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880162010 0x7f88801549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 -- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880162010 msgr2=0x7f88801549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880162010 0x7f88801549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 -- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8880154f20 msgr2=0x7f8880155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8880154f20 0x7f8880155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 -- 172.21.15.67:0/3518636522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8880162800 con 0x7f8868003b70 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.912+0000 7f8885773640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8880161ad0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8878034040 tx=0x7f8878004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.913+0000 7f887dffb640 1 -- 172.21.15.67:0/3518636522 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878040020 con 0x7f8868003b70 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.913+0000 7f887dffb640 1 -- 172.21.15.67:0/3518636522 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8878003bc0 con 0x7f8868003b70 2024-04-25T15:26:46.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.913+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8880162ae0 con 0x7f8868003b70 2024-04-25T15:26:46.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.913+0000 7f887dffb640 1 -- 172.21.15.67:0/3518636522 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878036040 con 0x7f8868003b70 2024-04-25T15:26:46.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.913+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88801630a0 con 0x7f8868003b70 2024-04-25T15:26:46.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.914+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8880050740 con 0x7f8868003b70 2024-04-25T15:26:46.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.914+0000 7f887dffb640 1 -- 172.21.15.67:0/3518636522 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8878043020 con 0x7f8868003b70 2024-04-25T15:26:46.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.915+0000 7f887dffb640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f886c0744b0 0x7f886c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:46.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.915+0000 7f887dffb640 1 -- 172.21.15.67:0/3518636522 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f88780b4fb0 con 0x7f8868003b70 2024-04-25T15:26:46.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.915+0000 7f8884f72640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f886c0744b0 0x7f886c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:46.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.916+0000 7f8884f72640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f886c0744b0 0x7f886c0769c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f888015d8a0 tx=0x7f8874008cd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:46.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:46.920+0000 7f887dffb640 1 -- 172.21.15.67:0/3518636522 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f887804a070 con 0x7f8868003b70 2024-04-25T15:26:47.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.106+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f88800d4170 con 0x7f8868003b70 2024-04-25T15:26:47.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.107+0000 7f887dffb640 1 -- 172.21.15.67:0/3518636522 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v40) ==== 84+0+65 (secure 0 0 0) 0x7f88780044e0 con 0x7f8868003b70 2024-04-25T15:26:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:47.108 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:26:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:26:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:26:47.108 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:26:47.109 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:26:47.109 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:26:47.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f886c0744b0 msgr2=0x7f886c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f886c0744b0 0x7f886c0769c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f888015d8a0 tx=0x7f8874008cd0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 msgr2=0x7f8880161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8880161ad0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8878034040 tx=0x7f8878004730 comp rx=0 tx=0).stop 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 shutdown_connections 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f886c0744b0 0x7f886c0769c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8880154f20 0x7f8880155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8880162010 0x7f88801549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 --2- 172.21.15.67:0/3518636522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8868003b70 0x7f8880161ad0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 >> 172.21.15.67:0/3518636522 conn(0x7f88800d2bc0 msgr2=0x7f88800b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 shutdown_connections 2024-04-25T15:26:47.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.109+0000 7f88879d7640 1 -- 172.21.15.67:0/3518636522 wait complete. 2024-04-25T15:26:47.121 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:26:47.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:26:47.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.236+0000 7f6c39684640 1 Processor -- start 2024-04-25T15:26:47.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.236+0000 7f6c39684640 1 -- start start 2024-04-25T15:26:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c39684640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c340cd9a0 0x7f6c3415f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c39684640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6c3406dce0 0x7f6c3415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c39684640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c340cdea0 con 0x7f6c340b0af0 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c39684640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6c340ceab0 con 0x7f6c340cd9a0 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c39684640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6c340ce180 con 0x7f6c3406dce0 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c32ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c340cd9a0 0x7f6c3415f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c2a7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6c340b0af0 0x7f6c3406d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36300/0 (socket says 172.21.15.67:36300) 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c2a7fc640 1 -- 172.21.15.67:0/1082367013 learned_addr learned my addr 172.21.15.67:0/1082367013 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.237+0000 7f6c337fe640 1 --2- 172.21.15.67:0/1082367013 >> v2:172.21.15.176:3300/0 conn(0x7f6c3406dce0 0x7f6c3415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.238+0000 7f6c32ffd640 1 -- 172.21.15.67:0/1082367013 >> v2:172.21.15.176:3300/0 conn(0x7f6c3406dce0 msgr2=0x7f6c3415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.238+0000 7f6c32ffd640 1 --2- 172.21.15.67:0/1082367013 >> v2:172.21.15.176:3300/0 conn(0x7f6c3406dce0 0x7f6c3415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.238+0000 7f6c32ffd640 1 -- 172.21.15.67:0/1082367013 >> v1:172.21.15.67:6789/0 conn(0x7f6c340b0af0 legacy=0x7f6c3406d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.238+0000 7f6c32ffd640 1 -- 172.21.15.67:0/1082367013 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c340ce460 con 0x7f6c340cd9a0 2024-04-25T15:26:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.238+0000 7f6c327fc640 1 -- 172.21.15.67:0/1082367013 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3295336623 0 0) 0x7f6c340cdea0 con 0x7f6c340b0af0 2024-04-25T15:26:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.238+0000 7f6c337fe640 1 --2- 172.21.15.67:0/1082367013 >> v2:172.21.15.176:3300/0 conn(0x7f6c3406dce0 0x7f6c3415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.238+0000 7f6c32ffd640 1 --2- 172.21.15.67:0/1082367013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c340cd9a0 0x7f6c3415f360 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6c34148020 tx=0x7f6c2402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc82203ab41ea2ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c327fc640 1 -- 172.21.15.67:0/1082367013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24037070 con 0x7f6c340cd9a0 2024-04-25T15:26:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c327fc640 1 -- 172.21.15.67:0/1082367013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c24032440 con 0x7f6c340cd9a0 2024-04-25T15:26:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c327fc640 1 -- 172.21.15.67:0/1082367013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24032810 con 0x7f6c340cd9a0 2024-04-25T15:26:47.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 -- 172.21.15.67:0/1082367013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c340cd9a0 msgr2=0x7f6c3415f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 --2- 172.21.15.67:0/1082367013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c340cd9a0 0x7f6c3415f360 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6c34148020 tx=0x7f6c2402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 -- 172.21.15.67:0/1082367013 shutdown_connections 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 --2- 172.21.15.67:0/1082367013 >> v2:172.21.15.176:3300/0 conn(0x7f6c3406dce0 0x7f6c3415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 --2- 172.21.15.67:0/1082367013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c340cd9a0 0x7f6c3415f360 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 -- 172.21.15.67:0/1082367013 >> 172.21.15.67:0/1082367013 conn(0x7f6c34073090 msgr2=0x7f6c340734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 -- 172.21.15.67:0/1082367013 shutdown_connections 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.239+0000 7f6c39684640 1 -- 172.21.15.67:0/1082367013 wait complete. 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.241+0000 7f6c39684640 1 Processor -- start 2024-04-25T15:26:47.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.241+0000 7f6c39684640 1 -- start start 2024-04-25T15:26:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c39684640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c340b0af0 0x7f6c3413ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c39684640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c340cd9a0 0x7f6c3413cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c39684640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c3413d2b0 0x7f6c3413b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c39684640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c340cef40 con 0x7f6c3413d2b0 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c39684640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6c340b0f00 con 0x7f6c340b0af0 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c39684640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6c340cf0c0 con 0x7f6c340cd9a0 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c3413d2b0 0x7f6c3413b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c3413d2b0 0x7f6c3413b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49798/0 (socket says 172.21.15.67:49798) 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 -- 172.21.15.67:0/2319132188 learned_addr learned my addr 172.21.15.67:0/2319132188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c32ffd640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c340b0af0 0x7f6c3413ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c2a7fc640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c340cd9a0 0x7f6c3413cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 -- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c340cd9a0 msgr2=0x7f6c3413cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c340cd9a0 0x7f6c3413cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 -- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c340b0af0 msgr2=0x7f6c3413ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c340b0af0 0x7f6c3413ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 -- 172.21.15.67:0/2319132188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c34140500 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c2a7fc640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c340cd9a0 0x7f6c3413cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c337fe640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c3413d2b0 0x7f6c3413b780 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007e70 tx=0x7f6c2c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c2bfff640 1 -- 172.21.15.67:0/2319132188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c01a070 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c2bfff640 1 -- 172.21.15.67:0/2319132188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c2c003bb0 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.243+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3413f760 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.244+0000 7f6c2bfff640 1 -- 172.21.15.67:0/2319132188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c004030 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.244+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3413fd20 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.244+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c34050740 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.245+0000 7f6c2bfff640 1 -- 172.21.15.67:0/2319132188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6c2c01d070 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.245+0000 7f6c2bfff640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6c140744b0 0x7f6c140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.246+0000 7f6c2bfff640 1 -- 172.21.15.67:0/2319132188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f6c2c0950d0 con 0x7f6c3413d2b0 2024-04-25T15:26:47.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.246+0000 7f6c32ffd640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6c140744b0 0x7f6c140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.246+0000 7f6c32ffd640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6c140744b0 0x7f6c140769c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6c24034000 tx=0x7f6c2402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:47.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.250+0000 7f6c2bfff640 1 -- 172.21.15.67:0/2319132188 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6c2c010070 con 0x7f6c3413d2b0 2024-04-25T15:26:47.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.434+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f6c3413f4f0 con 0x7f6c3413d2b0 2024-04-25T15:26:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.723+0000 7f6c2bfff640 1 -- 172.21.15.67:0/2319132188 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v41) ==== 138+0+0 (secure 0 0 0) 0x7f6c2c05fc40 con 0x7f6c3413d2b0 2024-04-25T15:26:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6c140744b0 msgr2=0x7f6c140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6c140744b0 0x7f6c140769c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6c24034000 tx=0x7f6c2402e000 comp rx=0 tx=0).stop 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c3413d2b0 msgr2=0x7f6c3413b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c3413d2b0 0x7f6c3413b780 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c007e70 tx=0x7f6c2c00d4a0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 shutdown_connections 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6c140744b0 0x7f6c140769c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c3413d2b0 0x7f6c3413b780 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c340cd9a0 0x7f6c3413cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 --2- 172.21.15.67:0/2319132188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c340b0af0 0x7f6c3413ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 >> 172.21.15.67:0/2319132188 conn(0x7f6c34073090 msgr2=0x7f6c3415fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 shutdown_connections 2024-04-25T15:26:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.726+0000 7f6c39684640 1 -- 172.21.15.67:0/2319132188 wait complete. 2024-04-25T15:26:47.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:26:47.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.860+0000 7fe0a14b1640 1 Processor -- start 2024-04-25T15:26:47.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.860+0000 7fe0a14b1640 1 -- start start 2024-04-25T15:26:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe0a14b1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe09c06dce0 0x7fe09c0704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe0a14b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe0a14b1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe09c06fa20 con 0x7fe09c0b0c90 2024-04-25T15:26:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe0a14b1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe09c070fd0 con 0x7fe09c06dce0 2024-04-25T15:26:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe0a14b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe09c0712b0 con 0x7fe09c15adf0 2024-04-25T15:26:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe093fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe09c06dce0 0x7fe09c0704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe093fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe09c06dce0 0x7fe09c0704c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44740/0 (socket says 172.21.15.67:44740) 2024-04-25T15:26:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe093fff640 1 -- 172.21.15.67:0/2828104672 learned_addr learned my addr 172.21.15.67:0/2828104672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe09b7fe640 1 --2- 172.21.15.67:0/2828104672 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe093fff640 1 -- 172.21.15.67:0/2828104672 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 msgr2=0x7fe09c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe093fff640 1 --2- 172.21.15.67:0/2828104672 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe093fff640 1 -- 172.21.15.67:0/2828104672 >> v1:172.21.15.176:6789/0 conn(0x7fe09c0b0c90 legacy=0x7fe09c06d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe093fff640 1 -- 172.21.15.67:0/2828104672 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09c06fcc0 con 0x7fe09c06dce0 2024-04-25T15:26:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.862+0000 7fe09b7fe640 1 --2- 172.21.15.67:0/2828104672 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.863+0000 7fe093fff640 1 --2- 172.21.15.67:0/2828104672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c06dce0 0x7fe09c0704c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe084009870 tx=0x7fe08402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d021b17767a414b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.863+0000 7fe09a7fc640 1 -- 172.21.15.67:0/2828104672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe084037070 con 0x7fe09c06dce0 2024-04-25T15:26:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.863+0000 7fe09a7fc640 1 -- 172.21.15.67:0/2828104672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe084032440 con 0x7fe09c06dce0 2024-04-25T15:26:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.863+0000 7fe09a7fc640 1 -- 172.21.15.67:0/2828104672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe084032810 con 0x7fe09c06dce0 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2828104672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c06dce0 msgr2=0x7fe09c0704c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2828104672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c06dce0 0x7fe09c0704c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe084009870 tx=0x7fe08402a270 comp rx=0 tx=0).stop 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2828104672 shutdown_connections 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2828104672 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2828104672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c06dce0 0x7fe09c0704c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2828104672 >> 172.21.15.67:0/2828104672 conn(0x7fe09c073090 msgr2=0x7fe09c0734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2828104672 shutdown_connections 2024-04-25T15:26:47.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.864+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2828104672 wait complete. 2024-04-25T15:26:47.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.866+0000 7fe0a14b1640 1 Processor -- start 2024-04-25T15:26:47.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.866+0000 7fe0a14b1640 1 -- start start 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe0a14b1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c0b0c90 0x7fe09c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe0a14b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe0a14b1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe09c0afcb0 0x7fe09c0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe0a14b1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c15d790 con 0x7fe09c0b0c90 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe0a14b1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c070c80 con 0x7fe09c0afcb0 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe0a14b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe09c15d910 con 0x7fe09c15adf0 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c0b0c90 0x7fe09c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c0b0c90 0x7fe09c14f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44750/0 (socket says 172.21.15.67:44750) 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 -- 172.21.15.67:0/2811838334 learned_addr learned my addr 172.21.15.67:0/2811838334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:47.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09b7fe640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe09c0afcb0 0x7fe09c0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe093fff640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 -- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 msgr2=0x7fe09c0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 -- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe09c0afcb0 msgr2=0x7fe09c0b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe09c0afcb0 0x7fe09c0b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.867+0000 7fe09affd640 1 -- 172.21.15.67:0/2811838334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09c0ae640 con 0x7fe09c0b0c90 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.868+0000 7fe09affd640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c0b0c90 0x7fe09c14f530 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c470 tx=0x7fe08c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.868+0000 7fe0937fe640 1 -- 172.21.15.67:0/2811838334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0052d0 con 0x7fe09c0b0c90 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.868+0000 7fe0937fe640 1 -- 172.21.15.67:0/2811838334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe08c005490 con 0x7fe09c0b0c90 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.868+0000 7fe0937fe640 1 -- 172.21.15.67:0/2811838334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c007260 con 0x7fe09c0b0c90 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.868+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09c14fa70 con 0x7fe09c0b0c90 2024-04-25T15:26:47.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.868+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe09c0aee20 con 0x7fe09c0b0c90 2024-04-25T15:26:47.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.869+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe09c158470 con 0x7fe09c0b0c90 2024-04-25T15:26:47.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.870+0000 7fe0937fe640 1 -- 172.21.15.67:0/2811838334 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe08c007400 con 0x7fe09c0b0c90 2024-04-25T15:26:47.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.870+0000 7fe0937fe640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe07c0744b0 0x7fe07c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:47.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.870+0000 7fe0937fe640 1 -- 172.21.15.67:0/2811838334 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5227+0+0 (secure 0 0 0) 0x7fe08c094420 con 0x7fe09c0b0c90 2024-04-25T15:26:47.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.870+0000 7fe093fff640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe07c0744b0 0x7fe07c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:47.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.871+0000 7fe093fff640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe07c0744b0 0x7fe07c0769c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe084034000 tx=0x7fe08402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:47.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:47.875+0000 7fe0937fe640 1 -- 172.21.15.67:0/2811838334 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe08c05e740 con 0x7fe09c0b0c90 2024-04-25T15:26:48.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.060+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe09c050740 con 0x7fe09c0b0c90 2024-04-25T15:26:48.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.725+0000 7fe0937fe640 1 -- 172.21.15.67:0/2811838334 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v42) ==== 170+0+0 (secure 0 0 0) 0x7fe08c062a40 con 0x7fe09c0b0c90 2024-04-25T15:26:48.727 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:26:48.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.731+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe07c0744b0 msgr2=0x7fe07c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:48.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.731+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe07c0744b0 0x7fe07c0769c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe084034000 tx=0x7fe08402e000 comp rx=0 tx=0).stop 2024-04-25T15:26:48.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.731+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c0b0c90 msgr2=0x7fe09c14f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:48.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.731+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c0b0c90 0x7fe09c14f530 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c470 tx=0x7fe08c00c930 comp rx=0 tx=0).stop 2024-04-25T15:26:48.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.734+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 shutdown_connections 2024-04-25T15:26:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.734+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe07c0744b0 0x7fe07c0769c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.734+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe09c0afcb0 0x7fe09c0b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.734+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe09c15adf0 0x7fe09c0b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.734+0000 7fe0a14b1640 1 --2- 172.21.15.67:0/2811838334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe09c0b0c90 0x7fe09c14f530 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.734+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 >> 172.21.15.67:0/2811838334 conn(0x7fe09c073090 msgr2=0x7fe09c0734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.735+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 shutdown_connections 2024-04-25T15:26:48.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.736+0000 7fe0a14b1640 1 -- 172.21.15.67:0/2811838334 wait complete. 2024-04-25T15:26:48.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:26:48.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.877+0000 7ff2924af640 1 Processor -- start 2024-04-25T15:26:48.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.877+0000 7ff2924af640 1 -- start start 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff2924af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff28c06ccd0 0x7ff28c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff2924af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff28c0d1850 con 0x7ff28c0d0e00 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff2924af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff28c160de0 con 0x7ff28c0d4a50 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff2924af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff28c1610c0 con 0x7ff28c06ccd0 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff28c06ccd0 0x7ff28c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff28c06ccd0 0x7ff28c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44758/0 (socket says 172.21.15.67:44758) 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28b7fe640 1 -- 172.21.15.67:0/157098253 learned_addr learned my addr 172.21.15.67:0/157098253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28bfff640 1 --1- 172.21.15.67:0/157098253 >> v1:172.21.15.67:6789/0 conn(0x7ff28c0d0e00 0x7ff28c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43234/0 (socket says 172.21.15.67:43234) 2024-04-25T15:26:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28b7fe640 1 -- 172.21.15.67:0/157098253 >> v1:172.21.15.176:6789/0 conn(0x7ff28c0d4a50 legacy=0x7ff28c0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28b7fe640 1 -- 172.21.15.67:0/157098253 >> v1:172.21.15.67:6789/0 conn(0x7ff28c0d0e00 legacy=0x7ff28c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28b7fe640 1 -- 172.21.15.67:0/157098253 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c1613a0 con 0x7ff28c06ccd0 2024-04-25T15:26:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.879+0000 7ff28affd640 1 -- 172.21.15.67:0/157098253 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319079409 0 0) 0x7ff28c0d1850 con 0x7ff28c0d0e00 2024-04-25T15:26:48.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.880+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/157098253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ccd0 0x7ff28c0d1210 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009870 tx=0x7ff27c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268e4c13f610e4a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:48.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.880+0000 7ff28affd640 1 -- 172.21.15.67:0/157098253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c037070 con 0x7ff28c06ccd0 2024-04-25T15:26:48.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.880+0000 7ff28affd640 1 -- 172.21.15.67:0/157098253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff27c032440 con 0x7ff28c06ccd0 2024-04-25T15:26:48.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.880+0000 7ff28affd640 1 -- 172.21.15.67:0/157098253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c032810 con 0x7ff28c06ccd0 2024-04-25T15:26:48.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.881+0000 7ff2924af640 1 -- 172.21.15.67:0/157098253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ccd0 msgr2=0x7ff28c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.881+0000 7ff2924af640 1 --2- 172.21.15.67:0/157098253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ccd0 0x7ff28c0d1210 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009870 tx=0x7ff27c02a270 comp rx=0 tx=0).stop 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.881+0000 7ff2924af640 1 -- 172.21.15.67:0/157098253 shutdown_connections 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.881+0000 7ff2924af640 1 --2- 172.21.15.67:0/157098253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c06ccd0 0x7ff28c0d1210 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.881+0000 7ff2924af640 1 -- 172.21.15.67:0/157098253 >> 172.21.15.67:0/157098253 conn(0x7ff28c0d5d90 msgr2=0x7ff28c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.881+0000 7ff2924af640 1 -- 172.21.15.67:0/157098253 shutdown_connections 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.881+0000 7ff2924af640 1 -- 172.21.15.67:0/157098253 wait complete. 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.883+0000 7ff2924af640 1 Processor -- start 2024-04-25T15:26:48.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.883+0000 7ff2924af640 1 -- start start 2024-04-25T15:26:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff2924af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0d0e00 0x7ff28c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff2924af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c0d4a50 0x7ff28c0cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff2924af640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c14e100 0x7ff28c15ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff2924af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c164e00 con 0x7ff28c0d0e00 2024-04-25T15:26:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff2924af640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c0b5320 con 0x7ff28c14e100 2024-04-25T15:26:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff2924af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff28c164f80 con 0x7ff28c0d4a50 2024-04-25T15:26:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff28bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0d0e00 0x7ff28c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff28bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0d0e00 0x7ff28c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44760/0 (socket says 172.21.15.67:44760) 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff28bfff640 1 -- 172.21.15.67:0/1113016345 learned_addr learned my addr 172.21.15.67:0/1113016345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff290a4c640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c14e100 0x7ff28c15ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.884+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c0d4a50 0x7ff28c0cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff28bfff640 1 -- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c0d4a50 msgr2=0x7ff28c0cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff28bfff640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c0d4a50 0x7ff28c0cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff28bfff640 1 -- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c14e100 msgr2=0x7ff28c15ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff28bfff640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c14e100 0x7ff28c15ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff28bfff640 1 -- 172.21.15.67:0/1113016345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c0cb340 con 0x7ff28c0d0e00 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c0d4a50 0x7ff28c0cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff28bfff640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0d0e00 0x7ff28c0cae00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff27400c3f0 tx=0x7ff27400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:48.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff288ff9640 1 -- 172.21.15.67:0/1113016345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740074e0 con 0x7ff28c0d0e00 2024-04-25T15:26:48.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff288ff9640 1 -- 172.21.15.67:0/1113016345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2740047e0 con 0x7ff28c0d0e00 2024-04-25T15:26:48.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c0c9830 con 0x7ff28c0d0e00 2024-04-25T15:26:48.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff288ff9640 1 -- 172.21.15.67:0/1113016345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff274004bd0 con 0x7ff28c0d0e00 2024-04-25T15:26:48.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.885+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c0ca040 con 0x7ff28c0d0e00 2024-04-25T15:26:48.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.886+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28c050740 con 0x7ff28c0d0e00 2024-04-25T15:26:48.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.887+0000 7ff288ff9640 1 -- 172.21.15.67:0/1113016345 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff274011070 con 0x7ff28c0d0e00 2024-04-25T15:26:48.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.887+0000 7ff288ff9640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff26c0744b0 0x7ff26c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:48.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.888+0000 7ff288ff9640 1 -- 172.21.15.67:0/1113016345 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5609+0+0 (secure 0 0 0) 0x7ff274094bf0 con 0x7ff28c0d0e00 2024-04-25T15:26:48.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.888+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff26c0744b0 0x7ff26c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:48.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.888+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff26c0744b0 0x7ff26c0769c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff27c034000 tx=0x7ff27c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:48.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:48.892+0000 7ff288ff9640 1 -- 172.21.15.67:0/1113016345 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff27405f470 con 0x7ff28c0d0e00 2024-04-25T15:26:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.119+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff28c163080 con 0x7ff28c0d0e00 2024-04-25T15:26:49.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.754+0000 7ff288ff9640 1 -- 172.21.15.67:0/1113016345 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) ==== 172+0+0 (secure 0 0 0) 0x7ff27405ed70 con 0x7ff28c0d0e00 2024-04-25T15:26:49.755 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 6 and data pool 7 2024-04-25T15:26:49.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff26c0744b0 msgr2=0x7ff26c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff26c0744b0 0x7ff26c0769c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff27c034000 tx=0x7ff27c02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0d0e00 msgr2=0x7ff28c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0d0e00 0x7ff28c0cae00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff27400c3f0 tx=0x7ff27400c8b0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 shutdown_connections 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff26c0744b0 0x7ff26c0769c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c14e100 0x7ff28c15ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c0d4a50 0x7ff28c0cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 --2- 172.21.15.67:0/1113016345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0d0e00 0x7ff28c0cae00 secure :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff27400c3f0 tx=0x7ff27400c8b0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.761+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 >> 172.21.15.67:0/1113016345 conn(0x7ff28c0d5d90 msgr2=0x7ff28c164470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:49.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.764+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 shutdown_connections 2024-04-25T15:26:49.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.765+0000 7ff2924af640 1 -- 172.21.15.67:0/1113016345 wait complete. 2024-04-25T15:26:49.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.903+0000 7fb33954d640 1 Processor -- start 2024-04-25T15:26:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.903+0000 7fb33954d640 1 -- start start 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb33954d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb334164b00 0x7fb3340b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb33954d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb334160fa0 0x7fb3340b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb33954d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3341615f0 con 0x7fb33415df80 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb33954d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb3341613b0 con 0x7fb334164b00 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb33954d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb334164670 con 0x7fb334160fa0 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb332ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb334164b00 0x7fb3340b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb332ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb334164b00 0x7fb3340b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44776/0 (socket says 172.21.15.67:44776) 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb332ffd640 1 -- 172.21.15.67:0/3153962516 learned_addr learned my addr 172.21.15.67:0/3153962516 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb332ffd640 1 -- 172.21.15.67:0/3153962516 >> v2:172.21.15.176:3300/0 conn(0x7fb334160fa0 msgr2=0x7fb3340b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb3337fe640 1 --2- 172.21.15.67:0/3153962516 >> v2:172.21.15.176:3300/0 conn(0x7fb334160fa0 0x7fb3340b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb332ffd640 1 --2- 172.21.15.67:0/3153962516 >> v2:172.21.15.176:3300/0 conn(0x7fb334160fa0 0x7fb3340b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb332ffd640 1 -- 172.21.15.67:0/3153962516 >> v1:172.21.15.176:6789/0 conn(0x7fb33415df80 legacy=0x7fb334160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb332ffd640 1 -- 172.21.15.67:0/3153962516 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb334164020 con 0x7fb334164b00 2024-04-25T15:26:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.905+0000 7fb3337fe640 1 --2- 172.21.15.67:0/3153962516 >> v2:172.21.15.176:3300/0 conn(0x7fb334160fa0 0x7fb3340b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.906+0000 7fb332ffd640 1 --2- 172.21.15.67:0/3153962516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334164b00 0x7fb3340b7b80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb3340d3e80 tx=0x7fb32c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d6337d64468c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.906+0000 7fb331ffb640 1 -- 172.21.15.67:0/3153962516 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c036070 con 0x7fb334164b00 2024-04-25T15:26:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.906+0000 7fb331ffb640 1 -- 172.21.15.67:0/3153962516 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb32c031600 con 0x7fb334164b00 2024-04-25T15:26:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.906+0000 7fb331ffb640 1 -- 172.21.15.67:0/3153962516 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0319d0 con 0x7fb334164b00 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 -- 172.21.15.67:0/3153962516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334164b00 msgr2=0x7fb3340b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 --2- 172.21.15.67:0/3153962516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334164b00 0x7fb3340b7b80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb3340d3e80 tx=0x7fb32c02a6f0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 -- 172.21.15.67:0/3153962516 shutdown_connections 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 --2- 172.21.15.67:0/3153962516 >> v2:172.21.15.176:3300/0 conn(0x7fb334160fa0 0x7fb3340b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 --2- 172.21.15.67:0/3153962516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334164b00 0x7fb3340b7b80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 -- 172.21.15.67:0/3153962516 >> 172.21.15.67:0/3153962516 conn(0x7fb3340d2bc0 msgr2=0x7fb3340d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 -- 172.21.15.67:0/3153962516 shutdown_connections 2024-04-25T15:26:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.907+0000 7fb33954d640 1 -- 172.21.15.67:0/3153962516 wait complete. 2024-04-25T15:26:49.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.909+0000 7fb33954d640 1 Processor -- start 2024-04-25T15:26:49.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.909+0000 7fb33954d640 1 -- start start 2024-04-25T15:26:49.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb33954d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb33415df80 0x7fb3340cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:49.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb33954d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334160fa0 0x7fb33414a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:49.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb33954d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb334164b00 0x7fb334149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:49.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb33954d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb3340b4a00 con 0x7fb334160fa0 2024-04-25T15:26:49.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb33954d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb3340b4880 con 0x7fb33415df80 2024-04-25T15:26:49.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb33954d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb3340b4b80 con 0x7fb334164b00 2024-04-25T15:26:49.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334160fa0 0x7fb33414a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334160fa0 0x7fb33414a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44780/0 (socket says 172.21.15.67:44780) 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 -- 172.21.15.67:0/152076961 learned_addr learned my addr 172.21.15.67:0/152076961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb332ffd640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb33415df80 0x7fb3340cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3337fe640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb334164b00 0x7fb334149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 -- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb334164b00 msgr2=0x7fb334149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb334164b00 0x7fb334149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 -- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb33415df80 msgr2=0x7fb3340cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb33415df80 0x7fb3340cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.910+0000 7fb3327fc640 1 -- 172.21.15.67:0/152076961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33414a160 con 0x7fb334160fa0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.911+0000 7fb3327fc640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334160fa0 0x7fb33414a460 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb32800b3f0 tx=0x7fb32800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.911+0000 7fb31b7fe640 1 -- 172.21.15.67:0/152076961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328015020 con 0x7fb334160fa0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.911+0000 7fb31b7fe640 1 -- 172.21.15.67:0/152076961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb328011830 con 0x7fb334160fa0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.911+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3341fa130 con 0x7fb334160fa0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.911+0000 7fb31b7fe640 1 -- 172.21.15.67:0/152076961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328011c20 con 0x7fb334160fa0 2024-04-25T15:26:49.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.911+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3341fa6f0 con 0x7fb334160fa0 2024-04-25T15:26:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.912+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3341fa8f0 con 0x7fb334160fa0 2024-04-25T15:26:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.913+0000 7fb31b7fe640 1 -- 172.21.15.67:0/152076961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb328018020 con 0x7fb334160fa0 2024-04-25T15:26:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.913+0000 7fb31b7fe640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3200744b0 0x7fb3200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.913+0000 7fb31b7fe640 1 -- 172.21.15.67:0/152076961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fb328094d70 con 0x7fb334160fa0 2024-04-25T15:26:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.914+0000 7fb332ffd640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3200744b0 0x7fb3200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:49.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.914+0000 7fb332ffd640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3200744b0 0x7fb3200769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb32c033000 tx=0x7fb32c009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:49.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:49.918+0000 7fb31b7fe640 1 -- 172.21.15.67:0/152076961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb32801f030 con 0x7fb334160fa0 2024-04-25T15:26:50.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.103+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb334050740 con 0x7fb334160fa0 2024-04-25T15:26:50.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.104+0000 7fb31b7fe640 1 -- 172.21.15.67:0/152076961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16590 (secure 0 0 0) 0x7fb32805eeb0 con 0x7fb334160fa0 2024-04-25T15:26:50.106 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:50.106 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:49.120467+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3200744b0 msgr2=0x7fb3200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3200744b0 0x7fb3200769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb32c033000 tx=0x7fb32c009000 comp rx=0 tx=0).stop 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334160fa0 msgr2=0x7fb33414a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334160fa0 0x7fb33414a460 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb32800b3f0 tx=0x7fb32800b8b0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 shutdown_connections 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3200744b0 0x7fb3200769c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb334164b00 0x7fb334149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb334160fa0 0x7fb33414a460 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 --2- 172.21.15.67:0/152076961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb33415df80 0x7fb3340cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 >> 172.21.15.67:0/152076961 conn(0x7fb3340d2bc0 msgr2=0x7fb33415f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 shutdown_connections 2024-04-25T15:26:50.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.106+0000 7fb33954d640 1 -- 172.21.15.67:0/152076961 wait complete. 2024-04-25T15:26:50.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:50.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.230+0000 7fc7a1d01640 1 Processor -- start 2024-04-25T15:26:50.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.230+0000 7fc7a1d01640 1 -- start start 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc7a1d01640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc79c0d0e00 0x7fc79c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc7a1d01640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c06c5c0 0x7fc79c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc7a1d01640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc79c06cf10 con 0x7fc79c0b7810 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc7a1d01640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc79c0d5a60 con 0x7fc79c0d0e00 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc7a1d01640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc79c0d1850 con 0x7fc79c06c5c0 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc79b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc79c0d0e00 0x7fc79c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc79b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc79c0d0e00 0x7fc79c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44784/0 (socket says 172.21.15.67:44784) 2024-04-25T15:26:50.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc79b7fe640 1 -- 172.21.15.67:0/3877355931 learned_addr learned my addr 172.21.15.67:0/3877355931 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.231+0000 7fc79bfff640 1 --1- 172.21.15.67:0/3877355931 >> v1:172.21.15.67:6789/0 conn(0x7fc79c0b7810 0x7fc79c0b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43244/0 (socket says 172.21.15.67:43244) 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.232+0000 7fc79affd640 1 --2- 172.21.15.67:0/3877355931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c06c5c0 0x7fc79c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.232+0000 7fc79b7fe640 1 -- 172.21.15.67:0/3877355931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c06c5c0 msgr2=0x7fc79c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.232+0000 7fc79b7fe640 1 --2- 172.21.15.67:0/3877355931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c06c5c0 0x7fc79c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.232+0000 7fc79b7fe640 1 -- 172.21.15.67:0/3877355931 >> v1:172.21.15.67:6789/0 conn(0x7fc79c0b7810 legacy=0x7fc79c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.232+0000 7fc79b7fe640 1 -- 172.21.15.67:0/3877355931 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c160de0 con 0x7fc79c0d0e00 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.232+0000 7fc79a7fc640 1 -- 172.21.15.67:0/3877355931 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700954587 0 0) 0x7fc794001490 con 0x7fc79c0b7810 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.232+0000 7fc79affd640 1 --2- 172.21.15.67:0/3877355931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c06c5c0 0x7fc79c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc79b7fe640 1 --2- 172.21.15.67:0/3877355931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0d0e00 0x7fc79c0d1210 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc790009870 tx=0x7fc79002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb19546c0b887853 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:50.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc79a7fc640 1 -- 172.21.15.67:0/3877355931 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790037070 con 0x7fc79c0d0e00 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc79a7fc640 1 -- 172.21.15.67:0/3877355931 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc790032440 con 0x7fc79c0d0e00 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc79a7fc640 1 -- 172.21.15.67:0/3877355931 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790032810 con 0x7fc79c0d0e00 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc7a1d01640 1 -- 172.21.15.67:0/3877355931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0d0e00 msgr2=0x7fc79c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/3877355931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0d0e00 0x7fc79c0d1210 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc790009870 tx=0x7fc79002a250 comp rx=0 tx=0).stop 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc7a1d01640 1 -- 172.21.15.67:0/3877355931 shutdown_connections 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/3877355931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c06c5c0 0x7fc79c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/3877355931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0d0e00 0x7fc79c0d1210 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.233+0000 7fc7a1d01640 1 -- 172.21.15.67:0/3877355931 >> 172.21.15.67:0/3877355931 conn(0x7fc79c0d5d90 msgr2=0x7fc79c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.234+0000 7fc7a1d01640 1 -- 172.21.15.67:0/3877355931 shutdown_connections 2024-04-25T15:26:50.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.234+0000 7fc7a1d01640 1 -- 172.21.15.67:0/3877355931 wait complete. 2024-04-25T15:26:50.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.235+0000 7fc7a1d01640 1 Processor -- start 2024-04-25T15:26:50.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.235+0000 7fc7a1d01640 1 -- start start 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc7a1d01640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc79c06c5c0 0x7fc79c1f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc7a1d01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0b7810 0x7fc79c14f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc7a1d01640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c0d0e00 0x7fc79c14ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc7a1d01640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c160ad0 con 0x7fc79c0b7810 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc7a1d01640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c1613a0 con 0x7fc79c06c5c0 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc7a1d01640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc79c0b5320 con 0x7fc79c0d0e00 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0b7810 0x7fc79c14f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0b7810 0x7fc79c14f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44788/0 (socket says 172.21.15.67:44788) 2024-04-25T15:26:50.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 -- 172.21.15.67:0/4143791994 learned_addr learned my addr 172.21.15.67:0/4143791994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79bfff640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c0d0e00 0x7fc79c14ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79b7fe640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc79c06c5c0 0x7fc79c1f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 -- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c0d0e00 msgr2=0x7fc79c14ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c0d0e00 0x7fc79c14ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 -- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc79c06c5c0 msgr2=0x7fc79c1f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc79c06c5c0 0x7fc79c1f61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 -- 172.21.15.67:0/4143791994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c0acdd0 con 0x7fc79c0b7810 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.237+0000 7fc79affd640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0b7810 0x7fc79c14f9e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d800 tx=0x7fc78c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.238+0000 7fc783fff640 1 -- 172.21.15.67:0/4143791994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c004030 con 0x7fc79c0b7810 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.238+0000 7fc783fff640 1 -- 172.21.15.67:0/4143791994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc78c0041d0 con 0x7fc79c0b7810 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.238+0000 7fc783fff640 1 -- 172.21.15.67:0/4143791994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0096a0 con 0x7fc79c0b7810 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.238+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c0ce8d0 con 0x7fc79c0b7810 2024-04-25T15:26:50.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.238+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79c0cee90 con 0x7fc79c0b7810 2024-04-25T15:26:50.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.239+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79c050740 con 0x7fc79c0b7810 2024-04-25T15:26:50.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.239+0000 7fc783fff640 1 -- 172.21.15.67:0/4143791994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc78c009840 con 0x7fc79c0b7810 2024-04-25T15:26:50.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.240+0000 7fc783fff640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7880744b0 0x7fc7880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.240+0000 7fc783fff640 1 -- 172.21.15.67:0/4143791994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fc78c095290 con 0x7fc79c0b7810 2024-04-25T15:26:50.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.240+0000 7fc79b7fe640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7880744b0 0x7fc7880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.241+0000 7fc79b7fe640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7880744b0 0x7fc7880769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc790034000 tx=0x7fc79002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:50.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.245+0000 7fc783fff640 1 -- 172.21.15.67:0/4143791994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc78c05f4f0 con 0x7fc79c0b7810 2024-04-25T15:26:50.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.430+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc79c0d4320 con 0x7fc79c0b7810 2024-04-25T15:26:50.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.432+0000 7fc783fff640 1 -- 172.21.15.67:0/4143791994 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16590 (secure 0 0 0) 0x7fc78c0637f0 con 0x7fc79c0b7810 2024-04-25T15:26:50.433 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:50.433 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:49.120467+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7880744b0 msgr2=0x7fc7880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7880744b0 0x7fc7880769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc790034000 tx=0x7fc79002e000 comp rx=0 tx=0).stop 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0b7810 msgr2=0x7fc79c14f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0b7810 0x7fc79c14f9e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d800 tx=0x7fc78c00dcc0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 shutdown_connections 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7880744b0 0x7fc7880769c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc79c0d0e00 0x7fc79c14ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc79c0b7810 0x7fc79c14f9e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 --2- 172.21.15.67:0/4143791994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc79c06c5c0 0x7fc79c1f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 >> 172.21.15.67:0/4143791994 conn(0x7fc79c0d5d90 msgr2=0x7fc79c1601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:50.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 shutdown_connections 2024-04-25T15:26:50.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.434+0000 7fc7a1d01640 1 -- 172.21.15.67:0/4143791994 wait complete. 2024-04-25T15:26:50.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:26:50.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.560+0000 7f08749d7640 1 Processor -- start 2024-04-25T15:26:50.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.560+0000 7f08749d7640 1 -- start start 2024-04-25T15:26:50.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.561+0000 7f08749d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0870164640 0x7f08700b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.561+0000 7f08749d7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08700d3170 con 0x7f08700b0a70 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.561+0000 7f08749d7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08700d3450 con 0x7f0870164ad0 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.561+0000 7f08749d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f08700d2f40 con 0x7f0870164640 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f08700b0a70 0x7f08700b0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43250/0 (socket says 172.21.15.67:43250) 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086ed9d640 1 -- 172.21.15.67:0/3738538181 learned_addr learned my addr 172.21.15.67:0/3738538181 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f0867fff640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0870164640 0x7f08700b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308775360 0 0) 0x7f08700d3170 con 0x7f08700b0a70 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f085c003520 con 0x7f08700b0a70 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4259525431 0 0) 0x7f085c003520 con 0x7f08700b0a70 2024-04-25T15:26:50.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0870164640 msgr2=0x7f08700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0870164640 0x7f08700b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 >> v1:172.21.15.176:6789/0 conn(0x7f0870164ad0 legacy=0x7f0870163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087006d6d0 con 0x7f08700b0a70 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f08680019f0 con 0x7f08700b0a70 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 >> v1:172.21.15.67:6789/0 conn(0x7f08700b0a70 legacy=0x7f08700b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c0039e0 0x7f085c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c004800 0x7f085c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c005150 0x7f085c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f085c00bfb0 con 0x7f085c005150 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f085c00c130 con 0x7f085c004800 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f085c00c330 con 0x7f085c0039e0 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086ed9d640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c005150 0x7f085c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f086e59c640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c0039e0 0x7f085c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.562+0000 7f0867fff640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c004800 0x7f085c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.563+0000 7f086ed9d640 1 -- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c0039e0 msgr2=0x7f085c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.563+0000 7f086ed9d640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c0039e0 0x7f085c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.563+0000 7f086ed9d640 1 -- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c004800 msgr2=0x7f085c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.563+0000 7f086ed9d640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c004800 0x7f085c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.563+0000 7f086ed9d640 1 -- 172.21.15.67:0/3738538181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c003520 con 0x7f085c005150 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.563+0000 7f086ed9d640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c005150 0x7f085c00b9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f086800a7b0 tx=0x7f085c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f65ccf98b6ae2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086800a3d0 con 0x7f085c005150 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f086802f070 con 0x7f085c005150 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f086dd9b640 1 -- 172.21.15.67:0/3738538181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086802bca0 con 0x7f085c005150 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f08749d7640 1 -- 172.21.15.67:0/3738538181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c005150 msgr2=0x7f085c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f08749d7640 1 --2- 172.21.15.67:0/3738538181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c005150 0x7f085c00b9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f086800a7b0 tx=0x7f085c005830 comp rx=0 tx=0).stop 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f086ed9d640 1 -- 172.21.15.67:0/3738538181 reap_dead start 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f08749d7640 1 -- 172.21.15.67:0/3738538181 shutdown_connections 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f08749d7640 1 -- 172.21.15.67:0/3738538181 >> 172.21.15.67:0/3738538181 conn(0x7f08700d3aa0 msgr2=0x7f08700d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f08749d7640 1 -- 172.21.15.67:0/3738538181 shutdown_connections 2024-04-25T15:26:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.564+0000 7f08749d7640 1 -- 172.21.15.67:0/3738538181 wait complete. 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.566+0000 7f08749d7640 1 Processor -- start 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.566+0000 7f08749d7640 1 -- start start 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.567+0000 7f08749d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c005150 0x7f08701632d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f08749d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08700b4a90 0x7f08700b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f08749d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08701613e0 0x7f08701617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f08749d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f08700b9430 con 0x7f08700b4a90 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f08749d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0870164ee0 con 0x7f08701613e0 2024-04-25T15:26:50.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f08749d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f08700b95b0 con 0x7f085c005150 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08700b4a90 0x7f08700b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08700b4a90 0x7f08700b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44810/0 (socket says 172.21.15.67:44810) 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 -- 172.21.15.67:0/1416861841 learned_addr learned my addr 172.21.15.67:0/1416861841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f086e59c640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c005150 0x7f08701632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f086ed9d640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08701613e0 0x7f08701617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 -- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c005150 msgr2=0x7f08701632d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c005150 0x7f08701632d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 -- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08701613e0 msgr2=0x7f08701617f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08701613e0 0x7f08701617f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 -- 172.21.15.67:0/1416861841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08700b7e70 con 0x7f08700b4a90 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f086ed9d640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08701613e0 0x7f08701617f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f086e59c640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c005150 0x7f08701632d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0867fff640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08700b4a90 0x7f08700b4ea0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f08580119d0 tx=0x7f0858011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0866ffd640 1 -- 172.21.15.67:0/1416861841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085800fb50 con 0x7f08700b4a90 2024-04-25T15:26:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0866ffd640 1 -- 172.21.15.67:0/1416861841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085800fcf0 con 0x7f08700b4a90 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f0866ffd640 1 -- 172.21.15.67:0/1416861841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085800fea0 con 0x7f08700b4a90 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.568+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08700b8150 con 0x7f08700b4a90 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.569+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08700b8430 con 0x7f08700b4a90 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.569+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08700d4770 con 0x7f08700b4a90 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.570+0000 7f0866ffd640 1 -- 172.21.15.67:0/1416861841 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0858012040 con 0x7f08700b4a90 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.571+0000 7f0866ffd640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08500744b0 0x7f08500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.571+0000 7f0866ffd640 1 -- 172.21.15.67:0/1416861841 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0858095920 con 0x7f08700b4a90 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.571+0000 7f086e59c640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08500744b0 0x7f08500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.571+0000 7f086e59c640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08500744b0 0x7f08500769c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f08700746e0 tx=0x7f0860009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:50.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.575+0000 7f0866ffd640 1 -- 172.21.15.67:0/1416861841 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0858019030 con 0x7f08700b4a90 2024-04-25T15:26:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:50.808+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f08700b7bc0 con 0x7f08700b4a90 2024-04-25T15:26:51.723 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-25T15:26:51.722+0000 7fad41703640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:26:51.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.763+0000 7f0866ffd640 1 -- 172.21.15.67:0/1416861841 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f0858060140 con 0x7f08700b4a90 2024-04-25T15:26:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08500744b0 msgr2=0x7f08500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08500744b0 0x7f08500769c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f08700746e0 tx=0x7f0860009040 comp rx=0 tx=0).stop 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08700b4a90 msgr2=0x7f08700b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08700b4a90 0x7f08700b4ea0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f08580119d0 tx=0x7f0858011e90 comp rx=0 tx=0).stop 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 shutdown_connections 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f08500744b0 0x7f08500769c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08701613e0 0x7f08701617f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08700b4a90 0x7f08700b4ea0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.766+0000 7f08749d7640 1 --2- 172.21.15.67:0/1416861841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f085c005150 0x7f08701632d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.767+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 >> 172.21.15.67:0/1416861841 conn(0x7f08700d3aa0 msgr2=0x7f087015eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.767+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 shutdown_connections 2024-04-25T15:26:51.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.767+0000 7f08749d7640 1 -- 172.21.15.67:0/1416861841 wait complete. 2024-04-25T15:26:51.781 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:26:51.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.902+0000 7f04e295e640 1 Processor -- start 2024-04-25T15:26:51.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.902+0000 7f04e295e640 1 -- start start 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04e295e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f04dc164ad0 0x7f04dc0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04e295e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04dc163f30 0x7f04dc164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04e295e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04dc164880 con 0x7f04dc0d1f30 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04e295e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f04dc164ee0 con 0x7f04dc163f30 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04e295e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f04dc0b31e0 con 0x7f04dc164ad0 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04db7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04dc163f30 0x7f04dc164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04e0efb640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f04dc0d1f30 0x7f04dc0d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43264/0 (socket says 172.21.15.67:43264) 2024-04-25T15:26:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04e0efb640 1 -- 172.21.15.67:0/3905468551 learned_addr learned my addr 172.21.15.67:0/3905468551 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04dbfff640 1 --2- 172.21.15.67:0/3905468551 >> v2:172.21.15.176:3300/0 conn(0x7f04dc164ad0 0x7f04dc0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04db7fe640 1 -- 172.21.15.67:0/3905468551 >> v2:172.21.15.176:3300/0 conn(0x7f04dc164ad0 msgr2=0x7f04dc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04db7fe640 1 --2- 172.21.15.67:0/3905468551 >> v2:172.21.15.176:3300/0 conn(0x7f04dc164ad0 0x7f04dc0b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04db7fe640 1 -- 172.21.15.67:0/3905468551 >> v1:172.21.15.67:6789/0 conn(0x7f04dc0d1f30 legacy=0x7f04dc0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04db7fe640 1 -- 172.21.15.67:0/3905468551 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04dc0d3170 con 0x7f04dc163f30 2024-04-25T15:26:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.904+0000 7f04daffd640 1 -- 172.21.15.67:0/3905468551 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504999292 0 0) 0x7f04d4001490 con 0x7f04dc0d1f30 2024-04-25T15:26:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.905+0000 7f04db7fe640 1 --2- 172.21.15.67:0/3905468551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc163f30 0x7f04dc164340 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f04dc074530 tx=0x7f04cc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5c7064ca3e9927 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.905+0000 7f04daffd640 1 -- 172.21.15.67:0/3905468551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc037070 con 0x7f04dc163f30 2024-04-25T15:26:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.905+0000 7f04daffd640 1 -- 172.21.15.67:0/3905468551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04cc032440 con 0x7f04dc163f30 2024-04-25T15:26:51.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.905+0000 7f04daffd640 1 -- 172.21.15.67:0/3905468551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc032810 con 0x7f04dc163f30 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 -- 172.21.15.67:0/3905468551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc163f30 msgr2=0x7f04dc164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 --2- 172.21.15.67:0/3905468551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc163f30 0x7f04dc164340 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f04dc074530 tx=0x7f04cc02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 -- 172.21.15.67:0/3905468551 shutdown_connections 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 --2- 172.21.15.67:0/3905468551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc163f30 0x7f04dc164340 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 --2- 172.21.15.67:0/3905468551 >> v2:172.21.15.176:3300/0 conn(0x7f04dc164ad0 0x7f04dc0b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 -- 172.21.15.67:0/3905468551 >> 172.21.15.67:0/3905468551 conn(0x7f04dc0d3aa0 msgr2=0x7f04dc0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 -- 172.21.15.67:0/3905468551 shutdown_connections 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.906+0000 7f04e295e640 1 -- 172.21.15.67:0/3905468551 wait complete. 2024-04-25T15:26:51.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.908+0000 7f04e295e640 1 Processor -- start 2024-04-25T15:26:51.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.908+0000 7f04e295e640 1 -- start start 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04e295e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc0d1f30 0x7f04dc1f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04e295e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04dc163f30 0x7f04dc15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04e295e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04dc164ad0 0x7f04dc1fa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04e295e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc06d9b0 con 0x7f04dc0d1f30 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04e295e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04dc06d6d0 con 0x7f04dc163f30 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04e295e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f04dc0b9430 con 0x7f04dc164ad0 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc0d1f30 0x7f04dc1f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc0d1f30 0x7f04dc1f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44830/0 (socket says 172.21.15.67:44830) 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04dbfff640 1 -- 172.21.15.67:0/1556644233 learned_addr learned my addr 172.21.15.67:0/1556644233 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:51.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04db7fe640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04dc163f30 0x7f04dc15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.909+0000 7f04e0efb640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04dc164ad0 0x7f04dc1fa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04dbfff640 1 -- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04dc164ad0 msgr2=0x7f04dc1fa4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04dbfff640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04dc164ad0 0x7f04dc1fa4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04dbfff640 1 -- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04dc163f30 msgr2=0x7f04dc15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04dbfff640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04dc163f30 0x7f04dc15bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04dbfff640 1 -- 172.21.15.67:0/1556644233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04dc1f9ce0 con 0x7f04dc0d1f30 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04e0efb640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04dc164ad0 0x7f04dc1fa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04dbfff640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc0d1f30 0x7f04dc1f66a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f04d000d5a0 tx=0x7f04d000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04d8ff9640 1 -- 172.21.15.67:0/1556644233 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0019070 con 0x7f04dc0d1f30 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04d8ff9640 1 -- 172.21.15.67:0/1556644233 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d0004240 con 0x7f04dc0d1f30 2024-04-25T15:26:51.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04d8ff9640 1 -- 172.21.15.67:0/1556644233 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0003610 con 0x7f04dc0d1f30 2024-04-25T15:26:51.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04dc0cd6c0 con 0x7f04dc0d1f30 2024-04-25T15:26:51.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.910+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04dc0cdc80 con 0x7f04dc0d1f30 2024-04-25T15:26:51.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.911+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04dc0d4770 con 0x7f04dc0d1f30 2024-04-25T15:26:51.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.912+0000 7f04d8ff9640 1 -- 172.21.15.67:0/1556644233 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f04d001c070 con 0x7f04dc0d1f30 2024-04-25T15:26:51.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.912+0000 7f04d8ff9640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04c80744b0 0x7f04c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:51.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.912+0000 7f04d8ff9640 1 -- 172.21.15.67:0/1556644233 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f04d0094ab0 con 0x7f04dc0d1f30 2024-04-25T15:26:51.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.912+0000 7f04db7fe640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04c80744b0 0x7f04c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:51.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.913+0000 7f04db7fe640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04c80744b0 0x7f04c80769c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f04cc034000 tx=0x7f04cc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:51.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:51.917+0000 7f04d8ff9640 1 -- 172.21.15.67:0/1556644233 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04d005ebf0 con 0x7f04dc0d1f30 2024-04-25T15:26:52.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.139+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f04dc068820 con 0x7f04dc0d1f30 2024-04-25T15:26:52.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.765+0000 7f04d8ff9640 1 -- 172.21.15.67:0/1556644233 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7f04d0062ef0 con 0x7f04dc0d1f30 2024-04-25T15:26:52.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04c80744b0 msgr2=0x7f04c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:52.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04c80744b0 0x7f04c80769c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f04cc034000 tx=0x7f04cc02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:52.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc0d1f30 msgr2=0x7f04dc1f66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc0d1f30 0x7f04dc1f66a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f04d000d5a0 tx=0x7f04d000da60 comp rx=0 tx=0).stop 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 shutdown_connections 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04c80744b0 0x7f04c80769c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04dc164ad0 0x7f04dc1fa4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04dc163f30 0x7f04dc15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 --2- 172.21.15.67:0/1556644233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04dc0d1f30 0x7f04dc1f66a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 >> 172.21.15.67:0/1556644233 conn(0x7f04dc0d3aa0 msgr2=0x7f04dc0b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 shutdown_connections 2024-04-25T15:26:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.767+0000 7f04e295e640 1 -- 172.21.15.67:0/1556644233 wait complete. 2024-04-25T15:26:52.780 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:26:52.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:52.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.893+0000 7f8d2b0bd640 1 Processor -- start 2024-04-25T15:26:52.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.893+0000 7f8d2b0bd640 1 -- start start 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.894+0000 7f8d2b0bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d240d4570 0x7f8d240b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.894+0000 7f8d2b0bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d2406e8f0 0x7f8d240d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.894+0000 7f8d2b0bd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2406d870 con 0x7f8d240d09f0 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.894+0000 7f8d2b0bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8d2406e460 con 0x7f8d240d4570 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.894+0000 7f8d2b0bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2406db10 con 0x7f8d2406e8f0 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d28e59640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d240d4570 0x7f8d240b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d28e59640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d240d4570 0x7f8d240b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44844/0 (socket says 172.21.15.67:44844) 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d28e59640 1 -- 172.21.15.67:0/241633054 learned_addr learned my addr 172.21.15.67:0/241633054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d23fff640 1 --2- 172.21.15.67:0/241633054 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d2406e8f0 0x7f8d240d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d28e59640 1 -- 172.21.15.67:0/241633054 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d2406e8f0 msgr2=0x7f8d240d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d28e59640 1 --2- 172.21.15.67:0/241633054 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d2406e8f0 0x7f8d240d04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d28e59640 1 -- 172.21.15.67:0/241633054 >> v1:172.21.15.176:6789/0 conn(0x7f8d240d09f0 legacy=0x7f8d240b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d28e59640 1 -- 172.21.15.67:0/241633054 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2406e0f0 con 0x7f8d240d4570 2024-04-25T15:26:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.895+0000 7f8d23fff640 1 --2- 172.21.15.67:0/241633054 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d2406e8f0 0x7f8d240d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.896+0000 7f8d28e59640 1 --2- 172.21.15.67:0/241633054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d4570 0x7f8d240b4830 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8d2406da40 tx=0x7f8d1c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b11a55c7af69f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.896+0000 7f8d237fe640 1 -- 172.21.15.67:0/241633054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c02ae50 con 0x7f8d240d4570 2024-04-25T15:26:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.896+0000 7f8d237fe640 1 -- 172.21.15.67:0/241633054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d1c004030 con 0x7f8d240d4570 2024-04-25T15:26:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.896+0000 7f8d237fe640 1 -- 172.21.15.67:0/241633054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c004460 con 0x7f8d240d4570 2024-04-25T15:26:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.896+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/241633054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d4570 msgr2=0x7f8d240b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.896+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/241633054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d4570 0x7f8d240b4830 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8d2406da40 tx=0x7f8d1c02a140 comp rx=0 tx=0).stop 2024-04-25T15:26:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.897+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/241633054 shutdown_connections 2024-04-25T15:26:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.897+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/241633054 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d2406e8f0 0x7f8d240d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.897+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/241633054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d4570 0x7f8d240b4830 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.897+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/241633054 >> 172.21.15.67:0/241633054 conn(0x7f8d240d3a50 msgr2=0x7f8d240d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.897+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/241633054 shutdown_connections 2024-04-25T15:26:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.897+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/241633054 wait complete. 2024-04-25T15:26:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.898+0000 7f8d2b0bd640 1 Processor -- start 2024-04-25T15:26:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.898+0000 7f8d2b0bd640 1 -- start start 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2b0bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d2406e8f0 0x7f8d2414b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2b0bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d09f0 0x7f8d2414e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2b0bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d240d4570 0x7f8d240b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2b0bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d240b34d0 con 0x7f8d240d09f0 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2b0bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2406ed00 con 0x7f8d2406e8f0 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2b0bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d240b3650 con 0x7f8d240d4570 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d09f0 0x7f8d2414e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d09f0 0x7f8d2414e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44856/0 (socket says 172.21.15.67:44856) 2024-04-25T15:26:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 -- 172.21.15.67:0/2334331766 learned_addr learned my addr 172.21.15.67:0/2334331766 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d28e59640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d2406e8f0 0x7f8d2414b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2965a640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d240d4570 0x7f8d240b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 -- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d240d4570 msgr2=0x7f8d240b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d240d4570 0x7f8d240b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 -- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d2406e8f0 msgr2=0x7f8d2414b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d2406e8f0 0x7f8d2414b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d23fff640 1 -- 172.21.15.67:0/2334331766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2415d340 con 0x7f8d240d09f0 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.900+0000 7f8d2965a640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d240d4570 0x7f8d240b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.901+0000 7f8d23fff640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d09f0 0x7f8d2414e5c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400ce00 tx=0x7f8d1400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.901+0000 7f8d217fa640 1 -- 172.21.15.67:0/2334331766 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1400f890 con 0x7f8d240d09f0 2024-04-25T15:26:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.901+0000 7f8d217fa640 1 -- 172.21.15.67:0/2334331766 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d1400fa30 con 0x7f8d240d09f0 2024-04-25T15:26:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.901+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2415d620 con 0x7f8d240d09f0 2024-04-25T15:26:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.901+0000 7f8d217fa640 1 -- 172.21.15.67:0/2334331766 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1400fe40 con 0x7f8d240d09f0 2024-04-25T15:26:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.901+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2415dbe0 con 0x7f8d240d09f0 2024-04-25T15:26:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.902+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d2415d0a0 con 0x7f8d240d09f0 2024-04-25T15:26:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.902+0000 7f8d217fa640 1 -- 172.21.15.67:0/2334331766 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8d14018400 con 0x7f8d240d09f0 2024-04-25T15:26:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.903+0000 7f8d217fa640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d040744e0 0x7f8d040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.903+0000 7f8d217fa640 1 -- 172.21.15.67:0/2334331766 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f8d14095e90 con 0x7f8d240d09f0 2024-04-25T15:26:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.903+0000 7f8d28e59640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d040744e0 0x7f8d040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.904+0000 7f8d28e59640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d040744e0 0x7f8d040769f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c033000 tx=0x7f8d1c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:52.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:52.907+0000 7f8d217fa640 1 -- 172.21.15.67:0/2334331766 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d2415d0a0 con 0x7f8d240d09f0 2024-04-25T15:26:53.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.128+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d24050740 con 0x7f8d240d09f0 2024-04-25T15:26:53.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.129+0000 7f8d217fa640 1 -- 172.21.15.67:0/2334331766 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5187 (secure 0 0 0) 0x7f8d14014030 con 0x7f8d240d09f0 2024-04-25T15:26:53.130 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:53.130 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1556919488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1556919488},{"type":"v1","addr":"172.21.15.67:6833","nonce":1556919488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4071024412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4071024412},{"type":"v1","addr":"172.21.15.176:6837","nonce":4071024412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3791679818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3791679818},{"type":"v1","addr":"172.21.15.176:6836","nonce":3791679818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:49.120424+0000","modified":"2024-04-25T15:26:52.141230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1882238407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1882238407},{"type":"v1","addr":"172.21.15.67:6835","nonce":1882238407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":3}]} 2024-04-25T15:26:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.130+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d040744e0 msgr2=0x7f8d040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.130+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d040744e0 0x7f8d040769f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c033000 tx=0x7f8d1c02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.130+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d09f0 msgr2=0x7f8d2414e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.130+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d09f0 0x7f8d2414e5c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400ce00 tx=0x7f8d1400a650 comp rx=0 tx=0).stop 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 shutdown_connections 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d040744e0 0x7f8d040769f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d240d4570 0x7f8d240b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d240d09f0 0x7f8d2414e5c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 --2- 172.21.15.67:0/2334331766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d2406e8f0 0x7f8d2414b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 >> 172.21.15.67:0/2334331766 conn(0x7f8d240d3a50 msgr2=0x7f8d24163ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 shutdown_connections 2024-04-25T15:26:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.131+0000 7f8d2b0bd640 1 -- 172.21.15.67:0/2334331766 wait complete. 2024-04-25T15:26:53.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.255+0000 7fae8a2e0640 1 Processor -- start 2024-04-25T15:26:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.255+0000 7fae8a2e0640 1 -- start start 2024-04-25T15:26:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae8a2e0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae840b8290 0x7fae84160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae8a2e0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fae8415df80 0x7fae84160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae8a2e0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae841615f0 con 0x7fae84164b00 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae8a2e0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fae84164670 con 0x7fae840b8290 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae8a2e0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fae84164020 con 0x7fae8415df80 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae840b8290 0x7fae84160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae840b8290 0x7fae84160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44864/0 (socket says 172.21.15.67:44864) 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae837fe640 1 -- 172.21.15.67:0/3652416672 learned_addr learned my addr 172.21.15.67:0/3652416672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.256+0000 7fae8887d640 1 --2- 172.21.15.67:0/3652416672 >> v2:172.21.15.176:3300/0 conn(0x7fae8415df80 0x7fae84160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.257+0000 7fae837fe640 1 -- 172.21.15.67:0/3652416672 >> v2:172.21.15.176:3300/0 conn(0x7fae8415df80 msgr2=0x7fae84160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.257+0000 7fae837fe640 1 --2- 172.21.15.67:0/3652416672 >> v2:172.21.15.176:3300/0 conn(0x7fae8415df80 0x7fae84160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.257+0000 7fae837fe640 1 -- 172.21.15.67:0/3652416672 >> v1:172.21.15.176:6789/0 conn(0x7fae84164b00 legacy=0x7fae840b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.257+0000 7fae837fe640 1 -- 172.21.15.67:0/3652416672 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae84164300 con 0x7fae840b8290 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.257+0000 7fae837fe640 1 --2- 172.21.15.67:0/3652416672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae840b8290 0x7fae84160890 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fae840d3e80 tx=0x7fae7402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417cbc9c4293a2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae82ffd640 1 -- 172.21.15.67:0/3652416672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74037070 con 0x7fae840b8290 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae82ffd640 1 -- 172.21.15.67:0/3652416672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae74032440 con 0x7fae840b8290 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae82ffd640 1 -- 172.21.15.67:0/3652416672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74032810 con 0x7fae840b8290 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 -- 172.21.15.67:0/3652416672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae840b8290 msgr2=0x7fae84160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/3652416672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae840b8290 0x7fae84160890 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fae840d3e80 tx=0x7fae7402a270 comp rx=0 tx=0).stop 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 -- 172.21.15.67:0/3652416672 shutdown_connections 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/3652416672 >> v2:172.21.15.176:3300/0 conn(0x7fae8415df80 0x7fae84160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/3652416672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae840b8290 0x7fae84160890 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 -- 172.21.15.67:0/3652416672 >> 172.21.15.67:0/3652416672 conn(0x7fae840d2bc0 msgr2=0x7fae840d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 -- 172.21.15.67:0/3652416672 shutdown_connections 2024-04-25T15:26:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.258+0000 7fae8a2e0640 1 -- 172.21.15.67:0/3652416672 wait complete. 2024-04-25T15:26:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.260+0000 7fae8a2e0640 1 Processor -- start 2024-04-25T15:26:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.260+0000 7fae8a2e0640 1 -- start start 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae8a2e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8415df80 0x7fae840c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae8a2e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae84164b00 0x7fae840b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae8a2e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae840b70c0 0x7fae8415bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae8a2e0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae840b4a00 con 0x7fae84164b00 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae8a2e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae840b4880 con 0x7fae8415df80 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae8a2e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae840b4b80 con 0x7fae840b70c0 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae84164b00 0x7fae840b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae84164b00 0x7fae840b6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44878/0 (socket says 172.21.15.67:44878) 2024-04-25T15:26:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 -- 172.21.15.67:0/316508686 learned_addr learned my addr 172.21.15.67:0/316508686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae83fff640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8415df80 0x7fae840c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae8887d640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae840b70c0 0x7fae8415bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 -- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae840b70c0 msgr2=0x7fae8415bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae840b70c0 0x7fae8415bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 -- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8415df80 msgr2=0x7fae840c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8415df80 0x7fae840c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 -- 172.21.15.67:0/316508686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae840c9300 con 0x7fae84164b00 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.262+0000 7fae837fe640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae84164b00 0x7fae840b6b80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fae74000c00 tx=0x7fae740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.263+0000 7fae80ff9640 1 -- 172.21.15.67:0/316508686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74041030 con 0x7fae84164b00 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.263+0000 7fae80ff9640 1 -- 172.21.15.67:0/316508686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae74030440 con 0x7fae84164b00 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.263+0000 7fae80ff9640 1 -- 172.21.15.67:0/316508686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74037040 con 0x7fae84164b00 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.263+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae840b50e0 con 0x7fae84164b00 2024-04-25T15:26:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.263+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae840b56a0 con 0x7fae84164b00 2024-04-25T15:26:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.264+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae84050740 con 0x7fae84164b00 2024-04-25T15:26:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.264+0000 7fae80ff9640 1 -- 172.21.15.67:0/316508686 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fae74044070 con 0x7fae84164b00 2024-04-25T15:26:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.265+0000 7fae80ff9640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae700744e0 0x7fae700769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.265+0000 7fae80ff9640 1 -- 172.21.15.67:0/316508686 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fae740b58a0 con 0x7fae84164b00 2024-04-25T15:26:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.265+0000 7fae83fff640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae700744e0 0x7fae700769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.265+0000 7fae83fff640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae700744e0 0x7fae700769f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fae780016e0 tx=0x7fae78009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.269+0000 7fae80ff9640 1 -- 172.21.15.67:0/316508686 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae7407f960 con 0x7fae84164b00 2024-04-25T15:26:53.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.454+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae840d4170 con 0x7fae84164b00 2024-04-25T15:26:53.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.456+0000 7fae80ff9640 1 -- 172.21.15.67:0/316508686 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16577 (secure 0 0 0) 0x7fae74083c60 con 0x7fae84164b00 2024-04-25T15:26:53.457 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:53.457 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":44,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:50.738984+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.457+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae700744e0 msgr2=0x7fae700769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.457+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae700744e0 0x7fae700769f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fae780016e0 tx=0x7fae78009040 comp rx=0 tx=0).stop 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.457+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae84164b00 msgr2=0x7fae840b6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.457+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae84164b00 0x7fae840b6b80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fae74000c00 tx=0x7fae740046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 shutdown_connections 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fae700744e0 0x7fae700769f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae840b70c0 0x7fae8415bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae84164b00 0x7fae840b6b80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 --2- 172.21.15.67:0/316508686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8415df80 0x7fae840c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 >> 172.21.15.67:0/316508686 conn(0x7fae840d2bc0 msgr2=0x7fae840d5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 shutdown_connections 2024-04-25T15:26:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.458+0000 7fae8a2e0640 1 -- 172.21.15.67:0/316508686 wait complete. 2024-04-25T15:26:53.470 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:26:53.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:26:53.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.581+0000 7f7737cb1640 1 Processor -- start 2024-04-25T15:26:53.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.581+0000 7f7737cb1640 1 -- start start 2024-04-25T15:26:53.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7737cb1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f773006e8f0 0x7f77300b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7737cb1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773006d870 con 0x7f77300d4570 2024-04-25T15:26:53.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7737cb1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773006db10 con 0x7f77300b4830 2024-04-25T15:26:53.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7737cb1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f77300b4c40 con 0x7f773006e8f0 2024-04-25T15:26:53.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7735a4d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f77300d4570 0x7f77300d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43276/0 (socket says 172.21.15.67:43276) 2024-04-25T15:26:53.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7735a4d640 1 -- 172.21.15.67:0/1546276744 learned_addr learned my addr 172.21.15.67:0/1546276744 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f773624e640 1 --2- 172.21.15.67:0/1546276744 >> v2:172.21.15.176:3300/0 conn(0x7f773006e8f0 0x7f77300b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3013924546 0 0) 0x7f773006d870 con 0x7f77300d4570 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f771c003520 con 0x7f77300d4570 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021387294 0 0) 0x7f773006db10 con 0x7f77300b4830 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f773006d870 con 0x7f77300b4830 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2322441355 0 0) 0x7f771c003520 con 0x7f77300d4570 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f773624e640 1 -- 172.21.15.67:0/1546276744 >> v1:172.21.15.176:6789/0 conn(0x7f77300b4830 legacy=0x7f77300b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f773624e640 1 -- 172.21.15.67:0/1546276744 >> v1:172.21.15.67:6789/0 conn(0x7f77300d4570 legacy=0x7f77300d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.583+0000 7f773624e640 1 -- 172.21.15.67:0/1546276744 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773006e0f0 con 0x7f773006e8f0 2024-04-25T15:26:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.584+0000 7f773624e640 1 --2- 172.21.15.67:0/1546276744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773006e8f0 0x7f77300b34d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7728009d90 tx=0x7f772802a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eb8e062bc46568 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.584+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772c0017f0 con 0x7f773006e8f0 2024-04-25T15:26:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.584+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7728007540 con 0x7f773006e8f0 2024-04-25T15:26:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.584+0000 7f7734a4b640 1 -- 172.21.15.67:0/1546276744 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280078f0 con 0x7f773006e8f0 2024-04-25T15:26:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.585+0000 7f7737cb1640 1 -- 172.21.15.67:0/1546276744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773006e8f0 msgr2=0x7f77300b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.585+0000 7f7737cb1640 1 --2- 172.21.15.67:0/1546276744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773006e8f0 0x7f77300b34d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7728009d90 tx=0x7f772802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.585+0000 7f7737cb1640 1 -- 172.21.15.67:0/1546276744 shutdown_connections 2024-04-25T15:26:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.585+0000 7f7737cb1640 1 --2- 172.21.15.67:0/1546276744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773006e8f0 0x7f77300b34d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.585+0000 7f7737cb1640 1 -- 172.21.15.67:0/1546276744 >> 172.21.15.67:0/1546276744 conn(0x7f77300d3a50 msgr2=0x7f77300d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.585+0000 7f7737cb1640 1 -- 172.21.15.67:0/1546276744 shutdown_connections 2024-04-25T15:26:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.585+0000 7f7737cb1640 1 -- 172.21.15.67:0/1546276744 wait complete. 2024-04-25T15:26:53.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.587+0000 7f7737cb1640 1 Processor -- start 2024-04-25T15:26:53.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.587+0000 7f7737cb1640 1 -- start start 2024-04-25T15:26:53.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7737cb1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773006e8f0 0x7f77300a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7737cb1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77300b4830 0x7f773015ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7737cb1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77300d4570 0x7f77301f9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7737cb1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77300d0fb0 con 0x7f773006e8f0 2024-04-25T15:26:53.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7737cb1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f773006e460 con 0x7f77300b4830 2024-04-25T15:26:53.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7737cb1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f773006ed00 con 0x7f77300d4570 2024-04-25T15:26:53.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7735a4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773006e8f0 0x7f77300a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7735a4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773006e8f0 0x7f77300a0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44890/0 (socket says 172.21.15.67:44890) 2024-04-25T15:26:53.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.588+0000 7f7735a4d640 1 -- 172.21.15.67:0/2283998458 learned_addr learned my addr 172.21.15.67:0/2283998458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:53.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f773624e640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77300d4570 0x7f77301f9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f773524c640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77300b4830 0x7f773015ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7735a4d640 1 -- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77300d4570 msgr2=0x7f77301f9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7735a4d640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77300d4570 0x7f77301f9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7735a4d640 1 -- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77300b4830 msgr2=0x7f773015ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7735a4d640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77300b4830 0x7f773015ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7735a4d640 1 -- 172.21.15.67:0/2283998458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773015f600 con 0x7f773006e8f0 2024-04-25T15:26:53.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7735a4d640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773006e8f0 0x7f77300a0fe0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c5a0 tx=0x7f772c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f77267fc640 1 -- 172.21.15.67:0/2283998458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772c017070 con 0x7f773006e8f0 2024-04-25T15:26:53.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f77267fc640 1 -- 172.21.15.67:0/2283998458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f772c007500 con 0x7f773006e8f0 2024-04-25T15:26:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773015f8e0 con 0x7f773006e8f0 2024-04-25T15:26:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f77267fc640 1 -- 172.21.15.67:0/2283998458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772c0059a0 con 0x7f773006e8f0 2024-04-25T15:26:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.589+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f773015fea0 con 0x7f773006e8f0 2024-04-25T15:26:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.590+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7730050740 con 0x7f773006e8f0 2024-04-25T15:26:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.591+0000 7f77267fc640 1 -- 172.21.15.67:0/2283998458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f772c01a070 con 0x7f773006e8f0 2024-04-25T15:26:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.591+0000 7f77267fc640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77140744b0 0x7f77140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.591+0000 7f77267fc640 1 -- 172.21.15.67:0/2283998458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f772c0945b0 con 0x7f773006e8f0 2024-04-25T15:26:53.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.592+0000 7f773524c640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77140744b0 0x7f77140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.592+0000 7f773524c640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77140744b0 0x7f77140769c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f77300b0ef0 tx=0x7f7720009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.596+0000 7f77267fc640 1 -- 172.21.15.67:0/2283998458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f772c05e780 con 0x7f773006e8f0 2024-04-25T15:26:53.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.782+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f77300d1220 con 0x7f77140744b0 2024-04-25T15:26:53.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.788+0000 7f77267fc640 1 -- 172.21.15.67:0/2283998458 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7f77300d1220 con 0x7f77140744b0 2024-04-25T15:26:53.792 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:53.793 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.791+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77140744b0 msgr2=0x7f77140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.791+0000 7f7737cb1640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77140744b0 0x7f77140769c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f77300b0ef0 tx=0x7f7720009040 comp rx=0 tx=0).stop 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.791+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773006e8f0 msgr2=0x7f77300a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.791+0000 7f7737cb1640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773006e8f0 0x7f77300a0fe0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c5a0 tx=0x7f772c00ca60 comp rx=0 tx=0).stop 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 shutdown_connections 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77140744b0 0x7f77140769c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77300d4570 0x7f77301f9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77300b4830 0x7f773015ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 --2- 172.21.15.67:0/2283998458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773006e8f0 0x7f77300a0fe0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 >> 172.21.15.67:0/2283998458 conn(0x7f77300d3a50 msgr2=0x7f77301629f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:53.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 shutdown_connections 2024-04-25T15:26:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.792+0000 7f7737cb1640 1 -- 172.21.15.67:0/2283998458 wait complete. 2024-04-25T15:26:53.803 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-04-25T15:26:52.814855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221412,"kb_used_data":5244,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753308,"statfs":{"total":773094113280,"available":772867387392,"internally_reserved":0,"allocated":5369856,"data_stored":2857720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.089941"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911636+0000","last_change":"2024-04-25T15:26:49.767205+0000","last_active":"2024-04-25T15:26:50.911636+0000","last_peered":"2024-04-25T15:26:50.911636+0000","last_clean":"2024-04-25T15:26:50.911636+0000","last_became_active":"2024-04-25T15:26:49.766906+0000","last_became_peered":"2024-04-25T15:26:49.766906+0000","last_unstale":"2024-04-25T15:26:50.911636+0000","last_undegraded":"2024-04-25T15:26:50.911636+0000","last_fullsized":"2024-04-25T15:26:50.911636+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:47:55.546423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912063+0000","last_change":"2024-04-25T15:26:49.767237+0000","last_active":"2024-04-25T15:26:50.912063+0000","last_peered":"2024-04-25T15:26:50.912063+0000","last_clean":"2024-04-25T15:26:50.912063+0000","last_became_active":"2024-04-25T15:26:49.766878+0000","last_became_peered":"2024-04-25T15:26:49.766878+0000","last_unstale":"2024-04-25T15:26:50.912063+0000","last_undegraded":"2024-04-25T15:26:50.912063+0000","last_fullsized":"2024-04-25T15:26:50.912063+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:26:57.031744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766386+0000","last_change":"2024-04-25T15:26:49.757909+0000","last_active":"2024-04-25T15:26:50.766386+0000","last_peered":"2024-04-25T15:26:50.766386+0000","last_clean":"2024-04-25T15:26:50.766386+0000","last_became_active":"2024-04-25T15:26:49.757459+0000","last_became_peered":"2024-04-25T15:26:49.757459+0000","last_unstale":"2024-04-25T15:26:50.766386+0000","last_undegraded":"2024-04-25T15:26:50.766386+0000","last_fullsized":"2024-04-25T15:26:50.766386+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:15.418323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745881+0000","last_change":"2024-04-25T15:26:49.771297+0000","last_active":"2024-04-25T15:26:50.745881+0000","last_peered":"2024-04-25T15:26:50.745881+0000","last_clean":"2024-04-25T15:26:50.745881+0000","last_became_active":"2024-04-25T15:26:49.770206+0000","last_became_peered":"2024-04-25T15:26:49.770206+0000","last_unstale":"2024-04-25T15:26:50.745881+0000","last_undegraded":"2024-04-25T15:26:50.745881+0000","last_fullsized":"2024-04-25T15:26:50.745881+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:26:57.031744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911730+0000","last_change":"2024-04-25T15:26:49.767848+0000","last_active":"2024-04-25T15:26:50.911730+0000","last_peered":"2024-04-25T15:26:50.911730+0000","last_clean":"2024-04-25T15:26:50.911730+0000","last_became_active":"2024-04-25T15:26:49.767457+0000","last_became_peered":"2024-04-25T15:26:49.767457+0000","last_unstale":"2024-04-25T15:26:50.911730+0000","last_undegraded":"2024-04-25T15:26:50.911730+0000","last_fullsized":"2024-04-25T15:26:50.911730+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:32.612916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763389+0000","last_change":"2024-04-25T15:26:49.759879+0000","last_active":"2024-04-25T15:26:50.763389+0000","last_peered":"2024-04-25T15:26:50.763389+0000","last_clean":"2024-04-25T15:26:50.763389+0000","last_became_active":"2024-04-25T15:26:49.759548+0000","last_became_peered":"2024-04-25T15:26:49.759548+0000","last_unstale":"2024-04-25T15:26:50.763389+0000","last_undegraded":"2024-04-25T15:26:50.763389+0000","last_fullsized":"2024-04-25T15:26:50.763389+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:23:06.988004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911267+0000","last_change":"2024-04-25T15:26:49.762842+0000","last_active":"2024-04-25T15:26:50.911267+0000","last_peered":"2024-04-25T15:26:50.911267+0000","last_clean":"2024-04-25T15:26:50.911267+0000","last_became_active":"2024-04-25T15:26:49.762506+0000","last_became_peered":"2024-04-25T15:26:49.762506+0000","last_unstale":"2024-04-25T15:26:50.911267+0000","last_undegraded":"2024-04-25T15:26:50.911267+0000","last_fullsized":"2024-04-25T15:26:50.911267+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:31.327920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745373+0000","last_change":"2024-04-25T15:26:49.772980+0000","last_active":"2024-04-25T15:26:50.745373+0000","last_peered":"2024-04-25T15:26:50.745373+0000","last_clean":"2024-04-25T15:26:50.745373+0000","last_became_active":"2024-04-25T15:26:49.772682+0000","last_became_peered":"2024-04-25T15:26:49.772682+0000","last_unstale":"2024-04-25T15:26:50.745373+0000","last_undegraded":"2024-04-25T15:26:50.745373+0000","last_fullsized":"2024-04-25T15:26:50.745373+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:48.082839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746165+0000","last_change":"2024-04-25T15:26:49.766308+0000","last_active":"2024-04-25T15:26:50.746165+0000","last_peered":"2024-04-25T15:26:50.746165+0000","last_clean":"2024-04-25T15:26:50.746165+0000","last_became_active":"2024-04-25T15:26:49.766001+0000","last_became_peered":"2024-04-25T15:26:49.766001+0000","last_unstale":"2024-04-25T15:26:50.746165+0000","last_undegraded":"2024-04-25T15:26:50.746165+0000","last_fullsized":"2024-04-25T15:26:50.746165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:12.831550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746071+0000","last_change":"2024-04-25T15:26:49.771702+0000","last_active":"2024-04-25T15:26:50.746071+0000","last_peered":"2024-04-25T15:26:50.746071+0000","last_clean":"2024-04-25T15:26:50.746071+0000","last_became_active":"2024-04-25T15:26:49.771362+0000","last_became_peered":"2024-04-25T15:26:49.771362+0000","last_unstale":"2024-04-25T15:26:50.746071+0000","last_undegraded":"2024-04-25T15:26:50.746071+0000","last_fullsized":"2024-04-25T15:26:50.746071+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:47:55.546423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745949+0000","last_change":"2024-04-25T15:26:49.764937+0000","last_active":"2024-04-25T15:26:50.745949+0000","last_peered":"2024-04-25T15:26:50.745949+0000","last_clean":"2024-04-25T15:26:50.745949+0000","last_became_active":"2024-04-25T15:26:49.764072+0000","last_became_peered":"2024-04-25T15:26:49.764072+0000","last_unstale":"2024-04-25T15:26:50.745949+0000","last_undegraded":"2024-04-25T15:26:50.745949+0000","last_fullsized":"2024-04-25T15:26:50.745949+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:31.327920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912095+0000","last_change":"2024-04-25T15:26:49.762913+0000","last_active":"2024-04-25T15:26:50.912095+0000","last_peered":"2024-04-25T15:26:50.912095+0000","last_clean":"2024-04-25T15:26:50.912095+0000","last_became_active":"2024-04-25T15:26:49.762594+0000","last_became_peered":"2024-04-25T15:26:49.762594+0000","last_unstale":"2024-04-25T15:26:50.912095+0000","last_undegraded":"2024-04-25T15:26:50.912095+0000","last_fullsized":"2024-04-25T15:26:50.912095+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:04.867170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742798+0000","last_change":"2024-04-25T15:26:49.747707+0000","last_active":"2024-04-25T15:26:50.742798+0000","last_peered":"2024-04-25T15:26:50.742798+0000","last_clean":"2024-04-25T15:26:50.742798+0000","last_became_active":"2024-04-25T15:26:49.747326+0000","last_became_peered":"2024-04-25T15:26:49.747326+0000","last_unstale":"2024-04-25T15:26:50.742798+0000","last_undegraded":"2024-04-25T15:26:50.742798+0000","last_fullsized":"2024-04-25T15:26:50.742798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:57.941388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745539+0000","last_change":"2024-04-25T15:26:49.762906+0000","last_active":"2024-04-25T15:26:50.745539+0000","last_peered":"2024-04-25T15:26:50.745539+0000","last_clean":"2024-04-25T15:26:50.745539+0000","last_became_active":"2024-04-25T15:26:49.761926+0000","last_became_peered":"2024-04-25T15:26:49.761926+0000","last_unstale":"2024-04-25T15:26:50.745539+0000","last_undegraded":"2024-04-25T15:26:50.745539+0000","last_fullsized":"2024-04-25T15:26:50.745539+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:32.612916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763320+0000","last_change":"2024-04-25T15:26:49.765391+0000","last_active":"2024-04-25T15:26:50.763320+0000","last_peered":"2024-04-25T15:26:50.763320+0000","last_clean":"2024-04-25T15:26:50.763320+0000","last_became_active":"2024-04-25T15:26:49.761318+0000","last_became_peered":"2024-04-25T15:26:49.761318+0000","last_unstale":"2024-04-25T15:26:50.763320+0000","last_undegraded":"2024-04-25T15:26:50.763320+0000","last_fullsized":"2024-04-25T15:26:50.763320+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:21:08.396080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746487+0000","last_change":"2024-04-25T15:26:49.772836+0000","last_active":"2024-04-25T15:26:50.746487+0000","last_peered":"2024-04-25T15:26:50.746487+0000","last_clean":"2024-04-25T15:26:50.746487+0000","last_became_active":"2024-04-25T15:26:49.772345+0000","last_became_peered":"2024-04-25T15:26:49.772345+0000","last_unstale":"2024-04-25T15:26:50.746487+0000","last_undegraded":"2024-04-25T15:26:50.746487+0000","last_fullsized":"2024-04-25T15:26:50.746487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:03.943226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766267+0000","last_change":"2024-04-25T15:26:49.761556+0000","last_active":"2024-04-25T15:26:50.766267+0000","last_peered":"2024-04-25T15:26:50.766267+0000","last_clean":"2024-04-25T15:26:50.766267+0000","last_became_active":"2024-04-25T15:26:49.761226+0000","last_became_peered":"2024-04-25T15:26:49.761226+0000","last_unstale":"2024-04-25T15:26:50.766267+0000","last_undegraded":"2024-04-25T15:26:50.766267+0000","last_fullsized":"2024-04-25T15:26:50.766267+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:52.172769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766307+0000","last_change":"2024-04-25T15:26:49.768683+0000","last_active":"2024-04-25T15:26:50.766307+0000","last_peered":"2024-04-25T15:26:50.766307+0000","last_clean":"2024-04-25T15:26:50.766307+0000","last_became_active":"2024-04-25T15:26:49.768327+0000","last_became_peered":"2024-04-25T15:26:49.768327+0000","last_unstale":"2024-04-25T15:26:50.766307+0000","last_undegraded":"2024-04-25T15:26:50.766307+0000","last_fullsized":"2024-04-25T15:26:50.766307+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:48.082839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746514+0000","last_change":"2024-04-25T15:26:49.772764+0000","last_active":"2024-04-25T15:26:50.746514+0000","last_peered":"2024-04-25T15:26:50.746514+0000","last_clean":"2024-04-25T15:26:50.746514+0000","last_became_active":"2024-04-25T15:26:49.772410+0000","last_became_peered":"2024-04-25T15:26:49.772410+0000","last_unstale":"2024-04-25T15:26:50.746514+0000","last_undegraded":"2024-04-25T15:26:50.746514+0000","last_fullsized":"2024-04-25T15:26:50.746514+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:49.933480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.743073+0000","last_change":"2024-04-25T15:26:49.764578+0000","last_active":"2024-04-25T15:26:50.743073+0000","last_peered":"2024-04-25T15:26:50.743073+0000","last_clean":"2024-04-25T15:26:50.743073+0000","last_became_active":"2024-04-25T15:26:49.764250+0000","last_became_peered":"2024-04-25T15:26:49.764250+0000","last_unstale":"2024-04-25T15:26:50.743073+0000","last_undegraded":"2024-04-25T15:26:50.743073+0000","last_fullsized":"2024-04-25T15:26:50.743073+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:18:20.978355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763476+0000","last_change":"2024-04-25T15:26:49.754648+0000","last_active":"2024-04-25T15:26:50.763476+0000","last_peered":"2024-04-25T15:26:50.763476+0000","last_clean":"2024-04-25T15:26:50.763476+0000","last_became_active":"2024-04-25T15:26:49.754349+0000","last_became_peered":"2024-04-25T15:26:49.754349+0000","last_unstale":"2024-04-25T15:26:50.763476+0000","last_undegraded":"2024-04-25T15:26:50.763476+0000","last_fullsized":"2024-04-25T15:26:50.763476+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:54.869352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745727+0000","last_change":"2024-04-25T15:26:49.771579+0000","last_active":"2024-04-25T15:26:50.745727+0000","last_peered":"2024-04-25T15:26:50.745727+0000","last_clean":"2024-04-25T15:26:50.745727+0000","last_became_active":"2024-04-25T15:26:49.770862+0000","last_became_peered":"2024-04-25T15:26:49.770862+0000","last_unstale":"2024-04-25T15:26:50.745727+0000","last_undegraded":"2024-04-25T15:26:50.745727+0000","last_fullsized":"2024-04-25T15:26:50.745727+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:57.061251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745914+0000","last_change":"2024-04-25T15:26:49.763645+0000","last_active":"2024-04-25T15:26:50.745914+0000","last_peered":"2024-04-25T15:26:50.745914+0000","last_clean":"2024-04-25T15:26:50.745914+0000","last_became_active":"2024-04-25T15:26:49.762838+0000","last_became_peered":"2024-04-25T15:26:49.762838+0000","last_unstale":"2024-04-25T15:26:50.745914+0000","last_undegraded":"2024-04-25T15:26:50.745914+0000","last_fullsized":"2024-04-25T15:26:50.745914+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:46.479088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745931+0000","last_change":"2024-04-25T15:26:49.757459+0000","last_active":"2024-04-25T15:26:50.745931+0000","last_peered":"2024-04-25T15:26:50.745931+0000","last_clean":"2024-04-25T15:26:50.745931+0000","last_became_active":"2024-04-25T15:26:49.757137+0000","last_became_peered":"2024-04-25T15:26:49.757137+0000","last_unstale":"2024-04-25T15:26:50.745931+0000","last_undegraded":"2024-04-25T15:26:50.745931+0000","last_fullsized":"2024-04-25T15:26:50.745931+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:02.570663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746282+0000","last_change":"2024-04-25T15:26:49.762958+0000","last_active":"2024-04-25T15:26:50.746282+0000","last_peered":"2024-04-25T15:26:50.746282+0000","last_clean":"2024-04-25T15:26:50.746282+0000","last_became_active":"2024-04-25T15:26:49.762634+0000","last_became_peered":"2024-04-25T15:26:49.762634+0000","last_unstale":"2024-04-25T15:26:50.746282+0000","last_undegraded":"2024-04-25T15:26:50.746282+0000","last_fullsized":"2024-04-25T15:26:50.746282+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:37.306991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742668+0000","last_change":"2024-04-25T15:26:49.750802+0000","last_active":"2024-04-25T15:26:50.742668+0000","last_peered":"2024-04-25T15:26:50.742668+0000","last_clean":"2024-04-25T15:26:50.742668+0000","last_became_active":"2024-04-25T15:26:49.750495+0000","last_became_peered":"2024-04-25T15:26:49.750495+0000","last_unstale":"2024-04-25T15:26:50.742668+0000","last_undegraded":"2024-04-25T15:26:50.742668+0000","last_fullsized":"2024-04-25T15:26:50.742668+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:11.969655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766133+0000","last_change":"2024-04-25T15:26:49.760749+0000","last_active":"2024-04-25T15:26:50.766133+0000","last_peered":"2024-04-25T15:26:50.766133+0000","last_clean":"2024-04-25T15:26:50.766133+0000","last_became_active":"2024-04-25T15:26:49.760428+0000","last_became_peered":"2024-04-25T15:26:49.760428+0000","last_unstale":"2024-04-25T15:26:50.766133+0000","last_undegraded":"2024-04-25T15:26:50.766133+0000","last_fullsized":"2024-04-25T15:26:50.766133+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:25.735036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912161+0000","last_change":"2024-04-25T15:26:49.762439+0000","last_active":"2024-04-25T15:26:50.912161+0000","last_peered":"2024-04-25T15:26:50.912161+0000","last_clean":"2024-04-25T15:26:50.912161+0000","last_became_active":"2024-04-25T15:26:49.762083+0000","last_became_peered":"2024-04-25T15:26:49.762083+0000","last_unstale":"2024-04-25T15:26:50.912161+0000","last_undegraded":"2024-04-25T15:26:50.912161+0000","last_fullsized":"2024-04-25T15:26:50.912161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:31.327920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746516+0000","last_change":"2024-04-25T15:26:49.761029+0000","last_active":"2024-04-25T15:26:50.746516+0000","last_peered":"2024-04-25T15:26:50.746516+0000","last_clean":"2024-04-25T15:26:50.746516+0000","last_became_active":"2024-04-25T15:26:49.760706+0000","last_became_peered":"2024-04-25T15:26:49.760706+0000","last_unstale":"2024-04-25T15:26:50.746516+0000","last_undegraded":"2024-04-25T15:26:50.746516+0000","last_fullsized":"2024-04-25T15:26:50.746516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:19.733261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766060+0000","last_change":"2024-04-25T15:26:49.758922+0000","last_active":"2024-04-25T15:26:50.766060+0000","last_peered":"2024-04-25T15:26:50.766060+0000","last_clean":"2024-04-25T15:26:50.766060+0000","last_became_active":"2024-04-25T15:26:49.758416+0000","last_became_peered":"2024-04-25T15:26:49.758416+0000","last_unstale":"2024-04-25T15:26:50.766060+0000","last_undegraded":"2024-04-25T15:26:50.766060+0000","last_fullsized":"2024-04-25T15:26:50.766060+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:33.933493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763380+0000","last_change":"2024-04-25T15:26:49.765303+0000","last_active":"2024-04-25T15:26:50.763380+0000","last_peered":"2024-04-25T15:26:50.763380+0000","last_clean":"2024-04-25T15:26:50.763380+0000","last_became_active":"2024-04-25T15:26:49.761077+0000","last_became_peered":"2024-04-25T15:26:49.761077+0000","last_unstale":"2024-04-25T15:26:50.763380+0000","last_undegraded":"2024-04-25T15:26:50.763380+0000","last_fullsized":"2024-04-25T15:26:50.763380+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:06.710458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742639+0000","last_change":"2024-04-25T15:26:49.748956+0000","last_active":"2024-04-25T15:26:50.742639+0000","last_peered":"2024-04-25T15:26:50.742639+0000","last_clean":"2024-04-25T15:26:50.742639+0000","last_became_active":"2024-04-25T15:26:49.748629+0000","last_became_peered":"2024-04-25T15:26:49.748629+0000","last_unstale":"2024-04-25T15:26:50.742639+0000","last_undegraded":"2024-04-25T15:26:50.742639+0000","last_fullsized":"2024-04-25T15:26:50.742639+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:44:06.129747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910818+0000","last_change":"2024-04-25T15:26:49.750003+0000","last_active":"2024-04-25T15:26:50.910818+0000","last_peered":"2024-04-25T15:26:50.910818+0000","last_clean":"2024-04-25T15:26:50.910818+0000","last_became_active":"2024-04-25T15:26:49.749683+0000","last_became_peered":"2024-04-25T15:26:49.749683+0000","last_unstale":"2024-04-25T15:26:50.910818+0000","last_undegraded":"2024-04-25T15:26:50.910818+0000","last_fullsized":"2024-04-25T15:26:50.910818+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:49.933480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.764005+0000","last_change":"2024-04-25T15:26:48.748242+0000","last_active":"2024-04-25T15:26:50.764005+0000","last_peered":"2024-04-25T15:26:50.764005+0000","last_clean":"2024-04-25T15:26:50.764005+0000","last_became_active":"2024-04-25T15:26:48.747914+0000","last_became_peered":"2024-04-25T15:26:48.747914+0000","last_unstale":"2024-04-25T15:26:50.764005+0000","last_undegraded":"2024-04-25T15:26:50.764005+0000","last_fullsized":"2024-04-25T15:26:50.764005+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:31.607605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746497+0000","last_change":"2024-04-25T15:26:49.767001+0000","last_active":"2024-04-25T15:26:50.746497+0000","last_peered":"2024-04-25T15:26:50.746497+0000","last_clean":"2024-04-25T15:26:50.746497+0000","last_became_active":"2024-04-25T15:26:49.766691+0000","last_became_peered":"2024-04-25T15:26:49.766691+0000","last_unstale":"2024-04-25T15:26:50.746497+0000","last_undegraded":"2024-04-25T15:26:50.746497+0000","last_fullsized":"2024-04-25T15:26:50.746497+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:41:36.882975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766434+0000","last_change":"2024-04-25T15:26:48.739592+0000","last_active":"2024-04-25T15:26:50.766434+0000","last_peered":"2024-04-25T15:26:50.766434+0000","last_clean":"2024-04-25T15:26:50.766434+0000","last_became_active":"2024-04-25T15:26:48.739265+0000","last_became_peered":"2024-04-25T15:26:48.739265+0000","last_unstale":"2024-04-25T15:26:50.766434+0000","last_undegraded":"2024-04-25T15:26:50.766434+0000","last_fullsized":"2024-04-25T15:26:50.766434+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:55.259761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912277+0000","last_change":"2024-04-25T15:26:49.759953+0000","last_active":"2024-04-25T15:26:50.912277+0000","last_peered":"2024-04-25T15:26:50.912277+0000","last_clean":"2024-04-25T15:26:50.912277+0000","last_became_active":"2024-04-25T15:26:49.759600+0000","last_became_peered":"2024-04-25T15:26:49.759600+0000","last_unstale":"2024-04-25T15:26:50.912277+0000","last_undegraded":"2024-04-25T15:26:50.912277+0000","last_fullsized":"2024-04-25T15:26:50.912277+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:22.027488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910995+0000","last_change":"2024-04-25T15:26:48.740212+0000","last_active":"2024-04-25T15:26:50.910995+0000","last_peered":"2024-04-25T15:26:50.910995+0000","last_clean":"2024-04-25T15:26:50.910995+0000","last_became_active":"2024-04-25T15:26:48.739902+0000","last_became_peered":"2024-04-25T15:26:48.739902+0000","last_unstale":"2024-04-25T15:26:50.910995+0000","last_undegraded":"2024-04-25T15:26:50.910995+0000","last_fullsized":"2024-04-25T15:26:50.910995+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:40.127334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910898+0000","last_change":"2024-04-25T15:26:49.758962+0000","last_active":"2024-04-25T15:26:50.910898+0000","last_peered":"2024-04-25T15:26:50.910898+0000","last_clean":"2024-04-25T15:26:50.910898+0000","last_became_active":"2024-04-25T15:26:49.758084+0000","last_became_peered":"2024-04-25T15:26:49.758084+0000","last_unstale":"2024-04-25T15:26:50.910898+0000","last_undegraded":"2024-04-25T15:26:50.910898+0000","last_fullsized":"2024-04-25T15:26:50.910898+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:10.302228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911316+0000","last_change":"2024-04-25T15:26:48.750796+0000","last_active":"2024-04-25T15:26:50.911316+0000","last_peered":"2024-04-25T15:26:50.911316+0000","last_clean":"2024-04-25T15:26:50.911316+0000","last_became_active":"2024-04-25T15:26:48.750501+0000","last_became_peered":"2024-04-25T15:26:48.750501+0000","last_unstale":"2024-04-25T15:26:50.911316+0000","last_undegraded":"2024-04-25T15:26:50.911316+0000","last_fullsized":"2024-04-25T15:26:50.911316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:29.432968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766068+0000","last_change":"2024-04-25T15:26:49.757778+0000","last_active":"2024-04-25T15:26:50.766068+0000","last_peered":"2024-04-25T15:26:50.766068+0000","last_clean":"2024-04-25T15:26:50.766068+0000","last_became_active":"2024-04-25T15:26:49.754768+0000","last_became_peered":"2024-04-25T15:26:49.754768+0000","last_unstale":"2024-04-25T15:26:50.766068+0000","last_undegraded":"2024-04-25T15:26:50.766068+0000","last_fullsized":"2024-04-25T15:26:50.766068+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:10.302228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763617+0000","last_change":"2024-04-25T15:26:48.751930+0000","last_active":"2024-04-25T15:26:50.763617+0000","last_peered":"2024-04-25T15:26:50.763617+0000","last_clean":"2024-04-25T15:26:50.763617+0000","last_became_active":"2024-04-25T15:26:48.751613+0000","last_became_peered":"2024-04-25T15:26:48.751613+0000","last_unstale":"2024-04-25T15:26:50.763617+0000","last_undegraded":"2024-04-25T15:26:50.763617+0000","last_fullsized":"2024-04-25T15:26:50.763617+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:26:56.026433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745898+0000","last_change":"2024-04-25T15:26:49.766766+0000","last_active":"2024-04-25T15:26:50.745898+0000","last_peered":"2024-04-25T15:26:50.745898+0000","last_clean":"2024-04-25T15:26:50.745898+0000","last_became_active":"2024-04-25T15:26:49.764242+0000","last_became_peered":"2024-04-25T15:26:49.764242+0000","last_unstale":"2024-04-25T15:26:50.745898+0000","last_undegraded":"2024-04-25T15:26:50.745898+0000","last_fullsized":"2024-04-25T15:26:50.745898+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:04.867170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766233+0000","last_change":"2024-04-25T15:26:48.740859+0000","last_active":"2024-04-25T15:26:50.766233+0000","last_peered":"2024-04-25T15:26:50.766233+0000","last_clean":"2024-04-25T15:26:50.766233+0000","last_became_active":"2024-04-25T15:26:48.740558+0000","last_became_peered":"2024-04-25T15:26:48.740558+0000","last_unstale":"2024-04-25T15:26:50.766233+0000","last_undegraded":"2024-04-25T15:26:50.766233+0000","last_fullsized":"2024-04-25T15:26:50.766233+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:00.557944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745478+0000","last_change":"2024-04-25T15:26:49.770655+0000","last_active":"2024-04-25T15:26:50.745478+0000","last_peered":"2024-04-25T15:26:50.745478+0000","last_clean":"2024-04-25T15:26:50.745478+0000","last_became_active":"2024-04-25T15:26:49.770344+0000","last_became_peered":"2024-04-25T15:26:49.770344+0000","last_unstale":"2024-04-25T15:26:50.745478+0000","last_undegraded":"2024-04-25T15:26:50.745478+0000","last_fullsized":"2024-04-25T15:26:50.745478+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:47.843945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911418+0000","last_change":"2024-04-25T15:26:48.748965+0000","last_active":"2024-04-25T15:26:50.911418+0000","last_peered":"2024-04-25T15:26:50.911418+0000","last_clean":"2024-04-25T15:26:50.911418+0000","last_became_active":"2024-04-25T15:26:48.748726+0000","last_became_peered":"2024-04-25T15:26:48.748726+0000","last_unstale":"2024-04-25T15:26:50.911418+0000","last_undegraded":"2024-04-25T15:26:50.911418+0000","last_fullsized":"2024-04-25T15:26:50.911418+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:41:35.877664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745869+0000","last_change":"2024-04-25T15:26:49.763896+0000","last_active":"2024-04-25T15:26:50.745869+0000","last_peered":"2024-04-25T15:26:50.745869+0000","last_clean":"2024-04-25T15:26:50.745869+0000","last_became_active":"2024-04-25T15:26:49.763242+0000","last_became_peered":"2024-04-25T15:26:49.763242+0000","last_unstale":"2024-04-25T15:26:50.745869+0000","last_undegraded":"2024-04-25T15:26:50.745869+0000","last_fullsized":"2024-04-25T15:26:50.745869+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:32.546192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745485+0000","last_change":"2024-04-25T15:26:48.740325+0000","last_active":"2024-04-25T15:26:50.745485+0000","last_peered":"2024-04-25T15:26:50.745485+0000","last_clean":"2024-04-25T15:26:50.745485+0000","last_became_active":"2024-04-25T15:26:48.740026+0000","last_became_peered":"2024-04-25T15:26:48.740026+0000","last_unstale":"2024-04-25T15:26:50.745485+0000","last_undegraded":"2024-04-25T15:26:50.745485+0000","last_fullsized":"2024-04-25T15:26:50.745485+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:36:47.091312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766334+0000","last_change":"2024-04-25T15:26:49.767747+0000","last_active":"2024-04-25T15:26:50.766334+0000","last_peered":"2024-04-25T15:26:50.766334+0000","last_clean":"2024-04-25T15:26:50.766334+0000","last_became_active":"2024-04-25T15:26:49.767368+0000","last_became_peered":"2024-04-25T15:26:49.767368+0000","last_unstale":"2024-04-25T15:26:50.766334+0000","last_undegraded":"2024-04-25T15:26:50.766334+0000","last_fullsized":"2024-04-25T15:26:50.766334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:47.843945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766392+0000","last_change":"2024-04-25T15:26:48.740454+0000","last_active":"2024-04-25T15:26:50.766392+0000","last_peered":"2024-04-25T15:26:50.766392+0000","last_clean":"2024-04-25T15:26:50.766392+0000","last_became_active":"2024-04-25T15:26:48.739972+0000","last_became_peered":"2024-04-25T15:26:48.739972+0000","last_unstale":"2024-04-25T15:26:50.766392+0000","last_undegraded":"2024-04-25T15:26:50.766392+0000","last_fullsized":"2024-04-25T15:26:50.766392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:01.465456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911798+0000","last_change":"2024-04-25T15:26:49.765412+0000","last_active":"2024-04-25T15:26:50.911798+0000","last_peered":"2024-04-25T15:26:50.911798+0000","last_clean":"2024-04-25T15:26:50.911798+0000","last_became_active":"2024-04-25T15:26:49.765086+0000","last_became_peered":"2024-04-25T15:26:49.765086+0000","last_unstale":"2024-04-25T15:26:50.911798+0000","last_undegraded":"2024-04-25T15:26:50.911798+0000","last_fullsized":"2024-04-25T15:26:50.911798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:12.831550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746139+0000","last_change":"2024-04-25T15:26:48.745941+0000","last_active":"2024-04-25T15:26:50.746139+0000","last_peered":"2024-04-25T15:26:50.746139+0000","last_clean":"2024-04-25T15:26:50.746139+0000","last_became_active":"2024-04-25T15:26:48.745570+0000","last_became_peered":"2024-04-25T15:26:48.745570+0000","last_unstale":"2024-04-25T15:26:50.746139+0000","last_undegraded":"2024-04-25T15:26:50.746139+0000","last_fullsized":"2024-04-25T15:26:50.746139+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:01.465456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763722+0000","last_change":"2024-04-25T15:26:49.751396+0000","last_active":"2024-04-25T15:26:50.763722+0000","last_peered":"2024-04-25T15:26:50.763722+0000","last_clean":"2024-04-25T15:26:50.763722+0000","last_became_active":"2024-04-25T15:26:49.749466+0000","last_became_peered":"2024-04-25T15:26:49.749466+0000","last_unstale":"2024-04-25T15:26:50.763722+0000","last_undegraded":"2024-04-25T15:26:50.763722+0000","last_fullsized":"2024-04-25T15:26:50.763722+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:08.824618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911490+0000","last_change":"2024-04-25T15:26:48.751237+0000","last_active":"2024-04-25T15:26:50.911490+0000","last_peered":"2024-04-25T15:26:50.911490+0000","last_clean":"2024-04-25T15:26:50.911490+0000","last_became_active":"2024-04-25T15:26:48.750776+0000","last_became_peered":"2024-04-25T15:26:48.750776+0000","last_unstale":"2024-04-25T15:26:50.911490+0000","last_undegraded":"2024-04-25T15:26:50.911490+0000","last_fullsized":"2024-04-25T15:26:50.911490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:51.167458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911336+0000","last_change":"2024-04-25T15:26:49.763644+0000","last_active":"2024-04-25T15:26:50.911336+0000","last_peered":"2024-04-25T15:26:50.911336+0000","last_clean":"2024-04-25T15:26:50.911336+0000","last_became_active":"2024-04-25T15:26:49.763346+0000","last_became_peered":"2024-04-25T15:26:49.763346+0000","last_unstale":"2024-04-25T15:26:50.911336+0000","last_undegraded":"2024-04-25T15:26:50.911336+0000","last_fullsized":"2024-04-25T15:26:50.911336+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:22.027488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.743032+0000","last_change":"2024-04-25T15:26:48.739781+0000","last_active":"2024-04-25T15:26:50.743032+0000","last_peered":"2024-04-25T15:26:50.743032+0000","last_clean":"2024-04-25T15:26:50.743032+0000","last_became_active":"2024-04-25T15:26:48.739062+0000","last_became_peered":"2024-04-25T15:26:48.739062+0000","last_unstale":"2024-04-25T15:26:50.743032+0000","last_undegraded":"2024-04-25T15:26:50.743032+0000","last_fullsized":"2024-04-25T15:26:50.743032+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:10.529331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746589+0000","last_change":"2024-04-25T15:26:49.764866+0000","last_active":"2024-04-25T15:26:50.746589+0000","last_peered":"2024-04-25T15:26:50.746589+0000","last_clean":"2024-04-25T15:26:50.746589+0000","last_became_active":"2024-04-25T15:26:49.764549+0000","last_became_peered":"2024-04-25T15:26:49.764549+0000","last_unstale":"2024-04-25T15:26:50.746589+0000","last_undegraded":"2024-04-25T15:26:50.746589+0000","last_fullsized":"2024-04-25T15:26:50.746589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:10.302228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763736+0000","last_change":"2024-04-25T15:26:48.749935+0000","last_active":"2024-04-25T15:26:50.763736+0000","last_peered":"2024-04-25T15:26:50.763736+0000","last_clean":"2024-04-25T15:26:50.763736+0000","last_became_active":"2024-04-25T15:26:48.749588+0000","last_became_peered":"2024-04-25T15:26:48.749588+0000","last_unstale":"2024-04-25T15:26:50.763736+0000","last_undegraded":"2024-04-25T15:26:50.763736+0000","last_fullsized":"2024-04-25T15:26:50.763736+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:56.055940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745990+0000","last_change":"2024-04-25T15:26:49.767288+0000","last_active":"2024-04-25T15:26:50.745990+0000","last_peered":"2024-04-25T15:26:50.745990+0000","last_clean":"2024-04-25T15:26:50.745990+0000","last_became_active":"2024-04-25T15:26:49.766759+0000","last_became_peered":"2024-04-25T15:26:49.766759+0000","last_unstale":"2024-04-25T15:26:50.745990+0000","last_undegraded":"2024-04-25T15:26:50.745990+0000","last_fullsized":"2024-04-25T15:26:50.745990+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:13.459028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911233+0000","last_change":"2024-04-25T15:26:48.750106+0000","last_active":"2024-04-25T15:26:50.911233+0000","last_peered":"2024-04-25T15:26:50.911233+0000","last_clean":"2024-04-25T15:26:50.911233+0000","last_became_active":"2024-04-25T15:26:48.749526+0000","last_became_peered":"2024-04-25T15:26:48.749526+0000","last_unstale":"2024-04-25T15:26:50.911233+0000","last_undegraded":"2024-04-25T15:26:50.911233+0000","last_fullsized":"2024-04-25T15:26:50.911233+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:09.296917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763312+0000","last_change":"2024-04-25T15:26:49.750265+0000","last_active":"2024-04-25T15:26:50.763312+0000","last_peered":"2024-04-25T15:26:50.763312+0000","last_clean":"2024-04-25T15:26:50.763312+0000","last_became_active":"2024-04-25T15:26:49.749195+0000","last_became_peered":"2024-04-25T15:26:49.749195+0000","last_unstale":"2024-04-25T15:26:50.763312+0000","last_undegraded":"2024-04-25T15:26:50.763312+0000","last_fullsized":"2024-04-25T15:26:50.763312+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.978844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763435+0000","last_change":"2024-04-25T15:26:48.748609+0000","last_active":"2024-04-25T15:26:50.763435+0000","last_peered":"2024-04-25T15:26:50.763435+0000","last_clean":"2024-04-25T15:26:50.763435+0000","last_became_active":"2024-04-25T15:26:48.748295+0000","last_became_peered":"2024-04-25T15:26:48.748295+0000","last_unstale":"2024-04-25T15:26:50.763435+0000","last_undegraded":"2024-04-25T15:26:50.763435+0000","last_fullsized":"2024-04-25T15:26:50.763435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:14.413012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911749+0000","last_change":"2024-04-25T15:26:49.759956+0000","last_active":"2024-04-25T15:26:50.911749+0000","last_peered":"2024-04-25T15:26:50.911749+0000","last_clean":"2024-04-25T15:26:50.911749+0000","last_became_active":"2024-04-25T15:26:49.759603+0000","last_became_peered":"2024-04-25T15:26:49.759603+0000","last_unstale":"2024-04-25T15:26:50.911749+0000","last_undegraded":"2024-04-25T15:26:50.911749+0000","last_fullsized":"2024-04-25T15:26:50.911749+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:46.641900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":22,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745567+0000","last_change":"2024-04-25T15:26:48.740097+0000","last_active":"2024-04-25T15:26:50.745567+0000","last_peered":"2024-04-25T15:26:50.745567+0000","last_clean":"2024-04-25T15:26:50.745567+0000","last_became_active":"2024-04-25T15:26:48.739737+0000","last_became_peered":"2024-04-25T15:26:48.739737+0000","last_unstale":"2024-04-25T15:26:50.745567+0000","last_undegraded":"2024-04-25T15:26:50.745567+0000","last_fullsized":"2024-04-25T15:26:50.745567+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:33.242255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742818+0000","last_change":"2024-04-25T15:26:49.759946+0000","last_active":"2024-04-25T15:26:50.742818+0000","last_peered":"2024-04-25T15:26:50.742818+0000","last_clean":"2024-04-25T15:26:50.742818+0000","last_became_active":"2024-04-25T15:26:49.759636+0000","last_became_peered":"2024-04-25T15:26:49.759636+0000","last_unstale":"2024-04-25T15:26:50.742818+0000","last_undegraded":"2024-04-25T15:26:50.742818+0000","last_fullsized":"2024-04-25T15:26:50.742818+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:30.337482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742696+0000","last_change":"2024-04-25T15:26:48.749171+0000","last_active":"2024-04-25T15:26:50.742696+0000","last_peered":"2024-04-25T15:26:50.742696+0000","last_clean":"2024-04-25T15:26:50.742696+0000","last_became_active":"2024-04-25T15:26:48.748816+0000","last_became_peered":"2024-04-25T15:26:48.748816+0000","last_unstale":"2024-04-25T15:26:50.742696+0000","last_undegraded":"2024-04-25T15:26:50.742696+0000","last_fullsized":"2024-04-25T15:26:50.742696+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:47.294774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746227+0000","last_change":"2024-04-25T15:26:49.762695+0000","last_active":"2024-04-25T15:26:50.746227+0000","last_peered":"2024-04-25T15:26:50.746227+0000","last_clean":"2024-04-25T15:26:50.746227+0000","last_became_active":"2024-04-25T15:26:49.762065+0000","last_became_peered":"2024-04-25T15:26:49.762065+0000","last_unstale":"2024-04-25T15:26:50.746227+0000","last_undegraded":"2024-04-25T15:26:50.746227+0000","last_fullsized":"2024-04-25T15:26:50.746227+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:08.385597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766435+0000","last_change":"2024-04-25T15:26:48.744243+0000","last_active":"2024-04-25T15:26:50.766435+0000","last_peered":"2024-04-25T15:26:50.766435+0000","last_clean":"2024-04-25T15:26:50.766435+0000","last_became_active":"2024-04-25T15:26:48.743887+0000","last_became_peered":"2024-04-25T15:26:48.743887+0000","last_unstale":"2024-04-25T15:26:50.766435+0000","last_undegraded":"2024-04-25T15:26:50.766435+0000","last_fullsized":"2024-04-25T15:26:50.766435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:31.540881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746588+0000","last_change":"2024-04-25T15:26:49.756504+0000","last_active":"2024-04-25T15:26:50.746588+0000","last_peered":"2024-04-25T15:26:50.746588+0000","last_clean":"2024-04-25T15:26:50.746588+0000","last_became_active":"2024-04-25T15:26:49.756097+0000","last_became_peered":"2024-04-25T15:26:49.756097+0000","last_unstale":"2024-04-25T15:26:50.746588+0000","last_undegraded":"2024-04-25T15:26:50.746588+0000","last_fullsized":"2024-04-25T15:26:50.746588+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:33.933493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911610+0000","last_change":"2024-04-25T15:26:48.742966+0000","last_active":"2024-04-25T15:26:50.911610+0000","last_peered":"2024-04-25T15:26:50.911610+0000","last_clean":"2024-04-25T15:26:50.911610+0000","last_became_active":"2024-04-25T15:26:48.742641+0000","last_became_peered":"2024-04-25T15:26:48.742641+0000","last_unstale":"2024-04-25T15:26:50.911610+0000","last_undegraded":"2024-04-25T15:26:50.911610+0000","last_fullsized":"2024-04-25T15:26:50.911610+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:09.296917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746199+0000","last_change":"2024-04-25T15:26:49.762303+0000","last_active":"2024-04-25T15:26:50.746199+0000","last_peered":"2024-04-25T15:26:50.746199+0000","last_clean":"2024-04-25T15:26:50.746199+0000","last_became_active":"2024-04-25T15:26:49.761731+0000","last_became_peered":"2024-04-25T15:26:49.761731+0000","last_unstale":"2024-04-25T15:26:50.746199+0000","last_undegraded":"2024-04-25T15:26:50.746199+0000","last_fullsized":"2024-04-25T15:26:50.746199+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:41.132645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763682+0000","last_change":"2024-04-25T15:26:48.750895+0000","last_active":"2024-04-25T15:26:50.763682+0000","last_peered":"2024-04-25T15:26:50.763682+0000","last_clean":"2024-04-25T15:26:50.763682+0000","last_became_active":"2024-04-25T15:26:48.750541+0000","last_became_peered":"2024-04-25T15:26:48.750541+0000","last_unstale":"2024-04-25T15:26:50.763682+0000","last_undegraded":"2024-04-25T15:26:50.763682+0000","last_fullsized":"2024-04-25T15:26:50.763682+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:21.022177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763435+0000","last_change":"2024-04-25T15:26:49.750174+0000","last_active":"2024-04-25T15:26:50.763435+0000","last_peered":"2024-04-25T15:26:50.763435+0000","last_clean":"2024-04-25T15:26:50.763435+0000","last_became_active":"2024-04-25T15:26:49.749618+0000","last_became_peered":"2024-04-25T15:26:49.749618+0000","last_unstale":"2024-04-25T15:26:50.763435+0000","last_undegraded":"2024-04-25T15:26:50.763435+0000","last_fullsized":"2024-04-25T15:26:50.763435+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:23.615205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763955+0000","last_change":"2024-04-25T15:26:48.751868+0000","last_active":"2024-04-25T15:26:50.763955+0000","last_peered":"2024-04-25T15:26:50.763955+0000","last_clean":"2024-04-25T15:26:50.763955+0000","last_became_active":"2024-04-25T15:26:48.751534+0000","last_became_peered":"2024-04-25T15:26:48.751534+0000","last_unstale":"2024-04-25T15:26:50.763955+0000","last_undegraded":"2024-04-25T15:26:50.763955+0000","last_fullsized":"2024-04-25T15:26:50.763955+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:03.861859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910889+0000","last_change":"2024-04-25T15:26:49.760874+0000","last_active":"2024-04-25T15:26:50.910889+0000","last_peered":"2024-04-25T15:26:50.910889+0000","last_clean":"2024-04-25T15:26:50.910889+0000","last_became_active":"2024-04-25T15:26:49.760536+0000","last_became_peered":"2024-04-25T15:26:49.760536+0000","last_unstale":"2024-04-25T15:26:50.910889+0000","last_undegraded":"2024-04-25T15:26:50.910889+0000","last_fullsized":"2024-04-25T15:26:50.910889+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:33.933493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911274+0000","last_change":"2024-04-25T15:26:48.745888+0000","last_active":"2024-04-25T15:26:50.911274+0000","last_peered":"2024-04-25T15:26:50.911274+0000","last_clean":"2024-04-25T15:26:50.911274+0000","last_became_active":"2024-04-25T15:26:48.745549+0000","last_became_peered":"2024-04-25T15:26:48.745549+0000","last_unstale":"2024-04-25T15:26:50.911274+0000","last_undegraded":"2024-04-25T15:26:50.911274+0000","last_fullsized":"2024-04-25T15:26:50.911274+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:48.928169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742710+0000","last_change":"2024-04-25T15:26:49.758878+0000","last_active":"2024-04-25T15:26:50.742710+0000","last_peered":"2024-04-25T15:26:50.742710+0000","last_clean":"2024-04-25T15:26:50.742710+0000","last_became_active":"2024-04-25T15:26:49.758594+0000","last_became_peered":"2024-04-25T15:26:49.758594+0000","last_unstale":"2024-04-25T15:26:50.742710+0000","last_undegraded":"2024-04-25T15:26:50.742710+0000","last_fullsized":"2024-04-25T15:26:50.742710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:33.254158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745499+0000","last_change":"2024-04-25T15:26:48.743524+0000","last_active":"2024-04-25T15:26:50.745499+0000","last_peered":"2024-04-25T15:26:50.745499+0000","last_clean":"2024-04-25T15:26:50.745499+0000","last_became_active":"2024-04-25T15:26:48.742078+0000","last_became_peered":"2024-04-25T15:26:48.742078+0000","last_unstale":"2024-04-25T15:26:50.745499+0000","last_undegraded":"2024-04-25T15:26:50.745499+0000","last_fullsized":"2024-04-25T15:26:50.745499+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:55.259761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745496+0000","last_change":"2024-04-25T15:26:49.763521+0000","last_active":"2024-04-25T15:26:50.745496+0000","last_peered":"2024-04-25T15:26:50.745496+0000","last_clean":"2024-04-25T15:26:50.745496+0000","last_became_active":"2024-04-25T15:26:49.763071+0000","last_became_peered":"2024-04-25T15:26:49.763071+0000","last_unstale":"2024-04-25T15:26:50.745496+0000","last_undegraded":"2024-04-25T15:26:50.745496+0000","last_fullsized":"2024-04-25T15:26:50.745496+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:15.418323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911084+0000","last_change":"2024-04-25T15:26:48.750972+0000","last_active":"2024-04-25T15:26:50.911084+0000","last_peered":"2024-04-25T15:26:50.911084+0000","last_clean":"2024-04-25T15:26:50.911084+0000","last_became_active":"2024-04-25T15:26:48.750183+0000","last_became_peered":"2024-04-25T15:26:48.750183+0000","last_unstale":"2024-04-25T15:26:50.911084+0000","last_undegraded":"2024-04-25T15:26:50.911084+0000","last_fullsized":"2024-04-25T15:26:50.911084+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:32.928182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746612+0000","last_change":"2024-04-25T15:26:49.764719+0000","last_active":"2024-04-25T15:26:50.746612+0000","last_peered":"2024-04-25T15:26:50.746612+0000","last_clean":"2024-04-25T15:26:50.746612+0000","last_became_active":"2024-04-25T15:26:49.764392+0000","last_became_peered":"2024-04-25T15:26:49.764392+0000","last_unstale":"2024-04-25T15:26:50.746612+0000","last_undegraded":"2024-04-25T15:26:50.746612+0000","last_fullsized":"2024-04-25T15:26:50.746612+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:25.735036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"44'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.765498+0000","last_change":"2024-04-25T15:26:48.744777+0000","last_active":"2024-04-25T15:26:50.765498+0000","last_peered":"2024-04-25T15:26:50.765498+0000","last_clean":"2024-04-25T15:26:50.765498+0000","last_became_active":"2024-04-25T15:26:48.744464+0000","last_became_peered":"2024-04-25T15:26:48.744464+0000","last_unstale":"2024-04-25T15:26:50.765498+0000","last_undegraded":"2024-04-25T15:26:50.765498+0000","last_fullsized":"2024-04-25T15:26:50.765498+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:30.322609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745962+0000","last_change":"2024-04-25T15:26:49.771404+0000","last_active":"2024-04-25T15:26:50.745962+0000","last_peered":"2024-04-25T15:26:50.745962+0000","last_clean":"2024-04-25T15:26:50.745962+0000","last_became_active":"2024-04-25T15:26:49.770616+0000","last_became_peered":"2024-04-25T15:26:49.770616+0000","last_unstale":"2024-04-25T15:26:50.745962+0000","last_undegraded":"2024-04-25T15:26:50.745962+0000","last_fullsized":"2024-04-25T15:26:50.745962+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:22.027488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911551+0000","last_change":"2024-04-25T15:26:48.751668+0000","last_active":"2024-04-25T15:26:50.911551+0000","last_peered":"2024-04-25T15:26:50.911551+0000","last_clean":"2024-04-25T15:26:50.911551+0000","last_became_active":"2024-04-25T15:26:48.751344+0000","last_became_peered":"2024-04-25T15:26:48.751344+0000","last_unstale":"2024-04-25T15:26:50.911551+0000","last_undegraded":"2024-04-25T15:26:50.911551+0000","last_fullsized":"2024-04-25T15:26:50.911551+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:47.077528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911435+0000","last_change":"2024-04-25T15:26:49.762698+0000","last_active":"2024-04-25T15:26:50.911435+0000","last_peered":"2024-04-25T15:26:50.911435+0000","last_clean":"2024-04-25T15:26:50.911435+0000","last_became_active":"2024-04-25T15:26:49.762219+0000","last_became_peered":"2024-04-25T15:26:49.762219+0000","last_unstale":"2024-04-25T15:26:50.911435+0000","last_undegraded":"2024-04-25T15:26:50.911435+0000","last_fullsized":"2024-04-25T15:26:50.911435+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:13.459028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910734+0000","last_change":"2024-04-25T15:26:48.739991+0000","last_active":"2024-04-25T15:26:50.910734+0000","last_peered":"2024-04-25T15:26:50.910734+0000","last_clean":"2024-04-25T15:26:50.910734+0000","last_became_active":"2024-04-25T15:26:48.739680+0000","last_became_peered":"2024-04-25T15:26:48.739680+0000","last_unstale":"2024-04-25T15:26:50.910734+0000","last_undegraded":"2024-04-25T15:26:50.910734+0000","last_fullsized":"2024-04-25T15:26:50.910734+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:01.565352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763574+0000","last_change":"2024-04-25T15:26:49.751089+0000","last_active":"2024-04-25T15:26:50.763574+0000","last_peered":"2024-04-25T15:26:50.763574+0000","last_clean":"2024-04-25T15:26:50.763574+0000","last_became_active":"2024-04-25T15:26:49.750778+0000","last_became_peered":"2024-04-25T15:26:49.750778+0000","last_unstale":"2024-04-25T15:26:50.763574+0000","last_undegraded":"2024-04-25T15:26:50.763574+0000","last_fullsized":"2024-04-25T15:26:50.763574+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:08.197424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911663+0000","last_change":"2024-04-25T15:26:48.752374+0000","last_active":"2024-04-25T15:26:50.911663+0000","last_peered":"2024-04-25T15:26:50.911663+0000","last_clean":"2024-04-25T15:26:50.911663+0000","last_became_active":"2024-04-25T15:26:48.751988+0000","last_became_peered":"2024-04-25T15:26:48.751988+0000","last_unstale":"2024-04-25T15:26:50.911663+0000","last_undegraded":"2024-04-25T15:26:50.911663+0000","last_fullsized":"2024-04-25T15:26:50.911663+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:24.729725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763906+0000","last_change":"2024-04-25T15:26:49.750516+0000","last_active":"2024-04-25T15:26:50.763906+0000","last_peered":"2024-04-25T15:26:50.763906+0000","last_clean":"2024-04-25T15:26:50.763906+0000","last_became_active":"2024-04-25T15:26:49.750125+0000","last_became_peered":"2024-04-25T15:26:49.750125+0000","last_unstale":"2024-04-25T15:26:50.763906+0000","last_undegraded":"2024-04-25T15:26:50.763906+0000","last_fullsized":"2024-04-25T15:26:50.763906+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:16.447499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766351+0000","last_change":"2024-04-25T15:26:48.747835+0000","last_active":"2024-04-25T15:26:50.766351+0000","last_peered":"2024-04-25T15:26:50.766351+0000","last_clean":"2024-04-25T15:26:50.766351+0000","last_became_active":"2024-04-25T15:26:48.747571+0000","last_became_peered":"2024-04-25T15:26:48.747571+0000","last_unstale":"2024-04-25T15:26:50.766351+0000","last_undegraded":"2024-04-25T15:26:50.766351+0000","last_fullsized":"2024-04-25T15:26:50.766351+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:33.242255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911909+0000","last_change":"2024-04-25T15:26:49.763851+0000","last_active":"2024-04-25T15:26:50.911909+0000","last_peered":"2024-04-25T15:26:50.911909+0000","last_clean":"2024-04-25T15:26:50.911909+0000","last_became_active":"2024-04-25T15:26:49.763575+0000","last_became_peered":"2024-04-25T15:26:49.763575+0000","last_unstale":"2024-04-25T15:26:50.911909+0000","last_undegraded":"2024-04-25T15:26:50.911909+0000","last_fullsized":"2024-04-25T15:26:50.911909+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:57.061251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910715+0000","last_change":"2024-04-25T15:26:48.745812+0000","last_active":"2024-04-25T15:26:50.910715+0000","last_peered":"2024-04-25T15:26:50.910715+0000","last_clean":"2024-04-25T15:26:50.910715+0000","last_became_active":"2024-04-25T15:26:48.744167+0000","last_became_peered":"2024-04-25T15:26:48.744167+0000","last_unstale":"2024-04-25T15:26:50.910715+0000","last_undegraded":"2024-04-25T15:26:50.910715+0000","last_fullsized":"2024-04-25T15:26:50.910715+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:45.473777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766164+0000","last_change":"2024-04-25T15:26:49.754629+0000","last_active":"2024-04-25T15:26:50.766164+0000","last_peered":"2024-04-25T15:26:50.766164+0000","last_clean":"2024-04-25T15:26:50.766164+0000","last_became_active":"2024-04-25T15:26:49.754273+0000","last_became_peered":"2024-04-25T15:26:49.754273+0000","last_unstale":"2024-04-25T15:26:50.766164+0000","last_undegraded":"2024-04-25T15:26:50.766164+0000","last_fullsized":"2024-04-25T15:26:50.766164+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:30.438279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745584+0000","last_change":"2024-04-25T15:26:48.745019+0000","last_active":"2024-04-25T15:26:50.745584+0000","last_peered":"2024-04-25T15:26:50.745584+0000","last_clean":"2024-04-25T15:26:50.745584+0000","last_became_active":"2024-04-25T15:26:48.744145+0000","last_became_peered":"2024-04-25T15:26:48.744145+0000","last_unstale":"2024-04-25T15:26:50.745584+0000","last_undegraded":"2024-04-25T15:26:50.745584+0000","last_fullsized":"2024-04-25T15:26:50.745584+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:00.557944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746144+0000","last_change":"2024-04-25T15:26:49.766393+0000","last_active":"2024-04-25T15:26:50.746144+0000","last_peered":"2024-04-25T15:26:50.746144+0000","last_clean":"2024-04-25T15:26:50.746144+0000","last_became_active":"2024-04-25T15:26:49.764426+0000","last_became_peered":"2024-04-25T15:26:49.764426+0000","last_unstale":"2024-04-25T15:26:50.746144+0000","last_undegraded":"2024-04-25T15:26:50.746144+0000","last_fullsized":"2024-04-25T15:26:50.746144+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:46.641900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912055+0000","last_change":"2024-04-25T15:26:48.748644+0000","last_active":"2024-04-25T15:26:50.912055+0000","last_peered":"2024-04-25T15:26:50.912055+0000","last_clean":"2024-04-25T15:26:50.912055+0000","last_became_active":"2024-04-25T15:26:48.748280+0000","last_became_peered":"2024-04-25T15:26:48.748280+0000","last_unstale":"2024-04-25T15:26:50.912055+0000","last_undegraded":"2024-04-25T15:26:50.912055+0000","last_fullsized":"2024-04-25T15:26:50.912055+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:32.928182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":153,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911105+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:26:50.911105+0000","last_peered":"2024-04-25T15:26:50.911105+0000","last_clean":"2024-04-25T15:26:50.911105+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:26:50.911105+0000","last_undegraded":"2024-04-25T15:26:50.911105+0000","last_fullsized":"2024-04-25T15:26:50.911105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":800266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":210480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":203578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":211420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":210446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":211442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":209878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":800210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:26:53.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:26:53.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.954+0000 7fa7780f9640 1 Processor -- start 2024-04-25T15:26:53.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.954+0000 7fa7780f9640 1 -- start start 2024-04-25T15:26:53.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa7780f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7700cf5e0 0x7fa7700cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa7780f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa77006c5c0 0x7fa77006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa7780f9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7700cff30 con 0x7fa7701443c0 2024-04-25T15:26:53.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa7780f9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa7700708d0 con 0x7fa77006c5c0 2024-04-25T15:26:53.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa7780f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa77015f160 con 0x7fa7700cf5e0 2024-04-25T15:26:53.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa776696640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa77006c5c0 0x7fa77006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa775e95640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa7701443c0 0x7fa77006fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43278/0 (socket says 172.21.15.67:43278) 2024-04-25T15:26:53.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa775e95640 1 -- 172.21.15.67:0/1904050171 learned_addr learned my addr 172.21.15.67:0/1904050171 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:53.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa776696640 1 -- 172.21.15.67:0/1904050171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7700cf5e0 msgr2=0x7fa7700cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:53.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa776696640 1 --2- 172.21.15.67:0/1904050171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7700cf5e0 0x7fa7700cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa776696640 1 -- 172.21.15.67:0/1904050171 >> v1:172.21.15.67:6789/0 conn(0x7fa7701443c0 legacy=0x7fa77006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.956+0000 7fa776696640 1 -- 172.21.15.67:0/1904050171 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77015f5f0 con 0x7fa77006c5c0 2024-04-25T15:26:53.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.957+0000 7fa776696640 1 --2- 172.21.15.67:0/1904050171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77006c9d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa760009d90 tx=0x7fa76002a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b68ade1bd6213b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.957+0000 7fa774e93640 1 -- 172.21.15.67:0/1904050171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760007740 con 0x7fa77006c5c0 2024-04-25T15:26:53.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.957+0000 7fa774e93640 1 -- 172.21.15.67:0/1904050171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7600078e0 con 0x7fa77006c5c0 2024-04-25T15:26:53.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.957+0000 7fa774e93640 1 -- 172.21.15.67:0/1904050171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760007cb0 con 0x7fa77006c5c0 2024-04-25T15:26:53.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 -- 172.21.15.67:0/1904050171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 msgr2=0x7fa77006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 --2- 172.21.15.67:0/1904050171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77006c9d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa760009d90 tx=0x7fa76002a560 comp rx=0 tx=0).stop 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 -- 172.21.15.67:0/1904050171 shutdown_connections 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 --2- 172.21.15.67:0/1904050171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77006c9d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 --2- 172.21.15.67:0/1904050171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7700cf5e0 0x7fa7700cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 -- 172.21.15.67:0/1904050171 >> 172.21.15.67:0/1904050171 conn(0x7fa7700d1ae0 msgr2=0x7fa7700d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 -- 172.21.15.67:0/1904050171 shutdown_connections 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.958+0000 7fa7780f9640 1 -- 172.21.15.67:0/1904050171 wait complete. 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.960+0000 7fa7780f9640 1 Processor -- start 2024-04-25T15:26:53.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.960+0000 7fa7780f9640 1 -- start start 2024-04-25T15:26:53.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa7780f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77013a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa7780f9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7700cf5e0 0x7fa770151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa7780f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7701443c0 0x7fa770152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa7780f9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa77015edf0 con 0x7fa77006c5c0 2024-04-25T15:26:53.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa7780f9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa77015eb10 con 0x7fa7700cf5e0 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa7780f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa7701447d0 con 0x7fa7701443c0 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa775e95640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77013a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa775e95640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77013a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44904/0 (socket says 172.21.15.67:44904) 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa775e95640 1 -- 172.21.15.67:0/3483872536 learned_addr learned my addr 172.21.15.67:0/3483872536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa776696640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7701443c0 0x7fa770152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.961+0000 7fa775694640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7700cf5e0 0x7fa770151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa775e95640 1 -- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7701443c0 msgr2=0x7fa770152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa775e95640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7701443c0 0x7fa770152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa775e95640 1 -- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7700cf5e0 msgr2=0x7fa770151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa775e95640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7700cf5e0 0x7fa770151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa775e95640 1 -- 172.21.15.67:0/3483872536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77013a8f0 con 0x7fa77006c5c0 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa775694640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7700cf5e0 0x7fa770151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:53.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa775e95640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77013a3b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa76800c7b0 tx=0x7fa76800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa76e7fc640 1 -- 172.21.15.67:0/3483872536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768018070 con 0x7fa77006c5c0 2024-04-25T15:26:53.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa76e7fc640 1 -- 172.21.15.67:0/3483872536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7680074e0 con 0x7fa77006c5c0 2024-04-25T15:26:53.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7701e80b0 con 0x7fa77006c5c0 2024-04-25T15:26:53.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa76e7fc640 1 -- 172.21.15.67:0/3483872536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76800ce60 con 0x7fa77006c5c0 2024-04-25T15:26:53.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.962+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7701e8670 con 0x7fa77006c5c0 2024-04-25T15:26:53.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.963+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa770050740 con 0x7fa77006c5c0 2024-04-25T15:26:53.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.964+0000 7fa76e7fc640 1 -- 172.21.15.67:0/3483872536 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa76801b070 con 0x7fa77006c5c0 2024-04-25T15:26:53.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.964+0000 7fa76e7fc640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa75c0744b0 0x7fa75c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:53.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.964+0000 7fa76e7fc640 1 -- 172.21.15.67:0/3483872536 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fa7680946e0 con 0x7fa77006c5c0 2024-04-25T15:26:53.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.965+0000 7fa775694640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa75c0744b0 0x7fa75c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:53.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.965+0000 7fa775694640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa75c0744b0 0x7fa75c0769c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa77006d280 tx=0x7fa764008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:53.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:53.969+0000 7fa76e7fc640 1 -- 172.21.15.67:0/3483872536 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa77013a8f0 con 0x7fa77006c5c0 2024-04-25T15:26:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.153+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa77006f950 con 0x7fa75c0744b0 2024-04-25T15:26:54.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.159+0000 7fa76e7fc640 1 -- 172.21.15.67:0/3483872536 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7fa77006f950 con 0x7fa75c0744b0 2024-04-25T15:26:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:54.164 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa75c0744b0 msgr2=0x7fa75c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa75c0744b0 0x7fa75c0769c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa77006d280 tx=0x7fa764008040 comp rx=0 tx=0).stop 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 msgr2=0x7fa77013a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77013a3b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa76800c7b0 tx=0x7fa76800cc70 comp rx=0 tx=0).stop 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 shutdown_connections 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa75c0744b0 0x7fa75c0769c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7701443c0 0x7fa770152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7700cf5e0 0x7fa770151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 --2- 172.21.15.67:0/3483872536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa77006c5c0 0x7fa77013a3b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 >> 172.21.15.67:0/3483872536 conn(0x7fa7700d1ae0 msgr2=0x7fa77015f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:54.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 shutdown_connections 2024-04-25T15:26:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.162+0000 7fa7780f9640 1 -- 172.21.15.67:0/3483872536 wait complete. 2024-04-25T15:26:54.174 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-04-25T15:26:52.814855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221412,"kb_used_data":5244,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753308,"statfs":{"total":773094113280,"available":772867387392,"internally_reserved":0,"allocated":5369856,"data_stored":2857720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.089941"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911636+0000","last_change":"2024-04-25T15:26:49.767205+0000","last_active":"2024-04-25T15:26:50.911636+0000","last_peered":"2024-04-25T15:26:50.911636+0000","last_clean":"2024-04-25T15:26:50.911636+0000","last_became_active":"2024-04-25T15:26:49.766906+0000","last_became_peered":"2024-04-25T15:26:49.766906+0000","last_unstale":"2024-04-25T15:26:50.911636+0000","last_undegraded":"2024-04-25T15:26:50.911636+0000","last_fullsized":"2024-04-25T15:26:50.911636+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:47:55.546423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912063+0000","last_change":"2024-04-25T15:26:49.767237+0000","last_active":"2024-04-25T15:26:50.912063+0000","last_peered":"2024-04-25T15:26:50.912063+0000","last_clean":"2024-04-25T15:26:50.912063+0000","last_became_active":"2024-04-25T15:26:49.766878+0000","last_became_peered":"2024-04-25T15:26:49.766878+0000","last_unstale":"2024-04-25T15:26:50.912063+0000","last_undegraded":"2024-04-25T15:26:50.912063+0000","last_fullsized":"2024-04-25T15:26:50.912063+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:26:57.031744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766386+0000","last_change":"2024-04-25T15:26:49.757909+0000","last_active":"2024-04-25T15:26:50.766386+0000","last_peered":"2024-04-25T15:26:50.766386+0000","last_clean":"2024-04-25T15:26:50.766386+0000","last_became_active":"2024-04-25T15:26:49.757459+0000","last_became_peered":"2024-04-25T15:26:49.757459+0000","last_unstale":"2024-04-25T15:26:50.766386+0000","last_undegraded":"2024-04-25T15:26:50.766386+0000","last_fullsized":"2024-04-25T15:26:50.766386+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:15.418323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745881+0000","last_change":"2024-04-25T15:26:49.771297+0000","last_active":"2024-04-25T15:26:50.745881+0000","last_peered":"2024-04-25T15:26:50.745881+0000","last_clean":"2024-04-25T15:26:50.745881+0000","last_became_active":"2024-04-25T15:26:49.770206+0000","last_became_peered":"2024-04-25T15:26:49.770206+0000","last_unstale":"2024-04-25T15:26:50.745881+0000","last_undegraded":"2024-04-25T15:26:50.745881+0000","last_fullsized":"2024-04-25T15:26:50.745881+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:26:57.031744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911730+0000","last_change":"2024-04-25T15:26:49.767848+0000","last_active":"2024-04-25T15:26:50.911730+0000","last_peered":"2024-04-25T15:26:50.911730+0000","last_clean":"2024-04-25T15:26:50.911730+0000","last_became_active":"2024-04-25T15:26:49.767457+0000","last_became_peered":"2024-04-25T15:26:49.767457+0000","last_unstale":"2024-04-25T15:26:50.911730+0000","last_undegraded":"2024-04-25T15:26:50.911730+0000","last_fullsized":"2024-04-25T15:26:50.911730+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:32.612916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763389+0000","last_change":"2024-04-25T15:26:49.759879+0000","last_active":"2024-04-25T15:26:50.763389+0000","last_peered":"2024-04-25T15:26:50.763389+0000","last_clean":"2024-04-25T15:26:50.763389+0000","last_became_active":"2024-04-25T15:26:49.759548+0000","last_became_peered":"2024-04-25T15:26:49.759548+0000","last_unstale":"2024-04-25T15:26:50.763389+0000","last_undegraded":"2024-04-25T15:26:50.763389+0000","last_fullsized":"2024-04-25T15:26:50.763389+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:23:06.988004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911267+0000","last_change":"2024-04-25T15:26:49.762842+0000","last_active":"2024-04-25T15:26:50.911267+0000","last_peered":"2024-04-25T15:26:50.911267+0000","last_clean":"2024-04-25T15:26:50.911267+0000","last_became_active":"2024-04-25T15:26:49.762506+0000","last_became_peered":"2024-04-25T15:26:49.762506+0000","last_unstale":"2024-04-25T15:26:50.911267+0000","last_undegraded":"2024-04-25T15:26:50.911267+0000","last_fullsized":"2024-04-25T15:26:50.911267+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:31.327920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745373+0000","last_change":"2024-04-25T15:26:49.772980+0000","last_active":"2024-04-25T15:26:50.745373+0000","last_peered":"2024-04-25T15:26:50.745373+0000","last_clean":"2024-04-25T15:26:50.745373+0000","last_became_active":"2024-04-25T15:26:49.772682+0000","last_became_peered":"2024-04-25T15:26:49.772682+0000","last_unstale":"2024-04-25T15:26:50.745373+0000","last_undegraded":"2024-04-25T15:26:50.745373+0000","last_fullsized":"2024-04-25T15:26:50.745373+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:48.082839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746165+0000","last_change":"2024-04-25T15:26:49.766308+0000","last_active":"2024-04-25T15:26:50.746165+0000","last_peered":"2024-04-25T15:26:50.746165+0000","last_clean":"2024-04-25T15:26:50.746165+0000","last_became_active":"2024-04-25T15:26:49.766001+0000","last_became_peered":"2024-04-25T15:26:49.766001+0000","last_unstale":"2024-04-25T15:26:50.746165+0000","last_undegraded":"2024-04-25T15:26:50.746165+0000","last_fullsized":"2024-04-25T15:26:50.746165+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:12.831550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746071+0000","last_change":"2024-04-25T15:26:49.771702+0000","last_active":"2024-04-25T15:26:50.746071+0000","last_peered":"2024-04-25T15:26:50.746071+0000","last_clean":"2024-04-25T15:26:50.746071+0000","last_became_active":"2024-04-25T15:26:49.771362+0000","last_became_peered":"2024-04-25T15:26:49.771362+0000","last_unstale":"2024-04-25T15:26:50.746071+0000","last_undegraded":"2024-04-25T15:26:50.746071+0000","last_fullsized":"2024-04-25T15:26:50.746071+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:47:55.546423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745949+0000","last_change":"2024-04-25T15:26:49.764937+0000","last_active":"2024-04-25T15:26:50.745949+0000","last_peered":"2024-04-25T15:26:50.745949+0000","last_clean":"2024-04-25T15:26:50.745949+0000","last_became_active":"2024-04-25T15:26:49.764072+0000","last_became_peered":"2024-04-25T15:26:49.764072+0000","last_unstale":"2024-04-25T15:26:50.745949+0000","last_undegraded":"2024-04-25T15:26:50.745949+0000","last_fullsized":"2024-04-25T15:26:50.745949+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:31.327920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912095+0000","last_change":"2024-04-25T15:26:49.762913+0000","last_active":"2024-04-25T15:26:50.912095+0000","last_peered":"2024-04-25T15:26:50.912095+0000","last_clean":"2024-04-25T15:26:50.912095+0000","last_became_active":"2024-04-25T15:26:49.762594+0000","last_became_peered":"2024-04-25T15:26:49.762594+0000","last_unstale":"2024-04-25T15:26:50.912095+0000","last_undegraded":"2024-04-25T15:26:50.912095+0000","last_fullsized":"2024-04-25T15:26:50.912095+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:04.867170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742798+0000","last_change":"2024-04-25T15:26:49.747707+0000","last_active":"2024-04-25T15:26:50.742798+0000","last_peered":"2024-04-25T15:26:50.742798+0000","last_clean":"2024-04-25T15:26:50.742798+0000","last_became_active":"2024-04-25T15:26:49.747326+0000","last_became_peered":"2024-04-25T15:26:49.747326+0000","last_unstale":"2024-04-25T15:26:50.742798+0000","last_undegraded":"2024-04-25T15:26:50.742798+0000","last_fullsized":"2024-04-25T15:26:50.742798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:57.941388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745539+0000","last_change":"2024-04-25T15:26:49.762906+0000","last_active":"2024-04-25T15:26:50.745539+0000","last_peered":"2024-04-25T15:26:50.745539+0000","last_clean":"2024-04-25T15:26:50.745539+0000","last_became_active":"2024-04-25T15:26:49.761926+0000","last_became_peered":"2024-04-25T15:26:49.761926+0000","last_unstale":"2024-04-25T15:26:50.745539+0000","last_undegraded":"2024-04-25T15:26:50.745539+0000","last_fullsized":"2024-04-25T15:26:50.745539+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:32.612916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763320+0000","last_change":"2024-04-25T15:26:49.765391+0000","last_active":"2024-04-25T15:26:50.763320+0000","last_peered":"2024-04-25T15:26:50.763320+0000","last_clean":"2024-04-25T15:26:50.763320+0000","last_became_active":"2024-04-25T15:26:49.761318+0000","last_became_peered":"2024-04-25T15:26:49.761318+0000","last_unstale":"2024-04-25T15:26:50.763320+0000","last_undegraded":"2024-04-25T15:26:50.763320+0000","last_fullsized":"2024-04-25T15:26:50.763320+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:21:08.396080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746487+0000","last_change":"2024-04-25T15:26:49.772836+0000","last_active":"2024-04-25T15:26:50.746487+0000","last_peered":"2024-04-25T15:26:50.746487+0000","last_clean":"2024-04-25T15:26:50.746487+0000","last_became_active":"2024-04-25T15:26:49.772345+0000","last_became_peered":"2024-04-25T15:26:49.772345+0000","last_unstale":"2024-04-25T15:26:50.746487+0000","last_undegraded":"2024-04-25T15:26:50.746487+0000","last_fullsized":"2024-04-25T15:26:50.746487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:03.943226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766267+0000","last_change":"2024-04-25T15:26:49.761556+0000","last_active":"2024-04-25T15:26:50.766267+0000","last_peered":"2024-04-25T15:26:50.766267+0000","last_clean":"2024-04-25T15:26:50.766267+0000","last_became_active":"2024-04-25T15:26:49.761226+0000","last_became_peered":"2024-04-25T15:26:49.761226+0000","last_unstale":"2024-04-25T15:26:50.766267+0000","last_undegraded":"2024-04-25T15:26:50.766267+0000","last_fullsized":"2024-04-25T15:26:50.766267+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:52.172769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766307+0000","last_change":"2024-04-25T15:26:49.768683+0000","last_active":"2024-04-25T15:26:50.766307+0000","last_peered":"2024-04-25T15:26:50.766307+0000","last_clean":"2024-04-25T15:26:50.766307+0000","last_became_active":"2024-04-25T15:26:49.768327+0000","last_became_peered":"2024-04-25T15:26:49.768327+0000","last_unstale":"2024-04-25T15:26:50.766307+0000","last_undegraded":"2024-04-25T15:26:50.766307+0000","last_fullsized":"2024-04-25T15:26:50.766307+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:48.082839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746514+0000","last_change":"2024-04-25T15:26:49.772764+0000","last_active":"2024-04-25T15:26:50.746514+0000","last_peered":"2024-04-25T15:26:50.746514+0000","last_clean":"2024-04-25T15:26:50.746514+0000","last_became_active":"2024-04-25T15:26:49.772410+0000","last_became_peered":"2024-04-25T15:26:49.772410+0000","last_unstale":"2024-04-25T15:26:50.746514+0000","last_undegraded":"2024-04-25T15:26:50.746514+0000","last_fullsized":"2024-04-25T15:26:50.746514+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:49.933480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.743073+0000","last_change":"2024-04-25T15:26:49.764578+0000","last_active":"2024-04-25T15:26:50.743073+0000","last_peered":"2024-04-25T15:26:50.743073+0000","last_clean":"2024-04-25T15:26:50.743073+0000","last_became_active":"2024-04-25T15:26:49.764250+0000","last_became_peered":"2024-04-25T15:26:49.764250+0000","last_unstale":"2024-04-25T15:26:50.743073+0000","last_undegraded":"2024-04-25T15:26:50.743073+0000","last_fullsized":"2024-04-25T15:26:50.743073+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:18:20.978355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763476+0000","last_change":"2024-04-25T15:26:49.754648+0000","last_active":"2024-04-25T15:26:50.763476+0000","last_peered":"2024-04-25T15:26:50.763476+0000","last_clean":"2024-04-25T15:26:50.763476+0000","last_became_active":"2024-04-25T15:26:49.754349+0000","last_became_peered":"2024-04-25T15:26:49.754349+0000","last_unstale":"2024-04-25T15:26:50.763476+0000","last_undegraded":"2024-04-25T15:26:50.763476+0000","last_fullsized":"2024-04-25T15:26:50.763476+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:59:54.869352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745727+0000","last_change":"2024-04-25T15:26:49.771579+0000","last_active":"2024-04-25T15:26:50.745727+0000","last_peered":"2024-04-25T15:26:50.745727+0000","last_clean":"2024-04-25T15:26:50.745727+0000","last_became_active":"2024-04-25T15:26:49.770862+0000","last_became_peered":"2024-04-25T15:26:49.770862+0000","last_unstale":"2024-04-25T15:26:50.745727+0000","last_undegraded":"2024-04-25T15:26:50.745727+0000","last_fullsized":"2024-04-25T15:26:50.745727+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:57.061251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745914+0000","last_change":"2024-04-25T15:26:49.763645+0000","last_active":"2024-04-25T15:26:50.745914+0000","last_peered":"2024-04-25T15:26:50.745914+0000","last_clean":"2024-04-25T15:26:50.745914+0000","last_became_active":"2024-04-25T15:26:49.762838+0000","last_became_peered":"2024-04-25T15:26:49.762838+0000","last_unstale":"2024-04-25T15:26:50.745914+0000","last_undegraded":"2024-04-25T15:26:50.745914+0000","last_fullsized":"2024-04-25T15:26:50.745914+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:46.479088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745931+0000","last_change":"2024-04-25T15:26:49.757459+0000","last_active":"2024-04-25T15:26:50.745931+0000","last_peered":"2024-04-25T15:26:50.745931+0000","last_clean":"2024-04-25T15:26:50.745931+0000","last_became_active":"2024-04-25T15:26:49.757137+0000","last_became_peered":"2024-04-25T15:26:49.757137+0000","last_unstale":"2024-04-25T15:26:50.745931+0000","last_undegraded":"2024-04-25T15:26:50.745931+0000","last_fullsized":"2024-04-25T15:26:50.745931+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:02.570663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746282+0000","last_change":"2024-04-25T15:26:49.762958+0000","last_active":"2024-04-25T15:26:50.746282+0000","last_peered":"2024-04-25T15:26:50.746282+0000","last_clean":"2024-04-25T15:26:50.746282+0000","last_became_active":"2024-04-25T15:26:49.762634+0000","last_became_peered":"2024-04-25T15:26:49.762634+0000","last_unstale":"2024-04-25T15:26:50.746282+0000","last_undegraded":"2024-04-25T15:26:50.746282+0000","last_fullsized":"2024-04-25T15:26:50.746282+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:37.306991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742668+0000","last_change":"2024-04-25T15:26:49.750802+0000","last_active":"2024-04-25T15:26:50.742668+0000","last_peered":"2024-04-25T15:26:50.742668+0000","last_clean":"2024-04-25T15:26:50.742668+0000","last_became_active":"2024-04-25T15:26:49.750495+0000","last_became_peered":"2024-04-25T15:26:49.750495+0000","last_unstale":"2024-04-25T15:26:50.742668+0000","last_undegraded":"2024-04-25T15:26:50.742668+0000","last_fullsized":"2024-04-25T15:26:50.742668+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:11.969655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766133+0000","last_change":"2024-04-25T15:26:49.760749+0000","last_active":"2024-04-25T15:26:50.766133+0000","last_peered":"2024-04-25T15:26:50.766133+0000","last_clean":"2024-04-25T15:26:50.766133+0000","last_became_active":"2024-04-25T15:26:49.760428+0000","last_became_peered":"2024-04-25T15:26:49.760428+0000","last_unstale":"2024-04-25T15:26:50.766133+0000","last_undegraded":"2024-04-25T15:26:50.766133+0000","last_fullsized":"2024-04-25T15:26:50.766133+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:25.735036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912161+0000","last_change":"2024-04-25T15:26:49.762439+0000","last_active":"2024-04-25T15:26:50.912161+0000","last_peered":"2024-04-25T15:26:50.912161+0000","last_clean":"2024-04-25T15:26:50.912161+0000","last_became_active":"2024-04-25T15:26:49.762083+0000","last_became_peered":"2024-04-25T15:26:49.762083+0000","last_unstale":"2024-04-25T15:26:50.912161+0000","last_undegraded":"2024-04-25T15:26:50.912161+0000","last_fullsized":"2024-04-25T15:26:50.912161+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:31.327920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746516+0000","last_change":"2024-04-25T15:26:49.761029+0000","last_active":"2024-04-25T15:26:50.746516+0000","last_peered":"2024-04-25T15:26:50.746516+0000","last_clean":"2024-04-25T15:26:50.746516+0000","last_became_active":"2024-04-25T15:26:49.760706+0000","last_became_peered":"2024-04-25T15:26:49.760706+0000","last_unstale":"2024-04-25T15:26:50.746516+0000","last_undegraded":"2024-04-25T15:26:50.746516+0000","last_fullsized":"2024-04-25T15:26:50.746516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:19.733261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766060+0000","last_change":"2024-04-25T15:26:49.758922+0000","last_active":"2024-04-25T15:26:50.766060+0000","last_peered":"2024-04-25T15:26:50.766060+0000","last_clean":"2024-04-25T15:26:50.766060+0000","last_became_active":"2024-04-25T15:26:49.758416+0000","last_became_peered":"2024-04-25T15:26:49.758416+0000","last_unstale":"2024-04-25T15:26:50.766060+0000","last_undegraded":"2024-04-25T15:26:50.766060+0000","last_fullsized":"2024-04-25T15:26:50.766060+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:33.933493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763380+0000","last_change":"2024-04-25T15:26:49.765303+0000","last_active":"2024-04-25T15:26:50.763380+0000","last_peered":"2024-04-25T15:26:50.763380+0000","last_clean":"2024-04-25T15:26:50.763380+0000","last_became_active":"2024-04-25T15:26:49.761077+0000","last_became_peered":"2024-04-25T15:26:49.761077+0000","last_unstale":"2024-04-25T15:26:50.763380+0000","last_undegraded":"2024-04-25T15:26:50.763380+0000","last_fullsized":"2024-04-25T15:26:50.763380+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:06.710458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742639+0000","last_change":"2024-04-25T15:26:49.748956+0000","last_active":"2024-04-25T15:26:50.742639+0000","last_peered":"2024-04-25T15:26:50.742639+0000","last_clean":"2024-04-25T15:26:50.742639+0000","last_became_active":"2024-04-25T15:26:49.748629+0000","last_became_peered":"2024-04-25T15:26:49.748629+0000","last_unstale":"2024-04-25T15:26:50.742639+0000","last_undegraded":"2024-04-25T15:26:50.742639+0000","last_fullsized":"2024-04-25T15:26:50.742639+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:44:06.129747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910818+0000","last_change":"2024-04-25T15:26:49.750003+0000","last_active":"2024-04-25T15:26:50.910818+0000","last_peered":"2024-04-25T15:26:50.910818+0000","last_clean":"2024-04-25T15:26:50.910818+0000","last_became_active":"2024-04-25T15:26:49.749683+0000","last_became_peered":"2024-04-25T15:26:49.749683+0000","last_unstale":"2024-04-25T15:26:50.910818+0000","last_undegraded":"2024-04-25T15:26:50.910818+0000","last_fullsized":"2024-04-25T15:26:50.910818+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:49.933480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.764005+0000","last_change":"2024-04-25T15:26:48.748242+0000","last_active":"2024-04-25T15:26:50.764005+0000","last_peered":"2024-04-25T15:26:50.764005+0000","last_clean":"2024-04-25T15:26:50.764005+0000","last_became_active":"2024-04-25T15:26:48.747914+0000","last_became_peered":"2024-04-25T15:26:48.747914+0000","last_unstale":"2024-04-25T15:26:50.764005+0000","last_undegraded":"2024-04-25T15:26:50.764005+0000","last_fullsized":"2024-04-25T15:26:50.764005+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:31.607605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746497+0000","last_change":"2024-04-25T15:26:49.767001+0000","last_active":"2024-04-25T15:26:50.746497+0000","last_peered":"2024-04-25T15:26:50.746497+0000","last_clean":"2024-04-25T15:26:50.746497+0000","last_became_active":"2024-04-25T15:26:49.766691+0000","last_became_peered":"2024-04-25T15:26:49.766691+0000","last_unstale":"2024-04-25T15:26:50.746497+0000","last_undegraded":"2024-04-25T15:26:50.746497+0000","last_fullsized":"2024-04-25T15:26:50.746497+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:41:36.882975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766434+0000","last_change":"2024-04-25T15:26:48.739592+0000","last_active":"2024-04-25T15:26:50.766434+0000","last_peered":"2024-04-25T15:26:50.766434+0000","last_clean":"2024-04-25T15:26:50.766434+0000","last_became_active":"2024-04-25T15:26:48.739265+0000","last_became_peered":"2024-04-25T15:26:48.739265+0000","last_unstale":"2024-04-25T15:26:50.766434+0000","last_undegraded":"2024-04-25T15:26:50.766434+0000","last_fullsized":"2024-04-25T15:26:50.766434+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:55.259761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912277+0000","last_change":"2024-04-25T15:26:49.759953+0000","last_active":"2024-04-25T15:26:50.912277+0000","last_peered":"2024-04-25T15:26:50.912277+0000","last_clean":"2024-04-25T15:26:50.912277+0000","last_became_active":"2024-04-25T15:26:49.759600+0000","last_became_peered":"2024-04-25T15:26:49.759600+0000","last_unstale":"2024-04-25T15:26:50.912277+0000","last_undegraded":"2024-04-25T15:26:50.912277+0000","last_fullsized":"2024-04-25T15:26:50.912277+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:22.027488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910995+0000","last_change":"2024-04-25T15:26:48.740212+0000","last_active":"2024-04-25T15:26:50.910995+0000","last_peered":"2024-04-25T15:26:50.910995+0000","last_clean":"2024-04-25T15:26:50.910995+0000","last_became_active":"2024-04-25T15:26:48.739902+0000","last_became_peered":"2024-04-25T15:26:48.739902+0000","last_unstale":"2024-04-25T15:26:50.910995+0000","last_undegraded":"2024-04-25T15:26:50.910995+0000","last_fullsized":"2024-04-25T15:26:50.910995+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:40.127334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910898+0000","last_change":"2024-04-25T15:26:49.758962+0000","last_active":"2024-04-25T15:26:50.910898+0000","last_peered":"2024-04-25T15:26:50.910898+0000","last_clean":"2024-04-25T15:26:50.910898+0000","last_became_active":"2024-04-25T15:26:49.758084+0000","last_became_peered":"2024-04-25T15:26:49.758084+0000","last_unstale":"2024-04-25T15:26:50.910898+0000","last_undegraded":"2024-04-25T15:26:50.910898+0000","last_fullsized":"2024-04-25T15:26:50.910898+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:10.302228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911316+0000","last_change":"2024-04-25T15:26:48.750796+0000","last_active":"2024-04-25T15:26:50.911316+0000","last_peered":"2024-04-25T15:26:50.911316+0000","last_clean":"2024-04-25T15:26:50.911316+0000","last_became_active":"2024-04-25T15:26:48.750501+0000","last_became_peered":"2024-04-25T15:26:48.750501+0000","last_unstale":"2024-04-25T15:26:50.911316+0000","last_undegraded":"2024-04-25T15:26:50.911316+0000","last_fullsized":"2024-04-25T15:26:50.911316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:29.432968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766068+0000","last_change":"2024-04-25T15:26:49.757778+0000","last_active":"2024-04-25T15:26:50.766068+0000","last_peered":"2024-04-25T15:26:50.766068+0000","last_clean":"2024-04-25T15:26:50.766068+0000","last_became_active":"2024-04-25T15:26:49.754768+0000","last_became_peered":"2024-04-25T15:26:49.754768+0000","last_unstale":"2024-04-25T15:26:50.766068+0000","last_undegraded":"2024-04-25T15:26:50.766068+0000","last_fullsized":"2024-04-25T15:26:50.766068+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:10.302228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763617+0000","last_change":"2024-04-25T15:26:48.751930+0000","last_active":"2024-04-25T15:26:50.763617+0000","last_peered":"2024-04-25T15:26:50.763617+0000","last_clean":"2024-04-25T15:26:50.763617+0000","last_became_active":"2024-04-25T15:26:48.751613+0000","last_became_peered":"2024-04-25T15:26:48.751613+0000","last_unstale":"2024-04-25T15:26:50.763617+0000","last_undegraded":"2024-04-25T15:26:50.763617+0000","last_fullsized":"2024-04-25T15:26:50.763617+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:26:56.026433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745898+0000","last_change":"2024-04-25T15:26:49.766766+0000","last_active":"2024-04-25T15:26:50.745898+0000","last_peered":"2024-04-25T15:26:50.745898+0000","last_clean":"2024-04-25T15:26:50.745898+0000","last_became_active":"2024-04-25T15:26:49.764242+0000","last_became_peered":"2024-04-25T15:26:49.764242+0000","last_unstale":"2024-04-25T15:26:50.745898+0000","last_undegraded":"2024-04-25T15:26:50.745898+0000","last_fullsized":"2024-04-25T15:26:50.745898+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:04.867170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766233+0000","last_change":"2024-04-25T15:26:48.740859+0000","last_active":"2024-04-25T15:26:50.766233+0000","last_peered":"2024-04-25T15:26:50.766233+0000","last_clean":"2024-04-25T15:26:50.766233+0000","last_became_active":"2024-04-25T15:26:48.740558+0000","last_became_peered":"2024-04-25T15:26:48.740558+0000","last_unstale":"2024-04-25T15:26:50.766233+0000","last_undegraded":"2024-04-25T15:26:50.766233+0000","last_fullsized":"2024-04-25T15:26:50.766233+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:00.557944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745478+0000","last_change":"2024-04-25T15:26:49.770655+0000","last_active":"2024-04-25T15:26:50.745478+0000","last_peered":"2024-04-25T15:26:50.745478+0000","last_clean":"2024-04-25T15:26:50.745478+0000","last_became_active":"2024-04-25T15:26:49.770344+0000","last_became_peered":"2024-04-25T15:26:49.770344+0000","last_unstale":"2024-04-25T15:26:50.745478+0000","last_undegraded":"2024-04-25T15:26:50.745478+0000","last_fullsized":"2024-04-25T15:26:50.745478+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:47.843945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911418+0000","last_change":"2024-04-25T15:26:48.748965+0000","last_active":"2024-04-25T15:26:50.911418+0000","last_peered":"2024-04-25T15:26:50.911418+0000","last_clean":"2024-04-25T15:26:50.911418+0000","last_became_active":"2024-04-25T15:26:48.748726+0000","last_became_peered":"2024-04-25T15:26:48.748726+0000","last_unstale":"2024-04-25T15:26:50.911418+0000","last_undegraded":"2024-04-25T15:26:50.911418+0000","last_fullsized":"2024-04-25T15:26:50.911418+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:41:35.877664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745869+0000","last_change":"2024-04-25T15:26:49.763896+0000","last_active":"2024-04-25T15:26:50.745869+0000","last_peered":"2024-04-25T15:26:50.745869+0000","last_clean":"2024-04-25T15:26:50.745869+0000","last_became_active":"2024-04-25T15:26:49.763242+0000","last_became_peered":"2024-04-25T15:26:49.763242+0000","last_unstale":"2024-04-25T15:26:50.745869+0000","last_undegraded":"2024-04-25T15:26:50.745869+0000","last_fullsized":"2024-04-25T15:26:50.745869+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:32.546192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745485+0000","last_change":"2024-04-25T15:26:48.740325+0000","last_active":"2024-04-25T15:26:50.745485+0000","last_peered":"2024-04-25T15:26:50.745485+0000","last_clean":"2024-04-25T15:26:50.745485+0000","last_became_active":"2024-04-25T15:26:48.740026+0000","last_became_peered":"2024-04-25T15:26:48.740026+0000","last_unstale":"2024-04-25T15:26:50.745485+0000","last_undegraded":"2024-04-25T15:26:50.745485+0000","last_fullsized":"2024-04-25T15:26:50.745485+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:36:47.091312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766334+0000","last_change":"2024-04-25T15:26:49.767747+0000","last_active":"2024-04-25T15:26:50.766334+0000","last_peered":"2024-04-25T15:26:50.766334+0000","last_clean":"2024-04-25T15:26:50.766334+0000","last_became_active":"2024-04-25T15:26:49.767368+0000","last_became_peered":"2024-04-25T15:26:49.767368+0000","last_unstale":"2024-04-25T15:26:50.766334+0000","last_undegraded":"2024-04-25T15:26:50.766334+0000","last_fullsized":"2024-04-25T15:26:50.766334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:47.843945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766392+0000","last_change":"2024-04-25T15:26:48.740454+0000","last_active":"2024-04-25T15:26:50.766392+0000","last_peered":"2024-04-25T15:26:50.766392+0000","last_clean":"2024-04-25T15:26:50.766392+0000","last_became_active":"2024-04-25T15:26:48.739972+0000","last_became_peered":"2024-04-25T15:26:48.739972+0000","last_unstale":"2024-04-25T15:26:50.766392+0000","last_undegraded":"2024-04-25T15:26:50.766392+0000","last_fullsized":"2024-04-25T15:26:50.766392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:01.465456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911798+0000","last_change":"2024-04-25T15:26:49.765412+0000","last_active":"2024-04-25T15:26:50.911798+0000","last_peered":"2024-04-25T15:26:50.911798+0000","last_clean":"2024-04-25T15:26:50.911798+0000","last_became_active":"2024-04-25T15:26:49.765086+0000","last_became_peered":"2024-04-25T15:26:49.765086+0000","last_unstale":"2024-04-25T15:26:50.911798+0000","last_undegraded":"2024-04-25T15:26:50.911798+0000","last_fullsized":"2024-04-25T15:26:50.911798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:12.831550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746139+0000","last_change":"2024-04-25T15:26:48.745941+0000","last_active":"2024-04-25T15:26:50.746139+0000","last_peered":"2024-04-25T15:26:50.746139+0000","last_clean":"2024-04-25T15:26:50.746139+0000","last_became_active":"2024-04-25T15:26:48.745570+0000","last_became_peered":"2024-04-25T15:26:48.745570+0000","last_unstale":"2024-04-25T15:26:50.746139+0000","last_undegraded":"2024-04-25T15:26:50.746139+0000","last_fullsized":"2024-04-25T15:26:50.746139+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:01.465456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763722+0000","last_change":"2024-04-25T15:26:49.751396+0000","last_active":"2024-04-25T15:26:50.763722+0000","last_peered":"2024-04-25T15:26:50.763722+0000","last_clean":"2024-04-25T15:26:50.763722+0000","last_became_active":"2024-04-25T15:26:49.749466+0000","last_became_peered":"2024-04-25T15:26:49.749466+0000","last_unstale":"2024-04-25T15:26:50.763722+0000","last_undegraded":"2024-04-25T15:26:50.763722+0000","last_fullsized":"2024-04-25T15:26:50.763722+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:08.824618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911490+0000","last_change":"2024-04-25T15:26:48.751237+0000","last_active":"2024-04-25T15:26:50.911490+0000","last_peered":"2024-04-25T15:26:50.911490+0000","last_clean":"2024-04-25T15:26:50.911490+0000","last_became_active":"2024-04-25T15:26:48.750776+0000","last_became_peered":"2024-04-25T15:26:48.750776+0000","last_unstale":"2024-04-25T15:26:50.911490+0000","last_undegraded":"2024-04-25T15:26:50.911490+0000","last_fullsized":"2024-04-25T15:26:50.911490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:51.167458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911336+0000","last_change":"2024-04-25T15:26:49.763644+0000","last_active":"2024-04-25T15:26:50.911336+0000","last_peered":"2024-04-25T15:26:50.911336+0000","last_clean":"2024-04-25T15:26:50.911336+0000","last_became_active":"2024-04-25T15:26:49.763346+0000","last_became_peered":"2024-04-25T15:26:49.763346+0000","last_unstale":"2024-04-25T15:26:50.911336+0000","last_undegraded":"2024-04-25T15:26:50.911336+0000","last_fullsized":"2024-04-25T15:26:50.911336+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:22.027488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.743032+0000","last_change":"2024-04-25T15:26:48.739781+0000","last_active":"2024-04-25T15:26:50.743032+0000","last_peered":"2024-04-25T15:26:50.743032+0000","last_clean":"2024-04-25T15:26:50.743032+0000","last_became_active":"2024-04-25T15:26:48.739062+0000","last_became_peered":"2024-04-25T15:26:48.739062+0000","last_unstale":"2024-04-25T15:26:50.743032+0000","last_undegraded":"2024-04-25T15:26:50.743032+0000","last_fullsized":"2024-04-25T15:26:50.743032+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:10.529331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746589+0000","last_change":"2024-04-25T15:26:49.764866+0000","last_active":"2024-04-25T15:26:50.746589+0000","last_peered":"2024-04-25T15:26:50.746589+0000","last_clean":"2024-04-25T15:26:50.746589+0000","last_became_active":"2024-04-25T15:26:49.764549+0000","last_became_peered":"2024-04-25T15:26:49.764549+0000","last_unstale":"2024-04-25T15:26:50.746589+0000","last_undegraded":"2024-04-25T15:26:50.746589+0000","last_fullsized":"2024-04-25T15:26:50.746589+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:10.302228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763736+0000","last_change":"2024-04-25T15:26:48.749935+0000","last_active":"2024-04-25T15:26:50.763736+0000","last_peered":"2024-04-25T15:26:50.763736+0000","last_clean":"2024-04-25T15:26:50.763736+0000","last_became_active":"2024-04-25T15:26:48.749588+0000","last_became_peered":"2024-04-25T15:26:48.749588+0000","last_unstale":"2024-04-25T15:26:50.763736+0000","last_undegraded":"2024-04-25T15:26:50.763736+0000","last_fullsized":"2024-04-25T15:26:50.763736+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:56.055940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745990+0000","last_change":"2024-04-25T15:26:49.767288+0000","last_active":"2024-04-25T15:26:50.745990+0000","last_peered":"2024-04-25T15:26:50.745990+0000","last_clean":"2024-04-25T15:26:50.745990+0000","last_became_active":"2024-04-25T15:26:49.766759+0000","last_became_peered":"2024-04-25T15:26:49.766759+0000","last_unstale":"2024-04-25T15:26:50.745990+0000","last_undegraded":"2024-04-25T15:26:50.745990+0000","last_fullsized":"2024-04-25T15:26:50.745990+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:13.459028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911233+0000","last_change":"2024-04-25T15:26:48.750106+0000","last_active":"2024-04-25T15:26:50.911233+0000","last_peered":"2024-04-25T15:26:50.911233+0000","last_clean":"2024-04-25T15:26:50.911233+0000","last_became_active":"2024-04-25T15:26:48.749526+0000","last_became_peered":"2024-04-25T15:26:48.749526+0000","last_unstale":"2024-04-25T15:26:50.911233+0000","last_undegraded":"2024-04-25T15:26:50.911233+0000","last_fullsized":"2024-04-25T15:26:50.911233+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:09.296917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763312+0000","last_change":"2024-04-25T15:26:49.750265+0000","last_active":"2024-04-25T15:26:50.763312+0000","last_peered":"2024-04-25T15:26:50.763312+0000","last_clean":"2024-04-25T15:26:50.763312+0000","last_became_active":"2024-04-25T15:26:49.749195+0000","last_became_peered":"2024-04-25T15:26:49.749195+0000","last_unstale":"2024-04-25T15:26:50.763312+0000","last_undegraded":"2024-04-25T15:26:50.763312+0000","last_fullsized":"2024-04-25T15:26:50.763312+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.978844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763435+0000","last_change":"2024-04-25T15:26:48.748609+0000","last_active":"2024-04-25T15:26:50.763435+0000","last_peered":"2024-04-25T15:26:50.763435+0000","last_clean":"2024-04-25T15:26:50.763435+0000","last_became_active":"2024-04-25T15:26:48.748295+0000","last_became_peered":"2024-04-25T15:26:48.748295+0000","last_unstale":"2024-04-25T15:26:50.763435+0000","last_undegraded":"2024-04-25T15:26:50.763435+0000","last_fullsized":"2024-04-25T15:26:50.763435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:14.413012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911749+0000","last_change":"2024-04-25T15:26:49.759956+0000","last_active":"2024-04-25T15:26:50.911749+0000","last_peered":"2024-04-25T15:26:50.911749+0000","last_clean":"2024-04-25T15:26:50.911749+0000","last_became_active":"2024-04-25T15:26:49.759603+0000","last_became_peered":"2024-04-25T15:26:49.759603+0000","last_unstale":"2024-04-25T15:26:50.911749+0000","last_undegraded":"2024-04-25T15:26:50.911749+0000","last_fullsized":"2024-04-25T15:26:50.911749+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:46.641900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":22,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745567+0000","last_change":"2024-04-25T15:26:48.740097+0000","last_active":"2024-04-25T15:26:50.745567+0000","last_peered":"2024-04-25T15:26:50.745567+0000","last_clean":"2024-04-25T15:26:50.745567+0000","last_became_active":"2024-04-25T15:26:48.739737+0000","last_became_peered":"2024-04-25T15:26:48.739737+0000","last_unstale":"2024-04-25T15:26:50.745567+0000","last_undegraded":"2024-04-25T15:26:50.745567+0000","last_fullsized":"2024-04-25T15:26:50.745567+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:33.242255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742818+0000","last_change":"2024-04-25T15:26:49.759946+0000","last_active":"2024-04-25T15:26:50.742818+0000","last_peered":"2024-04-25T15:26:50.742818+0000","last_clean":"2024-04-25T15:26:50.742818+0000","last_became_active":"2024-04-25T15:26:49.759636+0000","last_became_peered":"2024-04-25T15:26:49.759636+0000","last_unstale":"2024-04-25T15:26:50.742818+0000","last_undegraded":"2024-04-25T15:26:50.742818+0000","last_fullsized":"2024-04-25T15:26:50.742818+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:55:30.337482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742696+0000","last_change":"2024-04-25T15:26:48.749171+0000","last_active":"2024-04-25T15:26:50.742696+0000","last_peered":"2024-04-25T15:26:50.742696+0000","last_clean":"2024-04-25T15:26:50.742696+0000","last_became_active":"2024-04-25T15:26:48.748816+0000","last_became_peered":"2024-04-25T15:26:48.748816+0000","last_unstale":"2024-04-25T15:26:50.742696+0000","last_undegraded":"2024-04-25T15:26:50.742696+0000","last_fullsized":"2024-04-25T15:26:50.742696+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:47.294774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746227+0000","last_change":"2024-04-25T15:26:49.762695+0000","last_active":"2024-04-25T15:26:50.746227+0000","last_peered":"2024-04-25T15:26:50.746227+0000","last_clean":"2024-04-25T15:26:50.746227+0000","last_became_active":"2024-04-25T15:26:49.762065+0000","last_became_peered":"2024-04-25T15:26:49.762065+0000","last_unstale":"2024-04-25T15:26:50.746227+0000","last_undegraded":"2024-04-25T15:26:50.746227+0000","last_fullsized":"2024-04-25T15:26:50.746227+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:08.385597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766435+0000","last_change":"2024-04-25T15:26:48.744243+0000","last_active":"2024-04-25T15:26:50.766435+0000","last_peered":"2024-04-25T15:26:50.766435+0000","last_clean":"2024-04-25T15:26:50.766435+0000","last_became_active":"2024-04-25T15:26:48.743887+0000","last_became_peered":"2024-04-25T15:26:48.743887+0000","last_unstale":"2024-04-25T15:26:50.766435+0000","last_undegraded":"2024-04-25T15:26:50.766435+0000","last_fullsized":"2024-04-25T15:26:50.766435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:31.540881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746588+0000","last_change":"2024-04-25T15:26:49.756504+0000","last_active":"2024-04-25T15:26:50.746588+0000","last_peered":"2024-04-25T15:26:50.746588+0000","last_clean":"2024-04-25T15:26:50.746588+0000","last_became_active":"2024-04-25T15:26:49.756097+0000","last_became_peered":"2024-04-25T15:26:49.756097+0000","last_unstale":"2024-04-25T15:26:50.746588+0000","last_undegraded":"2024-04-25T15:26:50.746588+0000","last_fullsized":"2024-04-25T15:26:50.746588+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:33.933493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911610+0000","last_change":"2024-04-25T15:26:48.742966+0000","last_active":"2024-04-25T15:26:50.911610+0000","last_peered":"2024-04-25T15:26:50.911610+0000","last_clean":"2024-04-25T15:26:50.911610+0000","last_became_active":"2024-04-25T15:26:48.742641+0000","last_became_peered":"2024-04-25T15:26:48.742641+0000","last_unstale":"2024-04-25T15:26:50.911610+0000","last_undegraded":"2024-04-25T15:26:50.911610+0000","last_fullsized":"2024-04-25T15:26:50.911610+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:09.296917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746199+0000","last_change":"2024-04-25T15:26:49.762303+0000","last_active":"2024-04-25T15:26:50.746199+0000","last_peered":"2024-04-25T15:26:50.746199+0000","last_clean":"2024-04-25T15:26:50.746199+0000","last_became_active":"2024-04-25T15:26:49.761731+0000","last_became_peered":"2024-04-25T15:26:49.761731+0000","last_unstale":"2024-04-25T15:26:50.746199+0000","last_undegraded":"2024-04-25T15:26:50.746199+0000","last_fullsized":"2024-04-25T15:26:50.746199+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:59:41.132645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763682+0000","last_change":"2024-04-25T15:26:48.750895+0000","last_active":"2024-04-25T15:26:50.763682+0000","last_peered":"2024-04-25T15:26:50.763682+0000","last_clean":"2024-04-25T15:26:50.763682+0000","last_became_active":"2024-04-25T15:26:48.750541+0000","last_became_peered":"2024-04-25T15:26:48.750541+0000","last_unstale":"2024-04-25T15:26:50.763682+0000","last_undegraded":"2024-04-25T15:26:50.763682+0000","last_fullsized":"2024-04-25T15:26:50.763682+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:21.022177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763435+0000","last_change":"2024-04-25T15:26:49.750174+0000","last_active":"2024-04-25T15:26:50.763435+0000","last_peered":"2024-04-25T15:26:50.763435+0000","last_clean":"2024-04-25T15:26:50.763435+0000","last_became_active":"2024-04-25T15:26:49.749618+0000","last_became_peered":"2024-04-25T15:26:49.749618+0000","last_unstale":"2024-04-25T15:26:50.763435+0000","last_undegraded":"2024-04-25T15:26:50.763435+0000","last_fullsized":"2024-04-25T15:26:50.763435+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:23.615205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763955+0000","last_change":"2024-04-25T15:26:48.751868+0000","last_active":"2024-04-25T15:26:50.763955+0000","last_peered":"2024-04-25T15:26:50.763955+0000","last_clean":"2024-04-25T15:26:50.763955+0000","last_became_active":"2024-04-25T15:26:48.751534+0000","last_became_peered":"2024-04-25T15:26:48.751534+0000","last_unstale":"2024-04-25T15:26:50.763955+0000","last_undegraded":"2024-04-25T15:26:50.763955+0000","last_fullsized":"2024-04-25T15:26:50.763955+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:46:03.861859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910889+0000","last_change":"2024-04-25T15:26:49.760874+0000","last_active":"2024-04-25T15:26:50.910889+0000","last_peered":"2024-04-25T15:26:50.910889+0000","last_clean":"2024-04-25T15:26:50.910889+0000","last_became_active":"2024-04-25T15:26:49.760536+0000","last_became_peered":"2024-04-25T15:26:49.760536+0000","last_unstale":"2024-04-25T15:26:50.910889+0000","last_undegraded":"2024-04-25T15:26:50.910889+0000","last_fullsized":"2024-04-25T15:26:50.910889+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:33.933493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911274+0000","last_change":"2024-04-25T15:26:48.745888+0000","last_active":"2024-04-25T15:26:50.911274+0000","last_peered":"2024-04-25T15:26:50.911274+0000","last_clean":"2024-04-25T15:26:50.911274+0000","last_became_active":"2024-04-25T15:26:48.745549+0000","last_became_peered":"2024-04-25T15:26:48.745549+0000","last_unstale":"2024-04-25T15:26:50.911274+0000","last_undegraded":"2024-04-25T15:26:50.911274+0000","last_fullsized":"2024-04-25T15:26:50.911274+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:48.928169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.742710+0000","last_change":"2024-04-25T15:26:49.758878+0000","last_active":"2024-04-25T15:26:50.742710+0000","last_peered":"2024-04-25T15:26:50.742710+0000","last_clean":"2024-04-25T15:26:50.742710+0000","last_became_active":"2024-04-25T15:26:49.758594+0000","last_became_peered":"2024-04-25T15:26:49.758594+0000","last_unstale":"2024-04-25T15:26:50.742710+0000","last_undegraded":"2024-04-25T15:26:50.742710+0000","last_fullsized":"2024-04-25T15:26:50.742710+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:33.254158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745499+0000","last_change":"2024-04-25T15:26:48.743524+0000","last_active":"2024-04-25T15:26:50.745499+0000","last_peered":"2024-04-25T15:26:50.745499+0000","last_clean":"2024-04-25T15:26:50.745499+0000","last_became_active":"2024-04-25T15:26:48.742078+0000","last_became_peered":"2024-04-25T15:26:48.742078+0000","last_unstale":"2024-04-25T15:26:50.745499+0000","last_undegraded":"2024-04-25T15:26:50.745499+0000","last_fullsized":"2024-04-25T15:26:50.745499+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:55.259761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745496+0000","last_change":"2024-04-25T15:26:49.763521+0000","last_active":"2024-04-25T15:26:50.745496+0000","last_peered":"2024-04-25T15:26:50.745496+0000","last_clean":"2024-04-25T15:26:50.745496+0000","last_became_active":"2024-04-25T15:26:49.763071+0000","last_became_peered":"2024-04-25T15:26:49.763071+0000","last_unstale":"2024-04-25T15:26:50.745496+0000","last_undegraded":"2024-04-25T15:26:50.745496+0000","last_fullsized":"2024-04-25T15:26:50.745496+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:15.418323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911084+0000","last_change":"2024-04-25T15:26:48.750972+0000","last_active":"2024-04-25T15:26:50.911084+0000","last_peered":"2024-04-25T15:26:50.911084+0000","last_clean":"2024-04-25T15:26:50.911084+0000","last_became_active":"2024-04-25T15:26:48.750183+0000","last_became_peered":"2024-04-25T15:26:48.750183+0000","last_unstale":"2024-04-25T15:26:50.911084+0000","last_undegraded":"2024-04-25T15:26:50.911084+0000","last_fullsized":"2024-04-25T15:26:50.911084+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:32.928182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746612+0000","last_change":"2024-04-25T15:26:49.764719+0000","last_active":"2024-04-25T15:26:50.746612+0000","last_peered":"2024-04-25T15:26:50.746612+0000","last_clean":"2024-04-25T15:26:50.746612+0000","last_became_active":"2024-04-25T15:26:49.764392+0000","last_became_peered":"2024-04-25T15:26:49.764392+0000","last_unstale":"2024-04-25T15:26:50.746612+0000","last_undegraded":"2024-04-25T15:26:50.746612+0000","last_fullsized":"2024-04-25T15:26:50.746612+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:25.735036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"44'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.765498+0000","last_change":"2024-04-25T15:26:48.744777+0000","last_active":"2024-04-25T15:26:50.765498+0000","last_peered":"2024-04-25T15:26:50.765498+0000","last_clean":"2024-04-25T15:26:50.765498+0000","last_became_active":"2024-04-25T15:26:48.744464+0000","last_became_peered":"2024-04-25T15:26:48.744464+0000","last_unstale":"2024-04-25T15:26:50.765498+0000","last_undegraded":"2024-04-25T15:26:50.765498+0000","last_fullsized":"2024-04-25T15:26:50.765498+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:56:30.322609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745962+0000","last_change":"2024-04-25T15:26:49.771404+0000","last_active":"2024-04-25T15:26:50.745962+0000","last_peered":"2024-04-25T15:26:50.745962+0000","last_clean":"2024-04-25T15:26:50.745962+0000","last_became_active":"2024-04-25T15:26:49.770616+0000","last_became_peered":"2024-04-25T15:26:49.770616+0000","last_unstale":"2024-04-25T15:26:50.745962+0000","last_undegraded":"2024-04-25T15:26:50.745962+0000","last_fullsized":"2024-04-25T15:26:50.745962+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:22.027488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911551+0000","last_change":"2024-04-25T15:26:48.751668+0000","last_active":"2024-04-25T15:26:50.911551+0000","last_peered":"2024-04-25T15:26:50.911551+0000","last_clean":"2024-04-25T15:26:50.911551+0000","last_became_active":"2024-04-25T15:26:48.751344+0000","last_became_peered":"2024-04-25T15:26:48.751344+0000","last_unstale":"2024-04-25T15:26:50.911551+0000","last_undegraded":"2024-04-25T15:26:50.911551+0000","last_fullsized":"2024-04-25T15:26:50.911551+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:47.077528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911435+0000","last_change":"2024-04-25T15:26:49.762698+0000","last_active":"2024-04-25T15:26:50.911435+0000","last_peered":"2024-04-25T15:26:50.911435+0000","last_clean":"2024-04-25T15:26:50.911435+0000","last_became_active":"2024-04-25T15:26:49.762219+0000","last_became_peered":"2024-04-25T15:26:49.762219+0000","last_unstale":"2024-04-25T15:26:50.911435+0000","last_undegraded":"2024-04-25T15:26:50.911435+0000","last_fullsized":"2024-04-25T15:26:50.911435+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:13.459028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910734+0000","last_change":"2024-04-25T15:26:48.739991+0000","last_active":"2024-04-25T15:26:50.910734+0000","last_peered":"2024-04-25T15:26:50.910734+0000","last_clean":"2024-04-25T15:26:50.910734+0000","last_became_active":"2024-04-25T15:26:48.739680+0000","last_became_peered":"2024-04-25T15:26:48.739680+0000","last_unstale":"2024-04-25T15:26:50.910734+0000","last_undegraded":"2024-04-25T15:26:50.910734+0000","last_fullsized":"2024-04-25T15:26:50.910734+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:01.565352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763574+0000","last_change":"2024-04-25T15:26:49.751089+0000","last_active":"2024-04-25T15:26:50.763574+0000","last_peered":"2024-04-25T15:26:50.763574+0000","last_clean":"2024-04-25T15:26:50.763574+0000","last_became_active":"2024-04-25T15:26:49.750778+0000","last_became_peered":"2024-04-25T15:26:49.750778+0000","last_unstale":"2024-04-25T15:26:50.763574+0000","last_undegraded":"2024-04-25T15:26:50.763574+0000","last_fullsized":"2024-04-25T15:26:50.763574+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:08.197424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911663+0000","last_change":"2024-04-25T15:26:48.752374+0000","last_active":"2024-04-25T15:26:50.911663+0000","last_peered":"2024-04-25T15:26:50.911663+0000","last_clean":"2024-04-25T15:26:50.911663+0000","last_became_active":"2024-04-25T15:26:48.751988+0000","last_became_peered":"2024-04-25T15:26:48.751988+0000","last_unstale":"2024-04-25T15:26:50.911663+0000","last_undegraded":"2024-04-25T15:26:50.911663+0000","last_fullsized":"2024-04-25T15:26:50.911663+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:24.729725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.763906+0000","last_change":"2024-04-25T15:26:49.750516+0000","last_active":"2024-04-25T15:26:50.763906+0000","last_peered":"2024-04-25T15:26:50.763906+0000","last_clean":"2024-04-25T15:26:50.763906+0000","last_became_active":"2024-04-25T15:26:49.750125+0000","last_became_peered":"2024-04-25T15:26:49.750125+0000","last_unstale":"2024-04-25T15:26:50.763906+0000","last_undegraded":"2024-04-25T15:26:50.763906+0000","last_fullsized":"2024-04-25T15:26:50.763906+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:16.447499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766351+0000","last_change":"2024-04-25T15:26:48.747835+0000","last_active":"2024-04-25T15:26:50.766351+0000","last_peered":"2024-04-25T15:26:50.766351+0000","last_clean":"2024-04-25T15:26:50.766351+0000","last_became_active":"2024-04-25T15:26:48.747571+0000","last_became_peered":"2024-04-25T15:26:48.747571+0000","last_unstale":"2024-04-25T15:26:50.766351+0000","last_undegraded":"2024-04-25T15:26:50.766351+0000","last_fullsized":"2024-04-25T15:26:50.766351+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:33.242255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911909+0000","last_change":"2024-04-25T15:26:49.763851+0000","last_active":"2024-04-25T15:26:50.911909+0000","last_peered":"2024-04-25T15:26:50.911909+0000","last_clean":"2024-04-25T15:26:50.911909+0000","last_became_active":"2024-04-25T15:26:49.763575+0000","last_became_peered":"2024-04-25T15:26:49.763575+0000","last_unstale":"2024-04-25T15:26:50.911909+0000","last_undegraded":"2024-04-25T15:26:50.911909+0000","last_fullsized":"2024-04-25T15:26:50.911909+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:57.061251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.910715+0000","last_change":"2024-04-25T15:26:48.745812+0000","last_active":"2024-04-25T15:26:50.910715+0000","last_peered":"2024-04-25T15:26:50.910715+0000","last_clean":"2024-04-25T15:26:50.910715+0000","last_became_active":"2024-04-25T15:26:48.744167+0000","last_became_peered":"2024-04-25T15:26:48.744167+0000","last_unstale":"2024-04-25T15:26:50.910715+0000","last_undegraded":"2024-04-25T15:26:50.910715+0000","last_fullsized":"2024-04-25T15:26:50.910715+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:45.473777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.766164+0000","last_change":"2024-04-25T15:26:49.754629+0000","last_active":"2024-04-25T15:26:50.766164+0000","last_peered":"2024-04-25T15:26:50.766164+0000","last_clean":"2024-04-25T15:26:50.766164+0000","last_became_active":"2024-04-25T15:26:49.754273+0000","last_became_peered":"2024-04-25T15:26:49.754273+0000","last_unstale":"2024-04-25T15:26:50.766164+0000","last_undegraded":"2024-04-25T15:26:50.766164+0000","last_fullsized":"2024-04-25T15:26:50.766164+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:30.438279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.745584+0000","last_change":"2024-04-25T15:26:48.745019+0000","last_active":"2024-04-25T15:26:50.745584+0000","last_peered":"2024-04-25T15:26:50.745584+0000","last_clean":"2024-04-25T15:26:50.745584+0000","last_became_active":"2024-04-25T15:26:48.744145+0000","last_became_peered":"2024-04-25T15:26:48.744145+0000","last_unstale":"2024-04-25T15:26:50.745584+0000","last_undegraded":"2024-04-25T15:26:50.745584+0000","last_fullsized":"2024-04-25T15:26:50.745584+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:00.557944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.746144+0000","last_change":"2024-04-25T15:26:49.766393+0000","last_active":"2024-04-25T15:26:50.746144+0000","last_peered":"2024-04-25T15:26:50.746144+0000","last_clean":"2024-04-25T15:26:50.746144+0000","last_became_active":"2024-04-25T15:26:49.764426+0000","last_became_peered":"2024-04-25T15:26:49.764426+0000","last_unstale":"2024-04-25T15:26:50.746144+0000","last_undegraded":"2024-04-25T15:26:50.746144+0000","last_fullsized":"2024-04-25T15:26:50.746144+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:48.724081+0000","last_clean_scrub_stamp":"2024-04-25T15:26:48.724081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:46.641900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.912055+0000","last_change":"2024-04-25T15:26:48.748644+0000","last_active":"2024-04-25T15:26:50.912055+0000","last_peered":"2024-04-25T15:26:50.912055+0000","last_clean":"2024-04-25T15:26:50.912055+0000","last_became_active":"2024-04-25T15:26:48.748280+0000","last_became_peered":"2024-04-25T15:26:48.748280+0000","last_unstale":"2024-04-25T15:26:50.912055+0000","last_undegraded":"2024-04-25T15:26:50.912055+0000","last_fullsized":"2024-04-25T15:26:50.912055+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:26:47.718770+0000","last_clean_scrub_stamp":"2024-04-25T15:26:47.718770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:32.928182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":153,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-25T15:26:50.911105+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:26:50.911105+0000","last_peered":"2024-04-25T15:26:50.911105+0000","last_clean":"2024-04-25T15:26:50.911105+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:26:50.911105+0000","last_undegraded":"2024-04-25T15:26:50.911105+0000","last_fullsized":"2024-04-25T15:26:50.911105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":800266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":210480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":203578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":211420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":210446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":211442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":209878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":800210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:26:54.178 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:26:54.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:26:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.332+0000 7fe7a899d640 1 Processor -- start 2024-04-25T15:26:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.333+0000 7fe7a899d640 1 -- start start 2024-04-25T15:26:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a899d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7a40b8290 0x7fe7a4160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a899d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a415df80 0x7fe7a4160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a899d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7a41615f0 con 0x7fe7a4164b00 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a899d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a4164670 con 0x7fe7a40b8290 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a899d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a4164020 con 0x7fe7a415df80 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a1d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7a40b8290 0x7fe7a4160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7a4164b00 0x7fe7a40b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43294/0 (socket says 172.21.15.67:43294) 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a259c640 1 -- 172.21.15.67:0/35961973 learned_addr learned my addr 172.21.15.67:0/35961973 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a1d9b640 1 --2- 172.21.15.67:0/35961973 >> v2:172.21.15.67:3300/0 conn(0x7fe7a40b8290 0x7fe7a4160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44914/0 (socket says 172.21.15.67:44914) 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.334+0000 7fe7a2d9d640 1 --2- 172.21.15.67:0/35961973 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a415df80 0x7fe7a4160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a1d9b640 1 -- 172.21.15.67:0/35961973 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a415df80 msgr2=0x7fe7a4160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a1d9b640 1 --2- 172.21.15.67:0/35961973 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a415df80 0x7fe7a4160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a1d9b640 1 -- 172.21.15.67:0/35961973 >> v1:172.21.15.67:6789/0 conn(0x7fe7a4164b00 legacy=0x7fe7a40b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a1d9b640 1 -- 172.21.15.67:0/35961973 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a4164300 con 0x7fe7a40b8290 2024-04-25T15:26:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a2d9d640 1 --2- 172.21.15.67:0/35961973 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a415df80 0x7fe7a4160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a159a640 1 -- 172.21.15.67:0/35961973 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951106544 0 0) 0x7fe7a41615f0 con 0x7fe7a4164b00 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a1d9b640 1 --2- 172.21.15.67:0/35961973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a40b8290 0x7fe7a4160890 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0097a0 tx=0x7fe78c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87c200638481457 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a159a640 1 -- 172.21.15.67:0/35961973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c037070 con 0x7fe7a40b8290 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a159a640 1 -- 172.21.15.67:0/35961973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c032440 con 0x7fe7a40b8290 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.335+0000 7fe7a159a640 1 -- 172.21.15.67:0/35961973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c032810 con 0x7fe7a40b8290 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 -- 172.21.15.67:0/35961973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a40b8290 msgr2=0x7fe7a4160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 --2- 172.21.15.67:0/35961973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a40b8290 0x7fe7a4160890 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0097a0 tx=0x7fe78c02a180 comp rx=0 tx=0).stop 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 -- 172.21.15.67:0/35961973 shutdown_connections 2024-04-25T15:26:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 --2- 172.21.15.67:0/35961973 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a415df80 0x7fe7a4160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 --2- 172.21.15.67:0/35961973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a40b8290 0x7fe7a4160890 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 -- 172.21.15.67:0/35961973 >> 172.21.15.67:0/35961973 conn(0x7fe7a40d2bc0 msgr2=0x7fe7a40d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 -- 172.21.15.67:0/35961973 shutdown_connections 2024-04-25T15:26:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.336+0000 7fe7a899d640 1 -- 172.21.15.67:0/35961973 wait complete. 2024-04-25T15:26:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.338+0000 7fe7a899d640 1 Processor -- start 2024-04-25T15:26:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.338+0000 7fe7a899d640 1 -- start start 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a899d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a415df80 0x7fe7a4155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a899d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a4164b00 0x7fe7a415b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a899d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a40b7ad0 0x7fe7a40b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a899d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a40b4a00 con 0x7fe7a415df80 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a899d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a40b4880 con 0x7fe7a4164b00 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a899d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a40b4b80 con 0x7fe7a40b7ad0 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a415df80 0x7fe7a4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a415df80 0x7fe7a4155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44930/0 (socket says 172.21.15.67:44930) 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 -- 172.21.15.67:0/2110713647 learned_addr learned my addr 172.21.15.67:0/2110713647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a1d9b640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a4164b00 0x7fe7a415b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a2d9d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a40b7ad0 0x7fe7a40b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 -- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a40b7ad0 msgr2=0x7fe7a40b7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a40b7ad0 0x7fe7a40b7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 -- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a4164b00 msgr2=0x7fe7a415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a4164b00 0x7fe7a415b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 -- 172.21.15.67:0/2110713647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a4155760 con 0x7fe7a415df80 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a1d9b640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a4164b00 0x7fe7a415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a2d9d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a40b7ad0 0x7fe7a40b7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a259c640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a415df80 0x7fe7a4155220 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00c460 tx=0x7fe79c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe79affd640 1 -- 172.21.15.67:0/2110713647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c018070 con 0x7fe7a415df80 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe79affd640 1 -- 172.21.15.67:0/2110713647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe79c007500 con 0x7fe7a415df80 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe79affd640 1 -- 172.21.15.67:0/2110713647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c004a20 con 0x7fe7a415df80 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.340+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a40b8420 con 0x7fe7a415df80 2024-04-25T15:26:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.341+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a415be80 con 0x7fe7a415df80 2024-04-25T15:26:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.341+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a4050740 con 0x7fe7a415df80 2024-04-25T15:26:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.342+0000 7fe79affd640 1 -- 172.21.15.67:0/2110713647 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe79c01b070 con 0x7fe7a415df80 2024-04-25T15:26:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.343+0000 7fe79affd640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7840744e0 0x7fe7840769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.343+0000 7fe79affd640 1 -- 172.21.15.67:0/2110713647 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fe79c0944a0 con 0x7fe7a415df80 2024-04-25T15:26:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.343+0000 7fe7a1d9b640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7840744e0 0x7fe7840769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.344+0000 7fe7a1d9b640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7840744e0 0x7fe7840769f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe78c034000 tx=0x7fe78c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.347+0000 7fe79affd640 1 -- 172.21.15.67:0/2110713647 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7a40b8420 con 0x7fe7a415df80 2024-04-25T15:26:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.603+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fe7a40d4170 con 0x7fe7a415df80 2024-04-25T15:26:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.606+0000 7fe79affd640 1 -- 172.21.15.67:0/2110713647 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7fe7a4155760 con 0x7fe7a415df80 2024-04-25T15:26:54.607 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:26:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:26:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-25T15:26:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-25T15:26:54.607 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:26:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-25T15:26:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7840744e0 msgr2=0x7fe7840769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7840744e0 0x7fe7840769f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe78c034000 tx=0x7fe78c02e000 comp rx=0 tx=0).stop 2024-04-25T15:26:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a415df80 msgr2=0x7fe7a4155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a415df80 0x7fe7a4155220 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00c460 tx=0x7fe79c00c920 comp rx=0 tx=0).stop 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 shutdown_connections 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7840744e0 0x7fe7840769f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7a40b7ad0 0x7fe7a40b7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7a4164b00 0x7fe7a415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 --2- 172.21.15.67:0/2110713647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7a415df80 0x7fe7a4155220 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 >> 172.21.15.67:0/2110713647 conn(0x7fe7a40d2bc0 msgr2=0x7fe7a40d5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 shutdown_connections 2024-04-25T15:26:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.608+0000 7fe7a899d640 1 -- 172.21.15.67:0/2110713647 wait complete. 2024-04-25T15:26:54.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:26:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.731+0000 7f9b7749f640 1 Processor -- start 2024-04-25T15:26:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.731+0000 7f9b7749f640 1 -- start start 2024-04-25T15:26:54.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b7749f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9b700d0ed0 0x7f9b700d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b7749f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b70160be0 0x7f9b70160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b7749f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b70161530 con 0x7f9b701434d0 2024-04-25T15:26:54.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b7749f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9b701438e0 con 0x7f9b70160be0 2024-04-25T15:26:54.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b7749f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9b7006f3e0 con 0x7f9b700d0ed0 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b75a3c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b70160be0 0x7f9b70160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b75a3c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b70160be0 0x7f9b70160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44932/0 (socket says 172.21.15.67:44932) 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b75a3c640 1 -- 172.21.15.67:0/3790583627 learned_addr learned my addr 172.21.15.67:0/3790583627 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b74a3a640 1 --2- 172.21.15.67:0/3790583627 >> v2:172.21.15.176:3300/0 conn(0x7f9b700d0ed0 0x7f9b700d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b75a3c640 1 -- 172.21.15.67:0/3790583627 >> v2:172.21.15.176:3300/0 conn(0x7f9b700d0ed0 msgr2=0x7f9b700d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b75a3c640 1 --2- 172.21.15.67:0/3790583627 >> v2:172.21.15.176:3300/0 conn(0x7f9b700d0ed0 0x7f9b700d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b75a3c640 1 -- 172.21.15.67:0/3790583627 >> v1:172.21.15.176:6789/0 conn(0x7f9b701434d0 legacy=0x7f9b70070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b75a3c640 1 -- 172.21.15.67:0/3790583627 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7006f6c0 con 0x7f9b70160be0 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.733+0000 7f9b74a3a640 1 --2- 172.21.15.67:0/3790583627 >> v2:172.21.15.176:3300/0 conn(0x7f9b700d0ed0 0x7f9b700d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.734+0000 7f9b75a3c640 1 --2- 172.21.15.67:0/3790583627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b70160be0 0x7f9b70160ff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009e50 tx=0x7f9b6002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2a29ef55a671d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.734+0000 7f9b6ffff640 1 -- 172.21.15.67:0/3790583627 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60036070 con 0x7f9b70160be0 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.734+0000 7f9b6ffff640 1 -- 172.21.15.67:0/3790583627 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b60031600 con 0x7f9b70160be0 2024-04-25T15:26:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.734+0000 7f9b6ffff640 1 -- 172.21.15.67:0/3790583627 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b600319d0 con 0x7f9b70160be0 2024-04-25T15:26:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 -- 172.21.15.67:0/3790583627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b70160be0 msgr2=0x7f9b70160ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3790583627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b70160be0 0x7f9b70160ff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009e50 tx=0x7f9b6002a710 comp rx=0 tx=0).stop 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 -- 172.21.15.67:0/3790583627 shutdown_connections 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3790583627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b70160be0 0x7f9b70160ff0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3790583627 >> v2:172.21.15.176:3300/0 conn(0x7f9b700d0ed0 0x7f9b700d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 -- 172.21.15.67:0/3790583627 >> 172.21.15.67:0/3790583627 conn(0x7f9b700d1ae0 msgr2=0x7f9b700d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 -- 172.21.15.67:0/3790583627 shutdown_connections 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.735+0000 7f9b7749f640 1 -- 172.21.15.67:0/3790583627 wait complete. 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.737+0000 7f9b7749f640 1 Processor -- start 2024-04-25T15:26:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.737+0000 7f9b7749f640 1 -- start start 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7749f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b700d0ed0 0x7f9b7015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7749f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b701434d0 0x7f9b7013fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7749f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b70160be0 0x7f9b70145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7749f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7006f090 con 0x7f9b700d0ed0 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7749f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7006fc80 con 0x7f9b701434d0 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7749f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b700b51a0 con 0x7f9b70160be0 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7523b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b700d0ed0 0x7f9b7015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7523b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b700d0ed0 0x7f9b7015a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44944/0 (socket says 172.21.15.67:44944) 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b7523b640 1 -- 172.21.15.67:0/3215460335 learned_addr learned my addr 172.21.15.67:0/3215460335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b74a3a640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b701434d0 0x7f9b7013fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.738+0000 7f9b75a3c640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b70160be0 0x7f9b70145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7523b640 1 -- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b70160be0 msgr2=0x7f9b70145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7523b640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b70160be0 0x7f9b70145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7523b640 1 -- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b701434d0 msgr2=0x7f9b7013fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7523b640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b701434d0 0x7f9b7013fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7523b640 1 -- 172.21.15.67:0/3215460335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b701403d0 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7523b640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b700d0ed0 0x7f9b7015a860 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800b420 tx=0x7f9b6800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b6dffb640 1 -- 172.21.15.67:0/3215460335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68011830 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b6dffb640 1 -- 172.21.15.67:0/3215460335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b68016070 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b70139030 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b6dffb640 1 -- 172.21.15.67:0/3215460335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68011c10 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.739+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b701395f0 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.740+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b7015d030 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.741+0000 7f9b6dffb640 1 -- 172.21.15.67:0/3215460335 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9b6801b460 con 0x7f9b700d0ed0 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.741+0000 7f9b6dffb640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b5c0744b0 0x7f9b5c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.741+0000 7f9b6dffb640 1 -- 172.21.15.67:0/3215460335 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f9b68094ba0 con 0x7f9b700d0ed0 2024-04-25T15:26:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.741+0000 7f9b74a3a640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b5c0744b0 0x7f9b5c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.742+0000 7f9b74a3a640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b5c0744b0 0x7f9b5c0769c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9b700cf730 tx=0x7f9b64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.746+0000 7f9b6dffb640 1 -- 172.21.15.67:0/3215460335 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b68020020 con 0x7f9b700d0ed0 2024-04-25T15:26:54.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.969+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b70068c30 con 0x7f9b700d0ed0 2024-04-25T15:26:54.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.969+0000 7f9b6dffb640 1 -- 172.21.15.67:0/3215460335 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5187 (secure 0 0 0) 0x7f9b6805f450 con 0x7f9b700d0ed0 2024-04-25T15:26:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:54.971 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1556919488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1556919488},{"type":"v1","addr":"172.21.15.67:6833","nonce":1556919488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4071024412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4071024412},{"type":"v1","addr":"172.21.15.176:6837","nonce":4071024412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3791679818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3791679818},{"type":"v1","addr":"172.21.15.176:6836","nonce":3791679818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:49.120424+0000","modified":"2024-04-25T15:26:52.141230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1882238407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1882238407},{"type":"v1","addr":"172.21.15.67:6835","nonce":1882238407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":3}]} 2024-04-25T15:26:54.971 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b5c0744b0 msgr2=0x7f9b5c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b5c0744b0 0x7f9b5c0769c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9b700cf730 tx=0x7f9b64008040 comp rx=0 tx=0).stop 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b700d0ed0 msgr2=0x7f9b7015a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b700d0ed0 0x7f9b7015a860 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800b420 tx=0x7f9b6800b8e0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 shutdown_connections 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b5c0744b0 0x7f9b5c0769c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b70160be0 0x7f9b70145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b701434d0 0x7f9b7013fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 --2- 172.21.15.67:0/3215460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b700d0ed0 0x7f9b7015a860 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 >> 172.21.15.67:0/3215460335 conn(0x7f9b700d1ae0 msgr2=0x7f9b700b64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 shutdown_connections 2024-04-25T15:26:54.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:54.972+0000 7f9b7749f640 1 -- 172.21.15.67:0/3215460335 wait complete. 2024-04-25T15:26:55.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:26:49.120424+0000', 'modified': '2024-04-25T15:26:52.141230+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5502}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5502': {'gid': 5502, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/1882238407', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 1882238407}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 1882238407}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5502, 'qdb_cluster': [5502]} 2024-04-25T15:26:55.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:26:55.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T15:26:55.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.503+0000 7f1007dd7640 1 Processor -- start 2024-04-25T15:26:55.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.503+0000 7f1007dd7640 1 -- start start 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.504+0000 7f1007dd7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f100006d8e0 0x7f10000d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.504+0000 7f1007dd7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f100006dcf0 0x7f10000d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.504+0000 7f1007dd7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10000d04a0 0x7f10000d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.504+0000 7f1007dd7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f10000b9520 con 0x7f100006dcf0 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.504+0000 7f1007dd7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f10000b4820 con 0x7f100006d8e0 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.504+0000 7f1007dd7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f10000b3c10 con 0x7f10000d04a0 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f100006dcf0 0x7f10000d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f100006dcf0 0x7f10000d3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44954/0 (socket says 172.21.15.67:44954) 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 -- 172.21.15.67:0/646546995 learned_addr learned my addr 172.21.15.67:0/646546995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005b73640 1 --2- 172.21.15.67:0/646546995 >> v2:172.21.15.176:3300/0 conn(0x7f100006d8e0 0x7f10000d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1006374640 1 --2- 172.21.15.67:0/646546995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10000d04a0 0x7f10000d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 -- 172.21.15.67:0/646546995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10000d04a0 msgr2=0x7f10000d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 --2- 172.21.15.67:0/646546995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10000d04a0 0x7f10000d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 -- 172.21.15.67:0/646546995 >> v2:172.21.15.176:3300/0 conn(0x7f100006d8e0 msgr2=0x7f10000d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 --2- 172.21.15.67:0/646546995 >> v2:172.21.15.176:3300/0 conn(0x7f100006d8e0 0x7f10000d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.505+0000 7f1005372640 1 -- 172.21.15.67:0/646546995 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10000b3ef0 con 0x7f100006dcf0 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1005372640 1 --2- 172.21.15.67:0/646546995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100006dcf0 0x7f10000d3f60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f10000b38b0 tx=0x7f0ff002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a76f8e2e7c27431 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1004b71640 1 -- 172.21.15.67:0/646546995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0037070 con 0x7f100006dcf0 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1004b71640 1 -- 172.21.15.67:0/646546995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff0032440 con 0x7f100006dcf0 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1004b71640 1 -- 172.21.15.67:0/646546995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0032810 con 0x7f100006dcf0 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1007dd7640 1 -- 172.21.15.67:0/646546995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100006dcf0 msgr2=0x7f10000d3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1007dd7640 1 --2- 172.21.15.67:0/646546995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100006dcf0 0x7f10000d3f60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f10000b38b0 tx=0x7f0ff002a140 comp rx=0 tx=0).stop 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1007dd7640 1 -- 172.21.15.67:0/646546995 shutdown_connections 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.506+0000 7f1007dd7640 1 --2- 172.21.15.67:0/646546995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10000d04a0 0x7f10000d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.507+0000 7f1007dd7640 1 --2- 172.21.15.67:0/646546995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f100006dcf0 0x7f10000d3f60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.507+0000 7f1007dd7640 1 --2- 172.21.15.67:0/646546995 >> v2:172.21.15.176:3300/0 conn(0x7f100006d8e0 0x7f10000d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.507+0000 7f1007dd7640 1 -- 172.21.15.67:0/646546995 >> 172.21.15.67:0/646546995 conn(0x7f100008d110 msgr2=0x7f100008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.507+0000 7f1007dd7640 1 -- 172.21.15.67:0/646546995 shutdown_connections 2024-04-25T15:26:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.507+0000 7f1007dd7640 1 -- 172.21.15.67:0/646546995 wait complete. 2024-04-25T15:26:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.508+0000 7f1007dd7640 1 Processor -- start 2024-04-25T15:26:55.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.508+0000 7f1007dd7640 1 -- start start 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1007dd7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f100006d8e0 0x7f1000154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1007dd7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f100006dcf0 0x7f10001e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1007dd7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10000d04a0 0x7f1000151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1007dd7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f10000b38e0 con 0x7f10000d04a0 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1007dd7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10000b44b0 con 0x7f100006d8e0 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1007dd7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f10000d0df0 con 0x7f100006dcf0 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10000d04a0 0x7f1000151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10000d04a0 0x7f1000151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44970/0 (socket says 172.21.15.67:44970) 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 -- 172.21.15.67:0/1536739769 learned_addr learned my addr 172.21.15.67:0/1536739769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1005372640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f100006dcf0 0x7f10001e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1005b73640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f100006d8e0 0x7f1000154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 -- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f100006dcf0 msgr2=0x7f10001e6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f100006dcf0 0x7f10001e6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 -- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f100006d8e0 msgr2=0x7f1000154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f100006d8e0 0x7f1000154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 -- 172.21.15.67:0/1536739769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1000154a90 con 0x7f10000d04a0 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1005372640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f100006dcf0 0x7f10001e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1006374640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10000d04a0 0x7f1000151a40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8007ec0 tx=0x7f0ff800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f0ff67fc640 1 -- 172.21.15.67:0/1536739769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8014620 con 0x7f10000d04a0 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f0ff67fc640 1 -- 172.21.15.67:0/1536739769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff8019070 con 0x7f10000d04a0 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10001e7680 con 0x7f10000d04a0 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.510+0000 7f0ff67fc640 1 -- 172.21.15.67:0/1536739769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8014a20 con 0x7f10000d04a0 2024-04-25T15:26:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.511+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001e7c40 con 0x7f10000d04a0 2024-04-25T15:26:55.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.512+0000 7f0ff67fc640 1 -- 172.21.15.67:0/1536739769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0ff8014bc0 con 0x7f10000d04a0 2024-04-25T15:26:55.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.513+0000 7f0ff67fc640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0fe40744e0 0x7f0fe40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.513+0000 7f0ff67fc640 1 -- 172.21.15.67:0/1536739769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f0ff8095af0 con 0x7f10000d04a0 2024-04-25T15:26:55.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.513+0000 7f1005b73640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0fe40744e0 0x7f0fe40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.513+0000 7f1005b73640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0fe40744e0 0x7f0fe40769f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006fd0 tx=0x7f0ffc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.514+0000 7f100ed67740 1 Processor -- start 2024-04-25T15:26:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.514+0000 7f100ed67740 1 -- start start 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f100ed67740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97b1e270 0x55ad97eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 0x55ad97ee6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97ec0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f100ed67740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ad97e24740 con 0x55ad97b1e270 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55ad97dc03a0 con 0x55ad97afd590 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55ad97e20700 con 0x55ad97af53b0 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97b1e270 0x55ad97eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97b1e270 0x55ad97eea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44980/0 (socket says 172.21.15.67:44980) 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 -- 172.21.15.67:0/890512013 learned_addr learned my addr 172.21.15.67:0/890512013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1006374640 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97ec0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 -- 172.21.15.67:0/890512013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 msgr2=0x55ad97ee6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 0x55ad97ee6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 -- 172.21.15.67:0/890512013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 msgr2=0x55ad97ec0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97ec0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.515+0000 7f1005b73640 1 -- 172.21.15.67:0/890512013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad97ddc750 con 0x55ad97b1e270 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.516+0000 7f1005b73640 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97b1e270 0x55ad97eea600 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00fa00 tx=0x7f0ffc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8558815ed5f5238 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.516+0000 7f0fe37fe640 1 -- 172.21.15.67:0/890512013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc01b050 con 0x55ad97b1e270 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.516+0000 7f0fe37fe640 1 -- 172.21.15.67:0/890512013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ffc025070 con 0x55ad97b1e270 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.516+0000 7f0fe37fe640 1 -- 172.21.15.67:0/890512013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc014de0 con 0x55ad97b1e270 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 -- 172.21.15.67:0/890512013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97b1e270 msgr2=0x55ad97eea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97b1e270 0x55ad97eea600 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00fa00 tx=0x7f0ffc012000 comp rx=0 tx=0).stop 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 -- 172.21.15.67:0/890512013 shutdown_connections 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97ec0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 0x55ad97ee6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 --2- 172.21.15.67:0/890512013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97b1e270 0x55ad97eea600 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 -- 172.21.15.67:0/890512013 >> 172.21.15.67:0/890512013 conn(0x55ad97b123c0 msgr2=0x55ad97ebf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 -- 172.21.15.67:0/890512013 shutdown_connections 2024-04-25T15:26:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.517+0000 7f100ed67740 1 -- 172.21.15.67:0/890512013 wait complete. 2024-04-25T15:26:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.518+0000 7f100ed67740 1 -- start start 2024-04-25T15:26:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.518+0000 7f100ed67740 1 Processor -- start 2024-04-25T15:26:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f100ed67740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97ed7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97eeb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97b1e270 0x55ad97ed41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f100ed67740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ad97dabfe0 con 0x55ad97af53b0 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55ad97ee7030 con 0x55ad97afd590 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55ad97eeb910 con 0x55ad97b1e270 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f100ed67740 10 client.0 _ll_register_callbacks cb 0x55ad97dc2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f1005b73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97ed7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f1005b73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97ed7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44988/0 (socket says 172.21.15.67:44988) 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f1005b73640 1 -- 172.21.15.67:0/3749092626 learned_addr learned my addr 172.21.15.67:0/3749092626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f1005372640 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97eeb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.519+0000 7f1006374640 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97b1e270 0x55ad97ed41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f1005b73640 1 -- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97b1e270 msgr2=0x55ad97ed41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f1005b73640 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97b1e270 0x55ad97ed41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f1005b73640 1 -- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 msgr2=0x55ad97eeb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f1005b73640 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97eeb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f1005b73640 1 -- 172.21.15.67:0/3749092626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad97b362d0 con 0x55ad97af53b0 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f1005b73640 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97ed7c60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x55ad97efc180 tx=0x7f0ffc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63031ca736c5fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f0fe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/3749092626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc037060 con 0x55ad97af53b0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/3749092626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f0ffc020a80 con 0x55ad97af53b0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ffc025070 con 0x55ad97af53b0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/3749092626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0ffc01b020 con 0x55ad97af53b0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f100ed67740 10 client.5649 fetch_fsmap learned FSMap version 25 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55ad97c006b0 con 0x55ad97af53b0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/3749092626 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ffc025070 con 0x55ad97af53b0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/3749092626 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f0ffc020c80 con 0x55ad97af53b0 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f100ed67740 10 client.5649 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.520+0000 7f100ed67740 10 client.5649 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.521+0000 7f100ed67740 20 client.5649 populate_metadata read hostname 'smithi067' 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.521+0000 7f100ed67740 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97adce40 0x55ad97f17af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.521+0000 7f100ed67740 4 client.5649 mds_command: new command op to 5502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.521+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 --> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad97e32350 con 0x55ad97adce40 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.521+0000 7f1005372640 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97adce40 0x55ad97f17af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.521+0000 7f0fe0ff9640 10 client.5649 ms_handle_connect on v2:172.21.15.67:6834/1882238407 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.521+0000 7f1005372640 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97adce40 0x55ad97f17af0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=269f2dc41e6f3192 server_cookie=3cc86e38a257c920 in_seq=0 out_seq=0 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/3749092626 <== mds.0 v2:172.21.15.67:6834/1882238407 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ad97e32350 con 0x55ad97adce40 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f0fe0ff9640 10 client.5649 handle_command_reply: tid=0 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 client.5649 shutdown 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 msgr2=0x55ad97ed7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97ed7c60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x55ad97efc180 tx=0x7f0ffc004730 comp rx=0 tx=0).stop 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 shutdown_connections 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97adce40 0x55ad97f17af0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97b1e270 0x55ad97ed41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97afd590 0x55ad97eeb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f0fe0ff9640 0 client.5649 ms_handle_reset on v2:172.21.15.67:6834/1882238407 2024-04-25T15:26:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 --2- 172.21.15.67:0/3749092626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97ed7c60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 >> 172.21.15.67:0/3749092626 conn(0x55ad97b123c0 msgr2=0x55ad97ec0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 shutdown_connections 2024-04-25T15:26:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 1 -- 172.21.15.67:0/3749092626 wait complete. 2024-04-25T15:26:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.522+0000 7f100ed67740 20 client.5649 trim_cache size 0 max 16384 2024-04-25T15:26:55.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.541+0000 7f100ed67740 1 Processor -- start 2024-04-25T15:26:55.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.541+0000 7f100ed67740 1 -- start start 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f100ed67740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97f07330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97afd590 0x55ad97eed260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97f28ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f100ed67740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1000201d80 con 0x55ad97af53b0 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10002010e0 con 0x55ad97b1e270 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1000201730 con 0x55ad97afd590 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97f07330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97f07330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45004/0 (socket says 172.21.15.67:45004) 2024-04-25T15:26:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 -- 172.21.15.67:0/1920316490 learned_addr learned my addr 172.21.15.67:0/1920316490 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005372640 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97afd590 0x55ad97eed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1006374640 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97f28ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 -- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97afd590 msgr2=0x55ad97eed260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97afd590 0x55ad97eed260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 -- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 msgr2=0x55ad97f28ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97f28ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 -- 172.21.15.67:0/1920316490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad97f05ed0 con 0x55ad97af53b0 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f1005b73640 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97f07330 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc005b30 tx=0x7f0ffc02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fba3a8877ca2dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.543+0000 7f0fe2ffd640 1 -- 172.21.15.67:0/1920316490 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc020c80 con 0x55ad97af53b0 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f0fe2ffd640 1 -- 172.21.15.67:0/1920316490 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ffc025070 con 0x55ad97af53b0 2024-04-25T15:26:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f0fe2ffd640 1 -- 172.21.15.67:0/1920316490 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc041680 con 0x55ad97af53b0 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 -- 172.21.15.67:0/1920316490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 msgr2=0x55ad97f07330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97f07330 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc005b30 tx=0x7f0ffc02c040 comp rx=0 tx=0).stop 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 -- 172.21.15.67:0/1920316490 shutdown_connections 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97f28ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97afd590 0x55ad97eed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 --2- 172.21.15.67:0/1920316490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97af53b0 0x55ad97f07330 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 -- 172.21.15.67:0/1920316490 >> 172.21.15.67:0/1920316490 conn(0x55ad97b123c0 msgr2=0x55ad97ed3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 -- 172.21.15.67:0/1920316490 shutdown_connections 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.544+0000 7f100ed67740 1 -- 172.21.15.67:0/1920316490 wait complete. 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.545+0000 7f100ed67740 1 -- start start 2024-04-25T15:26:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.545+0000 7f100ed67740 1 Processor -- start 2024-04-25T15:26:55.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 0x55ad97f7f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97afd590 0x55ad97f82b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97ee5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ad97f050a0 con 0x55ad97afd590 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55ad97ed4e30 con 0x55ad97b1e270 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55ad97f05630 con 0x55ad97af53b0 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 10 client.0 _ll_register_callbacks cb 0x7f0ffc00f8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97afd590 0x55ad97f82b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97afd590 0x55ad97f82b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45016/0 (socket says 172.21.15.67:45016) 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 -- 172.21.15.67:0/1356511005 learned_addr learned my addr 172.21.15.67:0/1356511005 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005b73640 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 0x55ad97f7f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1006374640 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97ee5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 -- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 msgr2=0x55ad97f7f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 0x55ad97f7f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 -- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 msgr2=0x55ad97ee5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97ee5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 -- 172.21.15.67:0/1356511005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad97eebe10 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f1005372640 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97afd590 0x55ad97f82b60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002a180 tx=0x7f0ff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60686761cb1671d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f0fe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/1356511005 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0037040 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/1356511005 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f0ff0032660 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.547+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ffc025070 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/1356511005 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0ff004f050 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 10 client.5655 fetch_fsmap learned FSMap version 25 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55ad97eb9630 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/1356511005 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ffc025070 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/1356511005 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f0ff0052020 con 0x55ad97afd590 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 10 client.5655 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T15:26:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 10 client.5655 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 20 client.5655 populate_metadata read hostname 'smithi067' 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97f863a0 0x55ad97ee4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 4 client.5655 mds_command: new command op to 5502 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 --> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] -- command(tid 0: {"prefix": "status"}) -- 0x55ad97e2c1c0 con 0x55ad97f863a0 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f1005b73640 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97f863a0 0x55ad97ee4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.548+0000 7f0fe0ff9640 10 client.5655 ms_handle_connect on v2:172.21.15.67:6834/1882238407 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f1005b73640 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97f863a0 0x55ad97ee4450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aded5971bb63aee7 server_cookie=e957f270747f6782 in_seq=0 out_seq=0 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f0fe0ff9640 1 -- 172.21.15.67:0/1356511005 <== mds.0 v2:172.21.15.67:6834/1882238407 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ad97e2c1c0 con 0x55ad97f863a0 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f0fe0ff9640 10 client.5655 handle_command_reply: tid=0 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 client.5655 shutdown 2024-04-25T15:26:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97afd590 msgr2=0x55ad97f82b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97afd590 0x55ad97f82b60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002a180 tx=0x7f0ff00046c0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 shutdown_connections 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:6834/1882238407,v1:172.21.15.67:6835/1882238407] conn(0x55ad97f863a0 0x55ad97ee4450 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55ad97b1e270 0x55ad97ee5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad97afd590 0x55ad97f82b60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f0fe0ff9640 0 client.5655 ms_handle_reset on v2:172.21.15.67:6834/1882238407 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 --2- 172.21.15.67:0/1356511005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55ad97af53b0 0x55ad97f7f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 >> 172.21.15.67:0/1356511005 conn(0x55ad97b123c0 msgr2=0x55ad97ed3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 shutdown_connections 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 1 -- 172.21.15.67:0/1356511005 wait complete. 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.549+0000 7f100ed67740 20 client.5655 trim_cache size 0 max 16384 2024-04-25T15:26:55.556 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b43d0acb-252a-4f7a-bf14-34d789b94b95", 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5502, 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 5.7983924650000001, 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 44, 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 44, 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 10.780118680999999 2024-04-25T15:26:55.557 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:26:55.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0fe40744e0 msgr2=0x7f0fe40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0fe40744e0 0x7f0fe40769f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc006fd0 tx=0x7f0ffc008040 comp rx=0 tx=0).stop 2024-04-25T15:26:55.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10000d04a0 msgr2=0x7f1000151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10000d04a0 0x7f1000151a40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8007ec0 tx=0x7f0ff800e4b0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 shutdown_connections 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0fe40744e0 0x7f0fe40769f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10000d04a0 0x7f1000151a40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f100006dcf0 0x7f10001e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 --2- 172.21.15.67:0/1536739769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f100006d8e0 0x7f1000154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 >> 172.21.15.67:0/1536739769 conn(0x7f100008d110 msgr2=0x7f10000b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 shutdown_connections 2024-04-25T15:26:55.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.550+0000 7f1007dd7640 1 -- 172.21.15.67:0/1536739769 wait complete. 2024-04-25T15:26:55.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:26:55.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.672+0000 7fe457b5e640 1 Processor -- start 2024-04-25T15:26:55.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.673+0000 7fe457b5e640 1 -- start start 2024-04-25T15:26:55.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe457b5e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4500b21a0 0x7fe4500ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe457b5e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4500b7840 0x7fe4500b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe457b5e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4500b8190 con 0x7fe4500cfd40 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe457b5e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe4500d27e0 con 0x7fe4500b7840 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe457b5e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe4500d1c10 con 0x7fe4500b21a0 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe4560fb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4500b7840 0x7fe4500b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe4560fb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4500b7840 0x7fe4500b7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45026/0 (socket says 172.21.15.67:45026) 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe4560fb640 1 -- 172.21.15.67:0/3369592924 learned_addr learned my addr 172.21.15.67:0/3369592924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.674+0000 7fe4550f9640 1 --2- 172.21.15.67:0/3369592924 >> v2:172.21.15.176:3300/0 conn(0x7fe4500b21a0 0x7fe4500ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4560fb640 1 -- 172.21.15.67:0/3369592924 >> v2:172.21.15.176:3300/0 conn(0x7fe4500b21a0 msgr2=0x7fe4500ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4560fb640 1 --2- 172.21.15.67:0/3369592924 >> v2:172.21.15.176:3300/0 conn(0x7fe4500b21a0 0x7fe4500ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4560fb640 1 -- 172.21.15.67:0/3369592924 >> v1:172.21.15.176:6789/0 conn(0x7fe4500cfd40 legacy=0x7fe4500b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4560fb640 1 -- 172.21.15.67:0/3369592924 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4500b25b0 con 0x7fe4500b7840 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4560fb640 1 --2- 172.21.15.67:0/3369592924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4500b7c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe450072bf0 tx=0x7fe44c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71f8b17334e5663 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4548f8640 1 -- 172.21.15.67:0/3369592924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c036070 con 0x7fe4500b7840 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4548f8640 1 -- 172.21.15.67:0/3369592924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44c031600 con 0x7fe4500b7840 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.675+0000 7fe4548f8640 1 -- 172.21.15.67:0/3369592924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0319d0 con 0x7fe4500b7840 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 -- 172.21.15.67:0/3369592924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 msgr2=0x7fe4500b7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 --2- 172.21.15.67:0/3369592924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4500b7c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe450072bf0 tx=0x7fe44c02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 -- 172.21.15.67:0/3369592924 shutdown_connections 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 --2- 172.21.15.67:0/3369592924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4500b7c50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 --2- 172.21.15.67:0/3369592924 >> v2:172.21.15.176:3300/0 conn(0x7fe4500b21a0 0x7fe4500ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 -- 172.21.15.67:0/3369592924 >> 172.21.15.67:0/3369592924 conn(0x7fe45009b7d0 msgr2=0x7fe45009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 -- 172.21.15.67:0/3369592924 shutdown_connections 2024-04-25T15:26:55.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.676+0000 7fe457b5e640 1 -- 172.21.15.67:0/3369592924 wait complete. 2024-04-25T15:26:55.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.678+0000 7fe457b5e640 1 Processor -- start 2024-04-25T15:26:55.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.678+0000 7fe457b5e640 1 -- start start 2024-04-25T15:26:55.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe457b5e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4500b21a0 0x7fe45015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe457b5e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4501635d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe457b5e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4500cfd40 0x7fe4500b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe457b5e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe4500d2470 con 0x7fe4500b7840 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe457b5e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe4500d2190 con 0x7fe4500cfd40 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe457b5e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe4500ce650 con 0x7fe4500b21a0 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe4550f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4501635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe4550f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4501635d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45034/0 (socket says 172.21.15.67:45034) 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.679+0000 7fe4550f9640 1 -- 172.21.15.67:0/1585504666 learned_addr learned my addr 172.21.15.67:0/1585504666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4560fb640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4500cfd40 0x7fe4500b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4558fa640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4500b21a0 0x7fe45015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4550f9640 1 -- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4500b21a0 msgr2=0x7fe45015d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4550f9640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4500b21a0 0x7fe45015d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4550f9640 1 -- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4500cfd40 msgr2=0x7fe4500b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4550f9640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4500cfd40 0x7fe4500b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4550f9640 1 -- 172.21.15.67:0/1585504666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4501f70f0 con 0x7fe4500b7840 2024-04-25T15:26:55.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4550f9640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4501635d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e8a0 tx=0x7fe44000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4467fc640 1 -- 172.21.15.67:0/1585504666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440017070 con 0x7fe4500b7840 2024-04-25T15:26:55.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4467fc640 1 -- 172.21.15.67:0/1585504666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44000c940 con 0x7fe4500b7840 2024-04-25T15:26:55.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe4467fc640 1 -- 172.21.15.67:0/1585504666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44000cd30 con 0x7fe4500b7840 2024-04-25T15:26:55.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4501f73d0 con 0x7fe4500b7840 2024-04-25T15:26:55.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.680+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4501f7990 con 0x7fe4500b7840 2024-04-25T15:26:55.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.681+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4501f6e60 con 0x7fe4500b7840 2024-04-25T15:26:55.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.682+0000 7fe4467fc640 1 -- 172.21.15.67:0/1585504666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe44001a070 con 0x7fe4500b7840 2024-04-25T15:26:55.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.682+0000 7fe4467fc640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4340744b0 0x7fe4340769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:55.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.682+0000 7fe4467fc640 1 -- 172.21.15.67:0/1585504666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fe440095410 con 0x7fe4500b7840 2024-04-25T15:26:55.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.683+0000 7fe4558fa640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4340744b0 0x7fe4340769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:55.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.683+0000 7fe4558fa640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4340744b0 0x7fe4340769c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe448001500 tx=0x7fe448009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:55.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.687+0000 7fe4467fc640 1 -- 172.21.15.67:0/1585504666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe44005fc30 con 0x7fe4500b7840 2024-04-25T15:26:55.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.916+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe450050740 con 0x7fe4500b7840 2024-04-25T15:26:55.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.916+0000 7fe4467fc640 1 -- 172.21.15.67:0/1585504666 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7fe44005f530 con 0x7fe4500b7840 2024-04-25T15:26:55.917 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:26:55.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4340744b0 msgr2=0x7fe4340769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4340744b0 0x7fe4340769c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe448001500 tx=0x7fe448009040 comp rx=0 tx=0).stop 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 msgr2=0x7fe4501635d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4501635d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe44000e8a0 tx=0x7fe44000ed60 comp rx=0 tx=0).stop 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 shutdown_connections 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe4340744b0 0x7fe4340769c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4500cfd40 0x7fe4500b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4500b7840 0x7fe4501635d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 --2- 172.21.15.67:0/1585504666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4500b21a0 0x7fe45015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.918+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 >> 172.21.15.67:0/1585504666 conn(0x7fe45009b7d0 msgr2=0x7fe450070340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.919+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 shutdown_connections 2024-04-25T15:26:55.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:55.919+0000 7fe457b5e640 1 -- 172.21.15.67:0/1585504666 wait complete. 2024-04-25T15:26:55.930 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:26:55.930 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:26:55.931 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:26:55.931 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:26:55.931 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:55.931 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:26:55.931 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:26:55.931 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:26:55.931 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:26:55.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:55.931 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:55.947 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42683sec preferred_lft 42683sec 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:55.948 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7482:42ff:fe30:5ffc/64 scope link 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::885:bff:fe04:f6cb/64 scope link 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ca:af:51:1f:d5:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8af:51ff:fe1f:d5b0/64 scope link 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:55.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:55.949 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:26:56.011 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:26:56.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:56.012 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:26:56.068 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:26:56.069 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:26:56.069 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:26:56.069 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:26:56.069 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:26:56.069 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:26:56.069 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:26:56.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:26:56.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:26:56.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:26:56.070 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:26:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:26:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:56.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:56.195 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:26:56.210 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:26:56.210 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:26:56.210 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:26:56.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:56.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:26:56.274 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:56.274 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:56.330 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:26:56.330 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:56.330 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:56.385 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T15:26:56.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:56.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:56.478 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:56.479 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:56.487 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:56.487 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:56.487 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:26:56.487 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:26:56.487 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:26:56.488 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T15:26:56.488 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:26:56.488 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:26:56.488 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:26:56.488 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:26:56.490 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:26:56.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:56.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:26:56.514 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:26:56.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:26:56.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:56.616 INFO:teuthology.orchestra.run.smithi067.stdout:client5141 2024-04-25T15:26:56.617 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:56.617 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:56.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:26:56.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.784+0000 7f301f3d7640 1 Processor -- start 2024-04-25T15:26:56.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.785+0000 7f301f3d7640 1 -- start start 2024-04-25T15:26:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f301f3d7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f30180b0a80 0x7f30180d16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f301f3d7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30180ce7e0 con 0x7f30180b0670 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f301f3d7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30180b0e90 con 0x7f30180d1c20 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f301f3d7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f30180d2030 con 0x7f30180b0a80 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f301d173640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f30180b0670 0x7f30180cf520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43298/0 (socket says 172.21.15.67:43298) 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f301d173640 1 -- 172.21.15.67:0/3606513 learned_addr learned my addr 172.21.15.67:0/3606513 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f301c972640 1 --2- 172.21.15.67:0/3606513 >> v2:172.21.15.176:3300/0 conn(0x7f30180b0a80 0x7f30180d16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892890946 0 0) 0x7f3010001110 con 0x7f30180b0670 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.786+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3000003520 con 0x7f30180b0670 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3694582804 0 0) 0x7f3000003520 con 0x7f30180b0670 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 >> v2:172.21.15.176:3300/0 conn(0x7f30180b0a80 msgr2=0x7f30180d16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:56.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 --2- 172.21.15.67:0/3606513 >> v2:172.21.15.176:3300/0 conn(0x7f30180b0a80 0x7f30180d16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 >> v1:172.21.15.176:6789/0 conn(0x7f30180d1c20 legacy=0x7f30180d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30180cea60 con 0x7f30180b0670 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f3010001840 con 0x7f30180b0670 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 >> v1:172.21.15.67:6789/0 conn(0x7f30180b0670 legacy=0x7f30180cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30000039b0 0x7f3000003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3000004800 0x7f3000004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30000051c0 0x7f300000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30180d2030 con 0x7f30000039b0 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f300000c160 con 0x7f3000004800 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f300000c360 con 0x7f30000051c0 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301c972640 1 --2- 172.21.15.67:0/3606513 >> v2:172.21.15.176:3300/0 conn(0x7f30180b0a80 0x7f30180d16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d173640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30000039b0 0x7f3000003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d974640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30000051c0 0x7f300000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301c972640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3000004800 0x7f3000004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d173640 1 -- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30000051c0 msgr2=0x7f300000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d173640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30000051c0 0x7f300000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d173640 1 -- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3000004800 msgr2=0x7f3000004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d173640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3000004800 0x7f3000004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d173640 1 -- 172.21.15.67:0/3606513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3000003520 con 0x7f30000039b0 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.787+0000 7f301d974640 1 -- 172.21.15.67:0/3606513 reap_dead start 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.788+0000 7f301d173640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30000039b0 0x7f3000003e20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f301808d0e0 tx=0x7f3000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45703d0d81b0dc3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.788+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010036070 con 0x7f30000039b0 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.788+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f301002d070 con 0x7f30000039b0 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.788+0000 7f3017fff640 1 -- 172.21.15.67:0/3606513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301000c0e0 con 0x7f30000039b0 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.789+0000 7f301f3d7640 1 -- 172.21.15.67:0/3606513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30000039b0 msgr2=0x7f3000003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.789+0000 7f301f3d7640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30000039b0 0x7f3000003e20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f301808d0e0 tx=0x7f3000005830 comp rx=0 tx=0).stop 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.789+0000 7f301f3d7640 1 -- 172.21.15.67:0/3606513 shutdown_connections 2024-04-25T15:26:56.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.789+0000 7f301f3d7640 1 --2- 172.21.15.67:0/3606513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30000039b0 0x7f3000003e20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:56.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.789+0000 7f301f3d7640 1 -- 172.21.15.67:0/3606513 >> 172.21.15.67:0/3606513 conn(0x7f30180d3a80 msgr2=0x7f30180d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:56.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.789+0000 7f301f3d7640 1 -- 172.21.15.67:0/3606513 shutdown_connections 2024-04-25T15:26:56.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.789+0000 7f301f3d7640 1 -- 172.21.15.67:0/3606513 wait complete. 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.791+0000 7f301f3d7640 1 Processor -- start 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.791+0000 7f301f3d7640 1 -- start start 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301f3d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30000039b0 0x7f3018159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301f3d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30180c9920 0x7f30181f9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301f3d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30181fa440 0x7f301806dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301f3d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30180ced20 con 0x7f30180c9920 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301f3d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f301815f650 con 0x7f30000039b0 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301f3d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f30180cf000 con 0x7f30181fa440 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30180c9920 0x7f30181f9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30180c9920 0x7f30181f9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45062/0 (socket says 172.21.15.67:45062) 2024-04-25T15:26:56.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 -- 172.21.15.67:0/1897803556 learned_addr learned my addr 172.21.15.67:0/1897803556 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301d173640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30000039b0 0x7f3018159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301d974640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30181fa440 0x7f301806dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 -- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30181fa440 msgr2=0x7f301806dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30181fa440 0x7f301806dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 -- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30000039b0 msgr2=0x7f3018159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30000039b0 0x7f3018159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301c972640 1 -- 172.21.15.67:0/1897803556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301815a3c0 con 0x7f30180c9920 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301d173640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30000039b0 0x7f3018159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.792+0000 7f301d974640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30181fa440 0x7f301806dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.793+0000 7f301c972640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30180c9920 0x7f30181f9f00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f300c011900 tx=0x7f300c011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.793+0000 7f3015ffb640 1 -- 172.21.15.67:0/1897803556 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c00fbd0 con 0x7f30180c9920 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.793+0000 7f3015ffb640 1 -- 172.21.15.67:0/1897803556 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f300c00fd70 con 0x7f30180c9920 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.793+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30181fa850 con 0x7f30180c9920 2024-04-25T15:26:56.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.793+0000 7f3015ffb640 1 -- 172.21.15.67:0/1897803556 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c015660 con 0x7f30180c9920 2024-04-25T15:26:56.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.793+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30180caed0 con 0x7f30180c9920 2024-04-25T15:26:56.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.794+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30180cb7c0 con 0x7f30180c9920 2024-04-25T15:26:56.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.795+0000 7f3015ffb640 1 -- 172.21.15.67:0/1897803556 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f300c015800 con 0x7f30180c9920 2024-04-25T15:26:56.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.795+0000 7f3015ffb640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30040744e0 0x7f30040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:56.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.796+0000 7f3015ffb640 1 -- 172.21.15.67:0/1897803556 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f300c095ca0 con 0x7f30180c9920 2024-04-25T15:26:56.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.796+0000 7f301d173640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30040744e0 0x7f30040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:56.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.796+0000 7f301d173640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30040744e0 0x7f30040769f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f30180b6570 tx=0x7f301002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:56.800+0000 7f3015ffb640 1 -- 172.21.15.67:0/1897803556 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f300c019030 con 0x7f30180c9920 2024-04-25T15:26:57.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.026+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3018050740 con 0x7f30180c9920 2024-04-25T15:26:57.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.026+0000 7f3015ffb640 1 -- 172.21.15.67:0/1897803556 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f300c0604c0 con 0x7f30180c9920 2024-04-25T15:26:57.027 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:26:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30040744e0 msgr2=0x7f30040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30040744e0 0x7f30040769f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f30180b6570 tx=0x7f301002e000 comp rx=0 tx=0).stop 2024-04-25T15:26:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30180c9920 msgr2=0x7f30181f9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30180c9920 0x7f30181f9f00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f300c011900 tx=0x7f300c011dc0 comp rx=0 tx=0).stop 2024-04-25T15:26:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 shutdown_connections 2024-04-25T15:26:57.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30040744e0 0x7f30040769f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30181fa440 0x7f301806dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30180c9920 0x7f30181f9f00 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 --2- 172.21.15.67:0/1897803556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30000039b0 0x7f3018159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 >> 172.21.15.67:0/1897803556 conn(0x7f30180d3a80 msgr2=0x7f30180cf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 shutdown_connections 2024-04-25T15:26:57.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:57.028+0000 7f301f3d7640 1 -- 172.21.15.67:0/1897803556 wait complete. 2024-04-25T15:26:57.040 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:26:57.041 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:26:57.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.041 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:57.061 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42671sec preferred_lft 42671sec 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:26:57.062 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::248e:2bff:fe69:7f63/64 scope link 2024-04-25T15:26:57.063 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:26:57.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.064 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:26:57.128 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:26:57.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.130 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:26:57.188 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:26:57.188 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:26:57.188 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:26:57.188 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:26:57.188 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:26:57.189 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:26:57.189 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:26:57.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:26:57.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:26:57.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:26:57.189 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:26:57.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:26:57.312 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:57.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.318 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:26:57.334 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:26:57.334 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:26:57.335 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:26:57.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.335 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:26:57.398 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:57.398 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:57.453 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:26:57.453 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:26:57.453 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:57.507 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T15:26:57.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:57.600 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:57.600 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:57.614 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:26:57.614 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:26:57.614 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:26:57.614 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:26:57.614 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:26:57.615 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:26:57.615 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:26:57.615 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:26:57.616 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:26:57.616 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:26:57.617 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:26:57.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:26:57.641 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:26:57.730 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:26:57.734 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:26:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:26:57.737 INFO:teuthology.orchestra.run.smithi176.stdout:client5670 2024-04-25T15:26:57.737 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:26:57.737 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:57.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.753 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-25T15:26:57.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.777 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10A ./subdir) 2024-04-25T15:26:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:57.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.837 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1y00Ki ./subdir) 2024-04-25T15:26:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:57.894 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:57.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.894 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v af00 ./subdir) 2024-04-25T15:26:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:57.951 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:57.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:57.952 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v G ./subdir) 2024-04-25T15:26:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:58.008 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:58.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:58.009 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v '' ./subdir) 2024-04-25T15:26:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:58.065 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:58.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:58.066 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v ' ' ./subdir) 2024-04-25T15:26:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:58.122 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:58.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:58.123 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v -1t ./subdir) 2024-04-25T15:26:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:58.182 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:58.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:26:58.183 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v -1 ./subdir) 2024-04-25T15:26:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:26:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:setfattr: ./subdir: Invalid argument 2024-04-25T15:26:58.242 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:26:58.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:26:58.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:26:58.359 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:26:58.359 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:26:58.360 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:58.360 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:58.380 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:26:58.380 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:26:58.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:26:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.588+0000 7f9f581ce640 1 Processor -- start 2024-04-25T15:26:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.588+0000 7f9f581ce640 1 -- start start 2024-04-25T15:26:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f581ce640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9f500d4130 0x7f9f500d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f581ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006dcf0 0x7f9f500d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f581ce640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f500b3c10 con 0x7f9f5006d8e0 2024-04-25T15:26:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f581ce640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9f500b9520 con 0x7f9f500d4130 2024-04-25T15:26:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f581ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9f500b4820 con 0x7f9f5006dcf0 2024-04-25T15:26:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f5676b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006dcf0 0x7f9f500d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f55769640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9f500d4130 0x7f9f500d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f55f6a640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9f5006d8e0 0x7f9f500d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:55480/0 (socket says 172.21.15.67:55480) 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f55769640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9f500d4130 0x7f9f500d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:46728/0 (socket says 172.21.15.67:46728) 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f5676b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006dcf0 0x7f9f500d22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52508/0 (socket says 172.21.15.67:52508) 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f55f6a640 1 -- 172.21.15.67:0/3042348677 learned_addr learned my addr 172.21.15.67:0/3042348677 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f5676b640 1 -- 172.21.15.67:0/3042348677 >> v2:172.21.15.176:3300/0 conn(0x7f9f500d4130 msgr2=0x7f9f500d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f5676b640 1 --2- 172.21.15.67:0/3042348677 >> v2:172.21.15.176:3300/0 conn(0x7f9f500d4130 0x7f9f500d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f5676b640 1 -- 172.21.15.67:0/3042348677 >> v1:172.21.15.176:6789/0 conn(0x7f9f5006d8e0 legacy=0x7f9f500d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f5676b640 1 -- 172.21.15.67:0/3042348677 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f500b3ef0 con 0x7f9f5006dcf0 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f54f68640 1 -- 172.21.15.67:0/3042348677 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1575297682 0 0) 0x7f9f500b3c10 con 0x7f9f5006d8e0 2024-04-25T15:26:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.590+0000 7f9f55769640 1 --2- 172.21.15.67:0/3042348677 >> v2:172.21.15.176:3300/0 conn(0x7f9f500d4130 0x7f9f500d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:58.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.591+0000 7f9f5676b640 1 --2- 172.21.15.67:0/3042348677 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006dcf0 0x7f9f500d22a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009d90 tx=0x7f9f4002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ec49ecd621b6de4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.591+0000 7f9f54f68640 1 -- 172.21.15.67:0/3042348677 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40031600 con 0x7f9f5006dcf0 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.591+0000 7f9f54f68640 1 -- 172.21.15.67:0/3042348677 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f400317a0 con 0x7f9f5006dcf0 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.591+0000 7f9f54f68640 1 -- 172.21.15.67:0/3042348677 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40031b70 con 0x7f9f5006dcf0 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 -- 172.21.15.67:0/3042348677 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006dcf0 msgr2=0x7f9f500d22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 --2- 172.21.15.67:0/3042348677 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006dcf0 0x7f9f500d22a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009d90 tx=0x7f9f4002a5d0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 -- 172.21.15.67:0/3042348677 shutdown_connections 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 --2- 172.21.15.67:0/3042348677 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006dcf0 0x7f9f500d22a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 --2- 172.21.15.67:0/3042348677 >> v2:172.21.15.176:3300/0 conn(0x7f9f500d4130 0x7f9f500d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 -- 172.21.15.67:0/3042348677 >> 172.21.15.67:0/3042348677 conn(0x7f9f5008d110 msgr2=0x7f9f5008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 -- 172.21.15.67:0/3042348677 shutdown_connections 2024-04-25T15:26:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.592+0000 7f9f581ce640 1 -- 172.21.15.67:0/3042348677 wait complete. 2024-04-25T15:26:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.594+0000 7f9f581ce640 1 Processor -- start 2024-04-25T15:26:58.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.594+0000 7f9f581ce640 1 -- start start 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f581ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006d8e0 0x7f9f5014da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f581ce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f5006dcf0 0x7f9f500cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f581ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015d750 0x7f9f5014c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f581ce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f500b38e0 con 0x7f9f5015d750 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f581ce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9f500b44b0 con 0x7f9f5006dcf0 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f581ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9f500d27e0 con 0x7f9f5006d8e0 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f5676b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015d750 0x7f9f5014c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f5676b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015d750 0x7f9f5014c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38730/0 (socket says 172.21.15.67:38730) 2024-04-25T15:26:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f5676b640 1 -- 172.21.15.67:0/5496552 learned_addr learned my addr 172.21.15.67:0/5496552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f55769640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f5006dcf0 0x7f9f500cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.595+0000 7f9f55f6a640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006d8e0 0x7f9f5014da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f5676b640 1 -- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006d8e0 msgr2=0x7f9f5014da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f5676b640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006d8e0 0x7f9f5014da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f5676b640 1 -- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f5006dcf0 msgr2=0x7f9f500cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f5676b640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f5006dcf0 0x7f9f500cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f5676b640 1 -- 172.21.15.67:0/5496552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f5014c9e0 con 0x7f9f5015d750 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f55769640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f5006dcf0 0x7f9f500cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f5676b640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015d750 0x7f9f5014c000 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009d90 tx=0x7f9f40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f4e7fc640 1 -- 172.21.15.67:0/5496552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4003f070 con 0x7f9f5015d750 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f4e7fc640 1 -- 172.21.15.67:0/5496552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f40030b60 con 0x7f9f5015d750 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f5014ccc0 con 0x7f9f5015d750 2024-04-25T15:26:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f4e7fc640 1 -- 172.21.15.67:0/5496552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40009450 con 0x7f9f5015d750 2024-04-25T15:26:58.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.596+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f5014d280 con 0x7f9f5015d750 2024-04-25T15:26:58.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.597+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f50050740 con 0x7f9f5015d750 2024-04-25T15:26:58.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.598+0000 7f9f4e7fc640 1 -- 172.21.15.67:0/5496552 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9f4003b410 con 0x7f9f5015d750 2024-04-25T15:26:58.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.599+0000 7f9f4e7fc640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f3c0744e0 0x7f9f3c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.599+0000 7f9f4e7fc640 1 -- 172.21.15.67:0/5496552 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f9f400b62f0 con 0x7f9f5015d750 2024-04-25T15:26:58.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.599+0000 7f9f55f6a640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f3c0744e0 0x7f9f3c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.599+0000 7f9f55f6a640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f3c0744e0 0x7f9f3c0769f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9f48007fd0 tx=0x7f9f48007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:58.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.603+0000 7f9f4e7fc640 1 -- 172.21.15.67:0/5496552 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f400804c0 con 0x7f9f5015d750 2024-04-25T15:26:58.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:26:58.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:26:58.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.790+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f500a15b0 con 0x7f9f5015d750 2024-04-25T15:26:58.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.791+0000 7f9f4e7fc640 1 -- 172.21.15.67:0/5496552 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16577 (secure 0 0 0) 0x7f9f400847c0 con 0x7f9f5015d750 2024-04-25T15:26:58.792 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:26:58.792 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":44,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:50.738984+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:26:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f3c0744e0 msgr2=0x7f9f3c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f3c0744e0 0x7f9f3c0769f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9f48007fd0 tx=0x7f9f48007de0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015d750 msgr2=0x7f9f5014c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015d750 0x7f9f5014c000 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009d90 tx=0x7f9f40004a40 comp rx=0 tx=0).stop 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 shutdown_connections 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f3c0744e0 0x7f9f3c0769f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f5015d750 0x7f9f5014c000 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f5006dcf0 0x7f9f500cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 --2- 172.21.15.67:0/5496552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f5006d8e0 0x7f9f5014da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 >> 172.21.15.67:0/5496552 conn(0x7f9f5008d110 msgr2=0x7f9f50072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 shutdown_connections 2024-04-25T15:26:58.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.793+0000 7f9f581ce640 1 -- 172.21.15.67:0/5496552 wait complete. 2024-04-25T15:26:58.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:26:47.438381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:26:48.064334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:26:58.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:26:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.960+0000 7fcf99b72640 1 Processor -- start 2024-04-25T15:26:58.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.960+0000 7fcf99b72640 1 -- start start 2024-04-25T15:26:58.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf99b72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 0x7fcf9416b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf99b72640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf94161970 con 0x7fcf9415e0c0 2024-04-25T15:26:58.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf99b72640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf940d1a40 con 0x7fcf9406e760 2024-04-25T15:26:58.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf99b72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf941614e0 con 0x7fcf940d1440 2024-04-25T15:26:58.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf937fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcf9415e0c0 0x7fcf9406e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57018/0 (socket says 172.21.15.67:57018) 2024-04-25T15:26:58.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf937fe640 1 -- 172.21.15.67:0/292426457 learned_addr learned my addr 172.21.15.67:0/292426457 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:58.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf93fff640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 0x7fcf9416b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.961+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823639247 0 0) 0x7fcf94161970 con 0x7fcf9415e0c0 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf7c003520 con 0x7fcf9415e0c0 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3619982931 0 0) 0x7fcf940d1a40 con 0x7fcf9406e760 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf94161970 con 0x7fcf9406e760 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 631100928 0 0) 0x7fcf7c003520 con 0x7fcf9415e0c0 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 msgr2=0x7fcf9416b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 0x7fcf9416b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 >> v1:172.21.15.176:6789/0 conn(0x7fcf9406e760 legacy=0x7fcf940d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf94160bb0 con 0x7fcf9415e0c0 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fcf84001670 con 0x7fcf9415e0c0 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 >> v1:172.21.15.67:6789/0 conn(0x7fcf9415e0c0 legacy=0x7fcf9406e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf7c003b70 0x7fcf7c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 0x7fcf7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c004d40 0x7fcf7c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf941614e0 con 0x7fcf7c004d40 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c00bac0 con 0x7fcf7c003b70 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf7c00bcc0 con 0x7fcf940d1440 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf937fe640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf7c003b70 0x7fcf7c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf93fff640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c004d40 0x7fcf7c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf92ffd640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 0x7fcf7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf93fff640 1 -- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 msgr2=0x7fcf7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:58.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf93fff640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940d1440 0x7fcf7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf93fff640 1 -- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf7c003b70 msgr2=0x7fcf7c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:58.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf93fff640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf7c003b70 0x7fcf7c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.962+0000 7fcf93fff640 1 -- 172.21.15.67:0/292426457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf7c003520 con 0x7fcf7c004d40 2024-04-25T15:26:58.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.963+0000 7fcf93fff640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c004d40 0x7fcf7c005150 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009f90 tx=0x7fcf7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3951130a8e2e9ea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:58.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.963+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c009c10 con 0x7fcf7c004d40 2024-04-25T15:26:58.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.963+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf8c036070 con 0x7fcf7c004d40 2024-04-25T15:26:58.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.963+0000 7fcf927fc640 1 -- 172.21.15.67:0/292426457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c031810 con 0x7fcf7c004d40 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.964+0000 7fcf99b72640 1 -- 172.21.15.67:0/292426457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c004d40 msgr2=0x7fcf7c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.964+0000 7fcf99b72640 1 --2- 172.21.15.67:0/292426457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf7c004d40 0x7fcf7c005150 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009f90 tx=0x7fcf7c005830 comp rx=0 tx=0).stop 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.964+0000 7fcf93fff640 1 -- 172.21.15.67:0/292426457 reap_dead start 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.964+0000 7fcf99b72640 1 -- 172.21.15.67:0/292426457 shutdown_connections 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.964+0000 7fcf99b72640 1 -- 172.21.15.67:0/292426457 >> 172.21.15.67:0/292426457 conn(0x7fcf940d2800 msgr2=0x7fcf940d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.964+0000 7fcf99b72640 1 -- 172.21.15.67:0/292426457 shutdown_connections 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.964+0000 7fcf99b72640 1 -- 172.21.15.67:0/292426457 wait complete. 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.966+0000 7fcf99b72640 1 Processor -- start 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.966+0000 7fcf99b72640 1 -- start start 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf99b72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf941fa730 0x7fcf94151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf99b72640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf941e9db0 0x7fcf940b7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf99b72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940b8150 0x7fcf94163f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf99b72640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf940d59c0 con 0x7fcf941fa730 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf99b72640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94161170 con 0x7fcf941e9db0 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf99b72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf940d5b40 con 0x7fcf940b8150 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf941fa730 0x7fcf94151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf941fa730 0x7fcf94151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38750/0 (socket says 172.21.15.67:38750) 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.967+0000 7fcf937fe640 1 -- 172.21.15.67:0/938381303 learned_addr learned my addr 172.21.15.67:0/938381303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf93fff640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940b8150 0x7fcf94163f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf92ffd640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf941e9db0 0x7fcf940b7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf937fe640 1 -- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940b8150 msgr2=0x7fcf94163f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf937fe640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940b8150 0x7fcf94163f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf937fe640 1 -- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf941e9db0 msgr2=0x7fcf940b7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf937fe640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf941e9db0 0x7fcf940b7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf937fe640 1 -- 172.21.15.67:0/938381303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf941523b0 con 0x7fcf941fa730 2024-04-25T15:26:58.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf937fe640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf941fa730 0x7fcf94151e70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcf84007290 tx=0x7fcf84007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:58.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf7bfff640 1 -- 172.21.15.67:0/938381303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8400f940 con 0x7fcf941fa730 2024-04-25T15:26:58.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf7bfff640 1 -- 172.21.15.67:0/938381303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf84018070 con 0x7fcf941fa730 2024-04-25T15:26:58.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf940b8560 con 0x7fcf941fa730 2024-04-25T15:26:58.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf7bfff640 1 -- 172.21.15.67:0/938381303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8400fd40 con 0x7fcf941fa730 2024-04-25T15:26:58.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.968+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf941ea8e0 con 0x7fcf941fa730 2024-04-25T15:26:58.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.969+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf94050740 con 0x7fcf941fa730 2024-04-25T15:26:58.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.970+0000 7fcf7bfff640 1 -- 172.21.15.67:0/938381303 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcf8401d460 con 0x7fcf941fa730 2024-04-25T15:26:58.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.970+0000 7fcf7bfff640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcf800744b0 0x7fcf800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:58.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.970+0000 7fcf7bfff640 1 -- 172.21.15.67:0/938381303 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fcf840951c0 con 0x7fcf941fa730 2024-04-25T15:26:58.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.970+0000 7fcf92ffd640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcf800744b0 0x7fcf800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:58.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.971+0000 7fcf92ffd640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcf800744b0 0x7fcf800769c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcf880018d0 tx=0x7fcf88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:58.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:58.975+0000 7fcf7bfff640 1 -- 172.21.15.67:0/938381303 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf940b8560 con 0x7fcf941fa730 2024-04-25T15:26:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.162+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcf9415e820 con 0x7fcf941fa730 2024-04-25T15:26:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.162+0000 7fcf7bfff640 1 -- 172.21.15.67:0/938381303 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7fcf941523b0 con 0x7fcf941fa730 2024-04-25T15:26:59.163 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:26:59.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcf800744b0 msgr2=0x7fcf800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcf800744b0 0x7fcf800769c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcf880018d0 tx=0x7fcf88009040 comp rx=0 tx=0).stop 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf941fa730 msgr2=0x7fcf94151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf941fa730 0x7fcf94151e70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcf84007290 tx=0x7fcf84007690 comp rx=0 tx=0).stop 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 shutdown_connections 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcf800744b0 0x7fcf800769c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf940b8150 0x7fcf94163f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf941e9db0 0x7fcf940b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 --2- 172.21.15.67:0/938381303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf941fa730 0x7fcf94151e70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 >> 172.21.15.67:0/938381303 conn(0x7fcf940d2800 msgr2=0x7fcf94158a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.164+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 shutdown_connections 2024-04-25T15:26:59.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.165+0000 7fcf99b72640 1 -- 172.21.15.67:0/938381303 wait complete. 2024-04-25T15:26:59.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:26:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.290+0000 7f6beede3640 1 Processor -- start 2024-04-25T15:26:59.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.290+0000 7f6beede3640 1 -- start start 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.291+0000 7f6beede3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.291+0000 7f6beede3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6be80b2e60 0x7f6be808d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.291+0000 7f6beede3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6be8164020 0x7f6be8164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.291+0000 7f6beede3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6be8164970 con 0x7f6be8164020 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.291+0000 7f6beede3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6be80d23f0 con 0x7f6be80b2e60 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.291+0000 7f6beede3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6be80d26d0 con 0x7f6be80d1e20 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.291+0000 7f6bed380640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6be8164020 0x7f6be8164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bed380640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6be8164020 0x7f6be8164430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38766/0 (socket says 172.21.15.67:38766) 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bed380640 1 -- 172.21.15.67:0/1348475712 learned_addr learned my addr 172.21.15.67:0/1348475712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bdffff640 1 --2- 172.21.15.67:0/1348475712 >> v2:172.21.15.176:3300/0 conn(0x7f6be80b2e60 0x7f6be808d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6becb7f640 1 --2- 172.21.15.67:0/1348475712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bed380640 1 -- 172.21.15.67:0/1348475712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 msgr2=0x7f6be80d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bed380640 1 --2- 172.21.15.67:0/1348475712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bed380640 1 -- 172.21.15.67:0/1348475712 >> v2:172.21.15.176:3300/0 conn(0x7f6be80b2e60 msgr2=0x7f6be808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bed380640 1 --2- 172.21.15.67:0/1348475712 >> v2:172.21.15.176:3300/0 conn(0x7f6be80b2e60 0x7f6be808d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.292+0000 7f6bed380640 1 -- 172.21.15.67:0/1348475712 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be80d29b0 con 0x7f6be8164020 2024-04-25T15:26:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6bed380640 1 --2- 172.21.15.67:0/1348475712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be8164020 0x7f6be8164430 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6be8074890 tx=0x7f6be002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9beb7ff0948dba25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6bdf7fe640 1 -- 172.21.15.67:0/1348475712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0036070 con 0x7f6be8164020 2024-04-25T15:26:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6bdf7fe640 1 -- 172.21.15.67:0/1348475712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be0031600 con 0x7f6be8164020 2024-04-25T15:26:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6bdf7fe640 1 -- 172.21.15.67:0/1348475712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be00319d0 con 0x7f6be8164020 2024-04-25T15:26:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6beede3640 1 -- 172.21.15.67:0/1348475712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be8164020 msgr2=0x7f6be8164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6beede3640 1 --2- 172.21.15.67:0/1348475712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be8164020 0x7f6be8164430 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6be8074890 tx=0x7f6be002a730 comp rx=0 tx=0).stop 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6beede3640 1 -- 172.21.15.67:0/1348475712 shutdown_connections 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6beede3640 1 --2- 172.21.15.67:0/1348475712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be8164020 0x7f6be8164430 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6beede3640 1 --2- 172.21.15.67:0/1348475712 >> v2:172.21.15.176:3300/0 conn(0x7f6be80b2e60 0x7f6be808d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6beede3640 1 --2- 172.21.15.67:0/1348475712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.293+0000 7f6beede3640 1 -- 172.21.15.67:0/1348475712 >> 172.21.15.67:0/1348475712 conn(0x7f6be80d0530 msgr2=0x7f6be80d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.294+0000 7f6beede3640 1 -- 172.21.15.67:0/1348475712 shutdown_connections 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.294+0000 7f6beede3640 1 -- 172.21.15.67:0/1348475712 wait complete. 2024-04-25T15:26:59.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.295+0000 7f6beede3640 1 Processor -- start 2024-04-25T15:26:59.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.295+0000 7f6beede3640 1 -- start start 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6beede3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be80b2e60 0x7f6be80b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6beede3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6beede3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be8164020 0x7f6be81fa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6beede3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6be808dcb0 con 0x7f6be80b2e60 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6beede3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6be80d2f70 con 0x7f6be8164020 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6beede3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6be80ce070 con 0x7f6be80d1e20 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be80b2e60 0x7f6be80b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be80b2e60 0x7f6be80b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38782/0 (socket says 172.21.15.67:38782) 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 -- 172.21.15.67:0/1178323374 learned_addr learned my addr 172.21.15.67:0/1178323374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6bdffff640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6bed380640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be8164020 0x7f6be81fa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 -- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 msgr2=0x7f6be80cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 -- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be8164020 msgr2=0x7f6be81fa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be8164020 0x7f6be81fa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.297+0000 7f6becb7f640 1 -- 172.21.15.67:0/1178323374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be80b75c0 con 0x7f6be80b2e60 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.298+0000 7f6becb7f640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be80b2e60 0x7f6be80b8660 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6be400d620 tx=0x7f6be400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.298+0000 7f6bdd7fa640 1 -- 172.21.15.67:0/1178323374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be400b620 con 0x7f6be80b2e60 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.298+0000 7f6bdd7fa640 1 -- 172.21.15.67:0/1178323374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be4019070 con 0x7f6be80b2e60 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.298+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be80b78a0 con 0x7f6be80b2e60 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.298+0000 7f6bdd7fa640 1 -- 172.21.15.67:0/1178323374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be400ba00 con 0x7f6be80b2e60 2024-04-25T15:26:59.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.298+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be80b7e60 con 0x7f6be80b2e60 2024-04-25T15:26:59.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.299+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be8050740 con 0x7f6be80b2e60 2024-04-25T15:26:59.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.299+0000 7f6bdd7fa640 1 -- 172.21.15.67:0/1178323374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6be400bba0 con 0x7f6be80b2e60 2024-04-25T15:26:59.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.300+0000 7f6bdd7fa640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6bd40744b0 0x7f6bd40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.300+0000 7f6bdd7fa640 1 -- 172.21.15.67:0/1178323374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f6be4095510 con 0x7f6be80b2e60 2024-04-25T15:26:59.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.300+0000 7f6bdffff640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6bd40744b0 0x7f6bd40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.300+0000 7f6bdffff640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6bd40744b0 0x7f6bd40769c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6be80cdaf0 tx=0x7f6bd8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.304+0000 7f6bdd7fa640 1 -- 172.21.15.67:0/1178323374 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6be405f650 con 0x7f6be80b2e60 2024-04-25T15:26:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.488+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6be8071000 con 0x7f6be80b2e60 2024-04-25T15:26:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.489+0000 7f6bdd7fa640 1 -- 172.21.15.67:0/1178323374 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7f6be4063950 con 0x7f6be80b2e60 2024-04-25T15:26:59.490 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6bd40744b0 msgr2=0x7f6bd40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6bd40744b0 0x7f6bd40769c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6be80cdaf0 tx=0x7f6bd8008040 comp rx=0 tx=0).stop 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be80b2e60 msgr2=0x7f6be80b8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be80b2e60 0x7f6be80b8660 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6be400d620 tx=0x7f6be400dae0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 shutdown_connections 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6bd40744b0 0x7f6bd40769c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be8164020 0x7f6be81fa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be80d1e20 0x7f6be80cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 --2- 172.21.15.67:0/1178323374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be80b2e60 0x7f6be80b8660 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 >> 172.21.15.67:0/1178323374 conn(0x7f6be80d0530 msgr2=0x7f6be815e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 shutdown_connections 2024-04-25T15:26:59.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.491+0000 7f6beede3640 1 -- 172.21.15.67:0/1178323374 wait complete. 2024-04-25T15:26:59.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:26:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.617+0000 7f66c6c30640 1 Processor -- start 2024-04-25T15:26:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.617+0000 7f66c6c30640 1 -- start start 2024-04-25T15:26:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66c6c30640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f66c015e080 0x7f66c01646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66c6c30640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c01609b0 con 0x7f66c0161710 2024-04-25T15:26:59.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66c6c30640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c0164cd0 con 0x7f66c006ccd0 2024-04-25T15:26:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66c6c30640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f66c0160c30 con 0x7f66c015e080 2024-04-25T15:26:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66c49cc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66c0161710 0x7f66c006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57032/0 (socket says 172.21.15.67:57032) 2024-04-25T15:26:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66c49cc640 1 -- 172.21.15.67:0/2875561974 learned_addr learned my addr 172.21.15.67:0/2875561974 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66c51cd640 1 --2- 172.21.15.67:0/2875561974 >> v2:172.21.15.176:3300/0 conn(0x7f66c015e080 0x7f66c01646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1071230582 0 0) 0x7f66b8001490 con 0x7f66c0161710 2024-04-25T15:26:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66ac003520 con 0x7f66c0161710 2024-04-25T15:26:59.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1540949072 0 0) 0x7f66ac003520 con 0x7f66c0161710 2024-04-25T15:26:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 >> v2:172.21.15.176:3300/0 conn(0x7f66c015e080 msgr2=0x7f66c01646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 --2- 172.21.15.67:0/2875561974 >> v2:172.21.15.176:3300/0 conn(0x7f66c015e080 0x7f66c01646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 >> v1:172.21.15.176:6789/0 conn(0x7f66c006ccd0 legacy=0x7f66c0163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c0160f10 con 0x7f66c0161710 2024-04-25T15:26:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f66b8001bc0 con 0x7f66c0161710 2024-04-25T15:26:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.619+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 >> v1:172.21.15.67:6789/0 conn(0x7f66c0161710 legacy=0x7f66c006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bf7fe640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0039b0 0x7f66ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bf7fe640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66ac004800 0x7f66ac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bf7fe640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac005250 0x7f66ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0160c30 con 0x7f66ac0039b0 2024-04-25T15:26:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac00c1e0 con 0x7f66ac005250 2024-04-25T15:26:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f66ac00c3e0 con 0x7f66ac004800 2024-04-25T15:26:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c49cc640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0039b0 0x7f66ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bffff640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66ac004800 0x7f66ac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c51cd640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac005250 0x7f66ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c49cc640 1 -- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66ac004800 msgr2=0x7f66ac004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c49cc640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66ac004800 0x7f66ac004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c49cc640 1 -- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac005250 msgr2=0x7f66ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c49cc640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac005250 0x7f66ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c49cc640 1 -- 172.21.15.67:0/2875561974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ac003520 con 0x7f66ac0039b0 2024-04-25T15:26:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66bffff640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66ac004800 0x7f66ac004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c51cd640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac005250 0x7f66ac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.620+0000 7f66c51cd640 1 -- 172.21.15.67:0/2875561974 reap_dead start 2024-04-25T15:26:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.621+0000 7f66c49cc640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0039b0 0x7f66ac003e20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f66b800aa20 tx=0x7f66ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff5946007ecc56e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.621+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b800a670 con 0x7f66ac0039b0 2024-04-25T15:26:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.621+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b802f070 con 0x7f66ac0039b0 2024-04-25T15:26:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.621+0000 7f66bf7fe640 1 -- 172.21.15.67:0/2875561974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8034470 con 0x7f66ac0039b0 2024-04-25T15:26:59.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.622+0000 7f66c6c30640 1 -- 172.21.15.67:0/2875561974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0039b0 msgr2=0x7f66ac003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.622+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0039b0 0x7f66ac003e20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f66b800aa20 tx=0x7f66ac005830 comp rx=0 tx=0).stop 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.622+0000 7f66c6c30640 1 -- 172.21.15.67:0/2875561974 shutdown_connections 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.622+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2875561974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ac0039b0 0x7f66ac003e20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.622+0000 7f66c6c30640 1 -- 172.21.15.67:0/2875561974 >> 172.21.15.67:0/2875561974 conn(0x7f66c00d5d90 msgr2=0x7f66c00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.622+0000 7f66c6c30640 1 -- 172.21.15.67:0/2875561974 shutdown_connections 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.622+0000 7f66c6c30640 1 -- 172.21.15.67:0/2875561974 wait complete. 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.623+0000 7f66c6c30640 1 Processor -- start 2024-04-25T15:26:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.623+0000 7f66c6c30640 1 -- start start 2024-04-25T15:26:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c6c30640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac0039b0 0x7f66c00b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c6c30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66c0147160 0x7f66c0147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c6c30640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c01548d0 0x7f66c0154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c6c30640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66c006c740 con 0x7f66c01548d0 2024-04-25T15:26:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c6c30640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f66c006c5c0 con 0x7f66ac0039b0 2024-04-25T15:26:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c6c30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f66c006c8c0 con 0x7f66c0147160 2024-04-25T15:26:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c01548d0 0x7f66c0154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c01548d0 0x7f66c0154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38804/0 (socket says 172.21.15.67:38804) 2024-04-25T15:26:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 -- 172.21.15.67:0/2791734202 learned_addr learned my addr 172.21.15.67:0/2791734202 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c49cc640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac0039b0 0x7f66c00b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66bffff640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66c0147160 0x7f66c0147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 -- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66c0147160 msgr2=0x7f66c0147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66c0147160 0x7f66c0147570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 -- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac0039b0 msgr2=0x7f66c00b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac0039b0 0x7f66c00b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.625+0000 7f66c51cd640 1 -- 172.21.15.67:0/2791734202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c00ba000 con 0x7f66c01548d0 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.626+0000 7f66c51cd640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c01548d0 0x7f66c0154ce0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f66c00d6d90 tx=0x7f66a8011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.626+0000 7f66bd7fa640 1 -- 172.21.15.67:0/2791734202 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8003610 con 0x7f66c01548d0 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.626+0000 7f66bd7fa640 1 -- 172.21.15.67:0/2791734202 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a801b070 con 0x7f66c01548d0 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.626+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c00ba2e0 con 0x7f66c01548d0 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.626+0000 7f66bd7fa640 1 -- 172.21.15.67:0/2791734202 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8003a10 con 0x7f66c01548d0 2024-04-25T15:26:59.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.626+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c00ba8a0 con 0x7f66c01548d0 2024-04-25T15:26:59.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.627+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c0050740 con 0x7f66c01548d0 2024-04-25T15:26:59.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.627+0000 7f66bd7fa640 1 -- 172.21.15.67:0/2791734202 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f66a8003bb0 con 0x7f66c01548d0 2024-04-25T15:26:59.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.628+0000 7f66bd7fa640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f66a00744b0 0x7f66a00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.628+0000 7f66bd7fa640 1 -- 172.21.15.67:0/2791734202 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f66a80951f0 con 0x7f66c01548d0 2024-04-25T15:26:59.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.628+0000 7f66c49cc640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f66a00744b0 0x7f66a00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.629+0000 7f66c49cc640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f66a00744b0 0x7f66a00769c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f66c008e250 tx=0x7f66b8009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.633+0000 7f66bd7fa640 1 -- 172.21.15.67:0/2791734202 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66a8025020 con 0x7f66c01548d0 2024-04-25T15:26:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.819+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f66c01624d0 con 0x7f66c01548d0 2024-04-25T15:26:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.819+0000 7f66bd7fa640 1 -- 172.21.15.67:0/2791734202 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7f66a805f3c0 con 0x7f66c01548d0 2024-04-25T15:26:59.820 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:26:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f66a00744b0 msgr2=0x7f66a00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f66a00744b0 0x7f66a00769c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f66c008e250 tx=0x7f66b8009030 comp rx=0 tx=0).stop 2024-04-25T15:26:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c01548d0 msgr2=0x7f66c0154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c01548d0 0x7f66c0154ce0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f66c00d6d90 tx=0x7f66a8011920 comp rx=0 tx=0).stop 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 shutdown_connections 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f66a00744b0 0x7f66a00769c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c01548d0 0x7f66c0154ce0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66c0147160 0x7f66c0147570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 --2- 172.21.15.67:0/2791734202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66ac0039b0 0x7f66c00b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.821+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 >> 172.21.15.67:0/2791734202 conn(0x7f66c00d5d90 msgr2=0x7f66c00bb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.822+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 shutdown_connections 2024-04-25T15:26:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.822+0000 7f66c6c30640 1 -- 172.21.15.67:0/2791734202 wait complete. 2024-04-25T15:26:59.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:26:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.946+0000 7ff93ff46640 1 Processor -- start 2024-04-25T15:26:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.946+0000 7ff93ff46640 1 -- start start 2024-04-25T15:26:59.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93ff46640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff9380d33d0 0x7ff9380b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93ff46640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9380d2330 con 0x7ff9380d0be0 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93ff46640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9380d2610 con 0x7ff9380d45e0 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93ff46640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff9380d28f0 con 0x7ff9380d33d0 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93e4e3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff9380d0be0 0x7ff9380ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57046/0 (socket says 172.21.15.67:57046) 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93e4e3640 1 -- 172.21.15.67:0/1263625159 learned_addr learned my addr 172.21.15.67:0/1263625159 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93dce2640 1 --2- 172.21.15.67:0/1263625159 >> v2:172.21.15.176:3300/0 conn(0x7ff9380d33d0 0x7ff9380b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991079170 0 0) 0x7ff9380d2330 con 0x7ff9380d0be0 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.948+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff920003520 con 0x7ff9380d0be0 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2778038667 0 0) 0x7ff920003520 con 0x7ff9380d0be0 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 >> v2:172.21.15.176:3300/0 conn(0x7ff9380d33d0 msgr2=0x7ff9380b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 --2- 172.21.15.67:0/1263625159 >> v2:172.21.15.176:3300/0 conn(0x7ff9380d33d0 0x7ff9380b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 >> v1:172.21.15.176:6789/0 conn(0x7ff9380d45e0 legacy=0x7ff9380d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9380d2bd0 con 0x7ff9380d0be0 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7ff9280019f0 con 0x7ff9380d0be0 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 >> v1:172.21.15.67:6789/0 conn(0x7ff9380d0be0 legacy=0x7ff9380ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9200039b0 0x7ff920003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff920004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff920005150 0x7ff92000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff9380d28f0 con 0x7ff920004800 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff92000c0c0 con 0x7ff9200039b0 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff92000c2c0 con 0x7ff920005150 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93d4e1640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff920004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93dce2640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9200039b0 0x7ff920003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93e4e3640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff920005150 0x7ff92000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93d4e1640 1 -- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff920005150 msgr2=0x7ff92000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93d4e1640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff920005150 0x7ff92000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93d4e1640 1 -- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9200039b0 msgr2=0x7ff920003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93d4e1640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9200039b0 0x7ff920003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93d4e1640 1 -- 172.21.15.67:0/1263625159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff920003520 con 0x7ff920004800 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93e4e3640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff920005150 0x7ff92000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.949+0000 7ff93e4e3640 1 -- 172.21.15.67:0/1263625159 reap_dead start 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.950+0000 7ff93d4e1640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff920004c10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff92c001860 tx=0x7ff920005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becc27f31f77f7ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.950+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c037070 con 0x7ff920004800 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.950+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff92c02abe0 con 0x7ff920004800 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.950+0000 7ff93cce0640 1 -- 172.21.15.67:0/1263625159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c001430 con 0x7ff920004800 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.951+0000 7ff93ff46640 1 -- 172.21.15.67:0/1263625159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 msgr2=0x7ff920004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.951+0000 7ff93ff46640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff920004c10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff92c001860 tx=0x7ff920005830 comp rx=0 tx=0).stop 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.951+0000 7ff93ff46640 1 -- 172.21.15.67:0/1263625159 shutdown_connections 2024-04-25T15:26:59.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.951+0000 7ff93ff46640 1 --2- 172.21.15.67:0/1263625159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff920004c10 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.951+0000 7ff93ff46640 1 -- 172.21.15.67:0/1263625159 >> 172.21.15.67:0/1263625159 conn(0x7ff9380d3ac0 msgr2=0x7ff9380d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:26:59.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.951+0000 7ff93ff46640 1 -- 172.21.15.67:0/1263625159 shutdown_connections 2024-04-25T15:26:59.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.951+0000 7ff93ff46640 1 -- 172.21.15.67:0/1263625159 wait complete. 2024-04-25T15:26:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.953+0000 7ff93ff46640 1 Processor -- start 2024-04-25T15:26:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.953+0000 7ff93ff46640 1 -- start start 2024-04-25T15:26:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93ff46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff9380c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93ff46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff93814db80 0x7ff9380cc980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93ff46640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9380ccec0 0x7ff93814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93ff46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff9380d2040 con 0x7ff920004800 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93ff46640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff9380d1ec0 con 0x7ff9380ccec0 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93ff46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff93806dc70 con 0x7ff93814db80 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff9380c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff9380c8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38818/0 (socket says 172.21.15.67:38818) 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 -- 172.21.15.67:0/2348132758 learned_addr learned my addr 172.21.15.67:0/2348132758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93d4e1640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff93814db80 0x7ff9380cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93e4e3640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9380ccec0 0x7ff93814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 -- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff93814db80 msgr2=0x7ff9380cc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff93814db80 0x7ff9380cc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 -- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9380ccec0 msgr2=0x7ff93814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9380ccec0 0x7ff93814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:26:59.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 -- 172.21.15.67:0/2348132758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9380c7690 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.954+0000 7ff93dce2640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff9380c8730 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff9300119a0 tx=0x7ff930011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.955+0000 7ff9367fc640 1 -- 172.21.15.67:0/2348132758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930005ba0 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.955+0000 7ff9367fc640 1 -- 172.21.15.67:0/2348132758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff930005d40 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.955+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9380c7970 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.955+0000 7ff9367fc640 1 -- 172.21.15.67:0/2348132758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93000fb00 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.955+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9380c7f30 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.956+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9380b4bb0 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.956+0000 7ff9367fc640 1 -- 172.21.15.67:0/2348132758 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff930012040 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.957+0000 7ff9367fc640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9240744e0 0x7ff9240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.957+0000 7ff9367fc640 1 -- 172.21.15.67:0/2348132758 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7ff930095920 con 0x7ff920004800 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.957+0000 7ff93d4e1640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9240744e0 0x7ff9240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:26:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.958+0000 7ff93d4e1640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9240744e0 0x7ff9240769f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff92c034000 tx=0x7ff92c009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:26:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:26:59.962+0000 7ff9367fc640 1 -- 172.21.15.67:0/2348132758 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9380b4bb0 con 0x7ff920004800 2024-04-25T15:27:00.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.158+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff93806e310 con 0x7ff920004800 2024-04-25T15:27:00.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.159+0000 7ff9367fc640 1 -- 172.21.15.67:0/2348132758 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) ==== 80+0+0 (secure 0 0 0) 0x7ff93806e310 con 0x7ff920004800 2024-04-25T15:27:00.160 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.160+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9240744e0 msgr2=0x7ff9240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.160+0000 7ff93ff46640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9240744e0 0x7ff9240769f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff92c034000 tx=0x7ff92c009030 comp rx=0 tx=0).stop 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.160+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 msgr2=0x7ff9380c8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.160+0000 7ff93ff46640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff9380c8730 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff9300119a0 tx=0x7ff930011e60 comp rx=0 tx=0).stop 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 shutdown_connections 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff9240744e0 0x7ff9240769f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9380ccec0 0x7ff93814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff93814db80 0x7ff9380cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 --2- 172.21.15.67:0/2348132758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff920004800 0x7ff9380c8730 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 >> 172.21.15.67:0/2348132758 conn(0x7ff9380d3ac0 msgr2=0x7ff9380b89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:00.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 shutdown_connections 2024-04-25T15:27:00.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.161+0000 7ff93ff46640 1 -- 172.21.15.67:0/2348132758 wait complete. 2024-04-25T15:27:00.172 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T15:27:00.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:00.173 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:00.201 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:27:00.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:00.201 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:27:00.256 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:27:00.256 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:00.256 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:00.275 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:00.275 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:00.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.444+0000 7f4541515640 1 Processor -- start 2024-04-25T15:27:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.445+0000 7f4541515640 1 -- start start 2024-04-25T15:27:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f4541515640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c161e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f4541515640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f453c1623c0 0x7f453c1627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f4541515640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f453c0cfc30 con 0x7f453c0ce760 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f4541515640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f453c0d1ff0 con 0x7f453c1623c0 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f4541515640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f453c0d2890 con 0x7f453c0b91d0 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f453affd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f453affd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c161e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52584/0 (socket says 172.21.15.67:52584) 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f453affd640 1 -- 172.21.15.67:0/1469556327 learned_addr learned my addr 172.21.15.67:0/1469556327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.446+0000 7f453a7fc640 1 --2- 172.21.15.67:0/1469556327 >> v2:172.21.15.176:3300/0 conn(0x7f453c1623c0 0x7f453c1627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.447+0000 7f453affd640 1 -- 172.21.15.67:0/1469556327 >> v2:172.21.15.176:3300/0 conn(0x7f453c1623c0 msgr2=0x7f453c1627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.447+0000 7f453affd640 1 --2- 172.21.15.67:0/1469556327 >> v2:172.21.15.176:3300/0 conn(0x7f453c1623c0 0x7f453c1627d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.447+0000 7f453affd640 1 -- 172.21.15.67:0/1469556327 >> v1:172.21.15.176:6789/0 conn(0x7f453c0ce760 legacy=0x7f453c0ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.447+0000 7f453affd640 1 -- 172.21.15.67:0/1469556327 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453c0d2240 con 0x7f453c0b91d0 2024-04-25T15:27:00.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.447+0000 7f453affd640 1 --2- 172.21.15.67:0/1469556327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c161e80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d40a5ef09b4d28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4539ffb640 1 -- 172.21.15.67:0/1469556327 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534037070 con 0x7f453c0b91d0 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4539ffb640 1 -- 172.21.15.67:0/1469556327 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4534032440 con 0x7f453c0b91d0 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4539ffb640 1 -- 172.21.15.67:0/1469556327 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534032810 con 0x7f453c0b91d0 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 -- 172.21.15.67:0/1469556327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 msgr2=0x7f453c161e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 --2- 172.21.15.67:0/1469556327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c161e80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453402a270 comp rx=0 tx=0).stop 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 -- 172.21.15.67:0/1469556327 shutdown_connections 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 --2- 172.21.15.67:0/1469556327 >> v2:172.21.15.176:3300/0 conn(0x7f453c1623c0 0x7f453c1627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 --2- 172.21.15.67:0/1469556327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c161e80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 -- 172.21.15.67:0/1469556327 >> 172.21.15.67:0/1469556327 conn(0x7f453c0d3aa0 msgr2=0x7f453c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 -- 172.21.15.67:0/1469556327 shutdown_connections 2024-04-25T15:27:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.448+0000 7f4541515640 1 -- 172.21.15.67:0/1469556327 wait complete. 2024-04-25T15:27:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.450+0000 7f4541515640 1 Processor -- start 2024-04-25T15:27:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.450+0000 7f4541515640 1 -- start start 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.451+0000 7f4541515640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f4541515640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f453c0ce760 0x7f453c15bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f4541515640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f453c1623c0 0x7f453c0cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f4541515640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f453c0b2320 con 0x7f453c1623c0 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f4541515640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f453c162d10 con 0x7f453c0ce760 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f4541515640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f453c0b24a0 con 0x7f453c0b91d0 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f453c1623c0 0x7f453c0cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f453c1623c0 0x7f453c0cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38834/0 (socket says 172.21.15.67:38834) 2024-04-25T15:27:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 -- 172.21.15.67:0/3607275701 learned_addr learned my addr 172.21.15.67:0/3607275701 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453a7fc640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f453c0ce760 0x7f453c15bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453affd640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 -- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 msgr2=0x7f453c0ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c0ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 -- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f453c0ce760 msgr2=0x7f453c15bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f453c0ce760 0x7f453c15bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 -- 172.21.15.67:0/3607275701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453c15c020 con 0x7f453c1623c0 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453a7fc640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f453c0ce760 0x7f453c15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f453b7fe640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f453c1623c0 0x7f453c0cb7d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f452c00c730 tx=0x7f452c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f45237fe640 1 -- 172.21.15.67:0/3607275701 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c003b70 con 0x7f453c1623c0 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f45237fe640 1 -- 172.21.15.67:0/3607275701 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f452c003d10 con 0x7f453c1623c0 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.452+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f453c0cd570 con 0x7f453c1623c0 2024-04-25T15:27:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.453+0000 7f45237fe640 1 -- 172.21.15.67:0/3607275701 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452c005310 con 0x7f453c1623c0 2024-04-25T15:27:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.453+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f453c0cafd0 con 0x7f453c1623c0 2024-04-25T15:27:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.454+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f453c0699f0 con 0x7f453c1623c0 2024-04-25T15:27:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.454+0000 7f45237fe640 1 -- 172.21.15.67:0/3607275701 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f452c0056c0 con 0x7f453c1623c0 2024-04-25T15:27:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.455+0000 7f45237fe640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f45280744e0 0x7f45280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.455+0000 7f45237fe640 1 -- 172.21.15.67:0/3607275701 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f452c095260 con 0x7f453c1623c0 2024-04-25T15:27:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.455+0000 7f453affd640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f45280744e0 0x7f45280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.456+0000 7f453affd640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f45280744e0 0x7f45280769f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4534034000 tx=0x7f453402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.460+0000 7f45237fe640 1 -- 172.21.15.67:0/3607275701 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f453c0699f0 con 0x7f453c1623c0 2024-04-25T15:27:00.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.647+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f453c0caae0 con 0x7f453c1623c0 2024-04-25T15:27:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.648+0000 7f45237fe640 1 -- 172.21.15.67:0/3607275701 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16577 (secure 0 0 0) 0x7f453c15c020 con 0x7f453c1623c0 2024-04-25T15:27:00.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:00.649 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":44,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:50.738984+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f45280744e0 msgr2=0x7f45280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f45280744e0 0x7f45280769f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4534034000 tx=0x7f453402e000 comp rx=0 tx=0).stop 2024-04-25T15:27:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f453c1623c0 msgr2=0x7f453c0cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f453c1623c0 0x7f453c0cb7d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f452c00c730 tx=0x7f452c00cbf0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 shutdown_connections 2024-04-25T15:27:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f45280744e0 0x7f45280769f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f453c1623c0 0x7f453c0cb7d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f453c0ce760 0x7f453c15bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 --2- 172.21.15.67:0/3607275701 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f453c0b91d0 0x7f453c0ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 >> 172.21.15.67:0/3607275701 conn(0x7f453c0d3aa0 msgr2=0x7f453c071b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 shutdown_connections 2024-04-25T15:27:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.650+0000 7f4541515640 1 -- 172.21.15.67:0/3607275701 wait complete. 2024-04-25T15:27:00.662 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:26:47.438381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:26:48.064334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:00.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:00.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.777+0000 7f155d300640 1 Processor -- start 2024-04-25T15:27:00.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.777+0000 7f155d300640 1 -- start start 2024-04-25T15:27:00.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f155d300640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15580b9230 0x7f15581647c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f155d300640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f15580b0d10 0x7f15580b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f155d300640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f155806e650 con 0x7f155806e0b0 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f155d300640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f155806e930 con 0x7f15580b9230 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f155d300640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f155806ec10 con 0x7f15580b0d10 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f15567fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15580b9230 0x7f15581647c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f15567fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15580b9230 0x7f15581647c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38848/0 (socket says 172.21.15.67:38848) 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f15567fc640 1 -- 172.21.15.67:0/534512223 learned_addr learned my addr 172.21.15.67:0/534512223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.778+0000 7f15577fe640 1 --2- 172.21.15.67:0/534512223 >> v2:172.21.15.176:3300/0 conn(0x7f15580b0d10 0x7f15580b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.779+0000 7f15567fc640 1 -- 172.21.15.67:0/534512223 >> v2:172.21.15.176:3300/0 conn(0x7f15580b0d10 msgr2=0x7f15580b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.779+0000 7f15567fc640 1 --2- 172.21.15.67:0/534512223 >> v2:172.21.15.176:3300/0 conn(0x7f15580b0d10 0x7f15580b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.779+0000 7f15567fc640 1 -- 172.21.15.67:0/534512223 >> v1:172.21.15.176:6789/0 conn(0x7f155806e0b0 legacy=0x7f15581640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.779+0000 7f15567fc640 1 -- 172.21.15.67:0/534512223 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15580b07f0 con 0x7f15580b9230 2024-04-25T15:27:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.779+0000 7f15567fc640 1 --2- 172.21.15.67:0/534512223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b9230 0x7f15581647c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1558073d10 tx=0x7f154002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e9dd062fb7ae93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.779+0000 7f1555ffb640 1 -- 172.21.15.67:0/534512223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1540037070 con 0x7f15580b9230 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.779+0000 7f1555ffb640 1 -- 172.21.15.67:0/534512223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1540032440 con 0x7f15580b9230 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f1555ffb640 1 -- 172.21.15.67:0/534512223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1540032810 con 0x7f15580b9230 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 -- 172.21.15.67:0/534512223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b9230 msgr2=0x7f15581647c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 --2- 172.21.15.67:0/534512223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b9230 0x7f15581647c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1558073d10 tx=0x7f154002a270 comp rx=0 tx=0).stop 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 -- 172.21.15.67:0/534512223 shutdown_connections 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 --2- 172.21.15.67:0/534512223 >> v2:172.21.15.176:3300/0 conn(0x7f15580b0d10 0x7f15580b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 --2- 172.21.15.67:0/534512223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b9230 0x7f15581647c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 -- 172.21.15.67:0/534512223 >> 172.21.15.67:0/534512223 conn(0x7f15580d3a60 msgr2=0x7f15580d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 -- 172.21.15.67:0/534512223 shutdown_connections 2024-04-25T15:27:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.780+0000 7f155d300640 1 -- 172.21.15.67:0/534512223 wait complete. 2024-04-25T15:27:00.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.782+0000 7f155d300640 1 Processor -- start 2024-04-25T15:27:00.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.782+0000 7f155d300640 1 -- start start 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.783+0000 7f155d300640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155806e0b0 0x7f15580b2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.783+0000 7f155d300640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b0d10 0x7f15580b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.783+0000 7f155d300640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15580b9230 0x7f15580b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.783+0000 7f155d300640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1558164d00 con 0x7f15580b0d10 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.783+0000 7f155d300640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1558164e80 con 0x7f15580b9230 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.783+0000 7f155d300640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15580b1660 con 0x7f155806e0b0 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b0d10 0x7f15580b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b0d10 0x7f15580b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38852/0 (socket says 172.21.15.67:38852) 2024-04-25T15:27:00.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 -- 172.21.15.67:0/1501245356 learned_addr learned my addr 172.21.15.67:0/1501245356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15577fe640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15580b9230 0x7f15580b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f1556ffd640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155806e0b0 0x7f15580b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 -- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155806e0b0 msgr2=0x7f15580b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155806e0b0 0x7f15580b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 -- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15580b9230 msgr2=0x7f15580b4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15580b9230 0x7f15580b4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 -- 172.21.15.67:0/1501245356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15580b32a0 con 0x7f15580b0d10 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f1556ffd640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155806e0b0 0x7f15580b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f15567fc640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b0d10 0x7f15580b4560 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f154002a2b0 tx=0x7f15400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f154f7fe640 1 -- 172.21.15.67:0/1501245356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1540037040 con 0x7f15580b0d10 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f154f7fe640 1 -- 172.21.15.67:0/1501245356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1540042070 con 0x7f15580b0d10 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f154f7fe640 1 -- 172.21.15.67:0/1501245356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1540030880 con 0x7f15580b0d10 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15580b3580 con 0x7f15580b0d10 2024-04-25T15:27:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.784+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15580b3d60 con 0x7f15580b0d10 2024-04-25T15:27:00.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.785+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1558050740 con 0x7f15580b0d10 2024-04-25T15:27:00.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.786+0000 7f154f7fe640 1 -- 172.21.15.67:0/1501245356 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1540030a20 con 0x7f15580b0d10 2024-04-25T15:27:00.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.787+0000 7f154f7fe640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15380744e0 0x7f15380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:00.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.787+0000 7f154f7fe640 1 -- 172.21.15.67:0/1501245356 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f15400b5a70 con 0x7f15580b0d10 2024-04-25T15:27:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.787+0000 7f1556ffd640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15380744e0 0x7f15380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.787+0000 7f1556ffd640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15380744e0 0x7f15380769f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1548000f60 tx=0x7f1548009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.791+0000 7f154f7fe640 1 -- 172.21.15.67:0/1501245356 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1540080290 con 0x7f15580b0d10 2024-04-25T15:27:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.976+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15580cdad0 con 0x7f15580b0d10 2024-04-25T15:27:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.976+0000 7f154f7fe640 1 -- 172.21.15.67:0/1501245356 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7f154007fb90 con 0x7f15580b0d10 2024-04-25T15:27:00.977 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:00.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15380744e0 msgr2=0x7f15380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15380744e0 0x7f15380769f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1548000f60 tx=0x7f1548009040 comp rx=0 tx=0).stop 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b0d10 msgr2=0x7f15580b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b0d10 0x7f15580b4560 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f154002a2b0 tx=0x7f15400046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 shutdown_connections 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15380744e0 0x7f15380769f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15580b9230 0x7f15580b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15580b0d10 0x7f15580b4560 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 --2- 172.21.15.67:0/1501245356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f155806e0b0 0x7f15580b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 >> 172.21.15.67:0/1501245356 conn(0x7f15580d3a60 msgr2=0x7f1558072660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 shutdown_connections 2024-04-25T15:27:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:00.978+0000 7f155d300640 1 -- 172.21.15.67:0/1501245356 wait complete. 2024-04-25T15:27:00.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:27:01.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.104+0000 7fe867557640 1 Processor -- start 2024-04-25T15:27:01.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.104+0000 7fe867557640 1 -- start start 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.105+0000 7fe867557640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe860164640 0x7fe8600b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.105+0000 7fe867557640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8600d3170 con 0x7fe860164ad0 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.105+0000 7fe867557640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8600d3450 con 0x7fe8600b0a70 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.105+0000 7fe867557640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe8600d2f40 con 0x7fe860164640 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.106+0000 7fe864af2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe860164640 0x7fe8600b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.106+0000 7fe8652f3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe860164ad0 0x7fe860163f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57048/0 (socket says 172.21.15.67:57048) 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.106+0000 7fe8652f3640 1 -- 172.21.15.67:0/220940343 learned_addr learned my addr 172.21.15.67:0/220940343 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.106+0000 7fe864af2640 1 -- 172.21.15.67:0/220940343 >> v1:172.21.15.176:6789/0 conn(0x7fe8600b0a70 legacy=0x7fe8600b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.106+0000 7fe864af2640 1 -- 172.21.15.67:0/220940343 >> v1:172.21.15.67:6789/0 conn(0x7fe860164ad0 legacy=0x7fe860163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.106+0000 7fe864af2640 1 -- 172.21.15.67:0/220940343 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86006d6d0 con 0x7fe860164640 2024-04-25T15:27:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.106+0000 7fe857fff640 1 -- 172.21.15.67:0/220940343 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574316253 0 0) 0x7fe8600d3170 con 0x7fe860164ad0 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe864af2640 1 --2- 172.21.15.67:0/220940343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe860164640 0x7fe8600b2ca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe8500097a0 tx=0x7fe85002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2854ef4522e32f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe857fff640 1 -- 172.21.15.67:0/220940343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850037070 con 0x7fe860164640 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe857fff640 1 -- 172.21.15.67:0/220940343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe850032440 con 0x7fe860164640 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe857fff640 1 -- 172.21.15.67:0/220940343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850032810 con 0x7fe860164640 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe867557640 1 -- 172.21.15.67:0/220940343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe860164640 msgr2=0x7fe8600b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe867557640 1 --2- 172.21.15.67:0/220940343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe860164640 0x7fe8600b2ca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe8500097a0 tx=0x7fe85002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe867557640 1 -- 172.21.15.67:0/220940343 shutdown_connections 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.107+0000 7fe867557640 1 --2- 172.21.15.67:0/220940343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe860164640 0x7fe8600b2ca0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.108+0000 7fe867557640 1 -- 172.21.15.67:0/220940343 >> 172.21.15.67:0/220940343 conn(0x7fe8600d3aa0 msgr2=0x7fe8600d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.108+0000 7fe867557640 1 -- 172.21.15.67:0/220940343 shutdown_connections 2024-04-25T15:27:01.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.108+0000 7fe867557640 1 -- 172.21.15.67:0/220940343 wait complete. 2024-04-25T15:27:01.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.109+0000 7fe867557640 1 Processor -- start 2024-04-25T15:27:01.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.109+0000 7fe867557640 1 -- start start 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe867557640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8600b0a70 0x7fe86015dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe867557640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe860164ad0 0x7fe8600b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe867557640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8601621b0 0x7fe8601625c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe867557640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8600b31e0 con 0x7fe8600b0a70 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe867557640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe860164ee0 con 0x7fe860164ad0 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe867557640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8600b9430 con 0x7fe8601621b0 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8600b0a70 0x7fe86015dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8600b0a70 0x7fe86015dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38874/0 (socket says 172.21.15.67:38874) 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 -- 172.21.15.67:0/505790316 learned_addr learned my addr 172.21.15.67:0/505790316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe864af2640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe860164ad0 0x7fe8600b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe865af4640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8601621b0 0x7fe8601625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 -- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8601621b0 msgr2=0x7fe8601625c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8601621b0 0x7fe8601625c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 -- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe860164ad0 msgr2=0x7fe8600b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe860164ad0 0x7fe8600b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 -- 172.21.15.67:0/505790316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe860155540 con 0x7fe8600b0a70 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.111+0000 7fe8652f3640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8600b0a70 0x7fe86015dd30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00c440 tx=0x7fe85c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.112+0000 7fe855ffb640 1 -- 172.21.15.67:0/505790316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c018070 con 0x7fe8600b0a70 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.112+0000 7fe855ffb640 1 -- 172.21.15.67:0/505790316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c007500 con 0x7fe8600b0a70 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.112+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8600b39e0 con 0x7fe8600b0a70 2024-04-25T15:27:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.112+0000 7fe855ffb640 1 -- 172.21.15.67:0/505790316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c004a20 con 0x7fe8600b0a70 2024-04-25T15:27:01.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.112+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8601632a0 con 0x7fe8600b0a70 2024-04-25T15:27:01.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.113+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8600d4770 con 0x7fe8600b0a70 2024-04-25T15:27:01.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.114+0000 7fe855ffb640 1 -- 172.21.15.67:0/505790316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe85c01b070 con 0x7fe8600b0a70 2024-04-25T15:27:01.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.114+0000 7fe855ffb640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8440744b0 0x7fe8440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.114+0000 7fe855ffb640 1 -- 172.21.15.67:0/505790316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fe85c0942e0 con 0x7fe8600b0a70 2024-04-25T15:27:01.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.114+0000 7fe864af2640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8440744b0 0x7fe8440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.115+0000 7fe864af2640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8440744b0 0x7fe8440769c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe850034000 tx=0x7fe85002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.119+0000 7fe855ffb640 1 -- 172.21.15.67:0/505790316 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe860155540 con 0x7fe8600b0a70 2024-04-25T15:27:01.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.302+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe860068680 con 0x7fe8600b0a70 2024-04-25T15:27:01.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.302+0000 7fe855ffb640 1 -- 172.21.15.67:0/505790316 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7fe85c05e4b0 con 0x7fe8600b0a70 2024-04-25T15:27:01.303 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:27:01.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8440744b0 msgr2=0x7fe8440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8440744b0 0x7fe8440769c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe850034000 tx=0x7fe85002e000 comp rx=0 tx=0).stop 2024-04-25T15:27:01.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8600b0a70 msgr2=0x7fe86015dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8600b0a70 0x7fe86015dd30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00c440 tx=0x7fe85c00c900 comp rx=0 tx=0).stop 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 shutdown_connections 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8440744b0 0x7fe8440769c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8601621b0 0x7fe8601625c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe860164ad0 0x7fe8600b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 --2- 172.21.15.67:0/505790316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8600b0a70 0x7fe86015dd30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 >> 172.21.15.67:0/505790316 conn(0x7fe8600d3aa0 msgr2=0x7fe860071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 shutdown_connections 2024-04-25T15:27:01.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.304+0000 7fe867557640 1 -- 172.21.15.67:0/505790316 wait complete. 2024-04-25T15:27:01.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:27:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.430+0000 7fa21d3b2640 1 Processor -- start 2024-04-25T15:27:01.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.430+0000 7fa21d3b2640 1 -- start start 2024-04-25T15:27:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.431+0000 7fa21d3b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa218070360 0x7fa21806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.431+0000 7fa21d3b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218160240 0x7fa21806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.431+0000 7fa21d3b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa21806df20 con 0x7fa21806db10 2024-04-25T15:27:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.431+0000 7fa21d3b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa21806ff90 con 0x7fa218070360 2024-04-25T15:27:01.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.431+0000 7fa21d3b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2180b5c30 con 0x7fa218160240 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218160240 0x7fa21806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa216ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa218070360 0x7fa21806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218160240 0x7fa21806d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52618/0 (socket says 172.21.15.67:52618) 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa2177fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa21806db10 0x7fa218163ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:55552/0 (socket says 172.21.15.67:55552) 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa20ffff640 1 -- 172.21.15.67:0/4181479012 learned_addr learned my addr 172.21.15.67:0/4181479012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa20ffff640 1 -- 172.21.15.67:0/4181479012 >> v2:172.21.15.176:3300/0 conn(0x7fa218070360 msgr2=0x7fa21806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa20ffff640 1 --2- 172.21.15.67:0/4181479012 >> v2:172.21.15.176:3300/0 conn(0x7fa218070360 0x7fa21806ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa20ffff640 1 -- 172.21.15.67:0/4181479012 >> v1:172.21.15.176:6789/0 conn(0x7fa21806db10 legacy=0x7fa218163ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.432+0000 7fa20ffff640 1 -- 172.21.15.67:0/4181479012 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21806f410 con 0x7fa218160240 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.433+0000 7fa20ffff640 1 --2- 172.21.15.67:0/4181479012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218160240 0x7fa21806d5d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa200009870 tx=0x7fa20002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee84077982011c6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.433+0000 7fa2167fc640 1 -- 172.21.15.67:0/4181479012 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200037070 con 0x7fa218160240 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.433+0000 7fa2167fc640 1 -- 172.21.15.67:0/4181479012 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa200032440 con 0x7fa218160240 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.433+0000 7fa2167fc640 1 -- 172.21.15.67:0/4181479012 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200032810 con 0x7fa218160240 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 -- 172.21.15.67:0/4181479012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218160240 msgr2=0x7fa21806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/4181479012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218160240 0x7fa21806d5d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa200009870 tx=0x7fa20002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 -- 172.21.15.67:0/4181479012 shutdown_connections 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/4181479012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218160240 0x7fa21806d5d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/4181479012 >> v2:172.21.15.176:3300/0 conn(0x7fa218070360 0x7fa21806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 -- 172.21.15.67:0/4181479012 >> 172.21.15.67:0/4181479012 conn(0x7fa2180730b0 msgr2=0x7fa2180734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 -- 172.21.15.67:0/4181479012 shutdown_connections 2024-04-25T15:27:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.434+0000 7fa21d3b2640 1 -- 172.21.15.67:0/4181479012 wait complete. 2024-04-25T15:27:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.436+0000 7fa21d3b2640 1 Processor -- start 2024-04-25T15:27:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.436+0000 7fa21d3b2640 1 -- start start 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa21d3b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21806db10 0x7fa2181f5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa21d3b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218070360 0x7fa21815f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa21d3b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa218160240 0x7fa2180afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa21d3b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa21806fcb0 con 0x7fa21806db10 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa21d3b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa21806f9d0 con 0x7fa218160240 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa21d3b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2180cdde0 con 0x7fa218070360 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21806db10 0x7fa2181f5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21806db10 0x7fa2181f5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38888/0 (socket says 172.21.15.67:38888) 2024-04-25T15:27:01.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 -- 172.21.15.67:0/3822412183 learned_addr learned my addr 172.21.15.67:0/3822412183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa2177fe640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa218160240 0x7fa2180afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa20ffff640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218070360 0x7fa21815f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 -- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218070360 msgr2=0x7fa21815f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218070360 0x7fa21815f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 -- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa218160240 msgr2=0x7fa2180afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa218160240 0x7fa2180afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.437+0000 7fa216ffd640 1 -- 172.21.15.67:0/3822412183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2181f9b90 con 0x7fa21806db10 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.438+0000 7fa216ffd640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21806db10 0x7fa2181f5d60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa20800e8a0 tx=0x7fa20800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.438+0000 7fa20f7fe640 1 -- 172.21.15.67:0/3822412183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208017070 con 0x7fa21806db10 2024-04-25T15:27:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.438+0000 7fa20f7fe640 1 -- 172.21.15.67:0/3822412183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20800c940 con 0x7fa21806db10 2024-04-25T15:27:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.438+0000 7fa20f7fe640 1 -- 172.21.15.67:0/3822412183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20800cd30 con 0x7fa21806db10 2024-04-25T15:27:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.438+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2181f9e50 con 0x7fa21806db10 2024-04-25T15:27:01.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.438+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2181fa410 con 0x7fa21806db10 2024-04-25T15:27:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.439+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa218050740 con 0x7fa21806db10 2024-04-25T15:27:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.439+0000 7fa20f7fe640 1 -- 172.21.15.67:0/3822412183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa20801a070 con 0x7fa21806db10 2024-04-25T15:27:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.440+0000 7fa20f7fe640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1f80744b0 0x7fa1f80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.440+0000 7fa20f7fe640 1 -- 172.21.15.67:0/3822412183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fa208095080 con 0x7fa21806db10 2024-04-25T15:27:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.440+0000 7fa20ffff640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1f80744b0 0x7fa1f80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.441+0000 7fa20ffff640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1f80744b0 0x7fa1f80769c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa200034000 tx=0x7fa20002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.445+0000 7fa20f7fe640 1 -- 172.21.15.67:0/3822412183 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa20805f1c0 con 0x7fa21806db10 2024-04-25T15:27:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.628+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa2180b0880 con 0x7fa21806db10 2024-04-25T15:27:01.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.628+0000 7fa20f7fe640 1 -- 172.21.15.67:0/3822412183 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7fa2080634c0 con 0x7fa21806db10 2024-04-25T15:27:01.629 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:27:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1f80744b0 msgr2=0x7fa1f80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1f80744b0 0x7fa1f80769c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa200034000 tx=0x7fa20002e000 comp rx=0 tx=0).stop 2024-04-25T15:27:01.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21806db10 msgr2=0x7fa2181f5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21806db10 0x7fa2181f5d60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa20800e8a0 tx=0x7fa20800ed60 comp rx=0 tx=0).stop 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 shutdown_connections 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1f80744b0 0x7fa1f80769c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa218160240 0x7fa2180afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa218070360 0x7fa21815f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 --2- 172.21.15.67:0/3822412183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21806db10 0x7fa2181f5d60 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 >> 172.21.15.67:0/3822412183 conn(0x7fa2180730b0 msgr2=0x7fa2180b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 shutdown_connections 2024-04-25T15:27:01.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.630+0000 7fa21d3b2640 1 -- 172.21.15.67:0/3822412183 wait complete. 2024-04-25T15:27:01.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:27:01.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.754+0000 7f746f346640 1 Processor -- start 2024-04-25T15:27:01.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.755+0000 7f746f346640 1 -- start start 2024-04-25T15:27:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f746f346640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 0x7f7468160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f746f346640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74681615f0 con 0x7f74680b8290 2024-04-25T15:27:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f746f346640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7468164020 con 0x7f7468164b00 2024-04-25T15:27:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f746f346640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7468164670 con 0x7f746815df80 2024-04-25T15:27:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f746c8e1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74680b8290 0x7f7468160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57064/0 (socket says 172.21.15.67:57064) 2024-04-25T15:27:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f746c8e1640 1 -- 172.21.15.67:0/1915317118 learned_addr learned my addr 172.21.15.67:0/1915317118 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f746d8e3640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 0x7f7468160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461028825 0 0) 0x7f74681615f0 con 0x7f74680b8290 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.756+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7454003520 con 0x7f74680b8290 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728033483 0 0) 0x7f7468164020 con 0x7f7468164b00 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74681615f0 con 0x7f7468164b00 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1788764544 0 0) 0x7f7454003520 con 0x7f74680b8290 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 msgr2=0x7f7468160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 0x7f7468160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 >> v1:172.21.15.176:6789/0 conn(0x7f7468164b00 legacy=0x7f74680b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7468164300 con 0x7f74680b8290 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f7458001670 con 0x7f74680b8290 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 >> v1:172.21.15.67:6789/0 conn(0x7f74680b8290 legacy=0x7f7468160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f7454003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 0x7f7454004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7454004d40 0x7f7454005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7468164670 con 0x7f7454003b70 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f745400bac0 con 0x7f7454004d40 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f745400bcc0 con 0x7f746815df80 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746d0e2640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f7454003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746c8e1640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 0x7f7454004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746d8e3640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7454004d40 0x7f7454005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746d0e2640 1 -- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 msgr2=0x7f7454004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746d0e2640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f746815df80 0x7f7454004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746d0e2640 1 -- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7454004d40 msgr2=0x7f7454005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746d0e2640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7454004d40 0x7f7454005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.757+0000 7f746d0e2640 1 -- 172.21.15.67:0/1915317118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7454003520 con 0x7f7454003b70 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.758+0000 7f746d0e2640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f7454003fc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f74640018d0 tx=0x7f7454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3dd3b20b3398bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.758+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74640014f0 con 0x7f7454003b70 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.758+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74640052d0 con 0x7f7454003b70 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.758+0000 7f745ffff640 1 -- 172.21.15.67:0/1915317118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746402aca0 con 0x7f7454003b70 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.759+0000 7f746f346640 1 -- 172.21.15.67:0/1915317118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 msgr2=0x7f7454003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.759+0000 7f746f346640 1 --2- 172.21.15.67:0/1915317118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f7454003fc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f74640018d0 tx=0x7f7454005830 comp rx=0 tx=0).stop 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.759+0000 7f746d8e3640 1 -- 172.21.15.67:0/1915317118 reap_dead start 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.759+0000 7f746f346640 1 -- 172.21.15.67:0/1915317118 shutdown_connections 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.759+0000 7f746f346640 1 -- 172.21.15.67:0/1915317118 >> 172.21.15.67:0/1915317118 conn(0x7f74680d2bc0 msgr2=0x7f74680d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.759+0000 7f746f346640 1 -- 172.21.15.67:0/1915317118 shutdown_connections 2024-04-25T15:27:01.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.759+0000 7f746f346640 1 -- 172.21.15.67:0/1915317118 wait complete. 2024-04-25T15:27:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.761+0000 7f746f346640 1 Processor -- start 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.761+0000 7f746f346640 1 -- start start 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746f346640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f74681515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746f346640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7468151e70 0x7f746815a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746f346640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7468157480 0x7f746815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746f346640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74681613b0 con 0x7f7454003b70 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746f346640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7468161230 con 0x7f7468151e70 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746f346640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f74680b4880 con 0x7f7468157480 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746d0e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f74681515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746d0e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f74681515c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38908/0 (socket says 172.21.15.67:38908) 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746d0e2640 1 -- 172.21.15.67:0/3308932640 learned_addr learned my addr 172.21.15.67:0/3308932640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746c8e1640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7468151e70 0x7f746815a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.762+0000 7f746d8e3640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7468157480 0x7f746815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746d0e2640 1 -- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7468157480 msgr2=0x7f746815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746d0e2640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7468157480 0x7f746815adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746d0e2640 1 -- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7468151e70 msgr2=0x7f746815a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746d0e2640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7468151e70 0x7f746815a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746d0e2640 1 -- 172.21.15.67:0/3308932640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7468159840 con 0x7f7454003b70 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746c8e1640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7468151e70 0x7f746815a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746d0e2640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f74681515c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7464005040 tx=0x7f74640012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f745dffb640 1 -- 172.21.15.67:0/3308932640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746403f020 con 0x7f7454003b70 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f745dffb640 1 -- 172.21.15.67:0/3308932640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7464006e50 con 0x7f7454003b70 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7468159ac0 con 0x7f7454003b70 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f745dffb640 1 -- 172.21.15.67:0/3308932640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464037040 con 0x7f7454003b70 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.763+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746815a080 con 0x7f7454003b70 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.764+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7468050740 con 0x7f7454003b70 2024-04-25T15:27:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.765+0000 7f745dffb640 1 -- 172.21.15.67:0/3308932640 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7464042020 con 0x7f7454003b70 2024-04-25T15:27:01.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.765+0000 7f745dffb640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f744c0744b0 0x7f744c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:01.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.765+0000 7f745dffb640 1 -- 172.21.15.67:0/3308932640 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f74640b4f50 con 0x7f7454003b70 2024-04-25T15:27:01.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.766+0000 7f746c8e1640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f744c0744b0 0x7f744c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:01.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.766+0000 7f746c8e1640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f744c0744b0 0x7f744c0769c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f74680c94a0 tx=0x7f7458009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.770+0000 7f745dffb640 1 -- 172.21.15.67:0/3308932640 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7464049030 con 0x7f7454003b70 2024-04-25T15:27:01.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.969+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f74680d4170 con 0x7f7454003b70 2024-04-25T15:27:01.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.969+0000 7f745dffb640 1 -- 172.21.15.67:0/3308932640 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v44) ==== 80+0+0 (secure 0 0 0) 0x7f7468159840 con 0x7f7454003b70 2024-04-25T15:27:01.970 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:27:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f744c0744b0 msgr2=0x7f744c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f744c0744b0 0x7f744c0769c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f74680c94a0 tx=0x7f7458009040 comp rx=0 tx=0).stop 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 msgr2=0x7f74681515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f74681515c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7464005040 tx=0x7f74640012d0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 shutdown_connections 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f744c0744b0 0x7f744c0769c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7468157480 0x7f746815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7468151e70 0x7f746815a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.971+0000 7f746f346640 1 --2- 172.21.15.67:0/3308932640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7454003b70 0x7f74681515c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.972+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 >> 172.21.15.67:0/3308932640 conn(0x7f74680d2bc0 msgr2=0x7f746815e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.972+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 shutdown_connections 2024-04-25T15:27:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:01.972+0000 7f746f346640 1 -- 172.21.15.67:0/3308932640 wait complete. 2024-04-25T15:27:01.983 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:27:01.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:01.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-25T15:27:02.016 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:27:02.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:02.016 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:27:02.075 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:27:02.075 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:02.075 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:02.091 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:27:02.092 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:27:02.092 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:02.092 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:02.133 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:27:02.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:02.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.244+0000 7f9cef23c640 1 Processor -- start 2024-04-25T15:27:02.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.245+0000 7f9cef23c640 1 -- start start 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9cef23c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9ce806d8e0 0x7f9ce80d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9cef23c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806dcf0 0x7f9ce80d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9cef23c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ce80b3c10 con 0x7f9ce80d04a0 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9cef23c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9ce80b9520 con 0x7f9ce806d8e0 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9cef23c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ce80b4820 con 0x7f9ce806dcf0 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9cecfd8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9ce806d8e0 0x7f9ce80d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9ce7fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806dcf0 0x7f9ce80d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9ce7fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806dcf0 0x7f9ce80d3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52660/0 (socket says 172.21.15.67:52660) 2024-04-25T15:27:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9cecfd8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9ce806d8e0 0x7f9ce80d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:46904/0 (socket says 172.21.15.67:46904) 2024-04-25T15:27:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9ced7d9640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9ce80d04a0 0x7f9ce80d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:55572/0 (socket says 172.21.15.67:55572) 2024-04-25T15:27:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.246+0000 7f9ce7fff640 1 -- 172.21.15.67:0/3135622246 learned_addr learned my addr 172.21.15.67:0/3135622246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.247+0000 7f9ce7fff640 1 -- 172.21.15.67:0/3135622246 >> v2:172.21.15.176:3300/0 conn(0x7f9ce806d8e0 msgr2=0x7f9ce80d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.247+0000 7f9ce7fff640 1 --2- 172.21.15.67:0/3135622246 >> v2:172.21.15.176:3300/0 conn(0x7f9ce806d8e0 0x7f9ce80d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.247+0000 7f9ce7fff640 1 -- 172.21.15.67:0/3135622246 >> v1:172.21.15.176:6789/0 conn(0x7f9ce80d04a0 legacy=0x7f9ce80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.247+0000 7f9ce7fff640 1 -- 172.21.15.67:0/3135622246 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce80b3ef0 con 0x7f9ce806dcf0 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.247+0000 7f9cecfd8640 1 --2- 172.21.15.67:0/3135622246 >> v2:172.21.15.176:3300/0 conn(0x7f9ce806d8e0 0x7f9ce80d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.248+0000 7f9ce7fff640 1 --2- 172.21.15.67:0/3135622246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806dcf0 0x7f9ce80d3f60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009870 tx=0x7f9cd402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b7938ca991ab9ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.248+0000 7f9ce77fe640 1 -- 172.21.15.67:0/3135622246 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4037070 con 0x7f9ce806dcf0 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.248+0000 7f9ce77fe640 1 -- 172.21.15.67:0/3135622246 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd4032440 con 0x7f9ce806dcf0 2024-04-25T15:27:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.248+0000 7f9ce77fe640 1 -- 172.21.15.67:0/3135622246 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4032810 con 0x7f9ce806dcf0 2024-04-25T15:27:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 -- 172.21.15.67:0/3135622246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806dcf0 msgr2=0x7f9ce80d3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 --2- 172.21.15.67:0/3135622246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806dcf0 0x7f9ce80d3f60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009870 tx=0x7f9cd402a1c0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 -- 172.21.15.67:0/3135622246 shutdown_connections 2024-04-25T15:27:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 --2- 172.21.15.67:0/3135622246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806dcf0 0x7f9ce80d3f60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 --2- 172.21.15.67:0/3135622246 >> v2:172.21.15.176:3300/0 conn(0x7f9ce806d8e0 0x7f9ce80d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 -- 172.21.15.67:0/3135622246 >> 172.21.15.67:0/3135622246 conn(0x7f9ce808d110 msgr2=0x7f9ce808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 -- 172.21.15.67:0/3135622246 shutdown_connections 2024-04-25T15:27:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.249+0000 7f9cef23c640 1 -- 172.21.15.67:0/3135622246 wait complete. 2024-04-25T15:27:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.250+0000 7f9cef23c640 1 Processor -- start 2024-04-25T15:27:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.251+0000 7f9cef23c640 1 -- start start 2024-04-25T15:27:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9cef23c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806d8e0 0x7f9ce809f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9cef23c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ce806dcf0 0x7f9ce8153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9cef23c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce80d04a0 0x7f9ce81e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9cef23c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce80b38e0 con 0x7f9ce80d04a0 2024-04-25T15:27:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9cef23c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce80b44b0 con 0x7f9ce806dcf0 2024-04-25T15:27:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9cef23c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ce80d0fc0 con 0x7f9ce806d8e0 2024-04-25T15:27:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce80d04a0 0x7f9ce81e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce80d04a0 0x7f9ce81e8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38922/0 (socket says 172.21.15.67:38922) 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 -- 172.21.15.67:0/2440967154 learned_addr learned my addr 172.21.15.67:0/2440967154 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ce7fff640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ce806dcf0 0x7f9ce8153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9cecfd8640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806d8e0 0x7f9ce809f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 -- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806d8e0 msgr2=0x7f9ce809f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806d8e0 0x7f9ce809f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 -- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ce806dcf0 msgr2=0x7f9ce8153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ce806dcf0 0x7f9ce8153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.252+0000 7f9ced7d9640 1 -- 172.21.15.67:0/2440967154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce81e6ad0 con 0x7f9ce80d04a0 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.253+0000 7f9ced7d9640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce80d04a0 0x7f9ce81e8e60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000c6f0 tx=0x7f9ce000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.253+0000 7f9ce57fa640 1 -- 172.21.15.67:0/2440967154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0017070 con 0x7f9ce80d04a0 2024-04-25T15:27:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.253+0000 7f9ce57fa640 1 -- 172.21.15.67:0/2440967154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ce0005da0 con 0x7f9ce80d04a0 2024-04-25T15:27:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.253+0000 7f9ce57fa640 1 -- 172.21.15.67:0/2440967154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000a840 con 0x7f9ce80d04a0 2024-04-25T15:27:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.253+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce809fa20 con 0x7f9ce80d04a0 2024-04-25T15:27:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.253+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce81e6d70 con 0x7f9ce80d04a0 2024-04-25T15:27:02.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.254+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce8050740 con 0x7f9ce80d04a0 2024-04-25T15:27:02.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.255+0000 7f9ce57fa640 1 -- 172.21.15.67:0/2440967154 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9ce001a070 con 0x7f9ce80d04a0 2024-04-25T15:27:02.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.255+0000 7f9ce57fa640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ccc0744b0 0x7f9ccc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.255+0000 7f9ce57fa640 1 -- 172.21.15.67:0/2440967154 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f9ce0094740 con 0x7f9ce80d04a0 2024-04-25T15:27:02.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.255+0000 7f9cecfd8640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ccc0744b0 0x7f9ccc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.256+0000 7f9cecfd8640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ccc0744b0 0x7f9ccc0769c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009560 tx=0x7f9cdc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.260+0000 7f9ce57fa640 1 -- 172.21.15.67:0/2440967154 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ce005e880 con 0x7f9ce80d04a0 2024-04-25T15:27:02.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.487+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ce80b8a10 con 0x7f9ce80d04a0 2024-04-25T15:27:02.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.487+0000 7f9ce57fa640 1 -- 172.21.15.67:0/2440967154 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5187 (secure 0 0 0) 0x7f9ce0062b80 con 0x7f9ce80d04a0 2024-04-25T15:27:02.488 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:02.488 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1556919488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1556919488},{"type":"v1","addr":"172.21.15.67:6833","nonce":1556919488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4071024412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4071024412},{"type":"v1","addr":"172.21.15.176:6837","nonce":4071024412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3791679818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3791679818},{"type":"v1","addr":"172.21.15.176:6836","nonce":3791679818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:49.120424+0000","modified":"2024-04-25T15:26:52.141230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1882238407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1882238407},{"type":"v1","addr":"172.21.15.67:6835","nonce":1882238407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":3}]} 2024-04-25T15:27:02.489 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.489+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ccc0744b0 msgr2=0x7f9ccc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.489+0000 7f9cef23c640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ccc0744b0 0x7f9ccc0769c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009560 tx=0x7f9cdc009190 comp rx=0 tx=0).stop 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.489+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce80d04a0 msgr2=0x7f9ce81e8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.489+0000 7f9cef23c640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce80d04a0 0x7f9ce81e8e60 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000c6f0 tx=0x7f9ce000cbb0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 shutdown_connections 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9ccc0744b0 0x7f9ccc0769c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce80d04a0 0x7f9ce81e8e60 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ce806dcf0 0x7f9ce8153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 --2- 172.21.15.67:0/2440967154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce806d8e0 0x7f9ce809f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 >> 172.21.15.67:0/2440967154 conn(0x7f9ce808d110 msgr2=0x7f9ce8072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 shutdown_connections 2024-04-25T15:27:02.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.490+0000 7f9cef23c640 1 -- 172.21.15.67:0/2440967154 wait complete. 2024-04-25T15:27:02.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:02.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.612+0000 7f2ae0734640 1 Processor -- start 2024-04-25T15:27:02.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.612+0000 7f2ae0734640 1 -- start start 2024-04-25T15:27:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2ae0734640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ad80d3f70 0x7f2ad80b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2ae0734640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ad80b5350 con 0x7f2ad80b4f40 2024-04-25T15:27:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2ae0734640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ad80b4180 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2ae0734640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2ad80b4ab0 con 0x7f2ad80d3f70 2024-04-25T15:27:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2adecd1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ad80d3f70 0x7f2ad80b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2adecd1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ad80d3f70 0x7f2ad80b9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38938/0 (socket says 172.21.15.67:38938) 2024-04-25T15:27:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2addccf640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2ad80b4f40 0x7f2ad80d3860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57070/0 (socket says 172.21.15.67:57070) 2024-04-25T15:27:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2adecd1640 1 -- 172.21.15.67:0/1354335130 learned_addr learned my addr 172.21.15.67:0/1354335130 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2adecd1640 1 -- 172.21.15.67:0/1354335130 >> v1:172.21.15.176:6789/0 conn(0x7f2ad80b2cc0 legacy=0x7f2ad80d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2adecd1640 1 -- 172.21.15.67:0/1354335130 >> v1:172.21.15.67:6789/0 conn(0x7f2ad80b4f40 legacy=0x7f2ad80d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.614+0000 7f2adecd1640 1 -- 172.21.15.67:0/1354335130 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad80b4460 con 0x7f2ad80d3f70 2024-04-25T15:27:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.615+0000 7f2adecd1640 1 --2- 172.21.15.67:0/1354335130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80d3f70 0x7f2ad80b9160 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009e50 tx=0x7f2ac802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533a1ca44b75d105 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.615+0000 7f2add4ce640 1 -- 172.21.15.67:0/1354335130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8036070 con 0x7f2ad80d3f70 2024-04-25T15:27:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.615+0000 7f2add4ce640 1 -- 172.21.15.67:0/1354335130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ac8031600 con 0x7f2ad80d3f70 2024-04-25T15:27:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.615+0000 7f2add4ce640 1 -- 172.21.15.67:0/1354335130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac80319d0 con 0x7f2ad80d3f70 2024-04-25T15:27:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.616+0000 7f2ae0734640 1 -- 172.21.15.67:0/1354335130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80d3f70 msgr2=0x7f2ad80b9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.616+0000 7f2ae0734640 1 --2- 172.21.15.67:0/1354335130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80d3f70 0x7f2ad80b9160 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009e50 tx=0x7f2ac802a730 comp rx=0 tx=0).stop 2024-04-25T15:27:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.616+0000 7f2ae0734640 1 -- 172.21.15.67:0/1354335130 shutdown_connections 2024-04-25T15:27:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.616+0000 7f2ae0734640 1 --2- 172.21.15.67:0/1354335130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80d3f70 0x7f2ad80b9160 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.616+0000 7f2ae0734640 1 -- 172.21.15.67:0/1354335130 >> 172.21.15.67:0/1354335130 conn(0x7f2ad808d110 msgr2=0x7f2ad808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.616+0000 7f2ae0734640 1 -- 172.21.15.67:0/1354335130 shutdown_connections 2024-04-25T15:27:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.616+0000 7f2ae0734640 1 -- 172.21.15.67:0/1354335130 wait complete. 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.618+0000 7f2ae0734640 1 Processor -- start 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.618+0000 7f2ae0734640 1 -- start start 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ae0734640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80b2cc0 0x7f2ad81f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ae0734640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ad80b4f40 0x7f2ad81e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ae0734640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ad81e9300 0x7f2ad8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ae0734640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad80b0740 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ae0734640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad80b05c0 con 0x7f2ad81e9300 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ae0734640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad80b08c0 con 0x7f2ad80b4f40 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80b2cc0 0x7f2ad81f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80b2cc0 0x7f2ad81f58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38942/0 (socket says 172.21.15.67:38942) 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 -- 172.21.15.67:0/4143832506 learned_addr learned my addr 172.21.15.67:0/4143832506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2addccf640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ad80b4f40 0x7f2ad81e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2adecd1640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ad81e9300 0x7f2ad8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 -- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ad80b4f40 msgr2=0x7f2ad81e8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ad80b4f40 0x7f2ad81e8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 -- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ad81e9300 msgr2=0x7f2ad8151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ad81e9300 0x7f2ad8151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2ade4d0640 1 -- 172.21.15.67:0/4143832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad81e9b60 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2addccf640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ad80b4f40 0x7f2ad81e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.619+0000 7f2adecd1640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ad81e9300 0x7f2ad8151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.620+0000 7f2ade4d0640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80b2cc0 0x7f2ad81f58f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000c3f0 tx=0x7f2ad000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.620+0000 7f2ad6ffd640 1 -- 172.21.15.67:0/4143832506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad00052d0 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.620+0000 7f2ad6ffd640 1 -- 172.21.15.67:0/4143832506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad0005490 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.620+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad81e9e40 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.620+0000 7f2ad6ffd640 1 -- 172.21.15.67:0/4143832506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0007260 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.620+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad81ea400 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.621+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad81ea640 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.621+0000 7f2ad6ffd640 1 -- 172.21.15.67:0/4143832506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2ad0007400 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.622+0000 7f2ad6ffd640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ac40744e0 0x7f2ac40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.622+0000 7f2ad6ffd640 1 -- 172.21.15.67:0/4143832506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f2ad0094660 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.622+0000 7f2addccf640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ac40744e0 0x7f2ac40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.623+0000 7f2addccf640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ac40744e0 0x7f2ac40769f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2ad809f520 tx=0x7f2acc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.627+0000 7f2ad6ffd640 1 -- 172.21.15.67:0/4143832506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ad81ea640 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.814+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ad8050740 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.815+0000 7f2ad6ffd640 1 -- 172.21.15.67:0/4143832506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16577 (secure 0 0 0) 0x7f2ad0065e20 con 0x7f2ad80b2cc0 2024-04-25T15:27:02.816 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:02.817 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":44,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:50.738984+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.817+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ac40744e0 msgr2=0x7f2ac40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ac40744e0 0x7f2ac40769f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2ad809f520 tx=0x7f2acc009040 comp rx=0 tx=0).stop 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80b2cc0 msgr2=0x7f2ad81f58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80b2cc0 0x7f2ad81f58f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000c3f0 tx=0x7f2ad000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 shutdown_connections 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ac40744e0 0x7f2ac40769f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ad81e9300 0x7f2ad8151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ad80b4f40 0x7f2ad81e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 --2- 172.21.15.67:0/4143832506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad80b2cc0 0x7f2ad81f58f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 >> 172.21.15.67:0/4143832506 conn(0x7f2ad808d110 msgr2=0x7f2ad8163250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:02.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 shutdown_connections 2024-04-25T15:27:02.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.818+0000 7f2ae0734640 1 -- 172.21.15.67:0/4143832506 wait complete. 2024-04-25T15:27:02.830 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:26:47.438381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:26:48.064334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:02.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:02.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.948+0000 7f47e2461640 1 Processor -- start 2024-04-25T15:27:02.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.948+0000 7f47e2461640 1 -- start start 2024-04-25T15:27:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47e2461640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc08d820 0x7f47dc08dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47e2461640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47dc161d10 0x7f47dc162120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47e2461640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47dc162660 con 0x7f47dc0d1990 2024-04-25T15:27:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47e2461640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f47dc0b9610 con 0x7f47dc161d10 2024-04-25T15:27:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47e2461640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc0cf6d0 con 0x7f47dc08d820 2024-04-25T15:27:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47db7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47dc161d10 0x7f47dc162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47db7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47dc161d10 0x7f47dc162120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38944/0 (socket says 172.21.15.67:38944) 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47db7fe640 1 -- 172.21.15.67:0/3926039574 learned_addr learned my addr 172.21.15.67:0/3926039574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47e09fe640 1 --1- 172.21.15.67:0/3926039574 >> v1:172.21.15.67:6789/0 conn(0x7f47dc0d1990 0x7f47dc0d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57084/0 (socket says 172.21.15.67:57084) 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47dbfff640 1 --2- 172.21.15.67:0/3926039574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc08d820 0x7f47dc08dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47db7fe640 1 -- 172.21.15.67:0/3926039574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc08d820 msgr2=0x7f47dc08dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47db7fe640 1 --2- 172.21.15.67:0/3926039574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc08d820 0x7f47dc08dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47db7fe640 1 -- 172.21.15.67:0/3926039574 >> v1:172.21.15.67:6789/0 conn(0x7f47dc0d1990 legacy=0x7f47dc0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47db7fe640 1 -- 172.21.15.67:0/3926039574 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc0ceac0 con 0x7f47dc161d10 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.949+0000 7f47daffd640 1 -- 172.21.15.67:0/3926039574 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2759820863 0 0) 0x7f47d4001490 con 0x7f47dc0d1990 2024-04-25T15:27:02.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.950+0000 7f47db7fe640 1 --2- 172.21.15.67:0/3926039574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc162120 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009870 tx=0x7f47cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73daf2da87934bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.950+0000 7f47daffd640 1 -- 172.21.15.67:0/3926039574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc037070 con 0x7f47dc161d10 2024-04-25T15:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.950+0000 7f47daffd640 1 -- 172.21.15.67:0/3926039574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47cc032440 con 0x7f47dc161d10 2024-04-25T15:27:02.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.950+0000 7f47daffd640 1 -- 172.21.15.67:0/3926039574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc032810 con 0x7f47dc161d10 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 -- 172.21.15.67:0/3926039574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 msgr2=0x7f47dc162120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 --2- 172.21.15.67:0/3926039574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc162120 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009870 tx=0x7f47cc02a270 comp rx=0 tx=0).stop 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 -- 172.21.15.67:0/3926039574 shutdown_connections 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 --2- 172.21.15.67:0/3926039574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc162120 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 --2- 172.21.15.67:0/3926039574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc08d820 0x7f47dc08dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 -- 172.21.15.67:0/3926039574 >> 172.21.15.67:0/3926039574 conn(0x7f47dc0d0530 msgr2=0x7f47dc0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 -- 172.21.15.67:0/3926039574 shutdown_connections 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.951+0000 7f47e2461640 1 -- 172.21.15.67:0/3926039574 wait complete. 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.953+0000 7f47e2461640 1 Processor -- start 2024-04-25T15:27:02.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.953+0000 7f47e2461640 1 -- start start 2024-04-25T15:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.954+0000 7f47e2461640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47dc08d820 0x7f47dc06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.954+0000 7f47e2461640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc0d1990 0x7f47dc072430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.954+0000 7f47e2461640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc072970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.954+0000 7f47e2461640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc0cf360 con 0x7f47dc161d10 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.954+0000 7f47e2461640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc0cf080 con 0x7f47dc08d820 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.954+0000 7f47e2461640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc0ce6e0 con 0x7f47dc0d1990 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc072970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc072970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38952/0 (socket says 172.21.15.67:38952) 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 -- 172.21.15.67:0/4286173420 learned_addr learned my addr 172.21.15.67:0/4286173420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47dbfff640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47dc08d820 0x7f47dc06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47db7fe640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc0d1990 0x7f47dc072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 -- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc0d1990 msgr2=0x7f47dc072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc0d1990 0x7f47dc072430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 -- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47dc08d820 msgr2=0x7f47dc06ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47dc08d820 0x7f47dc06ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 -- 172.21.15.67:0/4286173420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc15de70 con 0x7f47dc161d10 2024-04-25T15:27:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e09fe640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc072970 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f47d400c700 tx=0x7f47d400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47d8ff9640 1 -- 172.21.15.67:0/4286173420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4018070 con 0x7f47dc161d10 2024-04-25T15:27:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47d8ff9640 1 -- 172.21.15.67:0/4286173420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d400ce10 con 0x7f47dc161d10 2024-04-25T15:27:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47dc15e150 con 0x7f47dc161d10 2024-04-25T15:27:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47d8ff9640 1 -- 172.21.15.67:0/4286173420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d400a860 con 0x7f47dc161d10 2024-04-25T15:27:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.955+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47dc15e710 con 0x7f47dc161d10 2024-04-25T15:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.956+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47dc050740 con 0x7f47dc161d10 2024-04-25T15:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.957+0000 7f47d8ff9640 1 -- 172.21.15.67:0/4286173420 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f47d401b070 con 0x7f47dc161d10 2024-04-25T15:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.957+0000 7f47d8ff9640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47c80744b0 0x7f47c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.957+0000 7f47d8ff9640 1 -- 172.21.15.67:0/4286173420 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f47d40946e0 con 0x7f47dc161d10 2024-04-25T15:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.958+0000 7f47dbfff640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47c80744b0 0x7f47c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.958+0000 7f47dbfff640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47c80744b0 0x7f47c80769c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f47d00079a0 tx=0x7f47d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:02.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:02.962+0000 7f47d8ff9640 1 -- 172.21.15.67:0/4286173420 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47dc162660 con 0x7f47dc161d10 2024-04-25T15:27:03.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.146+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47dc0d11c0 con 0x7f47dc161d10 2024-04-25T15:27:03.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.147+0000 7f47d8ff9640 1 -- 172.21.15.67:0/4286173420 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7f47d40650d0 con 0x7f47dc161d10 2024-04-25T15:27:03.148 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47c80744b0 msgr2=0x7f47c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47c80744b0 0x7f47c80769c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f47d00079a0 tx=0x7f47d0008040 comp rx=0 tx=0).stop 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 msgr2=0x7f47dc072970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc072970 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f47d400c700 tx=0x7f47d400cbc0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 shutdown_connections 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47c80744b0 0x7f47c80769c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47dc161d10 0x7f47dc072970 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47dc0d1990 0x7f47dc072430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 --2- 172.21.15.67:0/4286173420 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47dc08d820 0x7f47dc06ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 >> 172.21.15.67:0/4286173420 conn(0x7f47dc0d0530 msgr2=0x7f47dc15a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 shutdown_connections 2024-04-25T15:27:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.149+0000 7f47e2461640 1 -- 172.21.15.67:0/4286173420 wait complete. 2024-04-25T15:27:03.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:27:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.275+0000 7f4a81a42640 1 Processor -- start 2024-04-25T15:27:03.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.275+0000 7f4a81a42640 1 -- start start 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a81a42640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4a7c0d0e00 0x7f4a7c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a81a42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a81a42640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7c0d1850 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a81a42640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4a7c160de0 con 0x7f4a7c0d0e00 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a81a42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7c1610c0 con 0x7f4a7c0d4a50 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a73fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4a7c06c5c0 0x7f4a7c06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57088/0 (socket says 172.21.15.67:57088) 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a73fff640 1 -- 172.21.15.67:0/1616852774 learned_addr learned my addr 172.21.15.67:0/1616852774 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7b7fe640 1 --2- 172.21.15.67:0/1616852774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7affd640 1 --2- 172.21.15.67:0/1616852774 >> v2:172.21.15.176:3300/0 conn(0x7f4a7c0d0e00 0x7f4a7c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7a7fc640 1 -- 172.21.15.67:0/1616852774 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134554540 0 0) 0x7f4a7c0d1850 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7a7fc640 1 -- 172.21.15.67:0/1616852774 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a68003520 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7b7fe640 1 -- 172.21.15.67:0/1616852774 >> v2:172.21.15.176:3300/0 conn(0x7f4a7c0d0e00 msgr2=0x7f4a7c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7b7fe640 1 --2- 172.21.15.67:0/1616852774 >> v2:172.21.15.176:3300/0 conn(0x7f4a7c0d0e00 0x7f4a7c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7b7fe640 1 -- 172.21.15.67:0/1616852774 >> v1:172.21.15.67:6789/0 conn(0x7f4a7c06c5c0 legacy=0x7f4a7c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7a7fc640 1 -- 172.21.15.67:0/1616852774 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3441915318 0 0) 0x7f4a68003520 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.277+0000 7f4a7b7fe640 1 -- 172.21.15.67:0/1616852774 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c1613a0 con 0x7f4a7c0d4a50 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.278+0000 7f4a7b7fe640 1 --2- 172.21.15.67:0/1616852774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0d4e60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3b34e1392cb413e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.278+0000 7f4a7a7fc640 1 -- 172.21.15.67:0/1616852774 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a64001670 con 0x7f4a7c0d4a50 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a7a7fc640 1 -- 172.21.15.67:0/1616852774 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a74007540 con 0x7f4a7c0d4a50 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a7a7fc640 1 -- 172.21.15.67:0/1616852774 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74007910 con 0x7f4a7c0d4a50 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 -- 172.21.15.67:0/1616852774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 msgr2=0x7f4a7c0d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 --2- 172.21.15.67:0/1616852774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0d4e60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7402a5f0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 -- 172.21.15.67:0/1616852774 shutdown_connections 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 --2- 172.21.15.67:0/1616852774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0d4e60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 --2- 172.21.15.67:0/1616852774 >> v2:172.21.15.176:3300/0 conn(0x7f4a7c0d0e00 0x7f4a7c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 -- 172.21.15.67:0/1616852774 >> 172.21.15.67:0/1616852774 conn(0x7f4a7c0d5d90 msgr2=0x7f4a7c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 -- 172.21.15.67:0/1616852774 shutdown_connections 2024-04-25T15:27:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.279+0000 7f4a81a42640 1 -- 172.21.15.67:0/1616852774 wait complete. 2024-04-25T15:27:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.281+0000 7f4a81a42640 1 Processor -- start 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.281+0000 7f4a81a42640 1 -- start start 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.282+0000 7f4a81a42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c5c0 0x7f4a7c1f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.282+0000 7f4a81a42640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a7c0d0e00 0x7f4a7c0cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.282+0000 7f4a81a42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.282+0000 7f4a81a42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c164e00 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.282+0000 7f4a81a42640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c0b5320 con 0x7f4a7c0d0e00 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.282+0000 7f4a81a42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7c164f80 con 0x7f4a7c0d4a50 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c5c0 0x7f4a7c1f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c5c0 0x7f4a7c1f63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38958/0 (socket says 172.21.15.67:38958) 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 -- 172.21.15.67:0/4237816912 learned_addr learned my addr 172.21.15.67:0/4237816912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a73fff640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a7c0d0e00 0x7f4a7c0cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 -- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 msgr2=0x7f4a7c0cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 -- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a7c0d0e00 msgr2=0x7f4a7c0cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a7c0d0e00 0x7f4a7c0cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 -- 172.21.15.67:0/4237816912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c0cb930 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a7affd640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c5c0 0x7f4a7c1f63e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00e870 tx=0x7f4a6c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a78ff9640 1 -- 172.21.15.67:0/4237816912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c017070 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a78ff9640 1 -- 172.21.15.67:0/4237816912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a6c00c940 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a78ff9640 1 -- 172.21.15.67:0/4237816912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c00cd30 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7c0cbc10 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.283+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a7c0cc1d0 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.284+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a7c050740 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.285+0000 7f4a78ff9640 1 -- 172.21.15.67:0/4237816912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4a6c01a070 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.285+0000 7f4a78ff9640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a5c0744b0 0x7f4a5c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.285+0000 7f4a78ff9640 1 -- 172.21.15.67:0/4237816912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f4a6c0950d0 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.286+0000 7f4a73fff640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a5c0744b0 0x7f4a5c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.286+0000 7f4a73fff640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a5c0744b0 0x7f4a5c0769c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0d5b40 tx=0x7f4a64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.290+0000 7f4a78ff9640 1 -- 172.21.15.67:0/4237816912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a6c05f8f0 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.475+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a7c0d4320 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.475+0000 7f4a78ff9640 1 -- 172.21.15.67:0/4237816912 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7f4a6c05f1f0 con 0x7f4a7c06c5c0 2024-04-25T15:27:03.477 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:27:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a5c0744b0 msgr2=0x7f4a5c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a5c0744b0 0x7f4a5c0769c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0d5b40 tx=0x7f4a64009040 comp rx=0 tx=0).stop 2024-04-25T15:27:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c5c0 msgr2=0x7f4a7c1f63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c5c0 0x7f4a7c1f63e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00e870 tx=0x7f4a6c00ed30 comp rx=0 tx=0).stop 2024-04-25T15:27:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 shutdown_connections 2024-04-25T15:27:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a5c0744b0 0x7f4a5c0769c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a7c0d4a50 0x7f4a7c0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a7c0d0e00 0x7f4a7c0cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 --2- 172.21.15.67:0/4237816912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c5c0 0x7f4a7c1f63e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 >> 172.21.15.67:0/4237816912 conn(0x7f4a7c0d5d90 msgr2=0x7f4a7c163180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.477+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 shutdown_connections 2024-04-25T15:27:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.478+0000 7f4a81a42640 1 -- 172.21.15.67:0/4237816912 wait complete. 2024-04-25T15:27:03.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:27:03.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.603+0000 7f8155692640 1 Processor -- start 2024-04-25T15:27:03.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.603+0000 7f8155692640 1 -- start start 2024-04-25T15:27:03.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f8155692640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8150164b00 0x7f81500b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f8155692640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f8155692640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81501615f0 con 0x7f815015df80 2024-04-25T15:27:03.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f8155692640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f81501613b0 con 0x7f8150164b00 2024-04-25T15:27:03.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f8155692640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8150164670 con 0x7f8150160fa0 2024-04-25T15:27:03.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8150164b00 0x7f81500b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8150164b00 0x7f81500b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38964/0 (socket says 172.21.15.67:38964) 2024-04-25T15:27:03.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814effd640 1 -- 172.21.15.67:0/1976897722 learned_addr learned my addr 172.21.15.67:0/1976897722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814f7fe640 1 --2- 172.21.15.67:0/1976897722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814effd640 1 -- 172.21.15.67:0/1976897722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 msgr2=0x7f81500b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814effd640 1 --2- 172.21.15.67:0/1976897722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814effd640 1 -- 172.21.15.67:0/1976897722 >> v1:172.21.15.176:6789/0 conn(0x7f815015df80 legacy=0x7f8150160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814effd640 1 -- 172.21.15.67:0/1976897722 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8150164020 con 0x7f8150164b00 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.605+0000 7f814f7fe640 1 --2- 172.21.15.67:0/1976897722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.606+0000 7f814effd640 1 --2- 172.21.15.67:0/1976897722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8150164b00 0x7f81500b7b80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8148009870 tx=0x7f814802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fafe4ae8aea06d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.606+0000 7f814dffb640 1 -- 172.21.15.67:0/1976897722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8148037070 con 0x7f8150164b00 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.606+0000 7f814dffb640 1 -- 172.21.15.67:0/1976897722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8148032440 con 0x7f8150164b00 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.606+0000 7f814dffb640 1 -- 172.21.15.67:0/1976897722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8148032810 con 0x7f8150164b00 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 -- 172.21.15.67:0/1976897722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8150164b00 msgr2=0x7f81500b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 --2- 172.21.15.67:0/1976897722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8150164b00 0x7f81500b7b80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8148009870 tx=0x7f814802a210 comp rx=0 tx=0).stop 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 -- 172.21.15.67:0/1976897722 shutdown_connections 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 --2- 172.21.15.67:0/1976897722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 --2- 172.21.15.67:0/1976897722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8150164b00 0x7f81500b7b80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 -- 172.21.15.67:0/1976897722 >> 172.21.15.67:0/1976897722 conn(0x7f81500d2bc0 msgr2=0x7f81500d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:03.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 -- 172.21.15.67:0/1976897722 shutdown_connections 2024-04-25T15:27:03.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.607+0000 7f8155692640 1 -- 172.21.15.67:0/1976897722 wait complete. 2024-04-25T15:27:03.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.609+0000 7f8155692640 1 Processor -- start 2024-04-25T15:27:03.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.609+0000 7f8155692640 1 -- start start 2024-04-25T15:27:03.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f8155692640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815015df80 0x7f815015b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f8155692640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f8155692640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8150164b00 0x7f81501f9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f8155692640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81500b4a00 con 0x7f815015df80 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f8155692640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f81500b4880 con 0x7f8150164b00 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f8155692640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f81500b4b80 con 0x7f8150160fa0 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f814effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815015df80 0x7f815015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f814effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815015df80 0x7f815015b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38976/0 (socket says 172.21.15.67:38976) 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f814effd640 1 -- 172.21.15.67:0/1432858187 learned_addr learned my addr 172.21.15.67:0/1432858187 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f814f7fe640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8150164b00 0x7f81501f9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.610+0000 7f814e7fc640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f814effd640 1 -- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 msgr2=0x7f81500cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f814effd640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f814effd640 1 -- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8150164b00 msgr2=0x7f81501f9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f814effd640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8150164b00 0x7f81501f9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f814effd640 1 -- 172.21.15.67:0/1432858187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815015b8c0 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f814effd640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815015df80 0x7f815015b380 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8148009870 tx=0x7f81480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f81477fe640 1 -- 172.21.15.67:0/1432858187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8148041020 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f81477fe640 1 -- 172.21.15.67:0/1432858187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8148030970 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81500ce880 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f81477fe640 1 -- 172.21.15.67:0/1432858187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8148037040 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.611+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81500cee40 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.612+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8150050740 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.613+0000 7f81477fe640 1 -- 172.21.15.67:0/1432858187 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8148044020 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.613+0000 7f81477fe640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f81300744b0 0x7f81300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.613+0000 7f81477fe640 1 -- 172.21.15.67:0/1432858187 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f81480b5aa0 con 0x7f815015df80 2024-04-25T15:27:03.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.614+0000 7f814e7fc640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f81300744b0 0x7f81300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.614+0000 7f814e7fc640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f81300744b0 0x7f81300769c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f81500cdb40 tx=0x7f8138009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.618+0000 7f81477fe640 1 -- 172.21.15.67:0/1432858187 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f814804b030 con 0x7f815015df80 2024-04-25T15:27:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.801+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f815015cf10 con 0x7f815015df80 2024-04-25T15:27:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.801+0000 7f81477fe640 1 -- 172.21.15.67:0/1432858187 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7f81480882b0 con 0x7f815015df80 2024-04-25T15:27:03.803 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:27:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f81300744b0 msgr2=0x7f81300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f81300744b0 0x7f81300769c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f81500cdb40 tx=0x7f8138009060 comp rx=0 tx=0).stop 2024-04-25T15:27:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815015df80 msgr2=0x7f815015b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815015df80 0x7f815015b380 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8148009870 tx=0x7f81480046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 shutdown_connections 2024-04-25T15:27:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f81300744b0 0x7f81300769c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8150164b00 0x7f81501f9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8150160fa0 0x7f81500cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 --2- 172.21.15.67:0/1432858187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f815015df80 0x7f815015b380 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 >> 172.21.15.67:0/1432858187 conn(0x7f81500d2bc0 msgr2=0x7f8150160890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:03.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 shutdown_connections 2024-04-25T15:27:03.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.803+0000 7f8155692640 1 -- 172.21.15.67:0/1432858187 wait complete. 2024-04-25T15:27:03.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:03.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.927+0000 7f17a6614640 1 Processor -- start 2024-04-25T15:27:03.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.928+0000 7f17a6614640 1 -- start start 2024-04-25T15:27:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a6614640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f17a0164b00 0x7f17a00b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a6614640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a015df80 0x7f17a00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a6614640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17a0160890 0x7f17a0160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a6614640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f17a0164670 con 0x7f17a0160890 2024-04-25T15:27:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a6614640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f17a01615f0 con 0x7f17a0164b00 2024-04-25T15:27:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a6614640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17a0164020 con 0x7f17a015df80 2024-04-25T15:27:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17a0160890 0x7f17a0160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17a0160890 0x7f17a0160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38984/0 (socket says 172.21.15.67:38984) 2024-04-25T15:27:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 -- 172.21.15.67:0/2231063856 learned_addr learned my addr 172.21.15.67:0/2231063856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f179ffff640 1 --2- 172.21.15.67:0/2231063856 >> v2:172.21.15.176:3300/0 conn(0x7f17a0164b00 0x7f17a00b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f179f7fe640 1 --2- 172.21.15.67:0/2231063856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a015df80 0x7f17a00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 -- 172.21.15.67:0/2231063856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a015df80 msgr2=0x7f17a00b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 --2- 172.21.15.67:0/2231063856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a015df80 0x7f17a00b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 -- 172.21.15.67:0/2231063856 >> v2:172.21.15.176:3300/0 conn(0x7f17a0164b00 msgr2=0x7f17a00b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 --2- 172.21.15.67:0/2231063856 >> v2:172.21.15.176:3300/0 conn(0x7f17a0164b00 0x7f17a00b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f17a4bb1640 1 -- 172.21.15.67:0/2231063856 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a0164300 con 0x7f17a0160890 2024-04-25T15:27:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f179ffff640 1 --2- 172.21.15.67:0/2231063856 >> v2:172.21.15.176:3300/0 conn(0x7f17a0164b00 0x7f17a00b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.929+0000 7f179f7fe640 1 --2- 172.21.15.67:0/2231063856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a015df80 0x7f17a00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.930+0000 7f17a4bb1640 1 --2- 172.21.15.67:0/2231063856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a0160890 0x7f17a0160ca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f17a008e410 tx=0x7f179802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f9bd184df69c11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.930+0000 7f179effd640 1 -- 172.21.15.67:0/2231063856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798036070 con 0x7f17a0160890 2024-04-25T15:27:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.930+0000 7f179effd640 1 -- 172.21.15.67:0/2231063856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1798031600 con 0x7f17a0160890 2024-04-25T15:27:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.930+0000 7f179effd640 1 -- 172.21.15.67:0/2231063856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17980319d0 con 0x7f17a0160890 2024-04-25T15:27:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 -- 172.21.15.67:0/2231063856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a0160890 msgr2=0x7f17a0160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 --2- 172.21.15.67:0/2231063856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a0160890 0x7f17a0160ca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f17a008e410 tx=0x7f179802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 -- 172.21.15.67:0/2231063856 shutdown_connections 2024-04-25T15:27:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 --2- 172.21.15.67:0/2231063856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a0160890 0x7f17a0160ca0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 --2- 172.21.15.67:0/2231063856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a015df80 0x7f17a00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 --2- 172.21.15.67:0/2231063856 >> v2:172.21.15.176:3300/0 conn(0x7f17a0164b00 0x7f17a00b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 -- 172.21.15.67:0/2231063856 >> 172.21.15.67:0/2231063856 conn(0x7f17a00d2bc0 msgr2=0x7f17a00d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 -- 172.21.15.67:0/2231063856 shutdown_connections 2024-04-25T15:27:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.931+0000 7f17a6614640 1 -- 172.21.15.67:0/2231063856 wait complete. 2024-04-25T15:27:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.933+0000 7f17a6614640 1 Processor -- start 2024-04-25T15:27:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.933+0000 7f17a6614640 1 -- start start 2024-04-25T15:27:03.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f17a6614640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a015df80 0x7f17a01eb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f17a6614640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a0160890 0x7f17a01f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f17a6614640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a0164b00 0x7f17a01f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f17a6614640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17a00b4a00 con 0x7f17a015df80 2024-04-25T15:27:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f17a6614640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f17a00b4880 con 0x7f17a0160890 2024-04-25T15:27:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f17a6614640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17a00b4b80 con 0x7f17a0164b00 2024-04-25T15:27:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f179ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a015df80 0x7f17a01eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f179ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a015df80 0x7f17a01eb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38990/0 (socket says 172.21.15.67:38990) 2024-04-25T15:27:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.934+0000 7f179ffff640 1 -- 172.21.15.67:0/1272427877 learned_addr learned my addr 172.21.15.67:0/1272427877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f17a4bb1640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a0164b00 0x7f17a01f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179f7fe640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a0160890 0x7f17a01f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179ffff640 1 -- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a0164b00 msgr2=0x7f17a01f9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179ffff640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a0164b00 0x7f17a01f9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179ffff640 1 -- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a0160890 msgr2=0x7f17a01f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179ffff640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a0160890 0x7f17a01f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179ffff640 1 -- 172.21.15.67:0/1272427877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a01ea0e0 con 0x7f17a015df80 2024-04-25T15:27:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f17a4bb1640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a0164b00 0x7f17a01f9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179ffff640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a015df80 0x7f17a01eb180 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f179000d800 tx=0x7f179000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179cff9640 1 -- 172.21.15.67:0/1272427877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790004030 con 0x7f17a015df80 2024-04-25T15:27:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179cff9640 1 -- 172.21.15.67:0/1272427877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17900041d0 con 0x7f17a015df80 2024-04-25T15:27:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a01ea3c0 con 0x7f17a015df80 2024-04-25T15:27:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f179cff9640 1 -- 172.21.15.67:0/1272427877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17900096a0 con 0x7f17a015df80 2024-04-25T15:27:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.935+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a01ea980 con 0x7f17a015df80 2024-04-25T15:27:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.936+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a0050740 con 0x7f17a015df80 2024-04-25T15:27:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.937+0000 7f179cff9640 1 -- 172.21.15.67:0/1272427877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1790009840 con 0x7f17a015df80 2024-04-25T15:27:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.937+0000 7f179cff9640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f178c0744b0 0x7f178c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.937+0000 7f179cff9640 1 -- 172.21.15.67:0/1272427877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f1790095430 con 0x7f17a015df80 2024-04-25T15:27:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.938+0000 7f179f7fe640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f178c0744b0 0x7f178c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.938+0000 7f179f7fe640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f178c0744b0 0x7f178c0769c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1794006fd0 tx=0x7f1794008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:03.942+0000 7f179cff9640 1 -- 172.21.15.67:0/1272427877 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f179005fce0 con 0x7f17a015df80 2024-04-25T15:27:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.165+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17a00d4170 con 0x7f17a015df80 2024-04-25T15:27:04.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.166+0000 7f179cff9640 1 -- 172.21.15.67:0/1272427877 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5187 (secure 0 0 0) 0x7f179005f5e0 con 0x7f17a015df80 2024-04-25T15:27:04.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:04.168 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1556919488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1556919488},{"type":"v1","addr":"172.21.15.67:6833","nonce":1556919488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4071024412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4071024412},{"type":"v1","addr":"172.21.15.176:6837","nonce":4071024412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3791679818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3791679818},{"type":"v1","addr":"172.21.15.176:6836","nonce":3791679818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:49.120424+0000","modified":"2024-04-25T15:26:52.141230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1882238407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1882238407},{"type":"v1","addr":"172.21.15.67:6835","nonce":1882238407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":3}]} 2024-04-25T15:27:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-04-25T15:27:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f178c0744b0 msgr2=0x7f178c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f178c0744b0 0x7f178c0769c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1794006fd0 tx=0x7f1794008040 comp rx=0 tx=0).stop 2024-04-25T15:27:04.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a015df80 msgr2=0x7f17a01eb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a015df80 0x7f17a01eb180 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f179000d800 tx=0x7f179000dcc0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 shutdown_connections 2024-04-25T15:27:04.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f178c0744b0 0x7f178c0769c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a0164b00 0x7f17a01f9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a0160890 0x7f17a01f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 --2- 172.21.15.67:0/1272427877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a015df80 0x7f17a01eb180 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 >> 172.21.15.67:0/1272427877 conn(0x7f17a00d2bc0 msgr2=0x7f17a00d5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:04.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 shutdown_connections 2024-04-25T15:27:04.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.168+0000 7f17a6614640 1 -- 172.21.15.67:0/1272427877 wait complete. 2024-04-25T15:27:04.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.292+0000 7f3d3a4f2640 1 Processor -- start 2024-04-25T15:27:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.292+0000 7f3d3a4f2640 1 -- start start 2024-04-25T15:27:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.293+0000 7f3d3a4f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d341612f0 0x7f3d340d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.293+0000 7f3d3a4f2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3406f3e0 con 0x7f3d341434d0 2024-04-25T15:27:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.293+0000 7f3d3a4f2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3406f6c0 con 0x7f3d340d0ed0 2024-04-25T15:27:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.293+0000 7f3d3a4f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d341438e0 con 0x7f3d341612f0 2024-04-25T15:27:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d33fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d341434d0 0x7f3d34070480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57102/0 (socket says 172.21.15.67:57102) 2024-04-25T15:27:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d33fff640 1 -- 172.21.15.67:0/970113218 learned_addr learned my addr 172.21.15.67:0/970113218 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d38a8f640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d341612f0 0x7f3d340d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083880558 0 0) 0x7f3d3406f3e0 con 0x7f3d341434d0 2024-04-25T15:27:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d20005730 con 0x7f3d341434d0 2024-04-25T15:27:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1120823695 0 0) 0x7f3d20005730 con 0x7f3d341434d0 2024-04-25T15:27:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d341612f0 msgr2=0x7f3d340d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d341612f0 0x7f3d340d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 >> v1:172.21.15.176:6789/0 conn(0x7f3d340d0ed0 legacy=0x7f3d34160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3406f9a0 con 0x7f3d341434d0 2024-04-25T15:27:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f3d1c001670 con 0x7f3d341434d0 2024-04-25T15:27:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 >> v1:172.21.15.67:6789/0 conn(0x7f3d341434d0 legacy=0x7f3d34070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20005bf0 0x7f3d20006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d20006a10 0x7f3d20006e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20007360 0x7f3d2000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341438e0 con 0x7f3d20005bf0 2024-04-25T15:27:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2000e110 con 0x7f3d20006a10 2024-04-25T15:27:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2000e290 con 0x7f3d20007360 2024-04-25T15:27:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.294+0000 7f3d33fff640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20005bf0 0x7f3d20006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.295+0000 7f3d337fe640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d20006a10 0x7f3d20006e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.295+0000 7f3d38a8f640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20007360 0x7f3d2000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.295+0000 7f3d33fff640 1 -- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20007360 msgr2=0x7f3d2000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.295+0000 7f3d33fff640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20007360 0x7f3d2000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.295+0000 7f3d33fff640 1 -- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d20006a10 msgr2=0x7f3d20006e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.295+0000 7f3d33fff640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d20006a10 0x7f3d20006e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.295+0000 7f3d33fff640 1 -- 172.21.15.67:0/970113218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d20005730 con 0x7f3d20005bf0 2024-04-25T15:27:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d33fff640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20005bf0 0x7f3d20006020 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00c160 tx=0x7f3d20007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3fa90abbe7f04c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c007ca0 con 0x7f3d20005bf0 2024-04-25T15:27:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c02e070 con 0x7f3d20005bf0 2024-04-25T15:27:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d32ffd640 1 -- 172.21.15.67:0/970113218 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c005d10 con 0x7f3d20005bf0 2024-04-25T15:27:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/970113218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20005bf0 msgr2=0x7f3d20006020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d3a4f2640 1 --2- 172.21.15.67:0/970113218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20005bf0 0x7f3d20006020 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00c160 tx=0x7f3d20007a40 comp rx=0 tx=0).stop 2024-04-25T15:27:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d38a8f640 1 -- 172.21.15.67:0/970113218 reap_dead start 2024-04-25T15:27:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/970113218 shutdown_connections 2024-04-25T15:27:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/970113218 >> 172.21.15.67:0/970113218 conn(0x7f3d340d1ae0 msgr2=0x7f3d340d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/970113218 shutdown_connections 2024-04-25T15:27:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.296+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/970113218 wait complete. 2024-04-25T15:27:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.298+0000 7f3d3a4f2640 1 Processor -- start 2024-04-25T15:27:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.298+0000 7f3d3a4f2640 1 -- start start 2024-04-25T15:27:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d3a4f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20005bf0 0x7f3d341f5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d3a4f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d34145380 0x7f3d34145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d3a4f2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340b1a80 0x7f3d340b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d3a4f2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341612f0 con 0x7f3d340b1a80 2024-04-25T15:27:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d3a4f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3406f090 con 0x7f3d34145380 2024-04-25T15:27:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d3a4f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d34161470 con 0x7f3d20005bf0 2024-04-25T15:27:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340b1a80 0x7f3d340b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340b1a80 0x7f3d340b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39010/0 (socket says 172.21.15.67:39010) 2024-04-25T15:27:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 -- 172.21.15.67:0/580786661 learned_addr learned my addr 172.21.15.67:0/580786661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d33fff640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20005bf0 0x7f3d341f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d337fe640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d34145380 0x7f3d34145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 -- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20005bf0 msgr2=0x7f3d341f5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20005bf0 0x7f3d341f5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 -- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d34145380 msgr2=0x7f3d34145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d34145380 0x7f3d34145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 -- 172.21.15.67:0/580786661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d340b23d0 con 0x7f3d340b1a80 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d38a8f640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340b1a80 0x7f3d340b1e90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3d34146e50 tx=0x7f3d2c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d30ff9640 1 -- 172.21.15.67:0/580786661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c003990 con 0x7f3d340b1a80 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.300+0000 7f3d30ff9640 1 -- 172.21.15.67:0/580786661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c01c070 con 0x7f3d340b1a80 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.301+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3415f2e0 con 0x7f3d340b1a80 2024-04-25T15:27:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.301+0000 7f3d30ff9640 1 -- 172.21.15.67:0/580786661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c003d50 con 0x7f3d340b1a80 2024-04-25T15:27:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.301+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3415f8a0 con 0x7f3d340b1a80 2024-04-25T15:27:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.302+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d34050740 con 0x7f3d340b1a80 2024-04-25T15:27:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.302+0000 7f3d30ff9640 1 -- 172.21.15.67:0/580786661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d2c004030 con 0x7f3d340b1a80 2024-04-25T15:27:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.302+0000 7f3d30ff9640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d140744e0 0x7f3d140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.303+0000 7f3d30ff9640 1 -- 172.21.15.67:0/580786661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f3d2c095c70 con 0x7f3d340b1a80 2024-04-25T15:27:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.303+0000 7f3d33fff640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d140744e0 0x7f3d140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.303+0000 7f3d33fff640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d140744e0 0x7f3d140769f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c032000 tx=0x7f3d1c007000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:04.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.307+0000 7f3d30ff9640 1 -- 172.21.15.67:0/580786661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d340b23d0 con 0x7f3d340b1a80 2024-04-25T15:27:04.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.492+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d340d23e0 con 0x7f3d340b1a80 2024-04-25T15:27:04.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.494+0000 7f3d30ff9640 1 -- 172.21.15.67:0/580786661 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16577 (secure 0 0 0) 0x7f3d2c010040 con 0x7f3d340b1a80 2024-04-25T15:27:04.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:04.496 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":44,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:50.738984+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:04.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.495+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d140744e0 msgr2=0x7f3d140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.495+0000 7f3d3a4f2640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d140744e0 0x7f3d140769f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c032000 tx=0x7f3d1c007000 comp rx=0 tx=0).stop 2024-04-25T15:27:04.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340b1a80 msgr2=0x7f3d340b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340b1a80 0x7f3d340b1e90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3d34146e50 tx=0x7f3d2c00f580 comp rx=0 tx=0).stop 2024-04-25T15:27:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 shutdown_connections 2024-04-25T15:27:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d140744e0 0x7f3d140769f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d340b1a80 0x7f3d340b1e90 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d34145380 0x7f3d34145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 --2- 172.21.15.67:0/580786661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d20005bf0 0x7f3d341f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 >> 172.21.15.67:0/580786661 conn(0x7f3d340d1ae0 msgr2=0x7f3d3415d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:04.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 shutdown_connections 2024-04-25T15:27:04.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.496+0000 7f3d3a4f2640 1 -- 172.21.15.67:0/580786661 wait complete. 2024-04-25T15:27:04.509 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:27:04.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:04.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.621+0000 7f54d92b6640 1 Processor -- start 2024-04-25T15:27:04.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.621+0000 7f54d92b6640 1 -- start start 2024-04-25T15:27:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d92b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40cd050 0x7f54d40b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d92b6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f54d415d400 0x7f54d40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d92b6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54d40b7ca0 0x7f54d406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d92b6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f54d40b80b0 con 0x7f54d40b7ca0 2024-04-25T15:27:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d92b6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f54d40d0010 con 0x7f54d415d400 2024-04-25T15:27:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d92b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f54d40d02f0 con 0x7f54d40cd050 2024-04-25T15:27:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54d40b7ca0 0x7f54d406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54d40b7ca0 0x7f54d406e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39018/0 (socket says 172.21.15.67:39018) 2024-04-25T15:27:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.622+0000 7f54d37fe640 1 -- 172.21.15.67:0/4024383173 learned_addr learned my addr 172.21.15.67:0/4024383173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54cbfff640 1 --2- 172.21.15.67:0/4024383173 >> v2:172.21.15.176:3300/0 conn(0x7f54d415d400 0x7f54d40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54d2ffd640 1 --2- 172.21.15.67:0/4024383173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40cd050 0x7f54d40b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54d37fe640 1 -- 172.21.15.67:0/4024383173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40cd050 msgr2=0x7f54d40b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:04.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54d37fe640 1 --2- 172.21.15.67:0/4024383173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40cd050 0x7f54d40b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54d37fe640 1 -- 172.21.15.67:0/4024383173 >> v2:172.21.15.176:3300/0 conn(0x7f54d415d400 msgr2=0x7f54d40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:04.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54d37fe640 1 --2- 172.21.15.67:0/4024383173 >> v2:172.21.15.176:3300/0 conn(0x7f54d415d400 0x7f54d40b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54d37fe640 1 -- 172.21.15.67:0/4024383173 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d406d770 con 0x7f54d40b7ca0 2024-04-25T15:27:04.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.623+0000 7f54d37fe640 1 --2- 172.21.15.67:0/4024383173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d40b7ca0 0x7f54d406e250 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f54d40b8ab0 tx=0x7f54cc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f8a25aa2f390b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:04.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d27fc640 1 -- 172.21.15.67:0/4024383173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc036070 con 0x7f54d40b7ca0 2024-04-25T15:27:04.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d27fc640 1 -- 172.21.15.67:0/4024383173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54cc031600 con 0x7f54d40b7ca0 2024-04-25T15:27:04.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d27fc640 1 -- 172.21.15.67:0/4024383173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0319d0 con 0x7f54d40b7ca0 2024-04-25T15:27:04.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 -- 172.21.15.67:0/4024383173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d40b7ca0 msgr2=0x7f54d406e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 --2- 172.21.15.67:0/4024383173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d40b7ca0 0x7f54d406e250 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f54d40b8ab0 tx=0x7f54cc02a6a0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 -- 172.21.15.67:0/4024383173 shutdown_connections 2024-04-25T15:27:04.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 --2- 172.21.15.67:0/4024383173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d40b7ca0 0x7f54d406e250 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 --2- 172.21.15.67:0/4024383173 >> v2:172.21.15.176:3300/0 conn(0x7f54d415d400 0x7f54d40b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 --2- 172.21.15.67:0/4024383173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40cd050 0x7f54d40b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 -- 172.21.15.67:0/4024383173 >> 172.21.15.67:0/4024383173 conn(0x7f54d409b7d0 msgr2=0x7f54d409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:04.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 -- 172.21.15.67:0/4024383173 shutdown_connections 2024-04-25T15:27:04.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.624+0000 7f54d92b6640 1 -- 172.21.15.67:0/4024383173 wait complete. 2024-04-25T15:27:04.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.626+0000 7f54d92b6640 1 Processor -- start 2024-04-25T15:27:04.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.626+0000 7f54d92b6640 1 -- start start 2024-04-25T15:27:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d92b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40b7ca0 0x7f54d40c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d92b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54d40cd050 0x7f54d40c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d92b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d415d400 0x7f54d415f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d92b6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54d40b2920 con 0x7f54d415d400 2024-04-25T15:27:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d92b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f54d406dd30 con 0x7f54d40cd050 2024-04-25T15:27:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d92b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f54d40cfc90 con 0x7f54d40b7ca0 2024-04-25T15:27:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d415d400 0x7f54d415f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d415d400 0x7f54d415f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39022/0 (socket says 172.21.15.67:39022) 2024-04-25T15:27:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 -- 172.21.15.67:0/7884010 learned_addr learned my addr 172.21.15.67:0/7884010 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d2ffd640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40b7ca0 0x7f54d40c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54cbfff640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54d40cd050 0x7f54d40c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 -- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40b7ca0 msgr2=0x7f54d40c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40b7ca0 0x7f54d40c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 -- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54d40cd050 msgr2=0x7f54d40c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54d40cd050 0x7f54d40c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 -- 172.21.15.67:0/7884010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d40d2a10 con 0x7f54d415d400 2024-04-25T15:27:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d37fe640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d415d400 0x7f54d415f6d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f54cc034040 tx=0x7f54cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54cb7fe640 1 -- 172.21.15.67:0/7884010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc036050 con 0x7f54d415d400 2024-04-25T15:27:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54cb7fe640 1 -- 172.21.15.67:0/7884010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54cc042070 con 0x7f54d415d400 2024-04-25T15:27:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d40d2170 con 0x7f54d415d400 2024-04-25T15:27:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.628+0000 7f54cb7fe640 1 -- 172.21.15.67:0/7884010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc009510 con 0x7f54d415d400 2024-04-25T15:27:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.629+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d40d2730 con 0x7f54d415d400 2024-04-25T15:27:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.629+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d4050740 con 0x7f54d415d400 2024-04-25T15:27:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.630+0000 7f54cb7fe640 1 -- 172.21.15.67:0/7884010 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f54cc007bd0 con 0x7f54d415d400 2024-04-25T15:27:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.631+0000 7f54cb7fe640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54b40744e0 0x7f54b40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.631+0000 7f54cb7fe640 1 -- 172.21.15.67:0/7884010 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f54cc0b6050 con 0x7f54d415d400 2024-04-25T15:27:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.631+0000 7f54d2ffd640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54b40744e0 0x7f54b40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.631+0000 7f54d2ffd640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54b40744e0 0x7f54b40769f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f54c4006fd0 tx=0x7f54c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:04.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.635+0000 7f54cb7fe640 1 -- 172.21.15.67:0/7884010 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54cc080870 con 0x7f54d415d400 2024-04-25T15:27:04.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.860+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54d406c6d0 con 0x7f54d415d400 2024-04-25T15:27:04.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.860+0000 7f54cb7fe640 1 -- 172.21.15.67:0/7884010 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5187 (secure 0 0 0) 0x7f54cc080170 con 0x7f54d415d400 2024-04-25T15:27:04.862 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:04.862 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1556919488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1556919488},{"type":"v1","addr":"172.21.15.67:6833","nonce":1556919488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4071024412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4071024412},{"type":"v1","addr":"172.21.15.176:6837","nonce":4071024412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3791679818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3791679818},{"type":"v1","addr":"172.21.15.176:6836","nonce":3791679818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:49.120424+0000","modified":"2024-04-25T15:26:52.141230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1882238407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1882238407},{"type":"v1","addr":"172.21.15.67:6835","nonce":1882238407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":3}]} 2024-04-25T15:27:04.863 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-04-25T15:27:04.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.862+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54b40744e0 msgr2=0x7f54b40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.862+0000 7f54d92b6640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54b40744e0 0x7f54b40769f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f54c4006fd0 tx=0x7f54c4008040 comp rx=0 tx=0).stop 2024-04-25T15:27:04.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d415d400 msgr2=0x7f54d415f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d415d400 0x7f54d415f6d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f54cc034040 tx=0x7f54cc004a40 comp rx=0 tx=0).stop 2024-04-25T15:27:04.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 shutdown_connections 2024-04-25T15:27:04.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54b40744e0 0x7f54b40769f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d415d400 0x7f54d415f6d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54d40cd050 0x7f54d40c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 --2- 172.21.15.67:0/7884010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54d40b7ca0 0x7f54d40c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 >> 172.21.15.67:0/7884010 conn(0x7f54d409b7d0 msgr2=0x7f54d40b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:04.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 shutdown_connections 2024-04-25T15:27:04.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.863+0000 7f54d92b6640 1 -- 172.21.15.67:0/7884010 wait complete. 2024-04-25T15:27:04.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.987+0000 7f158dbba640 1 Processor -- start 2024-04-25T15:27:04.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.987+0000 7f158dbba640 1 -- start start 2024-04-25T15:27:04.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.988+0000 7f158dbba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15880b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.988+0000 7f158dbba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f158815df80 0x7f15880b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f158dbba640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1588160890 0x7f1588160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:04.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f158dbba640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1588164670 con 0x7f158815df80 2024-04-25T15:27:04.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f158dbba640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f15881615f0 con 0x7f1588160890 2024-04-25T15:27:04.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f158dbba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1588164020 con 0x7f1588164b00 2024-04-25T15:27:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f158815df80 0x7f15880b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f158815df80 0x7f15880b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39036/0 (socket says 172.21.15.67:39036) 2024-04-25T15:27:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 -- 172.21.15.67:0/3489291936 learned_addr learned my addr 172.21.15.67:0/3489291936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1587fff640 1 --2- 172.21.15.67:0/3489291936 >> v2:172.21.15.176:3300/0 conn(0x7f1588160890 0x7f1588160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f15877fe640 1 --2- 172.21.15.67:0/3489291936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15880b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 -- 172.21.15.67:0/3489291936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 msgr2=0x7f15880b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 --2- 172.21.15.67:0/3489291936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15880b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 -- 172.21.15.67:0/3489291936 >> v2:172.21.15.176:3300/0 conn(0x7f1588160890 msgr2=0x7f1588160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 --2- 172.21.15.67:0/3489291936 >> v2:172.21.15.176:3300/0 conn(0x7f1588160890 0x7f1588160ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f1586ffd640 1 -- 172.21.15.67:0/3489291936 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1588164300 con 0x7f158815df80 2024-04-25T15:27:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.989+0000 7f15877fe640 1 --2- 172.21.15.67:0/3489291936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15880b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.990+0000 7f1586ffd640 1 --2- 172.21.15.67:0/3489291936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f15880b80c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f15880b87a0 tx=0x7f157c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadd7daf1b5a9439 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.990+0000 7f15867fc640 1 -- 172.21.15.67:0/3489291936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c037070 con 0x7f158815df80 2024-04-25T15:27:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.990+0000 7f15867fc640 1 -- 172.21.15.67:0/3489291936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f157c032440 con 0x7f158815df80 2024-04-25T15:27:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.990+0000 7f15867fc640 1 -- 172.21.15.67:0/3489291936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c032810 con 0x7f158815df80 2024-04-25T15:27:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 -- 172.21.15.67:0/3489291936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 msgr2=0x7f15880b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 --2- 172.21.15.67:0/3489291936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f15880b80c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f15880b87a0 tx=0x7f157c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 -- 172.21.15.67:0/3489291936 shutdown_connections 2024-04-25T15:27:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 --2- 172.21.15.67:0/3489291936 >> v2:172.21.15.176:3300/0 conn(0x7f1588160890 0x7f1588160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 --2- 172.21.15.67:0/3489291936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f15880b80c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 --2- 172.21.15.67:0/3489291936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15880b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 -- 172.21.15.67:0/3489291936 >> 172.21.15.67:0/3489291936 conn(0x7f15880d2bc0 msgr2=0x7f15880d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 -- 172.21.15.67:0/3489291936 shutdown_connections 2024-04-25T15:27:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.991+0000 7f158dbba640 1 -- 172.21.15.67:0/3489291936 wait complete. 2024-04-25T15:27:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.992+0000 7f158dbba640 1 Processor -- start 2024-04-25T15:27:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.993+0000 7f158dbba640 1 -- start start 2024-04-25T15:27:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f158dbba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f1588151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f158dbba640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1588160890 0x7f1588150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f158dbba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15881ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f158dbba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15880b4a00 con 0x7f158815df80 2024-04-25T15:27:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f158dbba640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15880b4880 con 0x7f1588160890 2024-04-25T15:27:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f158dbba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15880b4b80 con 0x7f1588164b00 2024-04-25T15:27:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f15877fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f1588151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f15877fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f1588151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39050/0 (socket says 172.21.15.67:39050) 2024-04-25T15:27:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f15877fe640 1 -- 172.21.15.67:0/385101 learned_addr learned my addr 172.21.15.67:0/385101 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f1586ffd640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1588160890 0x7f1588150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f1587fff640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15881ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f15877fe640 1 -- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 msgr2=0x7f15881ea7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f15877fe640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15881ea7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f15877fe640 1 -- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1588160890 msgr2=0x7f1588150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.994+0000 7f15877fe640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1588160890 0x7f1588150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f15877fe640 1 -- 172.21.15.67:0/385101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15881546a0 con 0x7f158815df80 2024-04-25T15:27:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f1586ffd640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1588160890 0x7f1588150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f15877fe640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f1588151d50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f157800d7d0 tx=0x7f157800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f156ffff640 1 -- 172.21.15.67:0/385101 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578004030 con 0x7f158815df80 2024-04-25T15:27:05.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f156ffff640 1 -- 172.21.15.67:0/385101 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15780041d0 con 0x7f158815df80 2024-04-25T15:27:05.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f156ffff640 1 -- 172.21.15.67:0/385101 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780096a0 con 0x7f158815df80 2024-04-25T15:27:05.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1588154980 con 0x7f158815df80 2024-04-25T15:27:05.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.995+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1588154f40 con 0x7f158815df80 2024-04-25T15:27:05.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.996+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1588050740 con 0x7f158815df80 2024-04-25T15:27:05.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.997+0000 7f156ffff640 1 -- 172.21.15.67:0/385101 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1578009840 con 0x7f158815df80 2024-04-25T15:27:05.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.997+0000 7f156ffff640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15740744b0 0x7f15740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.997+0000 7f156ffff640 1 -- 172.21.15.67:0/385101 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f1578095510 con 0x7f158815df80 2024-04-25T15:27:05.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.997+0000 7f1586ffd640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15740744b0 0x7f15740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:04.998+0000 7f1586ffd640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15740744b0 0x7f15740769c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f157c034000 tx=0x7f157c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.002+0000 7f156ffff640 1 -- 172.21.15.67:0/385101 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f157805fd30 con 0x7f158815df80 2024-04-25T15:27:05.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.229+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15880d4170 con 0x7f158815df80 2024-04-25T15:27:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.229+0000 7f156ffff640 1 -- 172.21.15.67:0/385101 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5187 (secure 0 0 0) 0x7f157805f630 con 0x7f158815df80 2024-04-25T15:27:05.230 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:05.230 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1556919488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1556919488},{"type":"v1","addr":"172.21.15.67:6833","nonce":1556919488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4071024412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4071024412},{"type":"v1","addr":"172.21.15.176:6837","nonce":4071024412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3791679818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3791679818},{"type":"v1","addr":"172.21.15.176:6836","nonce":3791679818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:26:49.120424+0000","modified":"2024-04-25T15:26:52.141230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1882238407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1882238407},{"type":"v1","addr":"172.21.15.67:6835","nonce":1882238407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":3}]} 2024-04-25T15:27:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-04-25T15:27:05.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15740744b0 msgr2=0x7f15740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15740744b0 0x7f15740769c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f157c034000 tx=0x7f157c02e000 comp rx=0 tx=0).stop 2024-04-25T15:27:05.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 msgr2=0x7f1588151d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f1588151d50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f157800d7d0 tx=0x7f157800dc90 comp rx=0 tx=0).stop 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 shutdown_connections 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f15740744b0 0x7f15740769c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1588164b00 0x7f15881ea7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1588160890 0x7f1588150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 --2- 172.21.15.67:0/385101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f158815df80 0x7f1588151d50 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 >> 172.21.15.67:0/385101 conn(0x7f15880d2bc0 msgr2=0x7f158815e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 shutdown_connections 2024-04-25T15:27:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.231+0000 7f158dbba640 1 -- 172.21.15.67:0/385101 wait complete. 2024-04-25T15:27:05.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.354+0000 7fc7aa830640 1 Processor -- start 2024-04-25T15:27:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.354+0000 7fc7aa830640 1 -- start start 2024-04-25T15:27:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7aa830640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7a406d560 0x7fc7a415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7aa830640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc7a415a2f0 0x7fc7a40b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7aa830640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7a41611e0 con 0x7fc7a40b6a00 2024-04-25T15:27:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7aa830640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc7a4160d50 con 0x7fc7a406d560 2024-04-25T15:27:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7aa830640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc7a415a700 con 0x7fc7a415a2f0 2024-04-25T15:27:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7a406d560 0x7fc7a415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc7a40b6a00 0x7fc7a406ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57116/0 (socket says 172.21.15.67:57116) 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7a406d560 0x7fc7a415d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39056/0 (socket says 172.21.15.67:39056) 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a3fff640 1 -- 172.21.15.67:0/2441715117 learned_addr learned my addr 172.21.15.67:0/2441715117 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a8dcd640 1 --2- 172.21.15.67:0/2441715117 >> v2:172.21.15.176:3300/0 conn(0x7fc7a415a2f0 0x7fc7a40b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a37fe640 1 -- 172.21.15.67:0/2441715117 >> v2:172.21.15.176:3300/0 conn(0x7fc7a415a2f0 msgr2=0x7fc7a40b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a37fe640 1 --2- 172.21.15.67:0/2441715117 >> v2:172.21.15.176:3300/0 conn(0x7fc7a415a2f0 0x7fc7a40b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a37fe640 1 -- 172.21.15.67:0/2441715117 >> v1:172.21.15.67:6789/0 conn(0x7fc7a40b6a00 legacy=0x7fc7a406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a37fe640 1 -- 172.21.15.67:0/2441715117 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a4160420 con 0x7fc7a406d560 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a2ffd640 1 -- 172.21.15.67:0/2441715117 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735206578 0 0) 0x7fc7a41611e0 con 0x7fc7a40b6a00 2024-04-25T15:27:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.356+0000 7fc7a8dcd640 1 --2- 172.21.15.67:0/2441715117 >> v2:172.21.15.176:3300/0 conn(0x7fc7a415a2f0 0x7fc7a40b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.357+0000 7fc7a37fe640 1 --2- 172.21.15.67:0/2441715117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a415d6c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc7a415cc90 tx=0x7fc79402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ad30e3ae1526d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.357+0000 7fc7a2ffd640 1 -- 172.21.15.67:0/2441715117 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794037070 con 0x7fc7a406d560 2024-04-25T15:27:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.357+0000 7fc7a2ffd640 1 -- 172.21.15.67:0/2441715117 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc794032440 con 0x7fc7a406d560 2024-04-25T15:27:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.357+0000 7fc7a2ffd640 1 -- 172.21.15.67:0/2441715117 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794032810 con 0x7fc7a406d560 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 -- 172.21.15.67:0/2441715117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 msgr2=0x7fc7a415d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2441715117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a415d6c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc7a415cc90 tx=0x7fc79402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 -- 172.21.15.67:0/2441715117 shutdown_connections 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2441715117 >> v2:172.21.15.176:3300/0 conn(0x7fc7a415a2f0 0x7fc7a40b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2441715117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a415d6c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 -- 172.21.15.67:0/2441715117 >> 172.21.15.67:0/2441715117 conn(0x7fc7a40d1ae0 msgr2=0x7fc7a40d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 -- 172.21.15.67:0/2441715117 shutdown_connections 2024-04-25T15:27:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.358+0000 7fc7aa830640 1 -- 172.21.15.67:0/2441715117 wait complete. 2024-04-25T15:27:05.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.360+0000 7fc7aa830640 1 Processor -- start 2024-04-25T15:27:05.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.360+0000 7fc7aa830640 1 -- start start 2024-04-25T15:27:05.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7aa830640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a4168f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7aa830640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7a40b6a00 0x7fc7a415ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7aa830640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a415a2f0 0x7fc7a41572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7aa830640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a40ce580 con 0x7fc7a406d560 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7aa830640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a41609e0 con 0x7fc7a40b6a00 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7aa830640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a40ce700 con 0x7fc7a415a2f0 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a4168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a4168f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39068/0 (socket says 172.21.15.67:39068) 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.361+0000 7fc7a3fff640 1 -- 172.21.15.67:0/2958394330 learned_addr learned my addr 172.21.15.67:0/2958394330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a37fe640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7a40b6a00 0x7fc7a415ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a8dcd640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a415a2f0 0x7fc7a41572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a3fff640 1 -- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a415a2f0 msgr2=0x7fc7a41572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a3fff640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a415a2f0 0x7fc7a41572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a3fff640 1 -- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7a40b6a00 msgr2=0x7fc7a415ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a3fff640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7a40b6a00 0x7fc7a415ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a3fff640 1 -- 172.21.15.67:0/2958394330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a4167e80 con 0x7fc7a406d560 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a37fe640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7a40b6a00 0x7fc7a415ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:05.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a3fff640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a4168f20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c410 tx=0x7fc78c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a0ff9640 1 -- 172.21.15.67:0/2958394330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0074e0 con 0x7fc7a406d560 2024-04-25T15:27:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a0ff9640 1 -- 172.21.15.67:0/2958394330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc78c0047e0 con 0x7fc7a406d560 2024-04-25T15:27:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7a0ff9640 1 -- 172.21.15.67:0/2958394330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c004bd0 con 0x7fc7a406d560 2024-04-25T15:27:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a4168160 con 0x7fc7a406d560 2024-04-25T15:27:05.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.362+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a4168720 con 0x7fc7a406d560 2024-04-25T15:27:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.363+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a40cda50 con 0x7fc7a406d560 2024-04-25T15:27:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.364+0000 7fc7a0ff9640 1 -- 172.21.15.67:0/2958394330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc78c011070 con 0x7fc7a406d560 2024-04-25T15:27:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.364+0000 7fc7a0ff9640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7840744b0 0x7fc7840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.364+0000 7fc7a0ff9640 1 -- 172.21.15.67:0/2958394330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fc78c094eb0 con 0x7fc7a406d560 2024-04-25T15:27:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.364+0000 7fc7a37fe640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7840744b0 0x7fc7840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.365+0000 7fc7a37fe640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7840744b0 0x7fc7840769c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc794034000 tx=0x7fc79402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.369+0000 7fc7a0ff9640 1 -- 172.21.15.67:0/2958394330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc78c05f640 con 0x7fc7a406d560 2024-04-25T15:27:05.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.558+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7a40b6180 con 0x7fc7a406d560 2024-04-25T15:27:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.560+0000 7fc7a0ff9640 1 -- 172.21.15.67:0/2958394330 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16577 (secure 0 0 0) 0x7fc78c05ef40 con 0x7fc7a406d560 2024-04-25T15:27:05.561 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:05.561 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":44,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:26:50.738984+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:26:47.438381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-25T15:26:48.064334+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7840744b0 msgr2=0x7fc7840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7840744b0 0x7fc7840769c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc794034000 tx=0x7fc79402e000 comp rx=0 tx=0).stop 2024-04-25T15:27:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 msgr2=0x7fc7a4168f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a4168f20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00c410 tx=0x7fc78c00c8d0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 shutdown_connections 2024-04-25T15:27:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7840744b0 0x7fc7840769c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a415a2f0 0x7fc7a41572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7a40b6a00 0x7fc7a415ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 --2- 172.21.15.67:0/2958394330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7a406d560 0x7fc7a4168f20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 >> 172.21.15.67:0/2958394330 conn(0x7fc7a40d1ae0 msgr2=0x7fc7a40684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 shutdown_connections 2024-04-25T15:27:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.562+0000 7fc7aa830640 1 -- 172.21.15.67:0/2958394330 wait complete. 2024-04-25T15:27:05.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:27:05.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.687+0000 7f4e86ed8640 1 Processor -- start 2024-04-25T15:27:05.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.687+0000 7f4e86ed8640 1 -- start start 2024-04-25T15:27:05.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e86ed8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e80164b00 0x7f4e800b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e86ed8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4e80160fa0 0x7f4e800b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e86ed8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e801615f0 con 0x7f4e8015df80 2024-04-25T15:27:05.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e86ed8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4e801613b0 con 0x7f4e80160fa0 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e86ed8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4e80164670 con 0x7f4e80164b00 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e84c74640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e80164b00 0x7f4e800b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e85475640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4e80160fa0 0x7f4e800b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e84c74640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e80164b00 0x7f4e800b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:52864/0 (socket says 172.21.15.67:52864) 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e7ffff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f4e8015df80 0x7f4e80160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:55610/0 (socket says 172.21.15.67:55610) 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e84c74640 1 -- 172.21.15.67:0/1034758047 learned_addr learned my addr 172.21.15.67:0/1034758047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e85475640 1 --2- 172.21.15.67:0/1034758047 >> v2:172.21.15.176:3300/0 conn(0x7f4e80160fa0 0x7f4e800b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:47040/0 (socket says 172.21.15.67:47040) 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e7f7fe640 1 -- 172.21.15.67:0/1034758047 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3086852458 0 0) 0x7f4e801615f0 con 0x7f4e8015df80 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e84c74640 1 -- 172.21.15.67:0/1034758047 >> v2:172.21.15.176:3300/0 conn(0x7f4e80160fa0 msgr2=0x7f4e800b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e84c74640 1 --2- 172.21.15.67:0/1034758047 >> v2:172.21.15.176:3300/0 conn(0x7f4e80160fa0 0x7f4e800b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.689+0000 7f4e84c74640 1 -- 172.21.15.67:0/1034758047 >> v1:172.21.15.176:6789/0 conn(0x7f4e8015df80 legacy=0x7f4e80160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.690+0000 7f4e84c74640 1 -- 172.21.15.67:0/1034758047 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e80164020 con 0x7f4e80164b00 2024-04-25T15:27:05.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.690+0000 7f4e84c74640 1 --2- 172.21.15.67:0/1034758047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e80164b00 0x7f4e800b7b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009870 tx=0x7f4e7802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523ece6ce9cacd78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.690+0000 7f4e7f7fe640 1 -- 172.21.15.67:0/1034758047 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78037070 con 0x7f4e80164b00 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e7f7fe640 1 -- 172.21.15.67:0/1034758047 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e78032440 con 0x7f4e80164b00 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e7f7fe640 1 -- 172.21.15.67:0/1034758047 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78032810 con 0x7f4e80164b00 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 -- 172.21.15.67:0/1034758047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e80164b00 msgr2=0x7f4e800b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/1034758047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e80164b00 0x7f4e800b7b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009870 tx=0x7f4e7802a270 comp rx=0 tx=0).stop 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 -- 172.21.15.67:0/1034758047 shutdown_connections 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/1034758047 >> v2:172.21.15.176:3300/0 conn(0x7f4e80160fa0 0x7f4e800b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/1034758047 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e80164b00 0x7f4e800b7b80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 -- 172.21.15.67:0/1034758047 >> 172.21.15.67:0/1034758047 conn(0x7f4e800d2bc0 msgr2=0x7f4e800d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 -- 172.21.15.67:0/1034758047 shutdown_connections 2024-04-25T15:27:05.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.691+0000 7f4e86ed8640 1 -- 172.21.15.67:0/1034758047 wait complete. 2024-04-25T15:27:05.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.693+0000 7f4e86ed8640 1 Processor -- start 2024-04-25T15:27:05.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.693+0000 7f4e86ed8640 1 -- start start 2024-04-25T15:27:05.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.694+0000 7f4e86ed8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8015df80 0x7f4e801f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.694+0000 7f4e86ed8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e80160fa0 0x7f4e801615f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.694+0000 7f4e86ed8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e80164b00 0x7f4e80161b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.694+0000 7f4e86ed8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4e800b4a00 con 0x7f4e80160fa0 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.694+0000 7f4e86ed8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4e800b4880 con 0x7f4e80164b00 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.694+0000 7f4e86ed8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4e800b4b80 con 0x7f4e8015df80 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e80160fa0 0x7f4e801615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e80160fa0 0x7f4e801615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39080/0 (socket says 172.21.15.67:39080) 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 -- 172.21.15.67:0/446893669 learned_addr learned my addr 172.21.15.67:0/446893669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e85475640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e80164b00 0x7f4e80161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e84c74640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8015df80 0x7f4e801f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 -- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8015df80 msgr2=0x7f4e801f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8015df80 0x7f4e801f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 -- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e80164b00 msgr2=0x7f4e80161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e80164b00 0x7f4e80161b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 -- 172.21.15.67:0/446893669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e801592b0 con 0x7f4e80160fa0 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e84c74640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8015df80 0x7f4e801f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:05.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7ffff640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e80160fa0 0x7f4e801615f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c750 tx=0x7f4e7000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7d7fa640 1 -- 172.21.15.67:0/446893669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70018070 con 0x7f4e80160fa0 2024-04-25T15:27:05.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7d7fa640 1 -- 172.21.15.67:0/446893669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7000a5e0 con 0x7f4e80160fa0 2024-04-25T15:27:05.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e80159590 con 0x7f4e80160fa0 2024-04-25T15:27:05.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.695+0000 7f4e7d7fa640 1 -- 172.21.15.67:0/446893669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000ce60 con 0x7f4e80160fa0 2024-04-25T15:27:05.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.696+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e80153790 con 0x7f4e80160fa0 2024-04-25T15:27:05.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.697+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e80050740 con 0x7f4e80160fa0 2024-04-25T15:27:05.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.697+0000 7f4e7d7fa640 1 -- 172.21.15.67:0/446893669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4e7001b070 con 0x7f4e80160fa0 2024-04-25T15:27:05.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.698+0000 7f4e7d7fa640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e600744e0 0x7f4e600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:05.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.698+0000 7f4e7d7fa640 1 -- 172.21.15.67:0/446893669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f4e70094720 con 0x7f4e80160fa0 2024-04-25T15:27:05.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.698+0000 7f4e84c74640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e600744e0 0x7f4e600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:05.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.698+0000 7f4e84c74640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e600744e0 0x7f4e600769f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4e78034000 tx=0x7f4e7802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:05.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.702+0000 7f4e7d7fa640 1 -- 172.21.15.67:0/446893669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e7005e8f0 con 0x7f4e80160fa0 2024-04-25T15:27:05.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.935+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4e800d4170 con 0x7f4e80160fa0 2024-04-25T15:27:05.937 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:27:05.936+0000 7f3e02b50640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:27:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.937+0000 7f4e7d7fa640 1 -- 172.21.15.67:0/446893669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7f4e70062bf0 con 0x7f4e80160fa0 2024-04-25T15:27:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:27:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.940+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e600744e0 msgr2=0x7f4e600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.940+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e600744e0 0x7f4e600769f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4e78034000 tx=0x7f4e7802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.940+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e80160fa0 msgr2=0x7f4e801615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.940+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e80160fa0 0x7f4e801615f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c750 tx=0x7f4e7000cc10 comp rx=0 tx=0).stop 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.940+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 shutdown_connections 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.941+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e600744e0 0x7f4e600769f0 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4e78034000 tx=0x7f4e7802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.941+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e80164b00 0x7f4e80161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.941+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e80160fa0 0x7f4e801615f0 secure :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c750 tx=0x7f4e7000cc10 comp rx=0 tx=0).stop 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.941+0000 7f4e86ed8640 1 --2- 172.21.15.67:0/446893669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e8015df80 0x7f4e801f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.941+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 >> 172.21.15.67:0/446893669 conn(0x7f4e800d2bc0 msgr2=0x7f4e80160890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.941+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 shutdown_connections 2024-04-25T15:27:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:05.941+0000 7f4e86ed8640 1 -- 172.21.15.67:0/446893669 wait complete. 2024-04-25T15:27:05.943 INFO:tasks.ceph.mds.a.smithi067.stderr: -2933> 2024-04-25T15:26:51.722+0000 7fad41703640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:27:05.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:27:06.001 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:27:06.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.070+0000 7fadd069a640 1 Processor -- start 2024-04-25T15:27:06.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.070+0000 7fadd069a640 1 -- start start 2024-04-25T15:27:06.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadd069a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fadc8164730 0x7fadc80d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:06.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadd069a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadc80d23f0 con 0x7fadc80d1e20 2024-04-25T15:27:06.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadd069a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadc80d26d0 con 0x7fadc80b2e60 2024-04-25T15:27:06.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadd069a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fadc80d29b0 con 0x7fadc8164730 2024-04-25T15:27:06.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadcec37640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fadc8164730 0x7fadc80d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:06.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadcec37640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fadc8164730 0x7fadc80d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39104/0 (socket says 172.21.15.67:39104) 2024-04-25T15:27:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadcec37640 1 -- 172.21.15.67:0/1058411149 learned_addr learned my addr 172.21.15.67:0/1058411149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadce436640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fadc80d1e20 0x7fadc808d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57132/0 (socket says 172.21.15.67:57132) 2024-04-25T15:27:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadcec37640 1 -- 172.21.15.67:0/1058411149 >> v1:172.21.15.176:6789/0 conn(0x7fadc80b2e60 legacy=0x7fadc8164020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadcec37640 1 -- 172.21.15.67:0/1058411149 >> v1:172.21.15.67:6789/0 conn(0x7fadc80d1e20 legacy=0x7fadc808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadcd434640 1 -- 172.21.15.67:0/1058411149 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 871706212 0 0) 0x7fadc80d23f0 con 0x7fadc80d1e20 2024-04-25T15:27:06.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.072+0000 7fadcec37640 1 -- 172.21.15.67:0/1058411149 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc80d2c90 con 0x7fadc8164730 2024-04-25T15:27:06.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.073+0000 7fadcec37640 1 --2- 172.21.15.67:0/1058411149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc8164730 0x7fadc80d3490 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fadb8009e50 tx=0x7fadb802a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a6c559fb35e50d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:06.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.073+0000 7fadcd434640 1 -- 172.21.15.67:0/1058411149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8037070 con 0x7fadc8164730 2024-04-25T15:27:06.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.073+0000 7fadcd434640 1 -- 172.21.15.67:0/1058411149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb8032630 con 0x7fadc8164730 2024-04-25T15:27:06.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.073+0000 7fadcd434640 1 -- 172.21.15.67:0/1058411149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb802e050 con 0x7fadc8164730 2024-04-25T15:27:06.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.074+0000 7fadd069a640 1 -- 172.21.15.67:0/1058411149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc8164730 msgr2=0x7fadc80d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:06.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.074+0000 7fadd069a640 1 --2- 172.21.15.67:0/1058411149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc8164730 0x7fadc80d3490 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fadb8009e50 tx=0x7fadb802a840 comp rx=0 tx=0).stop 2024-04-25T15:27:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.074+0000 7fadd069a640 1 -- 172.21.15.67:0/1058411149 shutdown_connections 2024-04-25T15:27:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.074+0000 7fadd069a640 1 --2- 172.21.15.67:0/1058411149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc8164730 0x7fadc80d3490 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.074+0000 7fadd069a640 1 -- 172.21.15.67:0/1058411149 >> 172.21.15.67:0/1058411149 conn(0x7fadc80d0530 msgr2=0x7fadc80d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.074+0000 7fadd069a640 1 -- 172.21.15.67:0/1058411149 shutdown_connections 2024-04-25T15:27:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.074+0000 7fadd069a640 1 -- 172.21.15.67:0/1058411149 wait complete. 2024-04-25T15:27:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.076+0000 7fadd069a640 1 Processor -- start 2024-04-25T15:27:06.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.076+0000 7fadd069a640 1 -- start start 2024-04-25T15:27:06.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadd069a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc80b2e60 0x7fadc80b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:06.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadd069a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadc80d1e20 0x7fadc81f5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:06.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadd069a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadc81f6260 0x7fadc8162e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:06.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadd069a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fadc80ce1f0 con 0x7fadc80b2e60 2024-04-25T15:27:06.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadd069a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fadc80ce070 con 0x7fadc80d1e20 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadd069a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fadc80ce370 con 0x7fadc81f6260 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadce436640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc80b2e60 0x7fadc80b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadce436640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc80b2e60 0x7fadc80b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39108/0 (socket says 172.21.15.67:39108) 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadce436640 1 -- 172.21.15.67:0/1452742218 learned_addr learned my addr 172.21.15.67:0/1452742218 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadcec37640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadc81f6260 0x7fadc8162e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.077+0000 7fadcdc35640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadc80d1e20 0x7fadc81f5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadce436640 1 -- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadc81f6260 msgr2=0x7fadc8162e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadce436640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadc81f6260 0x7fadc8162e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadce436640 1 -- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadc80d1e20 msgr2=0x7fadc81f5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadce436640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadc80d1e20 0x7fadc81f5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadce436640 1 -- 172.21.15.67:0/1452742218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc80b5c90 con 0x7fadc80b2e60 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadcec37640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadc81f6260 0x7fadc8162e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadce436640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc80b2e60 0x7fadc80b3410 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fadc000c3f0 tx=0x7fadc000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadc6ffd640 1 -- 172.21.15.67:0/1452742218 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc00074e0 con 0x7fadc80b2e60 2024-04-25T15:27:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadc6ffd640 1 -- 172.21.15.67:0/1452742218 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc00047e0 con 0x7fadc80b2e60 2024-04-25T15:27:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc80b5f70 con 0x7fadc80b2e60 2024-04-25T15:27:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadc6ffd640 1 -- 172.21.15.67:0/1452742218 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0004c10 con 0x7fadc80b2e60 2024-04-25T15:27:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.078+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc80cbb40 con 0x7fadc80b2e60 2024-04-25T15:27:06.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.079+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc8050740 con 0x7fadc80b2e60 2024-04-25T15:27:06.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.080+0000 7fadc6ffd640 1 -- 172.21.15.67:0/1452742218 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fadc0011070 con 0x7fadc80b2e60 2024-04-25T15:27:06.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.080+0000 7fadc6ffd640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fadb40744b0 0x7fadb40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:06.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.080+0000 7fadc6ffd640 1 -- 172.21.15.67:0/1452742218 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5795+0+0 (secure 0 0 0) 0x7fadc0094d40 con 0x7fadc80b2e60 2024-04-25T15:27:06.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.081+0000 7fadcdc35640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fadb40744b0 0x7fadb40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:06.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.081+0000 7fadcdc35640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fadb40744b0 0x7fadb40769c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fadc81f9b60 tx=0x7fadbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.085+0000 7fadc6ffd640 1 -- 172.21.15.67:0/1452742218 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fadc005f500 con 0x7fadc80b2e60 2024-04-25T15:27:06.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.310+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fadc806ebd0 con 0x7fadc80b2e60 2024-04-25T15:27:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.945+0000 7fadc6ffd640 1 -- 172.21.15.67:0/1452742218 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fadc005ee00 con 0x7fadc80b2e60 2024-04-25T15:27:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fadb40744b0 msgr2=0x7fadb40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fadb40744b0 0x7fadb40769c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fadc81f9b60 tx=0x7fadbc009040 comp rx=0 tx=0).stop 2024-04-25T15:27:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc80b2e60 msgr2=0x7fadc80b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc80b2e60 0x7fadc80b3410 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fadc000c3f0 tx=0x7fadc000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 shutdown_connections 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fadb40744b0 0x7fadb40769c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadc81f6260 0x7fadc8162e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadc80d1e20 0x7fadc81f5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 --2- 172.21.15.67:0/1452742218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadc80b2e60 0x7fadc80b3410 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 >> 172.21.15.67:0/1452742218 conn(0x7fadc80d0530 msgr2=0x7fadc806d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 shutdown_connections 2024-04-25T15:27:06.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:06.947+0000 7fadd069a640 1 -- 172.21.15.67:0/1452742218 wait complete. 2024-04-25T15:27:06.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:27:07.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.078+0000 7f6ba9815640 1 Processor -- start 2024-04-25T15:27:07.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.078+0000 7f6ba9815640 1 -- start start 2024-04-25T15:27:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba9815640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ba4164b00 0x7f6ba40b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba9815640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba40b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba9815640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba41615f0 con 0x7f6ba4160890 2024-04-25T15:27:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba9815640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6ba41613b0 con 0x7f6ba4164b00 2024-04-25T15:27:07.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba9815640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba4164670 con 0x7f6ba415df80 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ba4164b00 0x7f6ba40b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ba4164b00 0x7f6ba40b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39116/0 (socket says 172.21.15.67:39116) 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ba4160890 0x7f6ba4160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57134/0 (socket says 172.21.15.67:57134) 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba2ffd640 1 -- 172.21.15.67:0/215700666 learned_addr learned my addr 172.21.15.67:0/215700666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba27fc640 1 --2- 172.21.15.67:0/215700666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba40b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba2ffd640 1 -- 172.21.15.67:0/215700666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 msgr2=0x7f6ba40b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba2ffd640 1 --2- 172.21.15.67:0/215700666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba40b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba2ffd640 1 -- 172.21.15.67:0/215700666 >> v1:172.21.15.67:6789/0 conn(0x7f6ba4160890 legacy=0x7f6ba4160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:07.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.080+0000 7f6ba2ffd640 1 -- 172.21.15.67:0/215700666 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba4164020 con 0x7f6ba4164b00 2024-04-25T15:27:07.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.081+0000 7f6ba2ffd640 1 --2- 172.21.15.67:0/215700666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba40b7b80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009870 tx=0x7f6b9c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6d83d16e20222 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.081+0000 7f6ba1ffb640 1 -- 172.21.15.67:0/215700666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c037070 con 0x7f6ba4164b00 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.081+0000 7f6ba1ffb640 1 -- 172.21.15.67:0/215700666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9c032440 con 0x7f6ba4164b00 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.081+0000 7f6ba1ffb640 1 -- 172.21.15.67:0/215700666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c032810 con 0x7f6ba4164b00 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 -- 172.21.15.67:0/215700666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 msgr2=0x7f6ba40b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 --2- 172.21.15.67:0/215700666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba40b7b80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009870 tx=0x7f6b9c02a270 comp rx=0 tx=0).stop 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 -- 172.21.15.67:0/215700666 shutdown_connections 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 --2- 172.21.15.67:0/215700666 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba40b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 --2- 172.21.15.67:0/215700666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba40b7b80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 -- 172.21.15.67:0/215700666 >> 172.21.15.67:0/215700666 conn(0x7f6ba40d2bc0 msgr2=0x7f6ba40d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 -- 172.21.15.67:0/215700666 shutdown_connections 2024-04-25T15:27:07.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.082+0000 7f6ba9815640 1 -- 172.21.15.67:0/215700666 wait complete. 2024-04-25T15:27:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.084+0000 7f6ba9815640 1 Processor -- start 2024-04-25T15:27:07.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.084+0000 7f6ba9815640 1 -- start start 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba9815640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba4145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba9815640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba4160890 0x7f6ba4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba9815640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba41f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba9815640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba40b4a00 con 0x7f6ba4164b00 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba9815640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba40b4880 con 0x7f6ba4160890 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba9815640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba40b4b80 con 0x7f6ba415df80 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba41f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:07.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba41f61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39126/0 (socket says 172.21.15.67:39126) 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba37fe640 1 -- 172.21.15.67:0/4120923856 learned_addr learned my addr 172.21.15.67:0/4120923856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba2ffd640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba4145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.085+0000 7f6ba27fc640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba4160890 0x7f6ba4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba37fe640 1 -- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 msgr2=0x7f6ba4145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba37fe640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba4145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba37fe640 1 -- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba4160890 msgr2=0x7f6ba4158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba37fe640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba4160890 0x7f6ba4158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba37fe640 1 -- 172.21.15.67:0/4120923856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba4146390 con 0x7f6ba4164b00 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba27fc640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba4160890 0x7f6ba4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba37fe640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba41f61f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c760 tx=0x7f6b9400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6b9bfff640 1 -- 172.21.15.67:0/4120923856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94017070 con 0x7f6ba4164b00 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6b9bfff640 1 -- 172.21.15.67:0/4120923856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9400ce10 con 0x7f6ba4164b00 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba4148d00 con 0x7f6ba4164b00 2024-04-25T15:27:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6b9bfff640 1 -- 172.21.15.67:0/4120923856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400a840 con 0x7f6ba4164b00 2024-04-25T15:27:07.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.086+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba4158200 con 0x7f6ba4164b00 2024-04-25T15:27:07.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.087+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba4158c00 con 0x7f6ba4164b00 2024-04-25T15:27:07.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.088+0000 7f6b9bfff640 1 -- 172.21.15.67:0/4120923856 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6b9401a070 con 0x7f6ba4164b00 2024-04-25T15:27:07.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.088+0000 7f6b9bfff640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b840744e0 0x7f6b840769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:07.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.088+0000 7f6b9bfff640 1 -- 172.21.15.67:0/4120923856 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5795+0+0 (secure 0 0 0) 0x7f6b940946c0 con 0x7f6ba4164b00 2024-04-25T15:27:07.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.088+0000 7f6ba2ffd640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b840744e0 0x7f6b840769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:07.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.089+0000 7f6ba2ffd640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b840744e0 0x7f6b840769f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c034000 tx=0x7f6b9c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:07.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.093+0000 7f6b9bfff640 1 -- 172.21.15.67:0/4120923856 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b9405e7b0 con 0x7f6ba4164b00 2024-04-25T15:27:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.276+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6ba4050740 con 0x7f6ba4164b00 2024-04-25T15:27:07.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.948+0000 7f6b9bfff640 1 -- 172.21.15.67:0/4120923856 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) ==== 181+0+0 (secure 0 0 0) 0x7f6b94062ab0 con 0x7f6ba4164b00 2024-04-25T15:27:07.949 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:27:07.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.951+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b840744e0 msgr2=0x7f6b840769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:07.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.951+0000 7f6ba9815640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b840744e0 0x7f6b840769f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c034000 tx=0x7f6b9c02e000 comp rx=0 tx=0).stop 2024-04-25T15:27:07.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.951+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 msgr2=0x7f6ba41f61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:07.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.951+0000 7f6ba9815640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba41f61f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c760 tx=0x7f6b9400cc20 comp rx=0 tx=0).stop 2024-04-25T15:27:07.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.952+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 shutdown_connections 2024-04-25T15:27:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.952+0000 7f6ba9815640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b840744e0 0x7f6b840769f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.952+0000 7f6ba9815640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba4164b00 0x7f6ba41f61f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.952+0000 7f6ba9815640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba4160890 0x7f6ba4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.952+0000 7f6ba9815640 1 --2- 172.21.15.67:0/4120923856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba415df80 0x7f6ba4145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.952+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 >> 172.21.15.67:0/4120923856 conn(0x7f6ba40d2bc0 msgr2=0x7f6ba415f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.953+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 shutdown_connections 2024-04-25T15:27:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:07.953+0000 7f6ba9815640 1 -- 172.21.15.67:0/4120923856 wait complete. 2024-04-25T15:27:07.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:27:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.091+0000 7f6ba54aa640 1 Processor -- start 2024-04-25T15:27:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.091+0000 7f6ba54aa640 1 -- start start 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6ba54aa640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6ba00d0e00 0x7f6ba00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6ba54aa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba006c5c0 0x7f6ba006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6ba54aa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba006cf10 con 0x7f6ba00b7810 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6ba54aa640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6ba00d5a60 con 0x7f6ba00d0e00 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6ba54aa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba00d1850 con 0x7f6ba006c5c0 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba006c5c0 0x7f6ba006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6b9effd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6ba00d0e00 0x7f6ba00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6b9effd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6ba00d0e00 0x7f6ba00d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:42804/0 (socket says 172.21.15.67:42804) 2024-04-25T15:27:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba006c5c0 0x7f6ba006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:46408/0 (socket says 172.21.15.67:46408) 2024-04-25T15:27:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.093+0000 7f6b9effd640 1 -- 172.21.15.67:0/2665024619 learned_addr learned my addr 172.21.15.67:0/2665024619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.094+0000 7f6b9effd640 1 -- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba006c5c0 msgr2=0x7f6ba006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.094+0000 7f6b9effd640 1 --2- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba006c5c0 0x7f6ba006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.094+0000 7f6b9effd640 1 -- 172.21.15.67:0/2665024619 >> v1:172.21.15.176:6789/0 conn(0x7f6ba00b7810 legacy=0x7f6ba00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.094+0000 7f6b9effd640 1 -- 172.21.15.67:0/2665024619 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba0160de0 con 0x7f6ba00d0e00 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.094+0000 7f6b9dffb640 1 -- 172.21.15.67:0/2665024619 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1798065197 0 0) 0x7f6b90001490 con 0x7f6ba00b7810 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.094+0000 7f6b9e7fc640 1 --2- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba006c5c0 0x7f6ba006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.094+0000 7f6b9effd640 1 --2- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba00d0e00 0x7f6ba00d1210 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009870 tx=0x7f6b9802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc72d28efaf37cf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6b9dffb640 1 -- 172.21.15.67:0/2665024619 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98037070 con 0x7f6ba00d0e00 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6b9dffb640 1 -- 172.21.15.67:0/2665024619 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b98032440 con 0x7f6ba00d0e00 2024-04-25T15:27:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6b9dffb640 1 -- 172.21.15.67:0/2665024619 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98032810 con 0x7f6ba00d0e00 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 -- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba00d0e00 msgr2=0x7f6ba00d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba00d0e00 0x7f6ba00d1210 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009870 tx=0x7f6b9802a270 comp rx=0 tx=0).stop 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 -- 172.21.15.67:0/2665024619 shutdown_connections 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba006c5c0 0x7f6ba006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/2665024619 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba00d0e00 0x7f6ba00d1210 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 -- 172.21.15.67:0/2665024619 >> 172.21.15.67:0/2665024619 conn(0x7f6ba00d5d90 msgr2=0x7f6ba00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 -- 172.21.15.67:0/2665024619 shutdown_connections 2024-04-25T15:27:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.095+0000 7f6ba54aa640 1 -- 172.21.15.67:0/2665024619 wait complete. 2024-04-25T15:27:08.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.097+0000 7f6ba54aa640 1 Processor -- start 2024-04-25T15:27:08.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.097+0000 7f6ba54aa640 1 -- start start 2024-04-25T15:27:08.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6ba54aa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba006c5c0 0x7f6ba00cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6ba54aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba00b7810 0x7f6ba014bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6ba54aa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba00d0e00 0x7f6ba00ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6ba54aa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0160ad0 con 0x7f6ba00b7810 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6ba54aa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba01613a0 con 0x7f6ba006c5c0 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6ba54aa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba00b5320 con 0x7f6ba00d0e00 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba00b7810 0x7f6ba014bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba00b7810 0x7f6ba014bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33522/0 (socket says 172.21.15.67:33522) 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 -- 172.21.15.67:0/3501203017 learned_addr learned my addr 172.21.15.67:0/3501203017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9effd640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba006c5c0 0x7f6ba00cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9f7fe640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba00d0e00 0x7f6ba00ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 -- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba00d0e00 msgr2=0x7f6ba00ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba00d0e00 0x7f6ba00ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 -- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba006c5c0 msgr2=0x7f6ba00cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba006c5c0 0x7f6ba00cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 -- 172.21.15.67:0/3501203017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba00c9e10 con 0x7f6ba00b7810 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.099+0000 7f6b9e7fc640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba00b7810 0x7f6ba014bfd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400e990 tx=0x7f6b9400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.100+0000 7f6b877fe640 1 -- 172.21.15.67:0/3501203017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94017070 con 0x7f6ba00b7810 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.100+0000 7f6b877fe640 1 -- 172.21.15.67:0/3501203017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9400c9e0 con 0x7f6ba00b7810 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.100+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba00ca0f0 con 0x7f6ba00b7810 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.100+0000 7f6b877fe640 1 -- 172.21.15.67:0/3501203017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400cdb0 con 0x7f6ba00b7810 2024-04-25T15:27:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.100+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba00ca6b0 con 0x7f6ba00b7810 2024-04-25T15:27:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.101+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba0050740 con 0x7f6ba00b7810 2024-04-25T15:27:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.101+0000 7f6b877fe640 1 -- 172.21.15.67:0/3501203017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6b9401a070 con 0x7f6ba00b7810 2024-04-25T15:27:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.102+0000 7f6b877fe640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b8c0744e0 0x7f6b8c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.102+0000 7f6b877fe640 1 -- 172.21.15.67:0/3501203017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5357+0+0 (secure 0 0 0) 0x7f6b94095280 con 0x7f6ba00b7810 2024-04-25T15:27:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.102+0000 7f6b9effd640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b8c0744e0 0x7f6b8c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.103+0000 7f6b9effd640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b8c0744e0 0x7f6b8c0769f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6b98034000 tx=0x7f6b9802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:08.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.107+0000 7f6b877fe640 1 -- 172.21.15.67:0/3501203017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b9405f520 con 0x7f6ba00b7810 2024-04-25T15:27:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.299+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6ba00c9ba0 con 0x7f6ba00b7810 2024-04-25T15:27:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.955+0000 7f6b877fe640 1 -- 172.21.15.67:0/3501203017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) ==== 169+0+0 (secure 0 0 0) 0x7f6b94063820 con 0x7f6ba00b7810 2024-04-25T15:27:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:27:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b8c0744e0 msgr2=0x7f6b8c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:08.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b8c0744e0 0x7f6b8c0769f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6b98034000 tx=0x7f6b9802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba00b7810 msgr2=0x7f6ba014bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba00b7810 0x7f6ba014bfd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400e990 tx=0x7f6b9400ee50 comp rx=0 tx=0).stop 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 shutdown_connections 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b8c0744e0 0x7f6b8c0769f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ba00d0e00 0x7f6ba00ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ba00b7810 0x7f6ba014bfd0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 --2- 172.21.15.67:0/3501203017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ba006c5c0 0x7f6ba00cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.957+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 >> 172.21.15.67:0/3501203017 conn(0x7f6ba00d5d90 msgr2=0x7f6ba00b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.958+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 shutdown_connections 2024-04-25T15:27:08.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:08.958+0000 7f6ba54aa640 1 -- 172.21.15.67:0/3501203017 wait complete. 2024-04-25T15:27:08.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:27:09.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.099+0000 7f77b94e0640 1 Processor -- start 2024-04-25T15:27:09.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.099+0000 7f77b94e0640 1 -- start start 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b94e0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f77b40b6510 0x7f77b40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b94e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77b406dce0 0x7f77b40b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b94e0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77b415cd80 con 0x7f77b4160d30 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b94e0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f77b415d690 con 0x7f77b40b6510 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b94e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77b40b6ab0 con 0x7f77b406dce0 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b27fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f77b4160d30 0x7f77b406d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52642/0 (socket says 172.21.15.67:52642) 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b27fc640 1 -- 172.21.15.67:0/3685338008 learned_addr learned my addr 172.21.15.67:0/3685338008 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b2ffd640 1 --2- 172.21.15.67:0/3685338008 >> v2:172.21.15.176:3300/0 conn(0x7f77b40b6510 0x7f77b40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b37fe640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77b406dce0 0x7f77b40b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3557646504 0 0) 0x7f77b415cd80 con 0x7f77b4160d30 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.101+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77a0003520 con 0x7f77b4160d30 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1914640952 0 0) 0x7f77a0003520 con 0x7f77b4160d30 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77b406dce0 msgr2=0x7f77b40b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77b406dce0 0x7f77b40b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 >> v2:172.21.15.176:3300/0 conn(0x7f77b40b6510 msgr2=0x7f77b40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 --2- 172.21.15.67:0/3685338008 >> v2:172.21.15.176:3300/0 conn(0x7f77b40b6510 0x7f77b40ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b415d040 con 0x7f77b4160d30 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f779c001670 con 0x7f77b4160d30 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 >> v1:172.21.15.67:6789/0 conn(0x7f77b4160d30 legacy=0x7f77b406d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a00039b0 0x7f77a0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0004360 0x7f77a0004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a0004f90 0x7f77a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77b415d690 con 0x7f77a0004360 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77b40b6ab0 con 0x7f77a0004f90 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77a000bf80 con 0x7f77a00039b0 2024-04-25T15:27:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b27fc640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0004360 0x7f77a0004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b2ffd640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a00039b0 0x7f77a0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b37fe640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a0004f90 0x7f77a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b27fc640 1 -- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a00039b0 msgr2=0x7f77a0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b27fc640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a00039b0 0x7f77a0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b27fc640 1 -- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a0004f90 msgr2=0x7f77a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b27fc640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a0004f90 0x7f77a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.102+0000 7f77b27fc640 1 -- 172.21.15.67:0/3685338008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a0003520 con 0x7f77a0004360 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.103+0000 7f77b27fc640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0004360 0x7f77a0004880 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f779c00e500 tx=0x7f77a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e7ba87bd724d25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.103+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c007da0 con 0x7f77a0004360 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.103+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f779c031070 con 0x7f77a0004360 2024-04-25T15:27:09.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.103+0000 7f77b1ffb640 1 -- 172.21.15.67:0/3685338008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c005d50 con 0x7f77a0004360 2024-04-25T15:27:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.104+0000 7f77b94e0640 1 -- 172.21.15.67:0/3685338008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0004360 msgr2=0x7f77a0004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.104+0000 7f77b94e0640 1 --2- 172.21.15.67:0/3685338008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77a0004360 0x7f77a0004880 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f779c00e500 tx=0x7f77a0005830 comp rx=0 tx=0).stop 2024-04-25T15:27:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.104+0000 7f77b37fe640 1 -- 172.21.15.67:0/3685338008 reap_dead start 2024-04-25T15:27:09.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.104+0000 7f77b94e0640 1 -- 172.21.15.67:0/3685338008 shutdown_connections 2024-04-25T15:27:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.104+0000 7f77b94e0640 1 -- 172.21.15.67:0/3685338008 >> 172.21.15.67:0/3685338008 conn(0x7f77b4073120 msgr2=0x7f77b4073530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.104+0000 7f77b94e0640 1 -- 172.21.15.67:0/3685338008 shutdown_connections 2024-04-25T15:27:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.104+0000 7f77b94e0640 1 -- 172.21.15.67:0/3685338008 wait complete. 2024-04-25T15:27:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.106+0000 7f77b94e0640 1 Processor -- start 2024-04-25T15:27:09.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.106+0000 7f77b94e0640 1 -- start start 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b94e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a0004360 0x7f77b4154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b94e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77b414efc0 0x7f77b414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b94e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b41f2400 0x7f77b41f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b94e0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41612c0 con 0x7f77b41f2400 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b94e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77b4161140 con 0x7f77b414efc0 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b94e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77b415db20 con 0x7f77a0004360 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b41f2400 0x7f77b41f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b41f2400 0x7f77b41f2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33544/0 (socket says 172.21.15.67:33544) 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b37fe640 1 -- 172.21.15.67:0/2907622926 learned_addr learned my addr 172.21.15.67:0/2907622926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b27fc640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77b414efc0 0x7f77b414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b2ffd640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a0004360 0x7f77b4154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b37fe640 1 -- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a0004360 msgr2=0x7f77b4154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b37fe640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a0004360 0x7f77b4154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b37fe640 1 -- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77b414efc0 msgr2=0x7f77b414e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.107+0000 7f77b37fe640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77b414efc0 0x7f77b414e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77b37fe640 1 -- 172.21.15.67:0/2907622926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b41f2d50 con 0x7f77b41f2400 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77b27fc640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77b414efc0 0x7f77b414e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77b2ffd640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a0004360 0x7f77b4154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77b37fe640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b41f2400 0x7f77b41f2810 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f77b41573d0 tx=0x7f77a400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77ab7fe640 1 -- 172.21.15.67:0/2907622926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a400fd80 con 0x7f77b41f2400 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77ab7fe640 1 -- 172.21.15.67:0/2907622926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a401d070 con 0x7f77b41f2400 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b4153a60 con 0x7f77b41f2400 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77ab7fe640 1 -- 172.21.15.67:0/2907622926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4015670 con 0x7f77b41f2400 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.108+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b4154020 con 0x7f77b41f2400 2024-04-25T15:27:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.109+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b4050740 con 0x7f77b41f2400 2024-04-25T15:27:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.110+0000 7f77ab7fe640 1 -- 172.21.15.67:0/2907622926 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f77a4015810 con 0x7f77b41f2400 2024-04-25T15:27:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.110+0000 7f77ab7fe640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77940744b0 0x7f77940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.110+0000 7f77ab7fe640 1 -- 172.21.15.67:0/2907622926 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7f77a4095840 con 0x7f77b41f2400 2024-04-25T15:27:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.111+0000 7f77b2ffd640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77940744b0 0x7f77940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.111+0000 7f77b2ffd640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77940744b0 0x7f77940769c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b62d0 tx=0x7f77ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.115+0000 7f77ab7fe640 1 -- 172.21.15.67:0/2907622926 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77a4010040 con 0x7f77b41f2400 2024-04-25T15:27:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.299+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f77b4070900 con 0x7f77b41f2400 2024-04-25T15:27:09.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.302+0000 7f77ab7fe640 1 -- 172.21.15.67:0/2907622926 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f77a40602d0 con 0x7f77b41f2400 2024-04-25T15:27:09.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77940744b0 msgr2=0x7f77940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77940744b0 0x7f77940769c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f77b40b62d0 tx=0x7f77ac008040 comp rx=0 tx=0).stop 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b41f2400 msgr2=0x7f77b41f2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b41f2400 0x7f77b41f2810 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f77b41573d0 tx=0x7f77a400f580 comp rx=0 tx=0).stop 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 shutdown_connections 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f77940744b0 0x7f77940769c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b41f2400 0x7f77b41f2810 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77b414efc0 0x7f77b414e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 --2- 172.21.15.67:0/2907622926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77a0004360 0x7f77b4154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 >> 172.21.15.67:0/2907622926 conn(0x7f77b4073120 msgr2=0x7f77b40b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.304+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 shutdown_connections 2024-04-25T15:27:09.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.305+0000 7f77b94e0640 1 -- 172.21.15.67:0/2907622926 wait complete. 2024-04-25T15:27:09.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values' 2024-04-25T15:27:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.427+0000 7fad5a6ee640 1 Processor -- start 2024-04-25T15:27:09.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.427+0000 7fad5a6ee640 1 -- start start 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.428+0000 7fad5a6ee640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fad540d1210 0x7fad540d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.428+0000 7fad5a6ee640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad540d1850 con 0x7fad5406ccd0 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.428+0000 7fad5a6ee640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad54160de0 con 0x7fad540d0e00 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.428+0000 7fad5a6ee640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fad540d1620 con 0x7fad540d1210 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.428+0000 7fad537fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fad5406ccd0 0x7fad540d4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52646/0 (socket says 172.21.15.67:52646) 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.428+0000 7fad537fe640 1 -- 172.21.15.67:0/758491856 learned_addr learned my addr 172.21.15.67:0/758491856 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad58c8b640 1 --2- 172.21.15.67:0/758491856 >> v2:172.21.15.176:3300/0 conn(0x7fad540d1210 0x7fad540d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2032790398 0 0) 0x7fad540d1850 con 0x7fad5406ccd0 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad3c003520 con 0x7fad5406ccd0 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2959643785 0 0) 0x7fad54160de0 con 0x7fad540d0e00 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad540d1850 con 0x7fad540d0e00 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4160160133 0 0) 0x7fad3c003520 con 0x7fad5406ccd0 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 >> v2:172.21.15.176:3300/0 conn(0x7fad540d1210 msgr2=0x7fad540d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 --2- 172.21.15.67:0/758491856 >> v2:172.21.15.176:3300/0 conn(0x7fad540d1210 0x7fad540d5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 >> v1:172.21.15.176:6789/0 conn(0x7fad540d0e00 legacy=0x7fad5406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad541610c0 con 0x7fad5406ccd0 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad58c8b640 1 --2- 172.21.15.67:0/758491856 >> v2:172.21.15.176:3300/0 conn(0x7fad540d1210 0x7fad540d5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fad44001670 con 0x7fad5406ccd0 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 >> v1:172.21.15.67:6789/0 conn(0x7fad5406ccd0 legacy=0x7fad540d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad3c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad3c004800 0x7fad3c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad3c005260 0x7fad3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fad3c00c190 con 0x7fad3c003b40 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fad3c00c340 con 0x7fad3c004800 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fad3c00c540 con 0x7fad3c005260 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad53fff640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad3c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad537fe640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad3c004800 0x7fad3c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.429+0000 7fad58c8b640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad3c005260 0x7fad3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad53fff640 1 -- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad3c005260 msgr2=0x7fad3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad53fff640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad3c005260 0x7fad3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad53fff640 1 -- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad3c004800 msgr2=0x7fad3c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad53fff640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad3c004800 0x7fad3c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad53fff640 1 -- 172.21.15.67:0/758491856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3c003520 con 0x7fad3c003b40 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad537fe640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad3c004800 0x7fad3c004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad58c8b640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad3c005260 0x7fad3c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad58c8b640 1 -- 172.21.15.67:0/758491856 reap_dead start 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad53fff640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad3c003f90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad480018d0 tx=0x7fad3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4d5c64f0a83714 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480014f0 con 0x7fad3c003b40 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad480052d0 con 0x7fad3c003b40 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.430+0000 7fad52ffd640 1 -- 172.21.15.67:0/758491856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4802ac40 con 0x7fad3c003b40 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.431+0000 7fad5a6ee640 1 -- 172.21.15.67:0/758491856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 msgr2=0x7fad3c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.431+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad3c003f90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fad480018d0 tx=0x7fad3c005830 comp rx=0 tx=0).stop 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.431+0000 7fad5a6ee640 1 -- 172.21.15.67:0/758491856 shutdown_connections 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.431+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/758491856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad3c003f90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.431+0000 7fad5a6ee640 1 -- 172.21.15.67:0/758491856 >> 172.21.15.67:0/758491856 conn(0x7fad540d5d90 msgr2=0x7fad540d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.431+0000 7fad5a6ee640 1 -- 172.21.15.67:0/758491856 shutdown_connections 2024-04-25T15:27:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.431+0000 7fad5a6ee640 1 -- 172.21.15.67:0/758491856 wait complete. 2024-04-25T15:27:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.433+0000 7fad5a6ee640 1 Processor -- start 2024-04-25T15:27:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.433+0000 7fad5a6ee640 1 -- start start 2024-04-25T15:27:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.434+0000 7fad5a6ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad540cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.434+0000 7fad5a6ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad54142910 0x7fad540cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.434+0000 7fad5a6ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad540cc480 0x7fad5415dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.434+0000 7fad5a6ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fad540b5320 con 0x7fad3c003b40 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.434+0000 7fad5a6ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fad54160ad0 con 0x7fad54142910 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.434+0000 7fad5a6ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fad54164e00 con 0x7fad540cc480 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.434+0000 7fad53fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad540cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad540cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33560/0 (socket says 172.21.15.67:33560) 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 -- 172.21.15.67:0/885449510 learned_addr learned my addr 172.21.15.67:0/885449510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad537fe640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad54142910 0x7fad540cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad58c8b640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad540cc480 0x7fad5415dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 -- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad540cc480 msgr2=0x7fad5415dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad540cc480 0x7fad5415dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 -- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad54142910 msgr2=0x7fad540cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad54142910 0x7fad540cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 -- 172.21.15.67:0/885449510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad540caea0 con 0x7fad3c003b40 2024-04-25T15:27:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad58c8b640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad540cc480 0x7fad5415dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad53fff640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad540cf4d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fad48005040 tx=0x7fad480012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad50ff9640 1 -- 172.21.15.67:0/885449510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48037040 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad50ff9640 1 -- 172.21.15.67:0/885449510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad48006e50 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad540cb180 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad50ff9640 1 -- 172.21.15.67:0/885449510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480014f0 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.435+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad540cb740 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.436+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad54050740 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.437+0000 7fad50ff9640 1 -- 172.21.15.67:0/885449510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fad48044020 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.438+0000 7fad50ff9640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad400744b0 0x7fad400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.438+0000 7fad50ff9640 1 -- 172.21.15.67:0/885449510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7fad480b4a90 con 0x7fad3c003b40 2024-04-25T15:27:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.438+0000 7fad537fe640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad400744b0 0x7fad400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.438+0000 7fad537fe640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad400744b0 0x7fad400769c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fad5415e900 tx=0x7fad44007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.442+0000 7fad50ff9640 1 -- 172.21.15.67:0/885449510 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad4804b020 con 0x7fad3c003b40 2024-04-25T15:27:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.701+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]} v 0) -- 0x7fad540d4320 con 0x7fad3c003b40 2024-04-25T15:27:09.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.956+0000 7fad50ff9640 1 -- 172.21.15.67:0/885449510 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]}]=0 v119) ==== 153+0+0 (secure 0 0 0) 0x7fad4807f520 con 0x7fad3c003b40 2024-04-25T15:27:09.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad400744b0 msgr2=0x7fad400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad400744b0 0x7fad400769c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fad5415e900 tx=0x7fad44007040 comp rx=0 tx=0).stop 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 msgr2=0x7fad540cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad540cf4d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fad48005040 tx=0x7fad480012d0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 shutdown_connections 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fad400744b0 0x7fad400769c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fad540cc480 0x7fad5415dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad54142910 0x7fad540cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 --2- 172.21.15.67:0/885449510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad3c003b40 0x7fad540cf4d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.958+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 >> 172.21.15.67:0/885449510 conn(0x7fad540d5d90 msgr2=0x7fad540d4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.959+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 shutdown_connections 2024-04-25T15:27:09.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:09.959+0000 7fad5a6ee640 1 -- 172.21.15.67:0/885449510 wait complete. 2024-04-25T15:27:09.971 INFO:tasks.cephfs_test_runner:test invalid values for ceph.quota.max_bytes ... ok 2024-04-25T15:27:09.971 INFO:tasks.cephfs_test_runner:Starting test: test_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) test human-readable values for setting ceph.quota.max_bytes 2024-04-25T15:27:09.971 INFO:tasks.cephfs_test_runner:test_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2024-04-25T15:27:09.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.083+0000 7f4805da8640 1 Processor -- start 2024-04-25T15:27:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.083+0000 7f4805da8640 1 -- start start 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f4805da8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4800164b00 0x7f48000b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f4805da8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48001615f0 con 0x7f480015df80 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f4805da8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4800164020 con 0x7f4800160fa0 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f4805da8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f48001613b0 con 0x7f4800164b00 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47feffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f480015df80 0x7f4800160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52656/0 (socket says 172.21.15.67:52656) 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47feffd640 1 -- 172.21.15.67:0/3342884198 learned_addr learned my addr 172.21.15.67:0/3342884198 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47ff7fe640 1 --2- 172.21.15.67:0/3342884198 >> v2:172.21.15.176:3300/0 conn(0x7f4800164b00 0x7f48000b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 917836800 0 0) 0x7f48001615f0 con 0x7f480015df80 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47e8003520 con 0x7f480015df80 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1577065301 0 0) 0x7f47e8003520 con 0x7f480015df80 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 >> v2:172.21.15.176:3300/0 conn(0x7f4800164b00 msgr2=0x7f48000b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 --2- 172.21.15.67:0/3342884198 >> v2:172.21.15.176:3300/0 conn(0x7f4800164b00 0x7f48000b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 >> v1:172.21.15.176:6789/0 conn(0x7f4800160fa0 legacy=0x7f48000b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4800164300 con 0x7f480015df80 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f47f4001670 con 0x7f480015df80 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 >> v1:172.21.15.67:6789/0 conn(0x7f480015df80 legacy=0x7f4800160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47e80039b0 0x7f47e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.085+0000 7f47fe7fc640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47e8004800 0x7f47e8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47fe7fc640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e80051c0 0x7f47e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f48001613b0 con 0x7f47e8004800 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f47e800c160 con 0x7f47e80051c0 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f47e800c360 con 0x7f47e80039b0 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47feffd640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47e8004800 0x7f47e8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47ff7fe640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47e80039b0 0x7f47e8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47fffff640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e80051c0 0x7f47e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47feffd640 1 -- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47e80039b0 msgr2=0x7f47e8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47feffd640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f47e80039b0 0x7f47e8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47feffd640 1 -- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e80051c0 msgr2=0x7f47e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47feffd640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e80051c0 0x7f47e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47feffd640 1 -- 172.21.15.67:0/3342884198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e8003520 con 0x7f47e8004800 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.086+0000 7f47fffff640 1 -- 172.21.15.67:0/3342884198 reap_dead start 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.087+0000 7f47feffd640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47e8004800 0x7f47e8004c10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47f400c210 tx=0x7f47e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dcd5acf52da002 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.087+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4007d10 con 0x7f47e8004800 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.087+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f40058d0 con 0x7f47e8004800 2024-04-25T15:27:10.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.087+0000 7f47fe7fc640 1 -- 172.21.15.67:0/3342884198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4005ce0 con 0x7f47e8004800 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.087+0000 7f4805da8640 1 -- 172.21.15.67:0/3342884198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47e8004800 msgr2=0x7f47e8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.087+0000 7f4805da8640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47e8004800 0x7f47e8004c10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47f400c210 tx=0x7f47e8005830 comp rx=0 tx=0).stop 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.088+0000 7f4805da8640 1 -- 172.21.15.67:0/3342884198 shutdown_connections 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.088+0000 7f4805da8640 1 --2- 172.21.15.67:0/3342884198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47e8004800 0x7f47e8004c10 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.088+0000 7f4805da8640 1 -- 172.21.15.67:0/3342884198 >> 172.21.15.67:0/3342884198 conn(0x7f48000d2bc0 msgr2=0x7f48000d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.088+0000 7f4805da8640 1 -- 172.21.15.67:0/3342884198 shutdown_connections 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.088+0000 7f4805da8640 1 -- 172.21.15.67:0/3342884198 wait complete. 2024-04-25T15:27:10.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.089+0000 7f4805da8640 1 Processor -- start 2024-04-25T15:27:10.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.090+0000 7f4805da8640 1 -- start start 2024-04-25T15:27:10.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f4805da8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e8004800 0x7f4800150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f4805da8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48001eb6c0 0x7f480015a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f4805da8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f480015aa90 0x7f48001f9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f4805da8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4800160c20 con 0x7f48001eb6c0 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f4805da8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4800160aa0 con 0x7f47e8004800 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f4805da8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4800160da0 con 0x7f480015aa90 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48001eb6c0 0x7f480015a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48001eb6c0 0x7f480015a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33574/0 (socket says 172.21.15.67:33574) 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 -- 172.21.15.67:0/2102954732 learned_addr learned my addr 172.21.15.67:0/2102954732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47fffff640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f480015aa90 0x7f48001f9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47ff7fe640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e8004800 0x7f4800150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 -- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f480015aa90 msgr2=0x7f48001f9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f480015aa90 0x7f48001f9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 -- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e8004800 msgr2=0x7f4800150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e8004800 0x7f4800150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47feffd640 1 -- 172.21.15.67:0/2102954732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480015aea0 con 0x7f48001eb6c0 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.091+0000 7f47ff7fe640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e8004800 0x7f4800150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.092+0000 7f47feffd640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48001eb6c0 0x7f480015a550 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f47f4001970 tx=0x7f48001645e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.092+0000 7f47e7fff640 1 -- 172.21.15.67:0/2102954732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4040030 con 0x7f48001eb6c0 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.092+0000 7f47e7fff640 1 -- 172.21.15.67:0/2102954732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f4005e30 con 0x7f48001eb6c0 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.092+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48001ec1d0 con 0x7f48001eb6c0 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.092+0000 7f47e7fff640 1 -- 172.21.15.67:0/2102954732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f40049f0 con 0x7f48001eb6c0 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.092+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48001580b0 con 0x7f48001eb6c0 2024-04-25T15:27:10.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.093+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4800050740 con 0x7f48001eb6c0 2024-04-25T15:27:10.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.094+0000 7f47e7fff640 1 -- 172.21.15.67:0/2102954732 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f47f4043070 con 0x7f48001eb6c0 2024-04-25T15:27:10.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.094+0000 7f47e7fff640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47ec0744b0 0x7f47ec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.094+0000 7f47e7fff640 1 -- 172.21.15.67:0/2102954732 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7f47f402f070 con 0x7f48001eb6c0 2024-04-25T15:27:10.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.094+0000 7f47ff7fe640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47ec0744b0 0x7f47ec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.095+0000 7f47ff7fe640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47ec0744b0 0x7f47ec0769c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009c40 tx=0x7f47f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.099+0000 7f47e7fff640 1 -- 172.21.15.67:0/2102954732 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47f404a070 con 0x7f48001eb6c0 2024-04-25T15:27:10.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.288+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48001ebf20 con 0x7f48001eb6c0 2024-04-25T15:27:10.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.289+0000 7f47e7fff640 1 -- 172.21.15.67:0/2102954732 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+12984 (secure 0 0 0) 0x7f480015aea0 con 0x7f48001eb6c0 2024-04-25T15:27:10.290 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:10.290 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:08.949533+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/1882238407":"2024-04-26T15:27:05.937048+0000","172.21.15.67:6834/1882238407":"2024-04-26T15:27:05.937048+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47ec0744b0 msgr2=0x7f47ec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47ec0744b0 0x7f47ec0769c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009c40 tx=0x7f47f0008040 comp rx=0 tx=0).stop 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48001eb6c0 msgr2=0x7f480015a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48001eb6c0 0x7f480015a550 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f47f4001970 tx=0x7f48001645e0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 shutdown_connections 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47ec0744b0 0x7f47ec0769c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f480015aa90 0x7f48001f9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48001eb6c0 0x7f480015a550 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 --2- 172.21.15.67:0/2102954732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f47e8004800 0x7f4800150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 >> 172.21.15.67:0/2102954732 conn(0x7f48000d2bc0 msgr2=0x7f480015efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:10.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 shutdown_connections 2024-04-25T15:27:10.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.291+0000 7f4805da8640 1 -- 172.21.15.67:0/2102954732 wait complete. 2024-04-25T15:27:10.303 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:10.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:10.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.417+0000 7f530ea92640 1 Processor -- start 2024-04-25T15:27:10.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.417+0000 7f530ea92640 1 -- start start 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530ea92640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f53080d1e20 0x7f53080d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530ea92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5308164020 0x7f5308164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530ea92640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5308164970 con 0x7f53080b2e60 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530ea92640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f53080d23f0 con 0x7f53080d1e20 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530ea92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f53080d26d0 con 0x7f5308164020 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f5307fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f53080b2e60 0x7f530808d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52660/0 (socket says 172.21.15.67:52660) 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f5307fff640 1 -- 172.21.15.67:0/1608083392 learned_addr learned my addr 172.21.15.67:0/1608083392 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530d02f640 1 --2- 172.21.15.67:0/1608083392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5308164020 0x7f5308164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530c82e640 1 --2- 172.21.15.67:0/1608083392 >> v2:172.21.15.176:3300/0 conn(0x7f53080d1e20 0x7f53080d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f53077fe640 1 -- 172.21.15.67:0/1608083392 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3398111173 0 0) 0x7f52f8001110 con 0x7f53080b2e60 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f53077fe640 1 -- 172.21.15.67:0/1608083392 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f4003520 con 0x7f53080b2e60 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530d02f640 1 -- 172.21.15.67:0/1608083392 >> v2:172.21.15.176:3300/0 conn(0x7f53080d1e20 msgr2=0x7f53080d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530d02f640 1 --2- 172.21.15.67:0/1608083392 >> v2:172.21.15.176:3300/0 conn(0x7f53080d1e20 0x7f53080d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530d02f640 1 -- 172.21.15.67:0/1608083392 >> v1:172.21.15.67:6789/0 conn(0x7f53080b2e60 legacy=0x7f530808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.419+0000 7f530d02f640 1 -- 172.21.15.67:0/1608083392 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53080d29b0 con 0x7f5308164020 2024-04-25T15:27:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.420+0000 7f530d02f640 1 --2- 172.21.15.67:0/1608083392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5308164020 0x7f5308164430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52f00097a0 tx=0x7f52f002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71ac0ad8e549ea92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.420+0000 7f53077fe640 1 -- 172.21.15.67:0/1608083392 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0037070 con 0x7f5308164020 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.420+0000 7f53077fe640 1 -- 172.21.15.67:0/1608083392 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f0032440 con 0x7f5308164020 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.420+0000 7f53077fe640 1 -- 172.21.15.67:0/1608083392 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f00327f0 con 0x7f5308164020 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 -- 172.21.15.67:0/1608083392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5308164020 msgr2=0x7f5308164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 --2- 172.21.15.67:0/1608083392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5308164020 0x7f5308164430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52f00097a0 tx=0x7f52f002a210 comp rx=0 tx=0).stop 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 -- 172.21.15.67:0/1608083392 shutdown_connections 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 --2- 172.21.15.67:0/1608083392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5308164020 0x7f5308164430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 --2- 172.21.15.67:0/1608083392 >> v2:172.21.15.176:3300/0 conn(0x7f53080d1e20 0x7f53080d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 -- 172.21.15.67:0/1608083392 >> 172.21.15.67:0/1608083392 conn(0x7f53080d0530 msgr2=0x7f53080d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 -- 172.21.15.67:0/1608083392 shutdown_connections 2024-04-25T15:27:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.421+0000 7f530ea92640 1 -- 172.21.15.67:0/1608083392 wait complete. 2024-04-25T15:27:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.423+0000 7f530ea92640 1 Processor -- start 2024-04-25T15:27:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.423+0000 7f530ea92640 1 -- start start 2024-04-25T15:27:10.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530ea92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080b2e60 0x7f53080722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530ea92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53080d1e20 0x7f5308156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530ea92640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5308164020 0x7f53080b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530ea92640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53080ce070 con 0x7f5308164020 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530ea92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f53080d2f70 con 0x7f53080b2e60 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530ea92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f53080ce1f0 con 0x7f53080d1e20 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530d02f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5308164020 0x7f53080b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.424+0000 7f530d02f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5308164020 0x7f53080b3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33582/0 (socket says 172.21.15.67:33582) 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530d02f640 1 -- 172.21.15.67:0/2088618329 learned_addr learned my addr 172.21.15.67:0/2088618329 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530c82e640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080b2e60 0x7f53080722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f5307fff640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53080d1e20 0x7f5308156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530d02f640 1 -- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53080d1e20 msgr2=0x7f5308156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530d02f640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53080d1e20 0x7f5308156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530d02f640 1 -- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080b2e60 msgr2=0x7f53080722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530d02f640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080b2e60 0x7f53080722c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530d02f640 1 -- 172.21.15.67:0/2088618329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5308072d50 con 0x7f5308164020 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f5307fff640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53080d1e20 0x7f5308156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530d02f640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5308164020 0x7f53080b3ae0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f52f002a250 tx=0x7f52f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f53057fa640 1 -- 172.21.15.67:0/2088618329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0037040 con 0x7f5308164020 2024-04-25T15:27:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f53057fa640 1 -- 172.21.15.67:0/2088618329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f003b430 con 0x7f5308164020 2024-04-25T15:27:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f53057fa640 1 -- 172.21.15.67:0/2088618329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f003b7e0 con 0x7f5308164020 2024-04-25T15:27:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530815d110 con 0x7f5308164020 2024-04-25T15:27:10.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.425+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530815d6d0 con 0x7f5308164020 2024-04-25T15:27:10.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.426+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5308050740 con 0x7f5308164020 2024-04-25T15:27:10.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.427+0000 7f53057fa640 1 -- 172.21.15.67:0/2088618329 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f52f0044020 con 0x7f5308164020 2024-04-25T15:27:10.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.428+0000 7f53057fa640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52e80744e0 0x7f52e80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.428+0000 7f53057fa640 1 -- 172.21.15.67:0/2088618329 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7f52f00b5140 con 0x7f5308164020 2024-04-25T15:27:10.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.428+0000 7f530c82e640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52e80744e0 0x7f52e80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.428+0000 7f530c82e640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52e80744e0 0x7f52e80769f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f53000073f0 tx=0x7f530000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.432+0000 7f53057fa640 1 -- 172.21.15.67:0/2088618329 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52f007fbd0 con 0x7f5308164020 2024-04-25T15:27:10.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.616+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5308072ae0 con 0x7f5308164020 2024-04-25T15:27:10.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.616+0000 7f53057fa640 1 -- 172.21.15.67:0/2088618329 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7f52f007f4d0 con 0x7f5308164020 2024-04-25T15:27:10.617 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52e80744e0 msgr2=0x7f52e80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52e80744e0 0x7f52e80769f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f53000073f0 tx=0x7f530000b040 comp rx=0 tx=0).stop 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5308164020 msgr2=0x7f53080b3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5308164020 0x7f53080b3ae0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f52f002a250 tx=0x7f52f00046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 shutdown_connections 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52e80744e0 0x7f52e80769f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5308164020 0x7f53080b3ae0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53080d1e20 0x7f5308156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 --2- 172.21.15.67:0/2088618329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080b2e60 0x7f53080722c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 >> 172.21.15.67:0/2088618329 conn(0x7f53080d0530 msgr2=0x7f53080b1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 shutdown_connections 2024-04-25T15:27:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.618+0000 7f530ea92640 1 -- 172.21.15.67:0/2088618329 wait complete. 2024-04-25T15:27:10.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values' 2024-04-25T15:27:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.745+0000 7f3932d6c640 1 Processor -- start 2024-04-25T15:27:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.745+0000 7f3932d6c640 1 -- start start 2024-04-25T15:27:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3932d6c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f392c0b91d0 0x7f392c161e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3932d6c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f392c0cf770 0x7f392c0b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3932d6c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f392c0cfc30 con 0x7f392c1623c0 2024-04-25T15:27:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3932d6c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f392c0d1ff0 con 0x7f392c0b91d0 2024-04-25T15:27:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3932d6c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f392c0d2890 con 0x7f392c0cf770 2024-04-25T15:27:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3930b08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f392c0b91d0 0x7f392c161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3930b08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f392c0b91d0 0x7f392c161e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33586/0 (socket says 172.21.15.67:33586) 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.746+0000 7f3930b08640 1 -- 172.21.15.67:0/2308805212 learned_addr learned my addr 172.21.15.67:0/2308805212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.747+0000 7f3931309640 1 --2- 172.21.15.67:0/2308805212 >> v2:172.21.15.176:3300/0 conn(0x7f392c0cf770 0x7f392c0b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.747+0000 7f3930b08640 1 -- 172.21.15.67:0/2308805212 >> v2:172.21.15.176:3300/0 conn(0x7f392c0cf770 msgr2=0x7f392c0b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.747+0000 7f3930b08640 1 --2- 172.21.15.67:0/2308805212 >> v2:172.21.15.176:3300/0 conn(0x7f392c0cf770 0x7f392c0b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.747+0000 7f3930b08640 1 -- 172.21.15.67:0/2308805212 >> v1:172.21.15.176:6789/0 conn(0x7f392c1623c0 legacy=0x7f392c1627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.747+0000 7f3930b08640 1 -- 172.21.15.67:0/2308805212 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392c0d2240 con 0x7f392c0b91d0 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.747+0000 7f3931309640 1 --2- 172.21.15.67:0/2308805212 >> v2:172.21.15.176:3300/0 conn(0x7f392c0cf770 0x7f392c0b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.747+0000 7f3930b08640 1 --2- 172.21.15.67:0/2308805212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c161e80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f392c0caa70 tx=0x7f392002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983d18a145462ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f392b7fe640 1 -- 172.21.15.67:0/2308805212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920037070 con 0x7f392c0b91d0 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f392b7fe640 1 -- 172.21.15.67:0/2308805212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3920032440 con 0x7f392c0b91d0 2024-04-25T15:27:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f392b7fe640 1 -- 172.21.15.67:0/2308805212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920032810 con 0x7f392c0b91d0 2024-04-25T15:27:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 -- 172.21.15.67:0/2308805212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 msgr2=0x7f392c161e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 --2- 172.21.15.67:0/2308805212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c161e80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f392c0caa70 tx=0x7f392002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 -- 172.21.15.67:0/2308805212 shutdown_connections 2024-04-25T15:27:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 --2- 172.21.15.67:0/2308805212 >> v2:172.21.15.176:3300/0 conn(0x7f392c0cf770 0x7f392c0b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 --2- 172.21.15.67:0/2308805212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c161e80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 -- 172.21.15.67:0/2308805212 >> 172.21.15.67:0/2308805212 conn(0x7f392c0d3aa0 msgr2=0x7f392c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 -- 172.21.15.67:0/2308805212 shutdown_connections 2024-04-25T15:27:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.748+0000 7f3932d6c640 1 -- 172.21.15.67:0/2308805212 wait complete. 2024-04-25T15:27:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.750+0000 7f3932d6c640 1 Processor -- start 2024-04-25T15:27:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.750+0000 7f3932d6c640 1 -- start start 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c15d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c1623c0 0x7f392c15dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c0ccfa0 0x7f392c0c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f392c0b24a0 con 0x7f392c0b91d0 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f392c0b2320 con 0x7f392c1623c0 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f392c0b0bd0 con 0x7f392c0ccfa0 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c15d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c15d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33598/0 (socket says 172.21.15.67:33598) 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 -- 172.21.15.67:0/1117459190 learned_addr learned my addr 172.21.15.67:0/1117459190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f392bfff640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c1623c0 0x7f392c15dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3931309640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c0ccfa0 0x7f392c0c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 -- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c0ccfa0 msgr2=0x7f392c0c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c0ccfa0 0x7f392c0c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 -- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c1623c0 msgr2=0x7f392c15dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c1623c0 0x7f392c15dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 -- 172.21.15.67:0/1117459190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392c15e3e0 con 0x7f392c0b91d0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3931309640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c0ccfa0 0x7f392c0c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3930b08640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c15d1f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3920000c00 tx=0x7f39200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f39297fa640 1 -- 172.21.15.67:0/1117459190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920041020 con 0x7f392c0b91d0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f39297fa640 1 -- 172.21.15.67:0/1117459190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3920030970 con 0x7f392c0b91d0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f392c15e6c0 con 0x7f392c0b91d0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f39297fa640 1 -- 172.21.15.67:0/1117459190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920037040 con 0x7f392c0b91d0 2024-04-25T15:27:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.752+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f392c15ec80 con 0x7f392c0b91d0 2024-04-25T15:27:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.754+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f392c050740 con 0x7f392c0b91d0 2024-04-25T15:27:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.754+0000 7f39297fa640 1 -- 172.21.15.67:0/1117459190 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3920044020 con 0x7f392c0b91d0 2024-04-25T15:27:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.754+0000 7f39297fa640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f39100744e0 0x7f39100769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.755+0000 7f39297fa640 1 -- 172.21.15.67:0/1117459190 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7f39200b59c0 con 0x7f392c0b91d0 2024-04-25T15:27:10.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.755+0000 7f392bfff640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f39100744e0 0x7f39100769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:10.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.755+0000 7f392bfff640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f39100744e0 0x7f39100769f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3918006fd0 tx=0x7f3918008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:10.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:10.759+0000 7f39297fa640 1 -- 172.21.15.67:0/1117459190 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39200b7e20 con 0x7f392c0b91d0 2024-04-25T15:27:11.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.025+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]} v 0) -- 0x7f392c0cf970 con 0x7f392c0b91d0 2024-04-25T15:27:11.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.963+0000 7f39297fa640 1 -- 172.21.15.67:0/1117459190 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]}]=0 v121) ==== 148+0+0 (secure 0 0 0) 0x7f392004b030 con 0x7f392c0b91d0 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f39100744e0 msgr2=0x7f39100769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f39100744e0 0x7f39100769f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3918006fd0 tx=0x7f3918008040 comp rx=0 tx=0).stop 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 msgr2=0x7f392c15d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c15d1f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3920000c00 tx=0x7f39200046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 shutdown_connections 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f39100744e0 0x7f39100769f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c0ccfa0 0x7f392c0c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c1623c0 0x7f392c15dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:11.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 --2- 172.21.15.67:0/1117459190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f392c0b91d0 0x7f392c15d1f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 >> 172.21.15.67:0/1117459190 conn(0x7f392c0d3aa0 msgr2=0x7f392c1613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 shutdown_connections 2024-04-25T15:27:11.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:11.966+0000 7f3932d6c640 1 -- 172.21.15.67:0/1117459190 wait complete. 2024-04-25T15:27:11.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:27:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.101+0000 7f726fdc0640 1 Processor -- start 2024-04-25T15:27:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.101+0000 7f726fdc0640 1 -- start start 2024-04-25T15:27:12.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726fdc0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 0x7f72680d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726fdc0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72680d23f0 con 0x7f7268164020 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726fdc0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72680d26d0 con 0x7f72680d1e20 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726fdc0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f72680d29b0 con 0x7f72680b2e60 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726e35d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7268164020 0x7f7268164430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52666/0 (socket says 172.21.15.67:52666) 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726e35d640 1 -- 172.21.15.67:0/113888083 learned_addr learned my addr 172.21.15.67:0/113888083 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726d35b640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 0x7f72680d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114770310 0 0) 0x7f72680d23f0 con 0x7f7268164020 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7250003520 con 0x7f7268164020 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167847096 0 0) 0x7f72680d26d0 con 0x7f72680d1e20 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72680d23f0 con 0x7f72680d1e20 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2834167145 0 0) 0x7f7250003520 con 0x7f7268164020 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 msgr2=0x7f72680d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 0x7f72680d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 >> v1:172.21.15.176:6789/0 conn(0x7f72680d1e20 legacy=0x7f726808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72680d2c90 con 0x7f7268164020 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f72580019f0 con 0x7f7268164020 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 >> v1:172.21.15.67:6789/0 conn(0x7f7268164020 legacy=0x7f7268164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f7250003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.103+0000 7f726cb5a640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 0x7f7250004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726cb5a640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7250004d40 0x7f7250005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72680d29b0 con 0x7f7250003b70 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f725000bac0 con 0x7f7250004d40 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f725000bcc0 con 0x7f72680b2e60 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726db5c640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f7250003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726e35d640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7250004d40 0x7f7250005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726d35b640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 0x7f7250004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726db5c640 1 -- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 msgr2=0x7f7250004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726db5c640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72680b2e60 0x7f7250004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726db5c640 1 -- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7250004d40 msgr2=0x7f7250005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726db5c640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7250004d40 0x7f7250005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.104+0000 7f726db5c640 1 -- 172.21.15.67:0/113888083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7250003520 con 0x7f7250003b70 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.105+0000 7f726db5c640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f7250003fc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f72600018d0 tx=0x7f7250005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef16a4d1b2ed4b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.105+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72600014f0 con 0x7f7250003b70 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.105+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72600052d0 con 0x7f7250003b70 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.105+0000 7f726cb5a640 1 -- 172.21.15.67:0/113888083 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726002aca0 con 0x7f7250003b70 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.105+0000 7f726fdc0640 1 -- 172.21.15.67:0/113888083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 msgr2=0x7f7250003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.105+0000 7f726fdc0640 1 --2- 172.21.15.67:0/113888083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f7250003fc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f72600018d0 tx=0x7f7250005830 comp rx=0 tx=0).stop 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.105+0000 7f726e35d640 1 -- 172.21.15.67:0/113888083 reap_dead start 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.106+0000 7f726fdc0640 1 -- 172.21.15.67:0/113888083 shutdown_connections 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.106+0000 7f726fdc0640 1 -- 172.21.15.67:0/113888083 >> 172.21.15.67:0/113888083 conn(0x7f72680d0530 msgr2=0x7f72680d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.106+0000 7f726fdc0640 1 -- 172.21.15.67:0/113888083 shutdown_connections 2024-04-25T15:27:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.106+0000 7f726fdc0640 1 -- 172.21.15.67:0/113888083 wait complete. 2024-04-25T15:27:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.107+0000 7f726fdc0640 1 Processor -- start 2024-04-25T15:27:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.108+0000 7f726fdc0640 1 -- start start 2024-04-25T15:27:12.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726fdc0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f726815d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726fdc0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f726815e150 0x7f72681f9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726fdc0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72681fa050 0x7f72680cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726fdc0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72680d3610 con 0x7f7250003b70 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726fdc0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f72680d3490 con 0x7f726815e150 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726fdc0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f72680d3790 con 0x7f72681fa050 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f726815d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f726815d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33620/0 (socket says 172.21.15.67:33620) 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 -- 172.21.15.67:0/1774939254 learned_addr learned my addr 172.21.15.67:0/1774939254 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726d35b640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f726815e150 0x7f72681f9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726e35d640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72681fa050 0x7f72680cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 -- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72681fa050 msgr2=0x7f72680cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72681fa050 0x7f72680cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 -- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f726815e150 msgr2=0x7f72681f9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f726815e150 0x7f72681f9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726db5c640 1 -- 172.21.15.67:0/1774939254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72681fa520 con 0x7f7250003b70 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.109+0000 7f726e35d640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72681fa050 0x7f72680cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.110+0000 7f726db5c640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f726815d220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f72600018d0 tx=0x7f72600012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.110+0000 7f72667fc640 1 -- 172.21.15.67:0/1774939254 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726003f020 con 0x7f7250003b70 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.110+0000 7f72667fc640 1 -- 172.21.15.67:0/1774939254 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7260006e50 con 0x7f7250003b70 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.110+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72681fa800 con 0x7f7250003b70 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.110+0000 7f72667fc640 1 -- 172.21.15.67:0/1774939254 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260037040 con 0x7f7250003b70 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.110+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72680cb850 con 0x7f7250003b70 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.111+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7268050740 con 0x7f7250003b70 2024-04-25T15:27:12.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.112+0000 7f72667fc640 1 -- 172.21.15.67:0/1774939254 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7260042020 con 0x7f7250003b70 2024-04-25T15:27:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.112+0000 7f72667fc640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f72540744e0 0x7f72540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.112+0000 7f72667fc640 1 -- 172.21.15.67:0/1774939254 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7f72600b4cd0 con 0x7f7250003b70 2024-04-25T15:27:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.112+0000 7f726d35b640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f72540744e0 0x7f72540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.113+0000 7f726d35b640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f72540744e0 0x7f72540769f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f726815d1d0 tx=0x7f725c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:12.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.117+0000 7f72667fc640 1 -- 172.21.15.67:0/1774939254 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7260049030 con 0x7f7250003b70 2024-04-25T15:27:12.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.301+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f7268071000 con 0x7f7250003b70 2024-04-25T15:27:12.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.305+0000 7f72667fc640 1 -- 172.21.15.67:0/1774939254 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7f726007f8b0 con 0x7f7250003b70 2024-04-25T15:27:12.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f72540744e0 msgr2=0x7f72540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f72540744e0 0x7f72540769f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f726815d1d0 tx=0x7f725c008040 comp rx=0 tx=0).stop 2024-04-25T15:27:12.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 msgr2=0x7f726815d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f726815d220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f72600018d0 tx=0x7f72600012d0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 shutdown_connections 2024-04-25T15:27:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f72540744e0 0x7f72540769f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72681fa050 0x7f72680cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f726815e150 0x7f72681f9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 --2- 172.21.15.67:0/1774939254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250003b70 0x7f726815d220 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 >> 172.21.15.67:0/1774939254 conn(0x7f72680d0530 msgr2=0x7f726815fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 shutdown_connections 2024-04-25T15:27:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.307+0000 7f726fdc0640 1 -- 172.21.15.67:0/1774939254 wait complete. 2024-04-25T15:27:12.319 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:27:12.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:27:12.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:27:12.407 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:27:12.408 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:12.408 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:12.423 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:27:12.424 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:12.424 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:12.465 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:27:12.465 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:12.465 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:12.481 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:27:12.481 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:12.481 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:12.522 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:27:12.523 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:27:12.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.634+0000 7f4205ce7640 1 Processor -- start 2024-04-25T15:27:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.634+0000 7f4205ce7640 1 -- start start 2024-04-25T15:27:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.635+0000 7f4205ce7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42001443c0 0x7f420006fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.635+0000 7f4205ce7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4200070370 0x7f42000cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.635+0000 7f4205ce7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f42000cfb20 0x7f42000ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.635+0000 7f4205ce7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f42000708d0 con 0x7f4200070370 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.635+0000 7f4205ce7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f420015f160 con 0x7f42000cfb20 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.635+0000 7f4205ce7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42000cff30 con 0x7f42001443c0 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4200070370 0x7f42000cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4200070370 0x7f42000cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33628/0 (socket says 172.21.15.67:33628) 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 -- 172.21.15.67:0/2354169506 learned_addr learned my addr 172.21.15.67:0/2354169506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41ff7fe640 1 --2- 172.21.15.67:0/2354169506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42001443c0 0x7f420006fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41fffff640 1 --2- 172.21.15.67:0/2354169506 >> v2:172.21.15.176:3300/0 conn(0x7f42000cfb20 0x7f42000ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 -- 172.21.15.67:0/2354169506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42001443c0 msgr2=0x7f420006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 --2- 172.21.15.67:0/2354169506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42001443c0 0x7f420006fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 -- 172.21.15.67:0/2354169506 >> v2:172.21.15.176:3300/0 conn(0x7f42000cfb20 msgr2=0x7f42000ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 --2- 172.21.15.67:0/2354169506 >> v2:172.21.15.176:3300/0 conn(0x7f42000cfb20 0x7f42000ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.636+0000 7f41feffd640 1 -- 172.21.15.67:0/2354169506 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420015f5f0 con 0x7f4200070370 2024-04-25T15:27:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.637+0000 7f41feffd640 1 --2- 172.21.15.67:0/2354169506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200070370 0x7f42000cf5e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f420006f8f0 tx=0x7f41f402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffa7ffaade83b13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.637+0000 7f41fe7fc640 1 -- 172.21.15.67:0/2354169506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4037070 con 0x7f4200070370 2024-04-25T15:27:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.637+0000 7f41fe7fc640 1 -- 172.21.15.67:0/2354169506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f4032440 con 0x7f4200070370 2024-04-25T15:27:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.637+0000 7f41fe7fc640 1 -- 172.21.15.67:0/2354169506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4032810 con 0x7f4200070370 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.637+0000 7f4205ce7640 1 -- 172.21.15.67:0/2354169506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200070370 msgr2=0x7f42000cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.637+0000 7f4205ce7640 1 --2- 172.21.15.67:0/2354169506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200070370 0x7f42000cf5e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f420006f8f0 tx=0x7f41f402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.638+0000 7f4205ce7640 1 -- 172.21.15.67:0/2354169506 shutdown_connections 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.638+0000 7f4205ce7640 1 --2- 172.21.15.67:0/2354169506 >> v2:172.21.15.176:3300/0 conn(0x7f42000cfb20 0x7f42000ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.638+0000 7f4205ce7640 1 --2- 172.21.15.67:0/2354169506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4200070370 0x7f42000cf5e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.638+0000 7f4205ce7640 1 --2- 172.21.15.67:0/2354169506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42001443c0 0x7f420006fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.638+0000 7f4205ce7640 1 -- 172.21.15.67:0/2354169506 >> 172.21.15.67:0/2354169506 conn(0x7f42000d1ae0 msgr2=0x7f42000d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.638+0000 7f4205ce7640 1 -- 172.21.15.67:0/2354169506 shutdown_connections 2024-04-25T15:27:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.638+0000 7f4205ce7640 1 -- 172.21.15.67:0/2354169506 wait complete. 2024-04-25T15:27:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.639+0000 7f4205ce7640 1 Processor -- start 2024-04-25T15:27:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.640+0000 7f4205ce7640 1 -- start start 2024-04-25T15:27:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f4205ce7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4200070370 0x7f420013e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f4205ce7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42000cfb20 0x7f42001e8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f4205ce7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42001443c0 0x7f42001e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f4205ce7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f420015edf0 con 0x7f42001443c0 2024-04-25T15:27:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f4205ce7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f420015eb10 con 0x7f42000cfb20 2024-04-25T15:27:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f4205ce7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42001447d0 con 0x7f4200070370 2024-04-25T15:27:12.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42001443c0 0x7f42001e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42001443c0 0x7f42001e8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33630/0 (socket says 172.21.15.67:33630) 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 -- 172.21.15.67:0/3278775863 learned_addr learned my addr 172.21.15.67:0/3278775863 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41ff7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4200070370 0x7f420013e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41feffd640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42000cfb20 0x7f42001e8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 -- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4200070370 msgr2=0x7f420013e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4200070370 0x7f420013e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 -- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42000cfb20 msgr2=0x7f42001e8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42000cfb20 0x7f42001e8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 -- 172.21.15.67:0/3278775863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420013ed00 con 0x7f42001443c0 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41ff7fe640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4200070370 0x7f420013e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.641+0000 7f41fffff640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42001443c0 0x7f42001e8fb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f41f8007d50 tx=0x7f41f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.642+0000 7f41e7fff640 1 -- 172.21.15.67:0/3278775863 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8014420 con 0x7f42001443c0 2024-04-25T15:27:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.642+0000 7f41e7fff640 1 -- 172.21.15.67:0/3278775863 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f8019070 con 0x7f42001443c0 2024-04-25T15:27:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.642+0000 7f41e7fff640 1 -- 172.21.15.67:0/3278775863 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8014820 con 0x7f42001443c0 2024-04-25T15:27:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.642+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420013efe0 con 0x7f42001443c0 2024-04-25T15:27:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.642+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420013f5a0 con 0x7f42001443c0 2024-04-25T15:27:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.643+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420013ea70 con 0x7f42001443c0 2024-04-25T15:27:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.643+0000 7f41e7fff640 1 -- 172.21.15.67:0/3278775863 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f41f80149c0 con 0x7f42001443c0 2024-04-25T15:27:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.644+0000 7f41e7fff640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f41ec0744e0 0x7f41ec0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.644+0000 7f41e7fff640 1 -- 172.21.15.67:0/3278775863 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7f41f80955c0 con 0x7f42001443c0 2024-04-25T15:27:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.644+0000 7f41ff7fe640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f41ec0744e0 0x7f41ec0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.645+0000 7f41ff7fe640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f41ec0744e0 0x7f41ec0769f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f42000d1860 tx=0x7f41f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:12.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.649+0000 7f41e7fff640 1 -- 172.21.15.67:0/3278775863 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41f805fa00 con 0x7f42001443c0 2024-04-25T15:27:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.874+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4200050740 con 0x7f42001443c0 2024-04-25T15:27:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.875+0000 7f41e7fff640 1 -- 172.21.15.67:0/3278775863 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3875 (secure 0 0 0) 0x7f41f8063d00 con 0x7f42001443c0 2024-04-25T15:27:12.876 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:12.876 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5054,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/1556919488","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1556919488},{"type":"v1","addr":"172.21.15.67:6833","nonce":1556919488}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5317,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/4071024412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4071024412},{"type":"v1","addr":"172.21.15.176:6837","nonce":4071024412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3791679818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3791679818},{"type":"v1","addr":"172.21.15.176:6836","nonce":3791679818}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/4240510974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":4240510974},{"type":"v1","addr":"172.21.15.67:6835","nonce":4240510974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[]} 2024-04-25T15:27:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 27 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f41ec0744e0 msgr2=0x7f41ec0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f41ec0744e0 0x7f41ec0769f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f42000d1860 tx=0x7f41f0008040 comp rx=0 tx=0).stop 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42001443c0 msgr2=0x7f42001e8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42001443c0 0x7f42001e8fb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f41f8007d50 tx=0x7f41f800e4b0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 shutdown_connections 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f41ec0744e0 0x7f41ec0769f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42001443c0 0x7f42001e8fb0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42000cfb20 0x7f42001e8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 --2- 172.21.15.67:0/3278775863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4200070370 0x7f420013e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.877+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 >> 172.21.15.67:0/3278775863 conn(0x7f42000d1ae0 msgr2=0x7f420006d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.878+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 shutdown_connections 2024-04-25T15:27:12.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:12.878+0000 7f4205ce7640 1 -- 172.21.15.67:0/3278775863 wait complete. 2024-04-25T15:27:12.890 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:27:12.890 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:27:12.890 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:27:12.890 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:12.890 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:27:12.890 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:27:12.890 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:27:12.891 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:12.891 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:27:12.891 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:27:12.891 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:27:12.891 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:12.891 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:27:12.891 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:27:12.892 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:27:12.892 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:12.901 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:27:12.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:27:12.903 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:27:12.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:27:12.904 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:27:12.904 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:27:12.905 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:27:12.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:27:12.945 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:27:12.946 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:12.947 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:27:12.947 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:12.948 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:27:12.948 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:12.949 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:27:12.949 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:12.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:27:13.034 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:27:13.035 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:27:13.039 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:27:13.039 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:27:13.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.069+0000 7fa4b4851640 1 Processor -- start 2024-04-25T15:27:13.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.069+0000 7fa4b4851640 1 -- start start 2024-04-25T15:27:13.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4b4851640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4b00b91d0 0x7fa4b0161e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:13.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4b4851640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa4b01623c0 0x7fa4b01627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:13.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4b4851640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:13.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4b4851640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa4b00d1ff0 con 0x7fa4b00b91d0 2024-04-25T15:27:13.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4b4851640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa4b00d2890 con 0x7fa4b01623c0 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4b4851640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b00cfc30 con 0x7fa4b00ce760 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4ae59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4b00b91d0 0x7fa4b0161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4ae59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4b00b91d0 0x7fa4b0161e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33676/0 (socket says 172.21.15.67:33676) 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4ae59c640 1 -- 172.21.15.67:0/781006841 learned_addr learned my addr 172.21.15.67:0/781006841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4aed9d640 1 --2- 172.21.15.67:0/781006841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.070+0000 7fa4add9b640 1 --2- 172.21.15.67:0/781006841 >> v2:172.21.15.176:3300/0 conn(0x7fa4b01623c0 0x7fa4b01627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.071+0000 7fa4ae59c640 1 -- 172.21.15.67:0/781006841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 msgr2=0x7fa4b00ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.071+0000 7fa4ae59c640 1 --2- 172.21.15.67:0/781006841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.071+0000 7fa4ae59c640 1 -- 172.21.15.67:0/781006841 >> v2:172.21.15.176:3300/0 conn(0x7fa4b01623c0 msgr2=0x7fa4b01627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.071+0000 7fa4ae59c640 1 --2- 172.21.15.67:0/781006841 >> v2:172.21.15.176:3300/0 conn(0x7fa4b01623c0 0x7fa4b01627d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.071+0000 7fa4ae59c640 1 -- 172.21.15.67:0/781006841 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b00d2240 con 0x7fa4b00b91d0 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.071+0000 7fa4ae59c640 1 --2- 172.21.15.67:0/781006841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b0161e80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0075fa0 tx=0x7fa4a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea33699581e7c815 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4ad59a640 1 -- 172.21.15.67:0/781006841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0037070 con 0x7fa4b00b91d0 2024-04-25T15:27:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4ad59a640 1 -- 172.21.15.67:0/781006841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a0032440 con 0x7fa4b00b91d0 2024-04-25T15:27:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4ad59a640 1 -- 172.21.15.67:0/781006841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0032810 con 0x7fa4b00b91d0 2024-04-25T15:27:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 -- 172.21.15.67:0/781006841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 msgr2=0x7fa4b0161e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:13.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 --2- 172.21.15.67:0/781006841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b0161e80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0075fa0 tx=0x7fa4a002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 -- 172.21.15.67:0/781006841 shutdown_connections 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 --2- 172.21.15.67:0/781006841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 --2- 172.21.15.67:0/781006841 >> v2:172.21.15.176:3300/0 conn(0x7fa4b01623c0 0x7fa4b01627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 --2- 172.21.15.67:0/781006841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b0161e80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 -- 172.21.15.67:0/781006841 >> 172.21.15.67:0/781006841 conn(0x7fa4b00d3aa0 msgr2=0x7fa4b00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 -- 172.21.15.67:0/781006841 shutdown_connections 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.072+0000 7fa4b4851640 1 -- 172.21.15.67:0/781006841 wait complete. 2024-04-25T15:27:13.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.074+0000 7fa4b4851640 1 Processor -- start 2024-04-25T15:27:13.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.074+0000 7fa4b4851640 1 -- start start 2024-04-25T15:27:13.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4b4851640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b00c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4b4851640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4b4851640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4b01623c0 0x7fa4b01e8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4b4851640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b00b2320 con 0x7fa4b00b91d0 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4b4851640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b0162d10 con 0x7fa4b01623c0 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4b4851640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b00b24a0 con 0x7fa4b00ce760 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b00c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b00c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33692/0 (socket says 172.21.15.67:33692) 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 -- 172.21.15.67:0/2473591019 learned_addr learned my addr 172.21.15.67:0/2473591019 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4aed9d640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4b01623c0 0x7fa4b01e8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4add9b640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 -- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 msgr2=0x7fa4b00c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 -- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4b01623c0 msgr2=0x7fa4b01e8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4b01623c0 0x7fa4b01e8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 -- 172.21.15.67:0/2473591019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b00c98c0 con 0x7fa4b00b91d0 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa4ae59c640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b00c8e20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa4a002a220 tx=0x7fa4a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.076+0000 7fa49effd640 1 -- 172.21.15.67:0/2473591019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0041020 con 0x7fa4b00b91d0 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.077+0000 7fa49effd640 1 -- 172.21.15.67:0/2473591019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a003d620 con 0x7fa4b00b91d0 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.077+0000 7fa49effd640 1 -- 172.21.15.67:0/2473591019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0037040 con 0x7fa4b00b91d0 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.077+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b00c9ba0 con 0x7fa4b00b91d0 2024-04-25T15:27:13.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.077+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b01e9700 con 0x7fa4b00b91d0 2024-04-25T15:27:13.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.077+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b0050740 con 0x7fa4b00b91d0 2024-04-25T15:27:13.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.078+0000 7fa49effd640 1 -- 172.21.15.67:0/2473591019 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa4a0044020 con 0x7fa4b00b91d0 2024-04-25T15:27:13.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.079+0000 7fa49effd640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4900744e0 0x7fa4900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:13.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.079+0000 7fa49effd640 1 -- 172.21.15.67:0/2473591019 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4943+0+0 (secure 0 0 0) 0x7fa4a00b5450 con 0x7fa4b00b91d0 2024-04-25T15:27:13.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.079+0000 7fa4add9b640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4900744e0 0x7fa4900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:13.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.080+0000 7fa4add9b640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4900744e0 0x7fa4900769f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa4b015c650 tx=0x7fa4a4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:13.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.084+0000 7fa49effd640 1 -- 172.21.15.67:0/2473591019 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4a004b030 con 0x7fa4b00b91d0 2024-04-25T15:27:13.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:13.287+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa4b00d4700 con 0x7fa4b00b91d0 2024-04-25T15:27:14.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.326+0000 7fa49effd640 1 -- 172.21.15.67:0/2473591019 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) ==== 97+0+0 (secure 0 0 0) 0x7fa4a007f920 con 0x7fa4b00b91d0 2024-04-25T15:27:14.327 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4900744e0 msgr2=0x7fa4900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4900744e0 0x7fa4900769f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa4b015c650 tx=0x7fa4a4009190 comp rx=0 tx=0).stop 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 msgr2=0x7fa4b00c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b00c8e20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa4a002a220 tx=0x7fa4a00046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 shutdown_connections 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4900744e0 0x7fa4900769f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4b01623c0 0x7fa4b01e8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4b00ce760 0x7fa4b00c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 --2- 172.21.15.67:0/2473591019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b00b91d0 0x7fa4b00c8e20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 >> 172.21.15.67:0/2473591019 conn(0x7fa4b00d3aa0 msgr2=0x7fa4b00d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 shutdown_connections 2024-04-25T15:27:14.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.329+0000 7fa4b4851640 1 -- 172.21.15.67:0/2473591019 wait complete. 2024-04-25T15:27:14.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:27:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.461+0000 7fa45f3e3640 1 Processor -- start 2024-04-25T15:27:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.461+0000 7fa45f3e3640 1 -- start start 2024-04-25T15:27:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45f3e3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa45806ccd0 0x7fa4580d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45f3e3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa4580d4a30 0x7fa4580d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45f3e3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4580d1830 con 0x7fa4580d0de0 2024-04-25T15:27:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45f3e3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa458160dc0 con 0x7fa45806ccd0 2024-04-25T15:27:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45f3e3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa4581610a0 con 0x7fa4580d4a30 2024-04-25T15:27:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45c97e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa45806ccd0 0x7fa4580d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45d17f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa4580d0de0 0x7fa45806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52674/0 (socket says 172.21.15.67:52674) 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45c97e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa45806ccd0 0x7fa4580d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33694/0 (socket says 172.21.15.67:33694) 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45d17f640 1 -- 172.21.15.67:0/2272900187 learned_addr learned my addr 172.21.15.67:0/2272900187 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45d980640 1 --2- 172.21.15.67:0/2272900187 >> v2:172.21.15.176:3300/0 conn(0x7fa4580d4a30 0x7fa4580d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45c97e640 1 -- 172.21.15.67:0/2272900187 >> v2:172.21.15.176:3300/0 conn(0x7fa4580d4a30 msgr2=0x7fa4580d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45c97e640 1 --2- 172.21.15.67:0/2272900187 >> v2:172.21.15.176:3300/0 conn(0x7fa4580d4a30 0x7fa4580d4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45c97e640 1 -- 172.21.15.67:0/2272900187 >> v1:172.21.15.67:6789/0 conn(0x7fa4580d0de0 legacy=0x7fa45806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa45c97e640 1 -- 172.21.15.67:0/2272900187 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa458161380 con 0x7fa45806ccd0 2024-04-25T15:27:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.463+0000 7fa457fff640 1 -- 172.21.15.67:0/2272900187 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3593834329 0 0) 0x7fa4580d1830 con 0x7fa4580d0de0 2024-04-25T15:27:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.464+0000 7fa45c97e640 1 --2- 172.21.15.67:0/2272900187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa45806ccd0 0x7fa4580d11f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa4580d1730 tx=0x7fa44c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7993e96d5536471b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.464+0000 7fa457fff640 1 -- 172.21.15.67:0/2272900187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c037070 con 0x7fa45806ccd0 2024-04-25T15:27:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.464+0000 7fa457fff640 1 -- 172.21.15.67:0/2272900187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa44c032440 con 0x7fa45806ccd0 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.464+0000 7fa457fff640 1 -- 172.21.15.67:0/2272900187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c02ae60 con 0x7fa45806ccd0 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 -- 172.21.15.67:0/2272900187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa45806ccd0 msgr2=0x7fa4580d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/2272900187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa45806ccd0 0x7fa4580d11f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa4580d1730 tx=0x7fa44c02a110 comp rx=0 tx=0).stop 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 -- 172.21.15.67:0/2272900187 shutdown_connections 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/2272900187 >> v2:172.21.15.176:3300/0 conn(0x7fa4580d4a30 0x7fa4580d4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/2272900187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa45806ccd0 0x7fa4580d11f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 -- 172.21.15.67:0/2272900187 >> 172.21.15.67:0/2272900187 conn(0x7fa4580d5d70 msgr2=0x7fa4580d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 -- 172.21.15.67:0/2272900187 shutdown_connections 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.465+0000 7fa45f3e3640 1 -- 172.21.15.67:0/2272900187 wait complete. 2024-04-25T15:27:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.467+0000 7fa45f3e3640 1 Processor -- start 2024-04-25T15:27:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.467+0000 7fa45f3e3640 1 -- start start 2024-04-25T15:27:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45f3e3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4580d0de0 0x7fa4580c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45f3e3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4580d4a30 0x7fa4580b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45f3e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4580cbc90 0x7fa4580cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45f3e3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa458164de0 con 0x7fa4580d4a30 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45f3e3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4580b5300 con 0x7fa4580d0de0 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45f3e3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa458164f60 con 0x7fa4580cbc90 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45c97e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4580d4a30 0x7fa4580b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45c97e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4580d4a30 0x7fa4580b7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33696/0 (socket says 172.21.15.67:33696) 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45c97e640 1 -- 172.21.15.67:0/1249093188 learned_addr learned my addr 172.21.15.67:0/1249093188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45d17f640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4580d0de0 0x7fa4580c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.468+0000 7fa45d980640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4580cbc90 0x7fa4580cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45c97e640 1 -- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4580cbc90 msgr2=0x7fa4580cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45c97e640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4580cbc90 0x7fa4580cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45c97e640 1 -- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4580d0de0 msgr2=0x7fa4580c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45c97e640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4580d0de0 0x7fa4580c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45c97e640 1 -- 172.21.15.67:0/1249093188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4580cc5e0 con 0x7fa4580d4a30 2024-04-25T15:27:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45d17f640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4580d0de0 0x7fa4580c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45c97e640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4580d4a30 0x7fa4580b7c40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa44c02a700 tx=0x7fa44c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa455ffb640 1 -- 172.21.15.67:0/1249093188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c037040 con 0x7fa4580d4a30 2024-04-25T15:27:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa455ffb640 1 -- 172.21.15.67:0/1249093188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa44c042070 con 0x7fa4580d4a30 2024-04-25T15:27:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa455ffb640 1 -- 172.21.15.67:0/1249093188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c030bd0 con 0x7fa4580d4a30 2024-04-25T15:27:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45815ccd0 con 0x7fa4580d4a30 2024-04-25T15:27:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.469+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4580ca480 con 0x7fa4580d4a30 2024-04-25T15:27:14.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.470+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45815ca40 con 0x7fa4580d4a30 2024-04-25T15:27:14.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.471+0000 7fa455ffb640 1 -- 172.21.15.67:0/1249093188 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa44c047460 con 0x7fa4580d4a30 2024-04-25T15:27:14.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.471+0000 7fa455ffb640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4440744b0 0x7fa4440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.471+0000 7fa455ffb640 1 -- 172.21.15.67:0/1249093188 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7fa44c0b5920 con 0x7fa4580d4a30 2024-04-25T15:27:14.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.472+0000 7fa45d17f640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4440744b0 0x7fa4440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.472+0000 7fa45d17f640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4440744b0 0x7fa4440769c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa450000f60 tx=0x7fa450007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:14.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.476+0000 7fa455ffb640 1 -- 172.21.15.67:0/1249093188 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4580cc5e0 con 0x7fa4580d4a30 2024-04-25T15:27:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.731+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa4580688a0 con 0x7fa4580d4a30 2024-04-25T15:27:14.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.732+0000 7fa455ffb640 1 -- 172.21.15.67:0/1249093188 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7fa44c032e20 con 0x7fa4580d4a30 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg==", 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA==", 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-25T15:27:14.734 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q==", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw==", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:14.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg==", 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw==", 2024-04-25T15:27:14.736 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA==", 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g==", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA==", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.738 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA==", 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ==", 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.739 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA==", 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.740 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q==", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ==", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:27:14.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg==", 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-25T15:27:14.742 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA==", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmVIPuGRAAswFIakCxyA7OvvoJIBYUiQ==", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg==", 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag==", 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:27:14.745 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:27:14.745 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:14.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.745 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:14.745 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:27:14.745 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:27:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4440744b0 msgr2=0x7fa4440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4440744b0 0x7fa4440769c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa450000f60 tx=0x7fa450007ca0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4580d4a30 msgr2=0x7fa4580b7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4580d4a30 0x7fa4580b7c40 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa44c02a700 tx=0x7fa44c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 shutdown_connections 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa4440744b0 0x7fa4440769c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4580cbc90 0x7fa4580cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4580d4a30 0x7fa4580b7c40 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 --2- 172.21.15.67:0/1249093188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4580d0de0 0x7fa4580c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 >> 172.21.15.67:0/1249093188 conn(0x7fa4580d5d70 msgr2=0x7fa45815ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 shutdown_connections 2024-04-25T15:27:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.734+0000 7fa45f3e3640 1 -- 172.21.15.67:0/1249093188 wait complete. 2024-04-25T15:27:14.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:14.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.859+0000 7f3d837d7640 1 Processor -- start 2024-04-25T15:27:14.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.859+0000 7f3d837d7640 1 -- start start 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d837d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c0d0e00 0x7f3d7c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d837d7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d7c06c5c0 0x7f3d7c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d837d7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3d7c0b7810 0x7f3d7c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d837d7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d7c06cf10 con 0x7f3d7c06c5c0 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d837d7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3d7c0d1850 con 0x7f3d7c0b7810 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d837d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7c160de0 con 0x7f3d7c0d0e00 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d80d72640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d7c06c5c0 0x7f3d7c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d80d72640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d7c06c5c0 0x7f3d7c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33708/0 (socket says 172.21.15.67:33708) 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.860+0000 7f3d80d72640 1 -- 172.21.15.67:0/3634980964 learned_addr learned my addr 172.21.15.67:0/3634980964 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d81d74640 1 --2- 172.21.15.67:0/3634980964 >> v2:172.21.15.176:3300/0 conn(0x7f3d7c0b7810 0x7f3d7c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d81573640 1 --2- 172.21.15.67:0/3634980964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c0d0e00 0x7f3d7c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d80d72640 1 -- 172.21.15.67:0/3634980964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c0d0e00 msgr2=0x7f3d7c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d80d72640 1 --2- 172.21.15.67:0/3634980964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c0d0e00 0x7f3d7c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d80d72640 1 -- 172.21.15.67:0/3634980964 >> v2:172.21.15.176:3300/0 conn(0x7f3d7c0b7810 msgr2=0x7f3d7c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d80d72640 1 --2- 172.21.15.67:0/3634980964 >> v2:172.21.15.176:3300/0 conn(0x7f3d7c0b7810 0x7f3d7c0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d80d72640 1 -- 172.21.15.67:0/3634980964 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7c1610c0 con 0x7f3d7c06c5c0 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d81d74640 1 --2- 172.21.15.67:0/3634980964 >> v2:172.21.15.176:3300/0 conn(0x7f3d7c0b7810 0x7f3d7c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:14.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.861+0000 7f3d80d72640 1 --2- 172.21.15.67:0/3634980964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c06c5c0 0x7f3d7c06c9d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c06d1e0 tx=0x7f3d6c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db13424e2e82c766 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:14.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d73fff640 1 -- 172.21.15.67:0/3634980964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c037070 con 0x7f3d7c06c5c0 2024-04-25T15:27:14.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d73fff640 1 -- 172.21.15.67:0/3634980964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d6c032440 con 0x7f3d7c06c5c0 2024-04-25T15:27:14.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d73fff640 1 -- 172.21.15.67:0/3634980964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c032810 con 0x7f3d7c06c5c0 2024-04-25T15:27:14.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 -- 172.21.15.67:0/3634980964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c06c5c0 msgr2=0x7f3d7c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 --2- 172.21.15.67:0/3634980964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c06c5c0 0x7f3d7c06c9d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c06d1e0 tx=0x7f3d6c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 -- 172.21.15.67:0/3634980964 shutdown_connections 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 --2- 172.21.15.67:0/3634980964 >> v2:172.21.15.176:3300/0 conn(0x7f3d7c0b7810 0x7f3d7c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 --2- 172.21.15.67:0/3634980964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c06c5c0 0x7f3d7c06c9d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 --2- 172.21.15.67:0/3634980964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c0d0e00 0x7f3d7c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 -- 172.21.15.67:0/3634980964 >> 172.21.15.67:0/3634980964 conn(0x7f3d7c0d5d90 msgr2=0x7f3d7c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 -- 172.21.15.67:0/3634980964 shutdown_connections 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.862+0000 7f3d837d7640 1 -- 172.21.15.67:0/3634980964 wait complete. 2024-04-25T15:27:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.864+0000 7f3d837d7640 1 Processor -- start 2024-04-25T15:27:14.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.864+0000 7f3d837d7640 1 -- start start 2024-04-25T15:27:14.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.865+0000 7f3d837d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c06c5c0 0x7f3d7c1eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.865+0000 7f3d837d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c0b7810 0x7f3d7c14dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.865+0000 7f3d837d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d7c0d0e00 0x7f3d7c14e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.865+0000 7f3d837d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7c0b5320 con 0x7f3d7c0b7810 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.865+0000 7f3d837d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7c160ad0 con 0x7f3d7c0d0e00 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.865+0000 7f3d837d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7c164e00 con 0x7f3d7c06c5c0 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c0b7810 0x7f3d7c14dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c0b7810 0x7f3d7c14dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33724/0 (socket says 172.21.15.67:33724) 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 -- 172.21.15.67:0/2071939112 learned_addr learned my addr 172.21.15.67:0/2071939112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d81d74640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d7c0d0e00 0x7f3d7c14e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d81573640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c06c5c0 0x7f3d7c1eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 -- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c06c5c0 msgr2=0x7f3d7c1eb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c06c5c0 0x7f3d7c1eb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 -- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d7c0d0e00 msgr2=0x7f3d7c14e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d7c0d0e00 0x7f3d7c14e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 -- 172.21.15.67:0/2071939112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7c1ebdd0 con 0x7f3d7c0b7810 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d80d72640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c0b7810 0x7f3d7c14dd80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c02a220 tx=0x7f3d6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d71ffb640 1 -- 172.21.15.67:0/2071939112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c041020 con 0x7f3d7c0b7810 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d71ffb640 1 -- 172.21.15.67:0/2071939112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d6c0306c0 con 0x7f3d7c0b7810 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7c1ec0b0 con 0x7f3d7c0b7810 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d71ffb640 1 -- 172.21.15.67:0/2071939112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c037040 con 0x7f3d7c0b7810 2024-04-25T15:27:14.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.866+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7c1ec670 con 0x7f3d7c0b7810 2024-04-25T15:27:14.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.867+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d7c14ba10 con 0x7f3d7c0b7810 2024-04-25T15:27:14.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.868+0000 7f3d71ffb640 1 -- 172.21.15.67:0/2071939112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d6c044020 con 0x7f3d7c0b7810 2024-04-25T15:27:14.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.868+0000 7f3d71ffb640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d600744b0 0x7f3d600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:14.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.869+0000 7f3d71ffb640 1 -- 172.21.15.67:0/2071939112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f3d6c0b56f0 con 0x7f3d7c0b7810 2024-04-25T15:27:14.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.869+0000 7f3d81573640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d600744b0 0x7f3d600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:14.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.869+0000 7f3d81573640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d600744b0 0x7f3d600769c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d78006fd0 tx=0x7f3d78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:14.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:14.873+0000 7f3d71ffb640 1 -- 172.21.15.67:0/2071939112 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d7c14ba10 con 0x7f3d7c0b7810 2024-04-25T15:27:15.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.062+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d7c1ebb60 con 0x7f3d7c0b7810 2024-04-25T15:27:15.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.063+0000 7f3d71ffb640 1 -- 172.21.15.67:0/2071939112 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+12855 (secure 0 0 0) 0x7f3d6c04b030 con 0x7f3d7c0b7810 2024-04-25T15:27:15.064 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:15.064 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":52,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:14.324863+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:15.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d600744b0 msgr2=0x7f3d600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d600744b0 0x7f3d600769c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d78006fd0 tx=0x7f3d78008040 comp rx=0 tx=0).stop 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c0b7810 msgr2=0x7f3d7c14dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c0b7810 0x7f3d7c14dd80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c02a220 tx=0x7f3d6c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 shutdown_connections 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d600744b0 0x7f3d600769c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d7c0d0e00 0x7f3d7c14e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d7c0b7810 0x7f3d7c14dd80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 --2- 172.21.15.67:0/2071939112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d7c06c5c0 0x7f3d7c1eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 >> 172.21.15.67:0/2071939112 conn(0x7f3d7c0d5d90 msgr2=0x7f3d7c0b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.065+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 shutdown_connections 2024-04-25T15:27:15.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.066+0000 7f3d837d7640 1 -- 172.21.15.67:0/2071939112 wait complete. 2024-04-25T15:27:15.078 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:15.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.192+0000 7fe6a9e67640 1 Processor -- start 2024-04-25T15:27:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.192+0000 7fe6a9e67640 1 -- start start 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.193+0000 7fe6a9e67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 0x7fe6a415e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.193+0000 7fe6a9e67640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe6a406c5c0 0x7fe6a406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.193+0000 7fe6a9e67640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6a41609b0 con 0x7fe6a4163fa0 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.193+0000 7fe6a9e67640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe6a406cf10 con 0x7fe6a406c5c0 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.193+0000 7fe6a9e67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a4164cd0 con 0x7fe6a4161710 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.193+0000 7fe6a3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe6a4163fa0 0x7fe6a41643b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52678/0 (socket says 172.21.15.67:52678) 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.193+0000 7fe6a3fff640 1 -- 172.21.15.67:0/3207392758 learned_addr learned my addr 172.21.15.67:0/3207392758 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a2ffd640 1 --2- 172.21.15.67:0/3207392758 >> v2:172.21.15.176:3300/0 conn(0x7fe6a406c5c0 0x7fe6a406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a37fe640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 0x7fe6a415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168361625 0 0) 0x7fe69c001490 con 0x7fe6a4163fa0 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe68c003520 con 0x7fe6a4163fa0 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1240267658 0 0) 0x7fe68c003520 con 0x7fe6a4163fa0 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 msgr2=0x7fe6a415e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 0x7fe6a415e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 >> v2:172.21.15.176:3300/0 conn(0x7fe6a406c5c0 msgr2=0x7fe6a406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 --2- 172.21.15.67:0/3207392758 >> v2:172.21.15.176:3300/0 conn(0x7fe6a406c5c0 0x7fe6a406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a4160c30 con 0x7fe6a4163fa0 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fe69c001bc0 con 0x7fe6a4163fa0 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 >> v1:172.21.15.67:6789/0 conn(0x7fe6a4163fa0 legacy=0x7fe6a41643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe68c0039b0 0x7fe68c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe68c004800 0x7fe68c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 0x7fe68c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a406cf10 con 0x7fe68c004800 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a4164cd0 con 0x7fe68c0039b0 2024-04-25T15:27:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe68c00ba70 con 0x7fe6a4161710 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a2ffd640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe68c004800 0x7fe68c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a37fe640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe68c0039b0 0x7fe68c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a3fff640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 0x7fe68c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a2ffd640 1 -- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 msgr2=0x7fe68c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a2ffd640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6a4161710 0x7fe68c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a2ffd640 1 -- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe68c0039b0 msgr2=0x7fe68c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a2ffd640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe68c0039b0 0x7fe68c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.194+0000 7fe6a2ffd640 1 -- 172.21.15.67:0/3207392758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c003520 con 0x7fe68c004800 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.195+0000 7fe6a2ffd640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe68c004800 0x7fe68c004c10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40d2770 tx=0x7fe68c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c77afbe79be04f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.195+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698036070 con 0x7fe68c004800 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.195+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe698031400 con 0x7fe68c004800 2024-04-25T15:27:15.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.195+0000 7fe6a27fc640 1 -- 172.21.15.67:0/3207392758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698009690 con 0x7fe68c004800 2024-04-25T15:27:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.196+0000 7fe6a9e67640 1 -- 172.21.15.67:0/3207392758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe68c004800 msgr2=0x7fe68c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.196+0000 7fe6a9e67640 1 --2- 172.21.15.67:0/3207392758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe68c004800 0x7fe68c004c10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40d2770 tx=0x7fe68c005830 comp rx=0 tx=0).stop 2024-04-25T15:27:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.196+0000 7fe6a3fff640 1 -- 172.21.15.67:0/3207392758 reap_dead start 2024-04-25T15:27:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.196+0000 7fe6a9e67640 1 -- 172.21.15.67:0/3207392758 shutdown_connections 2024-04-25T15:27:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.196+0000 7fe6a9e67640 1 -- 172.21.15.67:0/3207392758 >> 172.21.15.67:0/3207392758 conn(0x7fe6a40d5d90 msgr2=0x7fe6a40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:15.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.196+0000 7fe6a9e67640 1 -- 172.21.15.67:0/3207392758 shutdown_connections 2024-04-25T15:27:15.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.196+0000 7fe6a9e67640 1 -- 172.21.15.67:0/3207392758 wait complete. 2024-04-25T15:27:15.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.198+0000 7fe6a9e67640 1 Processor -- start 2024-04-25T15:27:15.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.198+0000 7fe6a9e67640 1 -- start start 2024-04-25T15:27:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a9e67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe68c004800 0x7fe6a40cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a9e67640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6a4155a30 0x7fe6a4144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a9e67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6a4144570 0x7fe6a4141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a9e67640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a415e080 con 0x7fe6a4144570 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a9e67640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a41611f0 con 0x7fe6a4155a30 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a9e67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a415e200 con 0x7fe68c004800 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6a4144570 0x7fe6a4141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6a4144570 0x7fe6a4141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33738/0 (socket says 172.21.15.67:33738) 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.199+0000 7fe6a3fff640 1 -- 172.21.15.67:0/102628837 learned_addr learned my addr 172.21.15.67:0/102628837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a37fe640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe68c004800 0x7fe6a40cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a2ffd640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6a4155a30 0x7fe6a4144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a3fff640 1 -- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe68c004800 msgr2=0x7fe6a40cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a3fff640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe68c004800 0x7fe6a40cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a3fff640 1 -- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6a4155a30 msgr2=0x7fe6a4144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a3fff640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6a4155a30 0x7fe6a4144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a3fff640 1 -- 172.21.15.67:0/102628837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a4144980 con 0x7fe6a4144570 2024-04-25T15:27:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a3fff640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6a4144570 0x7fe6a4141fa0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40b91d0 tx=0x7fe69c00aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe68bfff640 1 -- 172.21.15.67:0/102628837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c007800 con 0x7fe6a4144570 2024-04-25T15:27:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe68bfff640 1 -- 172.21.15.67:0/102628837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe69c018070 con 0x7fe6a4144570 2024-04-25T15:27:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a4156400 con 0x7fe6a4144570 2024-04-25T15:27:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe68bfff640 1 -- 172.21.15.67:0/102628837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c0133f0 con 0x7fe6a4144570 2024-04-25T15:27:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.200+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a41566e0 con 0x7fe6a4144570 2024-04-25T15:27:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.201+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a4050740 con 0x7fe6a4144570 2024-04-25T15:27:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.202+0000 7fe68bfff640 1 -- 172.21.15.67:0/102628837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe69c013590 con 0x7fe6a4144570 2024-04-25T15:27:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.203+0000 7fe68bfff640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6900744e0 0x7fe6900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.203+0000 7fe68bfff640 1 -- 172.21.15.67:0/102628837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7fe69c0954b0 con 0x7fe6a4144570 2024-04-25T15:27:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.203+0000 7fe6a37fe640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6900744e0 0x7fe6900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.203+0000 7fe6a37fe640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6900744e0 0x7fe6900769f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe6a408e250 tx=0x7fe694009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.207+0000 7fe68bfff640 1 -- 172.21.15.67:0/102628837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6a4144980 con 0x7fe6a4144570 2024-04-25T15:27:15.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.396+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe6a40ccb90 con 0x7fe6a4144570 2024-04-25T15:27:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.396+0000 7fe68bfff640 1 -- 172.21.15.67:0/102628837 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7fe6a41609b0 con 0x7fe6a4144570 2024-04-25T15:27:15.397 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6900744e0 msgr2=0x7fe6900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6900744e0 0x7fe6900769f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe6a408e250 tx=0x7fe694009190 comp rx=0 tx=0).stop 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6a4144570 msgr2=0x7fe6a4141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6a4144570 0x7fe6a4141fa0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40b91d0 tx=0x7fe69c00aec0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 shutdown_connections 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe6900744e0 0x7fe6900769f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6a4144570 0x7fe6a4141fa0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6a4155a30 0x7fe6a4144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.398+0000 7fe6a9e67640 1 --2- 172.21.15.67:0/102628837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe68c004800 0x7fe6a40cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.399+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 >> 172.21.15.67:0/102628837 conn(0x7fe6a40d5d90 msgr2=0x7fe6a415fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.399+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 shutdown_connections 2024-04-25T15:27:15.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.399+0000 7fe6a9e67640 1 -- 172.21.15.67:0/102628837 wait complete. 2024-04-25T15:27:15.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:27:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.522+0000 7fdb6724c640 1 Processor -- start 2024-04-25T15:27:15.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.522+0000 7fdb6724c640 1 -- start start 2024-04-25T15:27:15.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb6724c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdb600d0e00 0x7fdb600d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb6724c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb600d4a50 0x7fdb600d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb6724c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb600d1850 con 0x7fdb6006c5c0 2024-04-25T15:27:15.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb6724c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdb60160de0 con 0x7fdb600d4a50 2024-04-25T15:27:15.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb6724c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdb601610c0 con 0x7fdb600d0e00 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb657e9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb600d4a50 0x7fdb600d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb657e9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb600d4a50 0x7fdb600d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33754/0 (socket says 172.21.15.67:33754) 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb657e9640 1 -- 172.21.15.67:0/956164516 learned_addr learned my addr 172.21.15.67:0/956164516 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb64fe8640 1 --2- 172.21.15.67:0/956164516 >> v2:172.21.15.176:3300/0 conn(0x7fdb600d0e00 0x7fdb600d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb657e9640 1 -- 172.21.15.67:0/956164516 >> v2:172.21.15.176:3300/0 conn(0x7fdb600d0e00 msgr2=0x7fdb600d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb657e9640 1 --2- 172.21.15.67:0/956164516 >> v2:172.21.15.176:3300/0 conn(0x7fdb600d0e00 0x7fdb600d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb657e9640 1 -- 172.21.15.67:0/956164516 >> v1:172.21.15.176:6789/0 conn(0x7fdb6006c5c0 legacy=0x7fdb6006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.523+0000 7fdb657e9640 1 -- 172.21.15.67:0/956164516 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb601613a0 con 0x7fdb600d4a50 2024-04-25T15:27:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.524+0000 7fdb657e9640 1 --2- 172.21.15.67:0/956164516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb600d4e60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdb600d3e60 tx=0x7fdb5002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8669c8eafc66aa59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.524+0000 7fdb5f7fe640 1 -- 172.21.15.67:0/956164516 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb50036070 con 0x7fdb600d4a50 2024-04-25T15:27:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.524+0000 7fdb5f7fe640 1 -- 172.21.15.67:0/956164516 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb50031600 con 0x7fdb600d4a50 2024-04-25T15:27:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.524+0000 7fdb5f7fe640 1 -- 172.21.15.67:0/956164516 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb500319d0 con 0x7fdb600d4a50 2024-04-25T15:27:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 -- 172.21.15.67:0/956164516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 msgr2=0x7fdb600d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 --2- 172.21.15.67:0/956164516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb600d4e60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdb600d3e60 tx=0x7fdb5002a660 comp rx=0 tx=0).stop 2024-04-25T15:27:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 -- 172.21.15.67:0/956164516 shutdown_connections 2024-04-25T15:27:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 --2- 172.21.15.67:0/956164516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb600d4e60 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 --2- 172.21.15.67:0/956164516 >> v2:172.21.15.176:3300/0 conn(0x7fdb600d0e00 0x7fdb600d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 -- 172.21.15.67:0/956164516 >> 172.21.15.67:0/956164516 conn(0x7fdb600d5d90 msgr2=0x7fdb600d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 -- 172.21.15.67:0/956164516 shutdown_connections 2024-04-25T15:27:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.525+0000 7fdb6724c640 1 -- 172.21.15.67:0/956164516 wait complete. 2024-04-25T15:27:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.527+0000 7fdb6724c640 1 Processor -- start 2024-04-25T15:27:15.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.527+0000 7fdb6724c640 1 -- start start 2024-04-25T15:27:15.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb6724c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb6006c5c0 0x7fdb600cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb6724c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb600d0e00 0x7fdb601fe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb6724c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb601fe920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb6724c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb60164e00 con 0x7fdb600d4a50 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb6724c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdb600b5320 con 0x7fdb6006c5c0 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb6724c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdb60164f80 con 0x7fdb600d0e00 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb657e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb601fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.528+0000 7fdb657e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb601fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33760/0 (socket says 172.21.15.67:33760) 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb657e9640 1 -- 172.21.15.67:0/1299578714 learned_addr learned my addr 172.21.15.67:0/1299578714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb64fe8640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb6006c5c0 0x7fdb600cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb5ffff640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb600d0e00 0x7fdb601fe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb657e9640 1 -- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb600d0e00 msgr2=0x7fdb601fe3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb657e9640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb600d0e00 0x7fdb601fe3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb657e9640 1 -- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb6006c5c0 msgr2=0x7fdb600cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb657e9640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb6006c5c0 0x7fdb600cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb657e9640 1 -- 172.21.15.67:0/1299578714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb600cb1a0 con 0x7fdb600d4a50 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb64fe8640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb6006c5c0 0x7fdb600cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb5ffff640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb600d0e00 0x7fdb601fe3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:15.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb657e9640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb601fe920 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdb50009f40 tx=0x7fdb50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb5d7fa640 1 -- 172.21.15.67:0/1299578714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb50036050 con 0x7fdb600d4a50 2024-04-25T15:27:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb5d7fa640 1 -- 172.21.15.67:0/1299578714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb50042070 con 0x7fdb600d4a50 2024-04-25T15:27:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb5d7fa640 1 -- 172.21.15.67:0/1299578714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb50009510 con 0x7fdb600d4a50 2024-04-25T15:27:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb600ca100 con 0x7fdb600d4a50 2024-04-25T15:27:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.529+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb600ca6c0 con 0x7fdb600d4a50 2024-04-25T15:27:15.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.530+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb60050740 con 0x7fdb600d4a50 2024-04-25T15:27:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.531+0000 7fdb5d7fa640 1 -- 172.21.15.67:0/1299578714 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdb50007b90 con 0x7fdb600d4a50 2024-04-25T15:27:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.531+0000 7fdb5d7fa640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb400744b0 0x7fdb400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.532+0000 7fdb5d7fa640 1 -- 172.21.15.67:0/1299578714 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7fdb500b5d30 con 0x7fdb600d4a50 2024-04-25T15:27:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.532+0000 7fdb64fe8640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb400744b0 0x7fdb400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.532+0000 7fdb64fe8640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb400744b0 0x7fdb400769c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdb58006fd0 tx=0x7fdb58008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.536+0000 7fdb5d7fa640 1 -- 172.21.15.67:0/1299578714 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb600cb1a0 con 0x7fdb600d4a50 2024-04-25T15:27:15.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.727+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fdb600d4320 con 0x7fdb600d4a50 2024-04-25T15:27:15.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.727+0000 7fdb5d7fa640 1 -- 172.21.15.67:0/1299578714 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) ==== 84+0+65 (secure 0 0 0) 0x7fdb500808a0 con 0x7fdb600d4a50 2024-04-25T15:27:15.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:15.729 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:27:15.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:15.729 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:27:15.729 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:27:15.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:15.729 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.729+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb400744b0 msgr2=0x7fdb400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.729+0000 7fdb6724c640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb400744b0 0x7fdb400769c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdb58006fd0 tx=0x7fdb58008040 comp rx=0 tx=0).stop 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.729+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 msgr2=0x7fdb601fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.729+0000 7fdb6724c640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb601fe920 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdb50009f40 tx=0x7fdb50004a40 comp rx=0 tx=0).stop 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 shutdown_connections 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb400744b0 0x7fdb400769c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb600d4a50 0x7fdb601fe920 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb600d0e00 0x7fdb601fe3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 --2- 172.21.15.67:0/1299578714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb6006c5c0 0x7fdb600cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 >> 172.21.15.67:0/1299578714 conn(0x7fdb600d5d90 msgr2=0x7fdb6015c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:15.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 shutdown_connections 2024-04-25T15:27:15.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.730+0000 7fdb6724c640 1 -- 172.21.15.67:0/1299578714 wait complete. 2024-04-25T15:27:15.742 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:27:15.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:27:15.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.857+0000 7f0495042640 1 Processor -- start 2024-04-25T15:27:15.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.857+0000 7f0495042640 1 -- start start 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.858+0000 7f0495042640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04900b8290 0x7f0490160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.858+0000 7f0495042640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f049015df80 0x7f0490160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.858+0000 7f0495042640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04901615f0 con 0x7f0490164b00 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.858+0000 7f0495042640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0490164670 con 0x7f04900b8290 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f0495042640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0490164020 con 0x7f049015df80 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04900b8290 0x7f0490160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04900b8290 0x7f0490160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33766/0 (socket says 172.21.15.67:33766) 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0490164b00 0x7f04900b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52692/0 (socket says 172.21.15.67:52692) 2024-04-25T15:27:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048e59c640 1 -- 172.21.15.67:0/2278188737 learned_addr learned my addr 172.21.15.67:0/2278188737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048f59e640 1 --2- 172.21.15.67:0/2278188737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f049015df80 0x7f0490160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048e59c640 1 -- 172.21.15.67:0/2278188737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f049015df80 msgr2=0x7f0490160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048e59c640 1 --2- 172.21.15.67:0/2278188737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f049015df80 0x7f0490160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048e59c640 1 -- 172.21.15.67:0/2278188737 >> v1:172.21.15.67:6789/0 conn(0x7f0490164b00 legacy=0x7f04900b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048dd9b640 1 -- 172.21.15.67:0/2278188737 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2650985443 0 0) 0x7f04901615f0 con 0x7f0490164b00 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.859+0000 7f048e59c640 1 -- 172.21.15.67:0/2278188737 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0490164300 con 0x7f04900b8290 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.860+0000 7f048e59c640 1 --2- 172.21.15.67:0/2278188737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04900b8290 0x7f0490160890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0478009870 tx=0x7f047802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b586888450c20b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.860+0000 7f048dd9b640 1 -- 172.21.15.67:0/2278188737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478037070 con 0x7f04900b8290 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.860+0000 7f048dd9b640 1 -- 172.21.15.67:0/2278188737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0478032440 con 0x7f04900b8290 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.860+0000 7f048dd9b640 1 -- 172.21.15.67:0/2278188737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478032810 con 0x7f04900b8290 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.860+0000 7f0495042640 1 -- 172.21.15.67:0/2278188737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04900b8290 msgr2=0x7f0490160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.860+0000 7f0495042640 1 --2- 172.21.15.67:0/2278188737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04900b8290 0x7f0490160890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0478009870 tx=0x7f047802a270 comp rx=0 tx=0).stop 2024-04-25T15:27:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.861+0000 7f0495042640 1 -- 172.21.15.67:0/2278188737 shutdown_connections 2024-04-25T15:27:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.861+0000 7f0495042640 1 --2- 172.21.15.67:0/2278188737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f049015df80 0x7f0490160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.861+0000 7f0495042640 1 --2- 172.21.15.67:0/2278188737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04900b8290 0x7f0490160890 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.861+0000 7f0495042640 1 -- 172.21.15.67:0/2278188737 >> 172.21.15.67:0/2278188737 conn(0x7f04900d2bc0 msgr2=0x7f04900d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.861+0000 7f0495042640 1 -- 172.21.15.67:0/2278188737 shutdown_connections 2024-04-25T15:27:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.861+0000 7f0495042640 1 -- 172.21.15.67:0/2278188737 wait complete. 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.863+0000 7f0495042640 1 Processor -- start 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.863+0000 7f0495042640 1 -- start start 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f0495042640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f049015df80 0x7f04900c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f0495042640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490164b00 0x7f0490152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f0495042640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0490162070 0x7f0490162480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f0495042640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f04900b4a00 con 0x7f0490164b00 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f0495042640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04900b4880 con 0x7f049015df80 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f0495042640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f04900b4b80 con 0x7f0490162070 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490164b00 0x7f0490152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490164b00 0x7f0490152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33776/0 (socket says 172.21.15.67:33776) 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 -- 172.21.15.67:0/2557079963 learned_addr learned my addr 172.21.15.67:0/2557079963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048ed9d640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f049015df80 0x7f04900c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048f59e640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0490162070 0x7f0490162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 -- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0490162070 msgr2=0x7f0490162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0490162070 0x7f0490162480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 -- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f049015df80 msgr2=0x7f04900c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f049015df80 0x7f04900c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048e59c640 1 -- 172.21.15.67:0/2557079963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0490156b50 con 0x7f0490164b00 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.864+0000 7f048ed9d640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f049015df80 0x7f04900c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.865+0000 7f048e59c640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490164b00 0x7f0490152f20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0478009870 tx=0x7f04780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.865+0000 7f04877fe640 1 -- 172.21.15.67:0/2557079963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478041020 con 0x7f0490164b00 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.865+0000 7f04877fe640 1 -- 172.21.15.67:0/2557079963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04780306c0 con 0x7f0490164b00 2024-04-25T15:27:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.865+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0490153460 con 0x7f0490164b00 2024-04-25T15:27:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.865+0000 7f04877fe640 1 -- 172.21.15.67:0/2557079963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478037040 con 0x7f0490164b00 2024-04-25T15:27:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.865+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0490153970 con 0x7f0490164b00 2024-04-25T15:27:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.866+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04901536e0 con 0x7f0490164b00 2024-04-25T15:27:15.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.867+0000 7f04877fe640 1 -- 172.21.15.67:0/2557079963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0478044020 con 0x7f0490164b00 2024-04-25T15:27:15.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.867+0000 7f04877fe640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04700744b0 0x7f04700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:15.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.867+0000 7f04877fe640 1 -- 172.21.15.67:0/2557079963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f04780b56e0 con 0x7f0490164b00 2024-04-25T15:27:15.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.868+0000 7f048ed9d640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04700744b0 0x7f04700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:15.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.868+0000 7f048ed9d640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04700744b0 0x7f04700769c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0488000f60 tx=0x7f0488007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:15.872+0000 7f04877fe640 1 -- 172.21.15.67:0/2557079963 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f047804b030 con 0x7f0490164b00 2024-04-25T15:27:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.057+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0490050740 con 0x7f0490164b00 2024-04-25T15:27:16.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.985+0000 7f04877fe640 1 -- 172.21.15.67:0/2557079963 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) ==== 138+0+0 (secure 0 0 0) 0x7f0478080250 con 0x7f0490164b00 2024-04-25T15:27:16.987 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:27:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.989+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04700744b0 msgr2=0x7f04700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.989+0000 7f0495042640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04700744b0 0x7f04700769c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0488000f60 tx=0x7f0488007ca0 comp rx=0 tx=0).stop 2024-04-25T15:27:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.989+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490164b00 msgr2=0x7f0490152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.989+0000 7f0495042640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490164b00 0x7f0490152f20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0478009870 tx=0x7f04780046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.990+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 shutdown_connections 2024-04-25T15:27:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.990+0000 7f0495042640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04700744b0 0x7f04700769c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.990+0000 7f0495042640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0490162070 0x7f0490162480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.990+0000 7f0495042640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0490164b00 0x7f0490152f20 secure :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0478009870 tx=0x7f04780046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.990+0000 7f0495042640 1 --2- 172.21.15.67:0/2557079963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f049015df80 0x7f04900c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.990+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 >> 172.21.15.67:0/2557079963 conn(0x7f04900d2bc0 msgr2=0x7f049015eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.991+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 shutdown_connections 2024-04-25T15:27:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:16.991+0000 7f0495042640 1 -- 172.21.15.67:0/2557079963 wait complete. 2024-04-25T15:27:17.009 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:27:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.132+0000 7f6f60c84640 1 Processor -- start 2024-04-25T15:27:17.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.132+0000 7f6f60c84640 1 -- start start 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f60c84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c161710 0x7f6f5c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f60c84640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6f5c06c5c0 0x7f6f5c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f60c84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f5c163fa0 0x7f6f5c1643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f60c84640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6f5c1609b0 con 0x7f6f5c163fa0 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f60c84640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6f5c06cf10 con 0x7f6f5c06c5c0 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f60c84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6f5c164cd0 con 0x7f6f5c161710 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f5c163fa0 0x7f6f5c1643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:17.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f5c163fa0 0x7f6f5c1643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33782/0 (socket says 172.21.15.67:33782) 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1624067858 learned_addr learned my addr 172.21.15.67:0/1624067858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5a59c640 1 --2- 172.21.15.67:0/1624067858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c161710 0x7f6f5c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f59d9b640 1 --2- 172.21.15.67:0/1624067858 >> v2:172.21.15.176:3300/0 conn(0x7f6f5c06c5c0 0x7f6f5c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1624067858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c161710 msgr2=0x7f6f5c15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 --2- 172.21.15.67:0/1624067858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c161710 0x7f6f5c15e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1624067858 >> v2:172.21.15.176:3300/0 conn(0x7f6f5c06c5c0 msgr2=0x7f6f5c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 --2- 172.21.15.67:0/1624067858 >> v2:172.21.15.176:3300/0 conn(0x7f6f5c06c5c0 0x7f6f5c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1624067858 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5c160c30 con 0x7f6f5c163fa0 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.134+0000 7f6f59d9b640 1 --2- 172.21.15.67:0/1624067858 >> v2:172.21.15.176:3300/0 conn(0x7f6f5c06c5c0 0x7f6f5c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.135+0000 7f6f5ad9d640 1 --2- 172.21.15.67:0/1624067858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c1643b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c06efa0 tx=0x7f6f4c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2005aaf42851b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.135+0000 7f6f5959a640 1 -- 172.21.15.67:0/1624067858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c036070 con 0x7f6f5c163fa0 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.135+0000 7f6f5959a640 1 -- 172.21.15.67:0/1624067858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4c031600 con 0x7f6f5c163fa0 2024-04-25T15:27:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.135+0000 7f6f5959a640 1 -- 172.21.15.67:0/1624067858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c0319d0 con 0x7f6f5c163fa0 2024-04-25T15:27:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 -- 172.21.15.67:0/1624067858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 msgr2=0x7f6f5c1643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1624067858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c1643b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c06efa0 tx=0x7f6f4c02a610 comp rx=0 tx=0).stop 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 -- 172.21.15.67:0/1624067858 shutdown_connections 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1624067858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c1643b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1624067858 >> v2:172.21.15.176:3300/0 conn(0x7f6f5c06c5c0 0x7f6f5c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1624067858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c161710 0x7f6f5c15e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 -- 172.21.15.67:0/1624067858 >> 172.21.15.67:0/1624067858 conn(0x7f6f5c0d5d90 msgr2=0x7f6f5c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 -- 172.21.15.67:0/1624067858 shutdown_connections 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.136+0000 7f6f60c84640 1 -- 172.21.15.67:0/1624067858 wait complete. 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.138+0000 7f6f60c84640 1 Processor -- start 2024-04-25T15:27:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.138+0000 7f6f60c84640 1 -- start start 2024-04-25T15:27:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f60c84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c06c5c0 0x7f6f5c0ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f60c84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f5c161710 0x7f6f5c0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f60c84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c15cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f60c84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1648f0 con 0x7f6f5c163fa0 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f60c84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c1611f0 con 0x7f6f5c161710 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f60c84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6f5c164a70 con 0x7f6f5c06c5c0 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c15cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c15cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33784/0 (socket says 172.21.15.67:33784) 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1715901562 learned_addr learned my addr 172.21.15.67:0/1715901562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5a59c640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c06c5c0 0x7f6f5c0ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f59d9b640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f5c161710 0x7f6f5c0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c06c5c0 msgr2=0x7f6f5c0ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c06c5c0 0x7f6f5c0ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f5c161710 msgr2=0x7f6f5c0cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f5c161710 0x7f6f5c0cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 -- 172.21.15.67:0/1715901562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5c0cf7a0 con 0x7f6f5c163fa0 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f5ad9d640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c15cf40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009f40 tx=0x7f6f4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.140+0000 7f6f52ffd640 1 -- 172.21.15.67:0/1715901562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c03f020 con 0x7f6f5c163fa0 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.141+0000 7f6f52ffd640 1 -- 172.21.15.67:0/1715901562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4c037040 con 0x7f6f5c163fa0 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.141+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f5c0cfc10 con 0x7f6f5c163fa0 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.141+0000 7f6f52ffd640 1 -- 172.21.15.67:0/1715901562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c036070 con 0x7f6f5c163fa0 2024-04-25T15:27:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.141+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f5c0b7060 con 0x7f6f5c163fa0 2024-04-25T15:27:17.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.142+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f5c050740 con 0x7f6f5c163fa0 2024-04-25T15:27:17.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.143+0000 7f6f52ffd640 1 -- 172.21.15.67:0/1715901562 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6f4c042020 con 0x7f6f5c163fa0 2024-04-25T15:27:17.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.143+0000 7f6f52ffd640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6f3c0744b0 0x7f6f3c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:17.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.143+0000 7f6f52ffd640 1 -- 172.21.15.67:0/1715901562 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5227+0+0 (secure 0 0 0) 0x7f6f4c0b5d70 con 0x7f6f5c163fa0 2024-04-25T15:27:17.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.144+0000 7f6f5a59c640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6f3c0744b0 0x7f6f3c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.144+0000 7f6f5a59c640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6f3c0744b0 0x7f6f3c0769c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c08e250 tx=0x7f6f54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:17.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.147+0000 7f6f52ffd640 1 -- 172.21.15.67:0/1715901562 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f4c049020 con 0x7f6f5c163fa0 2024-04-25T15:27:17.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.333+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6f5c0cf970 con 0x7f6f5c163fa0 2024-04-25T15:27:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.987+0000 7f6f52ffd640 1 -- 172.21.15.67:0/1715901562 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) ==== 170+0+0 (secure 0 0 0) 0x7f6f4c080320 con 0x7f6f5c163fa0 2024-04-25T15:27:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:27:17.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.989+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6f3c0744b0 msgr2=0x7f6f3c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.989+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6f3c0744b0 0x7f6f3c0769c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c08e250 tx=0x7f6f54008040 comp rx=0 tx=0).stop 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.989+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 msgr2=0x7f6f5c15cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.989+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c15cf40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009f40 tx=0x7f6f4c004a40 comp rx=0 tx=0).stop 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 shutdown_connections 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6f3c0744b0 0x7f6f3c0769c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f5c163fa0 0x7f6f5c15cf40 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f5c161710 0x7f6f5c0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 --2- 172.21.15.67:0/1715901562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f5c06c5c0 0x7f6f5c0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 >> 172.21.15.67:0/1715901562 conn(0x7f6f5c0d5d90 msgr2=0x7f6f5c15ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 shutdown_connections 2024-04-25T15:27:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:17.990+0000 7f6f60c84640 1 -- 172.21.15.67:0/1715901562 wait complete. 2024-04-25T15:27:18.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.133+0000 7fc8de7f8640 1 Processor -- start 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.133+0000 7fc8de7f8640 1 -- start start 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8de7f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d808c720 0x7fc8d806f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8de7f8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc8d806fae0 0x7fc8d806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8de7f8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d8167870 con 0x7fc8d8167460 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8de7f8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d806ee30 con 0x7fc8d808c720 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8de7f8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d808cb30 con 0x7fc8d806fae0 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8d77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d808c720 0x7fc8d806f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8d77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8d808c720 0x7fc8d806f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37378/0 (socket says 172.21.15.67:37378) 2024-04-25T15:27:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8d77fe640 1 -- 172.21.15.67:0/2474949751 learned_addr learned my addr 172.21.15.67:0/2474949751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8d7fff640 1 --1- 172.21.15.67:0/2474949751 >> v1:172.21.15.67:6789/0 conn(0x7fc8d8167460 0x7fc8d816d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56162/0 (socket says 172.21.15.67:56162) 2024-04-25T15:27:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.135+0000 7fc8dcd95640 1 --2- 172.21.15.67:0/2474949751 >> v2:172.21.15.176:3300/0 conn(0x7fc8d806fae0 0x7fc8d806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.175+0000 7fc8d77fe640 1 -- 172.21.15.67:0/2474949751 >> v2:172.21.15.176:3300/0 conn(0x7fc8d806fae0 msgr2=0x7fc8d806d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.175+0000 7fc8d77fe640 1 --2- 172.21.15.67:0/2474949751 >> v2:172.21.15.176:3300/0 conn(0x7fc8d806fae0 0x7fc8d806d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.175+0000 7fc8d77fe640 1 -- 172.21.15.67:0/2474949751 >> v1:172.21.15.67:6789/0 conn(0x7fc8d8167460 legacy=0x7fc8d816d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.175+0000 7fc8d77fe640 1 -- 172.21.15.67:0/2474949751 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d816dc40 con 0x7fc8d808c720 2024-04-25T15:27:18.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.176+0000 7fc8d77fe640 1 --2- 172.21.15.67:0/2474949751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d808c720 0x7fc8d806f5a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8072eb0 tx=0x7fc8c002a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56eb2f3979622c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:18.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.176+0000 7fc8d6ffd640 1 -- 172.21.15.67:0/2474949751 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0039070 con 0x7fc8d808c720 2024-04-25T15:27:18.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.176+0000 7fc8d6ffd640 1 -- 172.21.15.67:0/2474949751 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c00316e0 con 0x7fc8d808c720 2024-04-25T15:27:18.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.176+0000 7fc8d6ffd640 1 -- 172.21.15.67:0/2474949751 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0031ab0 con 0x7fc8d808c720 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.176+0000 7fc8de7f8640 1 -- 172.21.15.67:0/2474949751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d808c720 msgr2=0x7fc8d806f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.176+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/2474949751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d808c720 0x7fc8d806f5a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8072eb0 tx=0x7fc8c002a870 comp rx=0 tx=0).stop 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.177+0000 7fc8de7f8640 1 -- 172.21.15.67:0/2474949751 shutdown_connections 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.177+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/2474949751 >> v2:172.21.15.176:3300/0 conn(0x7fc8d806fae0 0x7fc8d806d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.177+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/2474949751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d808c720 0x7fc8d806f5a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.177+0000 7fc8de7f8640 1 -- 172.21.15.67:0/2474949751 >> 172.21.15.67:0/2474949751 conn(0x7fc8d80730b0 msgr2=0x7fc8d80734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.177+0000 7fc8de7f8640 1 -- 172.21.15.67:0/2474949751 shutdown_connections 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.177+0000 7fc8de7f8640 1 -- 172.21.15.67:0/2474949751 wait complete. 2024-04-25T15:27:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.178+0000 7fc8de7f8640 1 Processor -- start 2024-04-25T15:27:18.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.179+0000 7fc8de7f8640 1 -- start start 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8de7f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d806fae0 0x7fc8d8167aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8de7f8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d808c720 0x7fc8d808b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8de7f8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8d8167460 0x7fc8d81694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8de7f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d806f080 con 0x7fc8d806fae0 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8de7f8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d816e200 con 0x7fc8d808c720 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8de7f8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d806db10 con 0x7fc8d8167460 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d806fae0 0x7fc8d8167aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d806fae0 0x7fc8d8167aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37394/0 (socket says 172.21.15.67:37394) 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 -- 172.21.15.67:0/1619323560 learned_addr learned my addr 172.21.15.67:0/1619323560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d77fe640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d808c720 0x7fc8d808b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8dcd95640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8d8167460 0x7fc8d81694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 -- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8d8167460 msgr2=0x7fc8d81694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8d8167460 0x7fc8d81694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 -- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d808c720 msgr2=0x7fc8d808b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d808c720 0x7fc8d808b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.180+0000 7fc8d7fff640 1 -- 172.21.15.67:0/1619323560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d8168410 con 0x7fc8d806fae0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.181+0000 7fc8d7fff640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d806fae0 0x7fc8d8167aa0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800b8d0 tx=0x7fc8c800bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.181+0000 7fc8d4ff9640 1 -- 172.21.15.67:0/1619323560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800cc10 con 0x7fc8d806fae0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.181+0000 7fc8d4ff9640 1 -- 172.21.15.67:0/1619323560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c800cdb0 con 0x7fc8d806fae0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.181+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d81686f0 con 0x7fc8d806fae0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.181+0000 7fc8d4ff9640 1 -- 172.21.15.67:0/1619323560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8006310 con 0x7fc8d806fae0 2024-04-25T15:27:18.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.181+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d8168cb0 con 0x7fc8d806fae0 2024-04-25T15:27:18.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.182+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d8050740 con 0x7fc8d806fae0 2024-04-25T15:27:18.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.183+0000 7fc8d4ff9640 1 -- 172.21.15.67:0/1619323560 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc8c80064b0 con 0x7fc8d806fae0 2024-04-25T15:27:18.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.183+0000 7fc8d4ff9640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8b80744b0 0x7fc8b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:18.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.183+0000 7fc8d4ff9640 1 -- 172.21.15.67:0/1619323560 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5609+0+0 (secure 0 0 0) 0x7fc8c8095560 con 0x7fc8d806fae0 2024-04-25T15:27:18.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.183+0000 7fc8d77fe640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8b80744b0 0x7fc8b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.184+0000 7fc8d77fe640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8b80744b0 0x7fc8b80769c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0036000 tx=0x7fc8c002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:18.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.187+0000 7fc8d4ff9640 1 -- 172.21.15.67:0/1619323560 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8c805f790 con 0x7fc8d806fae0 2024-04-25T15:27:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:18.411+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc8d806ff10 con 0x7fc8d806fae0 2024-04-25T15:27:19.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.019+0000 7fc8d4ff9640 1 -- 172.21.15.67:0/1619323560 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v29) ==== 172+0+0 (secure 0 0 0) 0x7fc8c8063a90 con 0x7fc8d806fae0 2024-04-25T15:27:19.022 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 8 and data pool 9 2024-04-25T15:27:19.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.034+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8b80744b0 msgr2=0x7fc8b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.034+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8b80744b0 0x7fc8b80769c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0036000 tx=0x7fc8c002f000 comp rx=0 tx=0).stop 2024-04-25T15:27:19.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.034+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d806fae0 msgr2=0x7fc8d8167aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.034+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d806fae0 0x7fc8d8167aa0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800b8d0 tx=0x7fc8c800bd90 comp rx=0 tx=0).stop 2024-04-25T15:27:19.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.035+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 shutdown_connections 2024-04-25T15:27:19.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.035+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8b80744b0 0x7fc8b80769c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.035+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8d8167460 0x7fc8d81694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.035+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d808c720 0x7fc8d808b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.035+0000 7fc8de7f8640 1 --2- 172.21.15.67:0/1619323560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8d806fae0 0x7fc8d8167aa0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.035+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 >> 172.21.15.67:0/1619323560 conn(0x7fc8d80730b0 msgr2=0x7fc8d816fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:19.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.037+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 shutdown_connections 2024-04-25T15:27:19.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.037+0000 7fc8de7f8640 1 -- 172.21.15.67:0/1619323560 wait complete. 2024-04-25T15:27:19.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.170+0000 7fb670fb2640 1 Processor -- start 2024-04-25T15:27:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.170+0000 7fb670fb2640 1 -- start start 2024-04-25T15:27:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb670fb2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb66c0d0e00 0x7fb66c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb670fb2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb66c06c5c0 0x7fb66c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb670fb2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb66c06cf10 con 0x7fb66c0b7810 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb670fb2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb66c0d5a60 con 0x7fb66c06c5c0 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb670fb2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb66c0d1850 con 0x7fb66c0d0e00 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb663fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb66c06c5c0 0x7fb66c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb663fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb66c06c5c0 0x7fb66c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37396/0 (socket says 172.21.15.67:37396) 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.171+0000 7fb663fff640 1 -- 172.21.15.67:0/1633182219 learned_addr learned my addr 172.21.15.67:0/1633182219 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.172+0000 7fb66a59c640 1 --2- 172.21.15.67:0/1633182219 >> v2:172.21.15.176:3300/0 conn(0x7fb66c0d0e00 0x7fb66c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.172+0000 7fb663fff640 1 -- 172.21.15.67:0/1633182219 >> v2:172.21.15.176:3300/0 conn(0x7fb66c0d0e00 msgr2=0x7fb66c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.172+0000 7fb663fff640 1 --2- 172.21.15.67:0/1633182219 >> v2:172.21.15.176:3300/0 conn(0x7fb66c0d0e00 0x7fb66c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.172+0000 7fb663fff640 1 -- 172.21.15.67:0/1633182219 >> v1:172.21.15.176:6789/0 conn(0x7fb66c0b7810 legacy=0x7fb66c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.172+0000 7fb663fff640 1 -- 172.21.15.67:0/1633182219 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c160de0 con 0x7fb66c06c5c0 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.172+0000 7fb663fff640 1 --2- 172.21.15.67:0/1633182219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c06c5c0 0x7fb66c06c9d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0d5be0 tx=0x7fb65402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949fe704c972ba71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb669d9b640 1 -- 172.21.15.67:0/1633182219 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654037070 con 0x7fb66c06c5c0 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb669d9b640 1 -- 172.21.15.67:0/1633182219 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb654032440 con 0x7fb66c06c5c0 2024-04-25T15:27:19.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb669d9b640 1 -- 172.21.15.67:0/1633182219 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654032810 con 0x7fb66c06c5c0 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 -- 172.21.15.67:0/1633182219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c06c5c0 msgr2=0x7fb66c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 --2- 172.21.15.67:0/1633182219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c06c5c0 0x7fb66c06c9d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0d5be0 tx=0x7fb65402a180 comp rx=0 tx=0).stop 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 -- 172.21.15.67:0/1633182219 shutdown_connections 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 --2- 172.21.15.67:0/1633182219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c06c5c0 0x7fb66c06c9d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 --2- 172.21.15.67:0/1633182219 >> v2:172.21.15.176:3300/0 conn(0x7fb66c0d0e00 0x7fb66c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 -- 172.21.15.67:0/1633182219 >> 172.21.15.67:0/1633182219 conn(0x7fb66c0d5d90 msgr2=0x7fb66c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 -- 172.21.15.67:0/1633182219 shutdown_connections 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.173+0000 7fb670fb2640 1 -- 172.21.15.67:0/1633182219 wait complete. 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.175+0000 7fb670fb2640 1 Processor -- start 2024-04-25T15:27:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.175+0000 7fb670fb2640 1 -- start start 2024-04-25T15:27:19.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.176+0000 7fb670fb2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb66c06c5c0 0x7fb66c0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb670fb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0b7810 0x7fb66c14d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb670fb2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb66c0d0e00 0x7fb66c1fa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb670fb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c160ad0 con 0x7fb66c0b7810 2024-04-25T15:27:19.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb670fb2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c1613a0 con 0x7fb66c06c5c0 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb670fb2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb66c0b5320 con 0x7fb66c0d0e00 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0b7810 0x7fb66c14d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0b7810 0x7fb66c14d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37400/0 (socket says 172.21.15.67:37400) 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 -- 172.21.15.67:0/2078388650 learned_addr learned my addr 172.21.15.67:0/2078388650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb66a59c640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb66c06c5c0 0x7fb66c0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb66ad9d640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb66c0d0e00 0x7fb66c1fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 -- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb66c0d0e00 msgr2=0x7fb66c1fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb66c0d0e00 0x7fb66c1fa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 -- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb66c06c5c0 msgr2=0x7fb66c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb66c06c5c0 0x7fb66c0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 -- 172.21.15.67:0/2078388650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c1fb270 con 0x7fb66c0b7810 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb663fff640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0b7810 0x7fb66c14d010 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb654000c00 tx=0x7fb6540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb662ffd640 1 -- 172.21.15.67:0/2078388650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654041020 con 0x7fb66c0b7810 2024-04-25T15:27:19.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb662ffd640 1 -- 172.21.15.67:0/2078388650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6540306c0 con 0x7fb66c0b7810 2024-04-25T15:27:19.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb662ffd640 1 -- 172.21.15.67:0/2078388650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654037040 con 0x7fb66c0b7810 2024-04-25T15:27:19.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.177+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66c0c97d0 con 0x7fb66c0b7810 2024-04-25T15:27:19.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.178+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66c0c9d90 con 0x7fb66c0b7810 2024-04-25T15:27:19.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.179+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66c050740 con 0x7fb66c0b7810 2024-04-25T15:27:19.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.179+0000 7fb662ffd640 1 -- 172.21.15.67:0/2078388650 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb654044020 con 0x7fb66c0b7810 2024-04-25T15:27:19.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.180+0000 7fb662ffd640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb64c0744e0 0x7fb64c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.180+0000 7fb662ffd640 1 -- 172.21.15.67:0/2078388650 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fb6540b59c0 con 0x7fb66c0b7810 2024-04-25T15:27:19.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.180+0000 7fb66a59c640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb64c0744e0 0x7fb64c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.180+0000 7fb66a59c640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb64c0744e0 0x7fb64c0769f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0079a0 tx=0x7fb65c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.184+0000 7fb662ffd640 1 -- 172.21.15.67:0/2078388650 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6540b7e20 con 0x7fb66c0b7810 2024-04-25T15:27:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.368+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb66c0d61a0 con 0x7fb66c0b7810 2024-04-25T15:27:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.370+0000 7fb662ffd640 1 -- 172.21.15.67:0/2078388650 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16642 (secure 0 0 0) 0x7fb65404b030 con 0x7fb66c0b7810 2024-04-25T15:27:19.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:19.372 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":55,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:18.413564+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb64c0744e0 msgr2=0x7fb64c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb64c0744e0 0x7fb64c0769f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0079a0 tx=0x7fb65c008040 comp rx=0 tx=0).stop 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0b7810 msgr2=0x7fb66c14d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0b7810 0x7fb66c14d010 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb654000c00 tx=0x7fb6540046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 shutdown_connections 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb64c0744e0 0x7fb64c0769f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb66c0d0e00 0x7fb66c1fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0b7810 0x7fb66c14d010 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.371+0000 7fb670fb2640 1 --2- 172.21.15.67:0/2078388650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb66c06c5c0 0x7fb66c0ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.372+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 >> 172.21.15.67:0/2078388650 conn(0x7fb66c0d5d90 msgr2=0x7fb66c162120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.372+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 shutdown_connections 2024-04-25T15:27:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.372+0000 7fb670fb2640 1 -- 172.21.15.67:0/2078388650 wait complete. 2024-04-25T15:27:19.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:19.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.495+0000 7ff5278a8640 1 Processor -- start 2024-04-25T15:27:19.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.495+0000 7ff5278a8640 1 -- start start 2024-04-25T15:27:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff5278a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5200b8290 0x7ff520160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff5278a8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff52015df80 0x7ff520160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff5278a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5201615f0 con 0x7ff520164b00 2024-04-25T15:27:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff5278a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff520164670 con 0x7ff5200b8290 2024-04-25T15:27:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff5278a8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff520164020 con 0x7ff52015df80 2024-04-25T15:27:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff524e43640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5200b8290 0x7ff520160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff525644640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff520164b00 0x7ff5200b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56174/0 (socket says 172.21.15.67:56174) 2024-04-25T15:27:19.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff525644640 1 -- 172.21.15.67:0/980538247 learned_addr learned my addr 172.21.15.67:0/980538247 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:19.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff525e45640 1 --2- 172.21.15.67:0/980538247 >> v2:172.21.15.176:3300/0 conn(0x7ff52015df80 0x7ff520160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff524e43640 1 -- 172.21.15.67:0/980538247 >> v2:172.21.15.176:3300/0 conn(0x7ff52015df80 msgr2=0x7ff520160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:19.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff524e43640 1 --2- 172.21.15.67:0/980538247 >> v2:172.21.15.176:3300/0 conn(0x7ff52015df80 0x7ff520160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff524e43640 1 -- 172.21.15.67:0/980538247 >> v1:172.21.15.67:6789/0 conn(0x7ff520164b00 legacy=0x7ff5200b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff51ffff640 1 -- 172.21.15.67:0/980538247 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150536232 0 0) 0x7ff5201615f0 con 0x7ff520164b00 2024-04-25T15:27:19.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.497+0000 7ff524e43640 1 -- 172.21.15.67:0/980538247 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff520164300 con 0x7ff5200b8290 2024-04-25T15:27:19.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.498+0000 7ff524e43640 1 --2- 172.21.15.67:0/980538247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200b8290 0x7ff520160890 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff5200d3e80 tx=0x7ff51402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229a06a0d1eab2af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.498+0000 7ff51ffff640 1 -- 172.21.15.67:0/980538247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514037070 con 0x7ff5200b8290 2024-04-25T15:27:19.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.498+0000 7ff51ffff640 1 -- 172.21.15.67:0/980538247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff514032440 con 0x7ff5200b8290 2024-04-25T15:27:19.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.498+0000 7ff51ffff640 1 -- 172.21.15.67:0/980538247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514032810 con 0x7ff5200b8290 2024-04-25T15:27:19.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 -- 172.21.15.67:0/980538247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200b8290 msgr2=0x7ff520160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 --2- 172.21.15.67:0/980538247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200b8290 0x7ff520160890 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff5200d3e80 tx=0x7ff51402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 -- 172.21.15.67:0/980538247 shutdown_connections 2024-04-25T15:27:19.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 --2- 172.21.15.67:0/980538247 >> v2:172.21.15.176:3300/0 conn(0x7ff52015df80 0x7ff520160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 --2- 172.21.15.67:0/980538247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200b8290 0x7ff520160890 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 -- 172.21.15.67:0/980538247 >> 172.21.15.67:0/980538247 conn(0x7ff5200d2bc0 msgr2=0x7ff5200d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:19.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 -- 172.21.15.67:0/980538247 shutdown_connections 2024-04-25T15:27:19.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.499+0000 7ff5278a8640 1 -- 172.21.15.67:0/980538247 wait complete. 2024-04-25T15:27:19.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.501+0000 7ff5278a8640 1 Processor -- start 2024-04-25T15:27:19.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.501+0000 7ff5278a8640 1 -- start start 2024-04-25T15:27:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff5278a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff52015df80 0x7ff5200b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff5278a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff520164b00 0x7ff5200c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff5278a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52015b8a0 0x7ff52015bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff5278a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5200b4a00 con 0x7ff52015b8a0 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff5278a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff5200b4880 con 0x7ff52015df80 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff5278a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff5200b4b80 con 0x7ff520164b00 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff525e45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52015b8a0 0x7ff52015bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff525e45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52015b8a0 0x7ff52015bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37432/0 (socket says 172.21.15.67:37432) 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff525e45640 1 -- 172.21.15.67:0/2936308172 learned_addr learned my addr 172.21.15.67:0/2936308172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff525644640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff52015df80 0x7ff5200b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.502+0000 7ff524e43640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff520164b00 0x7ff5200c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff525e45640 1 -- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff520164b00 msgr2=0x7ff5200c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff525e45640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff520164b00 0x7ff5200c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff525e45640 1 -- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff52015df80 msgr2=0x7ff5200b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff525e45640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff52015df80 0x7ff5200b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff525e45640 1 -- 172.21.15.67:0/2936308172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52015c1f0 con 0x7ff52015b8a0 2024-04-25T15:27:19.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff525e45640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52015b8a0 0x7ff52015bcb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff510009fd0 tx=0x7ff51000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff51dffb640 1 -- 172.21.15.67:0/2936308172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510014420 con 0x7ff52015b8a0 2024-04-25T15:27:19.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff51dffb640 1 -- 172.21.15.67:0/2936308172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff510019070 con 0x7ff52015b8a0 2024-04-25T15:27:19.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff51dffb640 1 -- 172.21.15.67:0/2936308172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510014800 con 0x7ff52015b8a0 2024-04-25T15:27:19.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5201f6300 con 0x7ff52015b8a0 2024-04-25T15:27:19.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.503+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5201561c0 con 0x7ff52015b8a0 2024-04-25T15:27:19.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.504+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff520050740 con 0x7ff52015b8a0 2024-04-25T15:27:19.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.505+0000 7ff51dffb640 1 -- 172.21.15.67:0/2936308172 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff5100149a0 con 0x7ff52015b8a0 2024-04-25T15:27:19.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.505+0000 7ff51dffb640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff50c0744b0 0x7ff50c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.505+0000 7ff51dffb640 1 -- 172.21.15.67:0/2936308172 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7ff5100958b0 con 0x7ff52015b8a0 2024-04-25T15:27:19.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.505+0000 7ff525644640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff50c0744b0 0x7ff50c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.506+0000 7ff525644640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff50c0744b0 0x7ff50c0769c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff518007fd0 tx=0x7ff518007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.510+0000 7ff51dffb640 1 -- 172.21.15.67:0/2936308172 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff52015c1f0 con 0x7ff52015b8a0 2024-04-25T15:27:19.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.696+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5200d4170 con 0x7ff52015b8a0 2024-04-25T15:27:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.697+0000 7ff51dffb640 1 -- 172.21.15.67:0/2936308172 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16642 (secure 0 0 0) 0x7ff51005fa80 con 0x7ff52015b8a0 2024-04-25T15:27:19.700 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:19.700 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":55,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:18.413564+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:19.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff50c0744b0 msgr2=0x7ff50c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff50c0744b0 0x7ff50c0769c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff518007fd0 tx=0x7ff518007de0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52015b8a0 msgr2=0x7ff52015bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52015b8a0 0x7ff52015bcb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff510009fd0 tx=0x7ff51000e4b0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 shutdown_connections 2024-04-25T15:27:19.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff50c0744b0 0x7ff50c0769c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52015b8a0 0x7ff52015bcb0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff520164b00 0x7ff5200c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 --2- 172.21.15.67:0/2936308172 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff52015df80 0x7ff5200b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 >> 172.21.15.67:0/2936308172 conn(0x7ff5200d2bc0 msgr2=0x7ff5200b7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:19.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 shutdown_connections 2024-04-25T15:27:19.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.700+0000 7ff5278a8640 1 -- 172.21.15.67:0/2936308172 wait complete. 2024-04-25T15:27:19.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:27:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.826+0000 7f4b142b6640 1 Processor -- start 2024-04-25T15:27:19.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.826+0000 7f4b142b6640 1 -- start start 2024-04-25T15:27:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b142b6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b0c0b2e60 0x7f4b0c0d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b142b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c164020 0x7f4b0c164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b142b6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0c164970 con 0x7f4b0c0d1e20 2024-04-25T15:27:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b142b6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4b0c0d23f0 con 0x7f4b0c0b2e60 2024-04-25T15:27:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b142b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b0c0d26d0 con 0x7f4b0c164020 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b11851640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b0c0b2e60 0x7f4b0c0d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b11851640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b0c0b2e60 0x7f4b0c0d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37434/0 (socket says 172.21.15.67:37434) 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b11851640 1 -- 172.21.15.67:0/2608236210 learned_addr learned my addr 172.21.15.67:0/2608236210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b12853640 1 --2- 172.21.15.67:0/2608236210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c164020 0x7f4b0c164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b11851640 1 -- 172.21.15.67:0/2608236210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c164020 msgr2=0x7f4b0c164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b11851640 1 --2- 172.21.15.67:0/2608236210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c164020 0x7f4b0c164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b11851640 1 -- 172.21.15.67:0/2608236210 >> v1:172.21.15.176:6789/0 conn(0x7f4b0c0d1e20 legacy=0x7f4b0c08d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.828+0000 7f4b11851640 1 -- 172.21.15.67:0/2608236210 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c0d29b0 con 0x7f4b0c0b2e60 2024-04-25T15:27:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.829+0000 7f4b11851640 1 --2- 172.21.15.67:0/2608236210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c0b2e60 0x7f4b0c0d3490 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009870 tx=0x7f4b0002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4069fc3cdadee73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.829+0000 7f4b11050640 1 -- 172.21.15.67:0/2608236210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00037070 con 0x7f4b0c0b2e60 2024-04-25T15:27:19.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.829+0000 7f4b11050640 1 -- 172.21.15.67:0/2608236210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b0002ae50 con 0x7f4b0c0b2e60 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.829+0000 7f4b11050640 1 -- 172.21.15.67:0/2608236210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00032860 con 0x7f4b0c0b2e60 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 -- 172.21.15.67:0/2608236210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c0b2e60 msgr2=0x7f4b0c0d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 --2- 172.21.15.67:0/2608236210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c0b2e60 0x7f4b0c0d3490 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009870 tx=0x7f4b0002a300 comp rx=0 tx=0).stop 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 -- 172.21.15.67:0/2608236210 shutdown_connections 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 --2- 172.21.15.67:0/2608236210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c164020 0x7f4b0c164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 --2- 172.21.15.67:0/2608236210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c0b2e60 0x7f4b0c0d3490 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 -- 172.21.15.67:0/2608236210 >> 172.21.15.67:0/2608236210 conn(0x7f4b0c0d0530 msgr2=0x7f4b0c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 -- 172.21.15.67:0/2608236210 shutdown_connections 2024-04-25T15:27:19.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.830+0000 7f4b142b6640 1 -- 172.21.15.67:0/2608236210 wait complete. 2024-04-25T15:27:19.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.832+0000 7f4b142b6640 1 Processor -- start 2024-04-25T15:27:19.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.832+0000 7f4b142b6640 1 -- start start 2024-04-25T15:27:19.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b142b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c0b2e60 0x7f4b0c0b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b142b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0c0d1e20 0x7f4b0c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b142b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c164020 0x7f4b0c15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b142b6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c0ce070 con 0x7f4b0c164020 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b142b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0c0d2f70 con 0x7f4b0c0d1e20 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b142b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b0c0ce1f0 con 0x7f4b0c0b2e60 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c164020 0x7f4b0c15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c164020 0x7f4b0c15e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37442/0 (socket says 172.21.15.67:37442) 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 -- 172.21.15.67:0/3186269846 learned_addr learned my addr 172.21.15.67:0/3186269846 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12052640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c0b2e60 0x7f4b0c0b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b11851640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0c0d1e20 0x7f4b0c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 -- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c0b2e60 msgr2=0x7f4b0c0b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c0b2e60 0x7f4b0c0b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 -- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0c0d1e20 msgr2=0x7f4b0c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0c0d1e20 0x7f4b0c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.833+0000 7f4b12853640 1 -- 172.21.15.67:0/3186269846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0c0b4a70 con 0x7f4b0c164020 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b12052640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c0b2e60 0x7f4b0c0b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:19.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b11851640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0c0d1e20 0x7f4b0c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:19.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b12853640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c164020 0x7f4b0c15e2f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4afc007d40 tx=0x7f4afc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b0affd640 1 -- 172.21.15.67:0/3186269846 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc014420 con 0x7f4b0c164020 2024-04-25T15:27:19.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b0affd640 1 -- 172.21.15.67:0/3186269846 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4afc019070 con 0x7f4b0c164020 2024-04-25T15:27:19.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0c1f71d0 con 0x7f4b0c164020 2024-04-25T15:27:19.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b0affd640 1 -- 172.21.15.67:0/3186269846 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc014820 con 0x7f4b0c164020 2024-04-25T15:27:19.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.834+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0c1f7790 con 0x7f4b0c164020 2024-04-25T15:27:19.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.835+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b0c050740 con 0x7f4b0c164020 2024-04-25T15:27:19.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.836+0000 7f4b0affd640 1 -- 172.21.15.67:0/3186269846 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4afc0149c0 con 0x7f4b0c164020 2024-04-25T15:27:19.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.836+0000 7f4b0affd640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4af80744e0 0x7f4af80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:19.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.836+0000 7f4b0affd640 1 -- 172.21.15.67:0/3186269846 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f4afc095970 con 0x7f4b0c164020 2024-04-25T15:27:19.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.836+0000 7f4b12052640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4af80744e0 0x7f4af80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:19.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.837+0000 7f4b12052640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4af80744e0 0x7f4af80769f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4b040015e0 tx=0x7f4b04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:19.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:19.841+0000 7f4b0affd640 1 -- 172.21.15.67:0/3186269846 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4afc05fb40 con 0x7f4b0c164020 2024-04-25T15:27:20.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:20.068+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4b0c071000 con 0x7f4b0c164020 2024-04-25T15:27:21.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.032+0000 7f4b0affd640 1 -- 172.21.15.67:0/3186269846 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) ==== 118+0+0 (secure 0 0 0) 0x7f4afc063e40 con 0x7f4b0c164020 2024-04-25T15:27:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.034+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4af80744e0 msgr2=0x7f4af80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.034+0000 7f4b142b6640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4af80744e0 0x7f4af80769f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4b040015e0 tx=0x7f4b04009040 comp rx=0 tx=0).stop 2024-04-25T15:27:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.034+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c164020 msgr2=0x7f4b0c15e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.034+0000 7f4b142b6640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c164020 0x7f4b0c15e2f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4afc007d40 tx=0x7f4afc00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 shutdown_connections 2024-04-25T15:27:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4af80744e0 0x7f4af80769f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b0c164020 0x7f4b0c15e2f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0c0d1e20 0x7f4b0c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 --2- 172.21.15.67:0/3186269846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0c0b2e60 0x7f4b0c0b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 >> 172.21.15.67:0/3186269846 conn(0x7f4b0c0d0530 msgr2=0x7f4b0c0cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 shutdown_connections 2024-04-25T15:27:21.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.035+0000 7f4b142b6640 1 -- 172.21.15.67:0/3186269846 wait complete. 2024-04-25T15:27:21.050 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:27:21.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:27:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.171+0000 7f65aa0af640 1 Processor -- start 2024-04-25T15:27:21.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.171+0000 7f65aa0af640 1 -- start start 2024-04-25T15:27:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65aa0af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a40b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65aa0af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65a4163f30 0x7f65a4164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:21.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65aa0af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a4164880 con 0x7f65a40d1f30 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65aa0af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f65a4164ee0 con 0x7f65a4163f30 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65aa0af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f65a40b31e0 con 0x7f65a4164ad0 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65a4163f30 0x7f65a4164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65a4163f30 0x7f65a4164340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37456/0 (socket says 172.21.15.67:37456) 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a2ffd640 1 -- 172.21.15.67:0/696873363 learned_addr learned my addr 172.21.15.67:0/696873363 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a37fe640 1 --2- 172.21.15.67:0/696873363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a40b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a2ffd640 1 -- 172.21.15.67:0/696873363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 msgr2=0x7f65a40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a2ffd640 1 --2- 172.21.15.67:0/696873363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a40b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a2ffd640 1 -- 172.21.15.67:0/696873363 >> v1:172.21.15.176:6789/0 conn(0x7f65a40d1f30 legacy=0x7f65a40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:21.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.173+0000 7f65a2ffd640 1 -- 172.21.15.67:0/696873363 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a40d3170 con 0x7f65a4163f30 2024-04-25T15:27:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.174+0000 7f65a2ffd640 1 --2- 172.21.15.67:0/696873363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a4164340 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6598009870 tx=0x7f659802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a5b954e3c1e1bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.174+0000 7f65a27fc640 1 -- 172.21.15.67:0/696873363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598037070 con 0x7f65a4163f30 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.174+0000 7f65a27fc640 1 -- 172.21.15.67:0/696873363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6598032440 con 0x7f65a4163f30 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.174+0000 7f65a27fc640 1 -- 172.21.15.67:0/696873363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598032810 con 0x7f65a4163f30 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 -- 172.21.15.67:0/696873363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 msgr2=0x7f65a4164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 --2- 172.21.15.67:0/696873363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a4164340 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6598009870 tx=0x7f659802a270 comp rx=0 tx=0).stop 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 -- 172.21.15.67:0/696873363 shutdown_connections 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 --2- 172.21.15.67:0/696873363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a4164340 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 --2- 172.21.15.67:0/696873363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a40b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 -- 172.21.15.67:0/696873363 >> 172.21.15.67:0/696873363 conn(0x7f65a40d3aa0 msgr2=0x7f65a40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 -- 172.21.15.67:0/696873363 shutdown_connections 2024-04-25T15:27:21.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.175+0000 7f65aa0af640 1 -- 172.21.15.67:0/696873363 wait complete. 2024-04-25T15:27:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.177+0000 7f65aa0af640 1 Processor -- start 2024-04-25T15:27:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.177+0000 7f65aa0af640 1 -- start start 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.178+0000 7f65aa0af640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65a40d1f30 0x7f65a40cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.178+0000 7f65aa0af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.178+0000 7f65aa0af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a415bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.178+0000 7f65aa0af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65a406d9b0 con 0x7f65a4163f30 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.178+0000 7f65aa0af640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f65a406d6d0 con 0x7f65a40d1f30 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.178+0000 7f65aa0af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f65a40b9430 con 0x7f65a4164ad0 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.178+0000 7f65a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a415d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37458/0 (socket says 172.21.15.67:37458) 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 -- 172.21.15.67:0/1367372946 learned_addr learned my addr 172.21.15.67:0/1367372946 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a37fe640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65a40d1f30 0x7f65a40cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a3fff640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a415bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 -- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 msgr2=0x7f65a415bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a415bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 -- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65a40d1f30 msgr2=0x7f65a40cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65a40d1f30 0x7f65a40cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 -- 172.21.15.67:0/1367372946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a40b8780 con 0x7f65a4163f30 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a3fff640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a415bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65a2ffd640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a415d280 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6598000c00 tx=0x7f65980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f658bfff640 1 -- 172.21.15.67:0/1367372946 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598037040 con 0x7f65a4163f30 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f658bfff640 1 -- 172.21.15.67:0/1367372946 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6598042070 con 0x7f65a4163f30 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a40b8a60 con 0x7f65a4163f30 2024-04-25T15:27:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f658bfff640 1 -- 172.21.15.67:0/1367372946 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598030880 con 0x7f65a4163f30 2024-04-25T15:27:21.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.179+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a40b6c70 con 0x7f65a4163f30 2024-04-25T15:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.181+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a40b84f0 con 0x7f65a4163f30 2024-04-25T15:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.181+0000 7f658bfff640 1 -- 172.21.15.67:0/1367372946 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6598030a20 con 0x7f65a4163f30 2024-04-25T15:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.181+0000 7f658bfff640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f65900744e0 0x7f65900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.182+0000 7f658bfff640 1 -- 172.21.15.67:0/1367372946 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f65980b5990 con 0x7f65a4163f30 2024-04-25T15:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.182+0000 7f65a37fe640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f65900744e0 0x7f65900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:21.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.182+0000 7f65a37fe640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f65900744e0 0x7f65900769f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f65a40746e0 tx=0x7f6594009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:21.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.186+0000 7f658bfff640 1 -- 172.21.15.67:0/1367372946 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65980801b0 con 0x7f65a4163f30 2024-04-25T15:27:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:21.408+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f65a40d4770 con 0x7f65a4163f30 2024-04-25T15:27:22.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.035+0000 7f658bfff640 1 -- 172.21.15.67:0/1367372946 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7f659807fab0 con 0x7f65a4163f30 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f65900744e0 msgr2=0x7f65900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f65900744e0 0x7f65900769f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f65a40746e0 tx=0x7f6594009190 comp rx=0 tx=0).stop 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 msgr2=0x7f65a415d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a415d280 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6598000c00 tx=0x7f65980046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 shutdown_connections 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f65900744e0 0x7f65900769f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f65a4164ad0 0x7f65a415bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65a4163f30 0x7f65a415d280 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 --2- 172.21.15.67:0/1367372946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65a40d1f30 0x7f65a40cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 >> 172.21.15.67:0/1367372946 conn(0x7f65a40d3aa0 msgr2=0x7f65a40d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 shutdown_connections 2024-04-25T15:27:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.038+0000 7f65aa0af640 1 -- 172.21.15.67:0/1367372946 wait complete. 2024-04-25T15:27:22.054 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:27:22.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:22.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.171+0000 7f8a9fb72640 1 Processor -- start 2024-04-25T15:27:22.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.171+0000 7f8a9fb72640 1 -- start start 2024-04-25T15:27:22.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9fb72640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8a98164750 0x7f8a9816b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9fb72640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a98164db0 con 0x7f8a9815abd0 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9fb72640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a98160d30 con 0x7f8a9806ccd0 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9fb72640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8a98161380 con 0x7f8a98164750 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9d90e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a9815abd0 0x7f8a9806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56180/0 (socket says 172.21.15.67:56180) 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9d90e640 1 -- 172.21.15.67:0/748255448 learned_addr learned my addr 172.21.15.67:0/748255448 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9e10f640 1 --2- 172.21.15.67:0/748255448 >> v2:172.21.15.176:3300/0 conn(0x7f8a98164750 0x7f8a9816b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1089548880 0 0) 0x7f8a98164db0 con 0x7f8a9815abd0 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a80003520 con 0x7f8a9815abd0 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1955871669 0 0) 0x7f8a80003520 con 0x7f8a9815abd0 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 >> v2:172.21.15.176:3300/0 conn(0x7f8a98164750 msgr2=0x7f8a9816b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 --2- 172.21.15.67:0/748255448 >> v2:172.21.15.176:3300/0 conn(0x7f8a98164750 0x7f8a9816b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 >> v1:172.21.15.176:6789/0 conn(0x7f8a9806ccd0 legacy=0x7f8a98164040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a98161010 con 0x7f8a9815abd0 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8a90001670 con 0x7f8a9815abd0 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 >> v1:172.21.15.67:6789/0 conn(0x7f8a9815abd0 legacy=0x7f8a9806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.173+0000 7f8a9c90c640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800039e0 0x7f8a80003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9c90c640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80004800 0x7f8a80004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9c90c640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80005270 0x7f8a8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98161380 con 0x7f8a80004800 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8000c190 con 0x7f8a80005270 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a8000c390 con 0x7f8a800039e0 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d10d640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80004800 0x7f8a80004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d90e640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800039e0 0x7f8a80003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9e10f640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80005270 0x7f8a8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d10d640 1 -- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800039e0 msgr2=0x7f8a80003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d10d640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800039e0 0x7f8a80003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d10d640 1 -- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80005270 msgr2=0x7f8a8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d10d640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80005270 0x7f8a8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d10d640 1 -- 172.21.15.67:0/748255448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a80003520 con 0x7f8a80004800 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9e10f640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80005270 0x7f8a8000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9d90e640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a800039e0 0x7f8a80003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.174+0000 7f8a9e10f640 1 -- 172.21.15.67:0/748255448 reap_dead start 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.175+0000 7f8a9d10d640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80004800 0x7f8a80004c10 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c001750 tx=0x7f8a80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938227443db9ca27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.175+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c001370 con 0x7f8a80004800 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.175+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a8c0052d0 con 0x7f8a80004800 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.175+0000 7f8a9c90c640 1 -- 172.21.15.67:0/748255448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c02abc0 con 0x7f8a80004800 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.175+0000 7f8a9fb72640 1 -- 172.21.15.67:0/748255448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80004800 msgr2=0x7f8a80004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.175+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80004800 0x7f8a80004c10 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c001750 tx=0x7f8a80005830 comp rx=0 tx=0).stop 2024-04-25T15:27:22.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.176+0000 7f8a9fb72640 1 -- 172.21.15.67:0/748255448 shutdown_connections 2024-04-25T15:27:22.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.176+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/748255448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80004800 0x7f8a80004c10 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.176+0000 7f8a9fb72640 1 -- 172.21.15.67:0/748255448 >> 172.21.15.67:0/748255448 conn(0x7f8a980d5df0 msgr2=0x7f8a980d6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:22.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.176+0000 7f8a9fb72640 1 -- 172.21.15.67:0/748255448 shutdown_connections 2024-04-25T15:27:22.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.176+0000 7f8a9fb72640 1 -- 172.21.15.67:0/748255448 wait complete. 2024-04-25T15:27:22.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.177+0000 7f8a9fb72640 1 Processor -- start 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.178+0000 7f8a9fb72640 1 -- start start 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9fb72640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80004800 0x7f8a9814c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9fb72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a9815b3d0 0x7f8a981fa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9fb72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a981fa680 0x7f8a981f5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9fb72640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98161810 con 0x7f8a981fa680 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9fb72640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98164b60 con 0x7f8a80004800 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9fb72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a98148670 con 0x7f8a9815b3d0 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a981fa680 0x7f8a981f5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a981fa680 0x7f8a981f5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37464/0 (socket says 172.21.15.67:37464) 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 -- 172.21.15.67:0/1151371423 learned_addr learned my addr 172.21.15.67:0/1151371423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9d10d640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a9815b3d0 0x7f8a981fa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9d90e640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80004800 0x7f8a9814c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 -- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a9815b3d0 msgr2=0x7f8a981fa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a9815b3d0 0x7f8a981fa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 -- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80004800 msgr2=0x7f8a9814c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80004800 0x7f8a9814c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 -- 172.21.15.67:0/1151371423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a981faa90 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9d10d640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a9815b3d0 0x7f8a981fa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.179+0000 7f8a9e10f640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a981fa680 0x7f8a981f5eb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a980d5da0 tx=0x7f8a88010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.180+0000 7f8a967fc640 1 -- 172.21.15.67:0/1151371423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88010dd0 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.180+0000 7f8a967fc640 1 -- 172.21.15.67:0/1151371423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a88003990 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.180+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a980cef90 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.180+0000 7f8a967fc640 1 -- 172.21.15.67:0/1151371423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88003d80 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.180+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a980cf550 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.181+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a98050740 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.181+0000 7f8a967fc640 1 -- 172.21.15.67:0/1151371423 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8a8801e020 con 0x7f8a981fa680 2024-04-25T15:27:22.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.182+0000 7f8a967fc640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a840744e0 0x7f8a840769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.182+0000 7f8a967fc640 1 -- 172.21.15.67:0/1151371423 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f8a880959a0 con 0x7f8a981fa680 2024-04-25T15:27:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.182+0000 7f8a9d90e640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a840744e0 0x7f8a840769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.182+0000 7f8a9d90e640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a840744e0 0x7f8a840769f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8a90000fb0 tx=0x7f8a90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.186+0000 7f8a967fc640 1 -- 172.21.15.67:0/1151371423 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a981faa90 con 0x7f8a981fa680 2024-04-25T15:27:22.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.408+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a980ced20 con 0x7f8a981fa680 2024-04-25T15:27:22.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.409+0000 7f8a967fc640 1 -- 172.21.15.67:0/1151371423 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5181 (secure 0 0 0) 0x7f8a88060130 con 0x7f8a981fa680 2024-04-25T15:27:22.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:22.411 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/610105566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":610105566},{"type":"v1","addr":"172.21.15.67:6835","nonce":610105566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/947236976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":947236976},{"type":"v1","addr":"172.21.15.176:6837","nonce":947236976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3211936625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3211936625},{"type":"v1","addr":"172.21.15.176:6836","nonce":3211936625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:18.413498+0000","modified":"2024-04-25T15:27:21.411023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5880":{"gid":5880,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3185771646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3185771646},{"type":"v1","addr":"172.21.15.67:6833","nonce":3185771646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5880,"qdb_cluster":[5880]},"id":4}]} 2024-04-25T15:27:22.411 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a840744e0 msgr2=0x7f8a840769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a840744e0 0x7f8a840769f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8a90000fb0 tx=0x7f8a90009040 comp rx=0 tx=0).stop 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a981fa680 msgr2=0x7f8a981f5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a981fa680 0x7f8a981f5eb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a980d5da0 tx=0x7f8a88010590 comp rx=0 tx=0).stop 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 shutdown_connections 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a840744e0 0x7f8a840769f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a981fa680 0x7f8a981f5eb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a9815b3d0 0x7f8a981fa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 --2- 172.21.15.67:0/1151371423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a80004800 0x7f8a9814c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 >> 172.21.15.67:0/1151371423 conn(0x7f8a980d5df0 msgr2=0x7f8a981631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:22.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 shutdown_connections 2024-04-25T15:27:22.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.411+0000 7f8a9fb72640 1 -- 172.21.15.67:0/1151371423 wait complete. 2024-04-25T15:27:22.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.533+0000 7f3baf4bd640 1 Processor -- start 2024-04-25T15:27:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.534+0000 7f3baf4bd640 1 -- start start 2024-04-25T15:27:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3baf4bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba80b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3baf4bd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3ba806e8f0 0x7f3ba80d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3baf4bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ba80d09f0 0x7f3ba80b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3baf4bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3ba806e460 con 0x7f3ba80d09f0 2024-04-25T15:27:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3baf4bd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3ba806d870 con 0x7f3ba806e8f0 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3baf4bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba806db10 con 0x7f3ba80d4570 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3bada5a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ba80d09f0 0x7f3ba80b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3bada5a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ba80d09f0 0x7f3ba80b2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37466/0 (socket says 172.21.15.67:37466) 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3bada5a640 1 -- 172.21.15.67:0/2250292733 learned_addr learned my addr 172.21.15.67:0/2250292733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3bad259640 1 --2- 172.21.15.67:0/2250292733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba80b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.535+0000 7f3baca58640 1 --2- 172.21.15.67:0/2250292733 >> v2:172.21.15.176:3300/0 conn(0x7f3ba806e8f0 0x7f3ba80d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.536+0000 7f3bada5a640 1 -- 172.21.15.67:0/2250292733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 msgr2=0x7f3ba80b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.536+0000 7f3bada5a640 1 --2- 172.21.15.67:0/2250292733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba80b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.536+0000 7f3bada5a640 1 -- 172.21.15.67:0/2250292733 >> v2:172.21.15.176:3300/0 conn(0x7f3ba806e8f0 msgr2=0x7f3ba80d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.536+0000 7f3bada5a640 1 --2- 172.21.15.67:0/2250292733 >> v2:172.21.15.176:3300/0 conn(0x7f3ba806e8f0 0x7f3ba80d04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.536+0000 7f3bada5a640 1 -- 172.21.15.67:0/2250292733 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba806e0f0 con 0x7f3ba80d09f0 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.536+0000 7f3bada5a640 1 --2- 172.21.15.67:0/2250292733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80b2dc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8074be0 tx=0x7f3ba002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8acca0aed596683 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.536+0000 7f3ba7fff640 1 -- 172.21.15.67:0/2250292733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0037070 con 0x7f3ba80d09f0 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3ba7fff640 1 -- 172.21.15.67:0/2250292733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ba0032440 con 0x7f3ba80d09f0 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3ba7fff640 1 -- 172.21.15.67:0/2250292733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0032810 con 0x7f3ba80d09f0 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 -- 172.21.15.67:0/2250292733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 msgr2=0x7f3ba80b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/2250292733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80b2dc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8074be0 tx=0x7f3ba002a140 comp rx=0 tx=0).stop 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 -- 172.21.15.67:0/2250292733 shutdown_connections 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/2250292733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80b2dc0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/2250292733 >> v2:172.21.15.176:3300/0 conn(0x7f3ba806e8f0 0x7f3ba80d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/2250292733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba80b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 -- 172.21.15.67:0/2250292733 >> 172.21.15.67:0/2250292733 conn(0x7f3ba80d3a50 msgr2=0x7f3ba80d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 -- 172.21.15.67:0/2250292733 shutdown_connections 2024-04-25T15:27:22.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.537+0000 7f3baf4bd640 1 -- 172.21.15.67:0/2250292733 wait complete. 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.539+0000 7f3baf4bd640 1 Processor -- start 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.539+0000 7f3baf4bd640 1 -- start start 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.540+0000 7f3baf4bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ba806e8f0 0x7f3ba8161fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.540+0000 7f3baf4bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.540+0000 7f3baf4bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.540+0000 7f3baf4bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba80b3300 con 0x7f3ba80d09f0 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.540+0000 7f3baf4bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba806ed00 con 0x7f3ba806e8f0 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.540+0000 7f3baf4bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba80b3480 con 0x7f3ba80d4570 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80c7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37470/0 (socket says 172.21.15.67:37470) 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 -- 172.21.15.67:0/515285128 learned_addr learned my addr 172.21.15.67:0/515285128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3bad259640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ba806e8f0 0x7f3ba8161fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3bada5a640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 -- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 msgr2=0x7f3ba809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 -- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ba806e8f0 msgr2=0x7f3ba8161fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ba806e8f0 0x7f3ba8161fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 -- 172.21.15.67:0/515285128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba81f8bf0 con 0x7f3ba80d09f0 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baca58640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80c7670 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00e870 tx=0x7f3b9c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3ba5ffb640 1 -- 172.21.15.67:0/515285128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c017070 con 0x7f3ba80d09f0 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3ba5ffb640 1 -- 172.21.15.67:0/515285128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b9c00ee60 con 0x7f3ba80d09f0 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba81f9900 con 0x7f3ba80d09f0 2024-04-25T15:27:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.541+0000 7f3ba5ffb640 1 -- 172.21.15.67:0/515285128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c00cb80 con 0x7f3ba80d09f0 2024-04-25T15:27:22.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.542+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba81f9340 con 0x7f3ba80d09f0 2024-04-25T15:27:22.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.542+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba8050740 con 0x7f3ba80d09f0 2024-04-25T15:27:22.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.543+0000 7f3ba5ffb640 1 -- 172.21.15.67:0/515285128 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3b9c01a070 con 0x7f3ba80d09f0 2024-04-25T15:27:22.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.544+0000 7f3ba5ffb640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b940744e0 0x7f3b940769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.544+0000 7f3ba5ffb640 1 -- 172.21.15.67:0/515285128 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f3b9c094f10 con 0x7f3ba80d09f0 2024-04-25T15:27:22.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.544+0000 7f3bad259640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b940744e0 0x7f3b940769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.545+0000 7f3bad259640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b940744e0 0x7f3b940769f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3b980113a0 tx=0x7f3b9800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.549+0000 7f3ba5ffb640 1 -- 172.21.15.67:0/515285128 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b9c05f0e0 con 0x7f3ba80d09f0 2024-04-25T15:27:22.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.735+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ba81f8dc0 con 0x7f3ba80d09f0 2024-04-25T15:27:22.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.737+0000 7f3ba5ffb640 1 -- 172.21.15.67:0/515285128 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16629 (secure 0 0 0) 0x7f3b9c0633e0 con 0x7f3ba80d09f0 2024-04-25T15:27:22.739 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:22.739 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:20.019287+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:22.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.738+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b940744e0 msgr2=0x7f3b940769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.738+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b940744e0 0x7f3b940769f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3b980113a0 tx=0x7f3b9800b040 comp rx=0 tx=0).stop 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.738+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 msgr2=0x7f3ba80c7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.738+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80c7670 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00e870 tx=0x7f3b9c00ed30 comp rx=0 tx=0).stop 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 shutdown_connections 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b940744e0 0x7f3b940769f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ba80d4570 0x7f3ba809dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ba80d09f0 0x7f3ba80c7670 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 --2- 172.21.15.67:0/515285128 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ba806e8f0 0x7f3ba8161fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 >> 172.21.15.67:0/515285128 conn(0x7f3ba80d3a50 msgr2=0x7f3ba8072b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 shutdown_connections 2024-04-25T15:27:22.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.739+0000 7f3baf4bd640 1 -- 172.21.15.67:0/515285128 wait complete. 2024-04-25T15:27:22.752 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:27:22.752 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:27:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.863+0000 7efd65430640 1 Processor -- start 2024-04-25T15:27:22.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.863+0000 7efd65430640 1 -- start start 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd65430640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd600cf770 0x7efd60162590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd65430640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7efd60162ad0 0x7efd600b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd65430640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd600cfc30 con 0x7efd600b91d0 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd65430640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efd600d1ff0 con 0x7efd600cf770 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd65430640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7efd600d2890 con 0x7efd60162ad0 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd57fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd600cf770 0x7efd60162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd57fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd600cf770 0x7efd60162590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37482/0 (socket says 172.21.15.67:37482) 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.864+0000 7efd57fff640 1 -- 172.21.15.67:0/2572076830 learned_addr learned my addr 172.21.15.67:0/2572076830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.865+0000 7efd5f7fe640 1 --2- 172.21.15.67:0/2572076830 >> v2:172.21.15.176:3300/0 conn(0x7efd60162ad0 0x7efd600b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.865+0000 7efd57fff640 1 -- 172.21.15.67:0/2572076830 >> v2:172.21.15.176:3300/0 conn(0x7efd60162ad0 msgr2=0x7efd600b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.865+0000 7efd57fff640 1 --2- 172.21.15.67:0/2572076830 >> v2:172.21.15.176:3300/0 conn(0x7efd60162ad0 0x7efd600b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.865+0000 7efd57fff640 1 -- 172.21.15.67:0/2572076830 >> v1:172.21.15.176:6789/0 conn(0x7efd600b91d0 legacy=0x7efd60161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.865+0000 7efd57fff640 1 -- 172.21.15.67:0/2572076830 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd600d2240 con 0x7efd600cf770 2024-04-25T15:27:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd57fff640 1 --2- 172.21.15.67:0/2572076830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600cf770 0x7efd60162590 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7efd600caa70 tx=0x7efd4802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b98550ead9a876 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd5e7fc640 1 -- 172.21.15.67:0/2572076830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48037070 con 0x7efd600cf770 2024-04-25T15:27:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd5e7fc640 1 -- 172.21.15.67:0/2572076830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd48032440 con 0x7efd600cf770 2024-04-25T15:27:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd5e7fc640 1 -- 172.21.15.67:0/2572076830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48032810 con 0x7efd600cf770 2024-04-25T15:27:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd65430640 1 -- 172.21.15.67:0/2572076830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600cf770 msgr2=0x7efd60162590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd65430640 1 --2- 172.21.15.67:0/2572076830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600cf770 0x7efd60162590 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7efd600caa70 tx=0x7efd4802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd65430640 1 -- 172.21.15.67:0/2572076830 shutdown_connections 2024-04-25T15:27:22.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.866+0000 7efd65430640 1 --2- 172.21.15.67:0/2572076830 >> v2:172.21.15.176:3300/0 conn(0x7efd60162ad0 0x7efd600b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.867+0000 7efd65430640 1 --2- 172.21.15.67:0/2572076830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600cf770 0x7efd60162590 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.867+0000 7efd65430640 1 -- 172.21.15.67:0/2572076830 >> 172.21.15.67:0/2572076830 conn(0x7efd600d3aa0 msgr2=0x7efd600d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:22.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.867+0000 7efd65430640 1 -- 172.21.15.67:0/2572076830 shutdown_connections 2024-04-25T15:27:22.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.867+0000 7efd65430640 1 -- 172.21.15.67:0/2572076830 wait complete. 2024-04-25T15:27:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.868+0000 7efd65430640 1 Processor -- start 2024-04-25T15:27:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.868+0000 7efd65430640 1 -- start start 2024-04-25T15:27:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600b91d0 0x7efd600ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd60162ad0 0x7efd600cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd600c9850 0x7efd600c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efd600b24a0 con 0x7efd600b91d0 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efd600b2320 con 0x7efd600c9850 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efd600b0bd0 con 0x7efd60162ad0 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600b91d0 0x7efd600ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600b91d0 0x7efd600ccf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37496/0 (socket says 172.21.15.67:37496) 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 -- 172.21.15.67:0/2728399465 learned_addr learned my addr 172.21.15.67:0/2728399465 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd57fff640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd60162ad0 0x7efd600cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5f7fe640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd600c9850 0x7efd600c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 -- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd60162ad0 msgr2=0x7efd600cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd60162ad0 0x7efd600cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 -- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd600c9850 msgr2=0x7efd600c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd600c9850 0x7efd600c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 -- 172.21.15.67:0/2728399465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd600cb3a0 con 0x7efd600b91d0 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd57fff640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd60162ad0 0x7efd600cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd5effd640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600b91d0 0x7efd600ccf70 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c3c0 tx=0x7efd5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd577fe640 1 -- 172.21.15.67:0/2728399465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd500052d0 con 0x7efd600b91d0 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd577fe640 1 -- 172.21.15.67:0/2728399465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd50005490 con 0x7efd600b91d0 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd600cb680 con 0x7efd600b91d0 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd577fe640 1 -- 172.21.15.67:0/2728399465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50007260 con 0x7efd600b91d0 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.870+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd600cbc40 con 0x7efd600b91d0 2024-04-25T15:27:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.872+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd60050740 con 0x7efd600b91d0 2024-04-25T15:27:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.872+0000 7efd577fe640 1 -- 172.21.15.67:0/2728399465 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efd50007400 con 0x7efd600b91d0 2024-04-25T15:27:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.873+0000 7efd577fe640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efd400744e0 0x7efd400769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.873+0000 7efd577fe640 1 -- 172.21.15.67:0/2728399465 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7efd500945a0 con 0x7efd600b91d0 2024-04-25T15:27:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.873+0000 7efd57fff640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efd400744e0 0x7efd400769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:22.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.873+0000 7efd57fff640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efd400744e0 0x7efd400769f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efd48034000 tx=0x7efd4802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:22.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:22.877+0000 7efd577fe640 1 -- 172.21.15.67:0/2728399465 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd5005e770 con 0x7efd600b91d0 2024-04-25T15:27:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.059+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd600d2c20 con 0x7efd400744e0 2024-04-25T15:27:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.066+0000 7efd577fe640 1 -- 172.21.15.67:0/2728399465 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7efd600d2c20 con 0x7efd400744e0 2024-04-25T15:27:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:23.072 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:27:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efd400744e0 msgr2=0x7efd400769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efd400744e0 0x7efd400769f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efd48034000 tx=0x7efd4802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600b91d0 msgr2=0x7efd600ccf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600b91d0 0x7efd600ccf70 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c3c0 tx=0x7efd5000c880 comp rx=0 tx=0).stop 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 shutdown_connections 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efd400744e0 0x7efd400769f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd600c9850 0x7efd600c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd60162ad0 0x7efd600cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 --2- 172.21.15.67:0/2728399465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd600b91d0 0x7efd600ccf70 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 >> 172.21.15.67:0/2728399465 conn(0x7efd600d3aa0 msgr2=0x7efd6006e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 shutdown_connections 2024-04-25T15:27:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.069+0000 7efd65430640 1 -- 172.21.15.67:0/2728399465 wait complete. 2024-04-25T15:27:23.082 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-25T15:27:22.822331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222672,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752048,"statfs":{"total":773094113280,"available":772866097152,"internally_reserved":0,"allocated":6627328,"data_stored":3412076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.834236"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082561+0000","last_change":"2024-04-25T15:27:19.027989+0000","last_active":"2024-04-25T15:27:20.082561+0000","last_peered":"2024-04-25T15:27:20.082561+0000","last_clean":"2024-04-25T15:27:20.082561+0000","last_became_active":"2024-04-25T15:27:19.027574+0000","last_became_peered":"2024-04-25T15:27:19.027574+0000","last_unstale":"2024-04-25T15:27:20.082561+0000","last_undegraded":"2024-04-25T15:27:20.082561+0000","last_fullsized":"2024-04-25T15:27:20.082561+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:11:19.196334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082807+0000","last_change":"2024-04-25T15:27:19.028322+0000","last_active":"2024-04-25T15:27:20.082807+0000","last_peered":"2024-04-25T15:27:20.082807+0000","last_clean":"2024-04-25T15:27:20.082807+0000","last_became_active":"2024-04-25T15:27:19.028023+0000","last_became_peered":"2024-04-25T15:27:19.028023+0000","last_unstale":"2024-04-25T15:27:20.082807+0000","last_undegraded":"2024-04-25T15:27:20.082807+0000","last_fullsized":"2024-04-25T15:27:20.082807+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:27.073882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083553+0000","last_change":"2024-04-25T15:27:19.028642+0000","last_active":"2024-04-25T15:27:20.083553+0000","last_peered":"2024-04-25T15:27:20.083553+0000","last_clean":"2024-04-25T15:27:20.083553+0000","last_became_active":"2024-04-25T15:27:19.028329+0000","last_became_peered":"2024-04-25T15:27:19.028329+0000","last_unstale":"2024-04-25T15:27:20.083553+0000","last_undegraded":"2024-04-25T15:27:20.083553+0000","last_fullsized":"2024-04-25T15:27:20.083553+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:44:38.823454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083176+0000","last_change":"2024-04-25T15:27:19.033445+0000","last_active":"2024-04-25T15:27:20.083176+0000","last_peered":"2024-04-25T15:27:20.083176+0000","last_clean":"2024-04-25T15:27:20.083176+0000","last_became_active":"2024-04-25T15:27:19.032634+0000","last_became_peered":"2024-04-25T15:27:19.032634+0000","last_unstale":"2024-04-25T15:27:20.083176+0000","last_undegraded":"2024-04-25T15:27:20.083176+0000","last_fullsized":"2024-04-25T15:27:20.083176+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:55.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082495+0000","last_change":"2024-04-25T15:27:19.031298+0000","last_active":"2024-04-25T15:27:20.082495+0000","last_peered":"2024-04-25T15:27:20.082495+0000","last_clean":"2024-04-25T15:27:20.082495+0000","last_became_active":"2024-04-25T15:27:19.030955+0000","last_became_peered":"2024-04-25T15:27:19.030955+0000","last_unstale":"2024-04-25T15:27:20.082495+0000","last_undegraded":"2024-04-25T15:27:20.082495+0000","last_fullsized":"2024-04-25T15:27:20.082495+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:06.145828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023486+0000","last_change":"2024-04-25T15:27:19.030506+0000","last_active":"2024-04-25T15:27:20.023486+0000","last_peered":"2024-04-25T15:27:20.023486+0000","last_clean":"2024-04-25T15:27:20.023486+0000","last_became_active":"2024-04-25T15:27:19.030180+0000","last_became_peered":"2024-04-25T15:27:19.030180+0000","last_unstale":"2024-04-25T15:27:20.023486+0000","last_undegraded":"2024-04-25T15:27:20.023486+0000","last_fullsized":"2024-04-25T15:27:20.023486+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:06.086869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023794+0000","last_change":"2024-04-25T15:27:19.032135+0000","last_active":"2024-04-25T15:27:20.023794+0000","last_peered":"2024-04-25T15:27:20.023794+0000","last_clean":"2024-04-25T15:27:20.023794+0000","last_became_active":"2024-04-25T15:27:19.031822+0000","last_became_peered":"2024-04-25T15:27:19.031822+0000","last_unstale":"2024-04-25T15:27:20.023794+0000","last_undegraded":"2024-04-25T15:27:20.023794+0000","last_fullsized":"2024-04-25T15:27:20.023794+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:37.460277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082591+0000","last_change":"2024-04-25T15:27:19.032917+0000","last_active":"2024-04-25T15:27:20.082591+0000","last_peered":"2024-04-25T15:27:20.082591+0000","last_clean":"2024-04-25T15:27:20.082591+0000","last_became_active":"2024-04-25T15:27:19.030486+0000","last_became_peered":"2024-04-25T15:27:19.030486+0000","last_unstale":"2024-04-25T15:27:20.082591+0000","last_undegraded":"2024-04-25T15:27:20.082591+0000","last_fullsized":"2024-04-25T15:27:20.082591+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:22.565243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024122+0000","last_change":"2024-04-25T15:27:19.029718+0000","last_active":"2024-04-25T15:27:20.024122+0000","last_peered":"2024-04-25T15:27:20.024122+0000","last_clean":"2024-04-25T15:27:20.024122+0000","last_became_active":"2024-04-25T15:27:19.029334+0000","last_became_peered":"2024-04-25T15:27:19.029334+0000","last_unstale":"2024-04-25T15:27:20.024122+0000","last_undegraded":"2024-04-25T15:27:20.024122+0000","last_fullsized":"2024-04-25T15:27:20.024122+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:45.710352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024418+0000","last_change":"2024-04-25T15:27:19.034934+0000","last_active":"2024-04-25T15:27:20.024418+0000","last_peered":"2024-04-25T15:27:20.024418+0000","last_clean":"2024-04-25T15:27:20.024418+0000","last_became_active":"2024-04-25T15:27:19.034556+0000","last_became_peered":"2024-04-25T15:27:19.034556+0000","last_unstale":"2024-04-25T15:27:20.024418+0000","last_undegraded":"2024-04-25T15:27:20.024418+0000","last_fullsized":"2024-04-25T15:27:20.024418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:24.132206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024406+0000","last_change":"2024-04-25T15:27:19.026266+0000","last_active":"2024-04-25T15:27:20.024406+0000","last_peered":"2024-04-25T15:27:20.024406+0000","last_clean":"2024-04-25T15:27:20.024406+0000","last_became_active":"2024-04-25T15:27:19.025958+0000","last_became_peered":"2024-04-25T15:27:19.025958+0000","last_unstale":"2024-04-25T15:27:20.024406+0000","last_undegraded":"2024-04-25T15:27:20.024406+0000","last_fullsized":"2024-04-25T15:27:20.024406+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:52.878059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083353+0000","last_change":"2024-04-25T15:27:19.041293+0000","last_active":"2024-04-25T15:27:20.083353+0000","last_peered":"2024-04-25T15:27:20.083353+0000","last_clean":"2024-04-25T15:27:20.083353+0000","last_became_active":"2024-04-25T15:27:19.040871+0000","last_became_peered":"2024-04-25T15:27:19.040871+0000","last_unstale":"2024-04-25T15:27:20.083353+0000","last_undegraded":"2024-04-25T15:27:20.083353+0000","last_fullsized":"2024-04-25T15:27:20.083353+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:36.346379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030231+0000","last_change":"2024-04-25T15:27:19.031072+0000","last_active":"2024-04-25T15:27:20.030231+0000","last_peered":"2024-04-25T15:27:20.030231+0000","last_clean":"2024-04-25T15:27:20.030231+0000","last_became_active":"2024-04-25T15:27:19.030774+0000","last_became_peered":"2024-04-25T15:27:19.030774+0000","last_unstale":"2024-04-25T15:27:20.030231+0000","last_undegraded":"2024-04-25T15:27:20.030231+0000","last_fullsized":"2024-04-25T15:27:20.030231+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:47.390421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024125+0000","last_change":"2024-04-25T15:27:19.033832+0000","last_active":"2024-04-25T15:27:20.024125+0000","last_peered":"2024-04-25T15:27:20.024125+0000","last_clean":"2024-04-25T15:27:20.024125+0000","last_became_active":"2024-04-25T15:27:19.033527+0000","last_became_peered":"2024-04-25T15:27:19.033527+0000","last_unstale":"2024-04-25T15:27:20.024125+0000","last_undegraded":"2024-04-25T15:27:20.024125+0000","last_fullsized":"2024-04-25T15:27:20.024125+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:14.471211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023923+0000","last_change":"2024-04-25T15:27:19.037925+0000","last_active":"2024-04-25T15:27:20.023923+0000","last_peered":"2024-04-25T15:27:20.023923+0000","last_clean":"2024-04-25T15:27:20.023923+0000","last_became_active":"2024-04-25T15:27:19.037565+0000","last_became_peered":"2024-04-25T15:27:19.037565+0000","last_unstale":"2024-04-25T15:27:20.023923+0000","last_undegraded":"2024-04-25T15:27:20.023923+0000","last_fullsized":"2024-04-25T15:27:20.023923+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:38.087471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023443+0000","last_change":"2024-04-25T15:27:19.028130+0000","last_active":"2024-04-25T15:27:20.023443+0000","last_peered":"2024-04-25T15:27:20.023443+0000","last_clean":"2024-04-25T15:27:20.023443+0000","last_became_active":"2024-04-25T15:27:19.027818+0000","last_became_peered":"2024-04-25T15:27:19.027818+0000","last_unstale":"2024-04-25T15:27:20.023443+0000","last_undegraded":"2024-04-25T15:27:20.023443+0000","last_fullsized":"2024-04-25T15:27:20.023443+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:09.754561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083027+0000","last_change":"2024-04-25T15:27:19.030503+0000","last_active":"2024-04-25T15:27:20.083027+0000","last_peered":"2024-04-25T15:27:20.083027+0000","last_clean":"2024-04-25T15:27:20.083027+0000","last_became_active":"2024-04-25T15:27:19.030160+0000","last_became_peered":"2024-04-25T15:27:19.030160+0000","last_unstale":"2024-04-25T15:27:20.083027+0000","last_undegraded":"2024-04-25T15:27:20.083027+0000","last_fullsized":"2024-04-25T15:27:20.083027+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:06.805733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083501+0000","last_change":"2024-04-25T15:27:19.031522+0000","last_active":"2024-04-25T15:27:20.083501+0000","last_peered":"2024-04-25T15:27:20.083501+0000","last_clean":"2024-04-25T15:27:20.083501+0000","last_became_active":"2024-04-25T15:27:19.031161+0000","last_became_peered":"2024-04-25T15:27:19.031161+0000","last_unstale":"2024-04-25T15:27:20.083501+0000","last_undegraded":"2024-04-25T15:27:20.083501+0000","last_fullsized":"2024-04-25T15:27:20.083501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:36.346379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082809+0000","last_change":"2024-04-25T15:27:19.027966+0000","last_active":"2024-04-25T15:27:20.082809+0000","last_peered":"2024-04-25T15:27:20.082809+0000","last_clean":"2024-04-25T15:27:20.082809+0000","last_became_active":"2024-04-25T15:27:19.027626+0000","last_became_peered":"2024-04-25T15:27:19.027626+0000","last_unstale":"2024-04-25T15:27:20.082809+0000","last_undegraded":"2024-04-25T15:27:20.082809+0000","last_fullsized":"2024-04-25T15:27:20.082809+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:39.565081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083235+0000","last_change":"2024-04-25T15:27:19.021908+0000","last_active":"2024-04-25T15:27:20.083235+0000","last_peered":"2024-04-25T15:27:20.083235+0000","last_clean":"2024-04-25T15:27:20.083235+0000","last_became_active":"2024-04-25T15:27:19.021582+0000","last_became_peered":"2024-04-25T15:27:19.021582+0000","last_unstale":"2024-04-25T15:27:20.083235+0000","last_undegraded":"2024-04-25T15:27:20.083235+0000","last_fullsized":"2024-04-25T15:27:20.083235+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:24.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083632+0000","last_change":"2024-04-25T15:27:19.032408+0000","last_active":"2024-04-25T15:27:20.083632+0000","last_peered":"2024-04-25T15:27:20.083632+0000","last_clean":"2024-04-25T15:27:20.083632+0000","last_became_active":"2024-04-25T15:27:19.032095+0000","last_became_peered":"2024-04-25T15:27:19.032095+0000","last_unstale":"2024-04-25T15:27:20.083632+0000","last_undegraded":"2024-04-25T15:27:20.083632+0000","last_fullsized":"2024-04-25T15:27:20.083632+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.352770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083184+0000","last_change":"2024-04-25T15:27:19.032672+0000","last_active":"2024-04-25T15:27:20.083184+0000","last_peered":"2024-04-25T15:27:20.083184+0000","last_clean":"2024-04-25T15:27:20.083184+0000","last_became_active":"2024-04-25T15:27:19.032402+0000","last_became_peered":"2024-04-25T15:27:19.032402+0000","last_unstale":"2024-04-25T15:27:20.083184+0000","last_undegraded":"2024-04-25T15:27:20.083184+0000","last_fullsized":"2024-04-25T15:27:20.083184+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:02.680672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083985+0000","last_change":"2024-04-25T15:27:19.022381+0000","last_active":"2024-04-25T15:27:20.083985+0000","last_peered":"2024-04-25T15:27:20.083985+0000","last_clean":"2024-04-25T15:27:20.083985+0000","last_became_active":"2024-04-25T15:27:19.020844+0000","last_became_peered":"2024-04-25T15:27:19.020844+0000","last_unstale":"2024-04-25T15:27:20.083985+0000","last_undegraded":"2024-04-25T15:27:20.083985+0000","last_fullsized":"2024-04-25T15:27:20.083985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:25:21.742614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030036+0000","last_change":"2024-04-25T15:27:19.032317+0000","last_active":"2024-04-25T15:27:20.030036+0000","last_peered":"2024-04-25T15:27:20.030036+0000","last_clean":"2024-04-25T15:27:20.030036+0000","last_became_active":"2024-04-25T15:27:19.031999+0000","last_became_peered":"2024-04-25T15:27:19.031999+0000","last_unstale":"2024-04-25T15:27:20.030036+0000","last_undegraded":"2024-04-25T15:27:20.030036+0000","last_fullsized":"2024-04-25T15:27:20.030036+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:51.395116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083171+0000","last_change":"2024-04-25T15:27:19.029264+0000","last_active":"2024-04-25T15:27:20.083171+0000","last_peered":"2024-04-25T15:27:20.083171+0000","last_clean":"2024-04-25T15:27:20.083171+0000","last_became_active":"2024-04-25T15:27:19.028366+0000","last_became_peered":"2024-04-25T15:27:19.028366+0000","last_unstale":"2024-04-25T15:27:20.083171+0000","last_undegraded":"2024-04-25T15:27:20.083171+0000","last_fullsized":"2024-04-25T15:27:20.083171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:54.997889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083846+0000","last_change":"2024-04-25T15:27:19.033075+0000","last_active":"2024-04-25T15:27:20.083846+0000","last_peered":"2024-04-25T15:27:20.083846+0000","last_clean":"2024-04-25T15:27:20.083846+0000","last_became_active":"2024-04-25T15:27:19.032699+0000","last_became_peered":"2024-04-25T15:27:19.032699+0000","last_unstale":"2024-04-25T15:27:20.083846+0000","last_undegraded":"2024-04-25T15:27:20.083846+0000","last_fullsized":"2024-04-25T15:27:20.083846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:09.754561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023732+0000","last_change":"2024-04-25T15:27:19.029775+0000","last_active":"2024-04-25T15:27:20.023732+0000","last_peered":"2024-04-25T15:27:20.023732+0000","last_clean":"2024-04-25T15:27:20.023732+0000","last_became_active":"2024-04-25T15:27:19.029366+0000","last_became_peered":"2024-04-25T15:27:19.029366+0000","last_unstale":"2024-04-25T15:27:20.023732+0000","last_undegraded":"2024-04-25T15:27:20.023732+0000","last_fullsized":"2024-04-25T15:27:20.023732+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:25.758868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082650+0000","last_change":"2024-04-25T15:27:19.031695+0000","last_active":"2024-04-25T15:27:20.082650+0000","last_peered":"2024-04-25T15:27:20.082650+0000","last_clean":"2024-04-25T15:27:20.082650+0000","last_became_active":"2024-04-25T15:27:19.031393+0000","last_became_peered":"2024-04-25T15:27:19.031393+0000","last_unstale":"2024-04-25T15:27:20.082650+0000","last_undegraded":"2024-04-25T15:27:20.082650+0000","last_fullsized":"2024-04-25T15:27:20.082650+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:03.196347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083162+0000","last_change":"2024-04-25T15:27:19.033619+0000","last_active":"2024-04-25T15:27:20.083162+0000","last_peered":"2024-04-25T15:27:20.083162+0000","last_clean":"2024-04-25T15:27:20.083162+0000","last_became_active":"2024-04-25T15:27:19.032907+0000","last_became_peered":"2024-04-25T15:27:19.032907+0000","last_unstale":"2024-04-25T15:27:20.083162+0000","last_undegraded":"2024-04-25T15:27:20.083162+0000","last_fullsized":"2024-04-25T15:27:20.083162+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:30.823727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083143+0000","last_change":"2024-04-25T15:27:19.030726+0000","last_active":"2024-04-25T15:27:20.083143+0000","last_peered":"2024-04-25T15:27:20.083143+0000","last_clean":"2024-04-25T15:27:20.083143+0000","last_became_active":"2024-04-25T15:27:19.030058+0000","last_became_peered":"2024-04-25T15:27:19.030058+0000","last_unstale":"2024-04-25T15:27:20.083143+0000","last_undegraded":"2024-04-25T15:27:20.083143+0000","last_fullsized":"2024-04-25T15:27:20.083143+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:55.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082329+0000","last_change":"2024-04-25T15:27:19.030427+0000","last_active":"2024-04-25T15:27:20.082329+0000","last_peered":"2024-04-25T15:27:20.082329+0000","last_clean":"2024-04-25T15:27:20.082329+0000","last_became_active":"2024-04-25T15:27:19.030096+0000","last_became_peered":"2024-04-25T15:27:19.030096+0000","last_unstale":"2024-04-25T15:27:20.082329+0000","last_undegraded":"2024-04-25T15:27:20.082329+0000","last_fullsized":"2024-04-25T15:27:20.082329+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:33.206079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023769+0000","last_change":"2024-04-25T15:27:19.023868+0000","last_active":"2024-04-25T15:27:20.023769+0000","last_peered":"2024-04-25T15:27:20.023769+0000","last_clean":"2024-04-25T15:27:20.023769+0000","last_became_active":"2024-04-25T15:27:19.023303+0000","last_became_peered":"2024-04-25T15:27:19.023303+0000","last_unstale":"2024-04-25T15:27:20.023769+0000","last_undegraded":"2024-04-25T15:27:20.023769+0000","last_fullsized":"2024-04-25T15:27:20.023769+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:25:21.742614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023903+0000","last_change":"2024-04-25T15:27:19.029245+0000","last_active":"2024-04-25T15:27:20.023903+0000","last_peered":"2024-04-25T15:27:20.023903+0000","last_clean":"2024-04-25T15:27:20.023903+0000","last_became_active":"2024-04-25T15:27:19.028883+0000","last_became_peered":"2024-04-25T15:27:19.028883+0000","last_unstale":"2024-04-25T15:27:20.023903+0000","last_undegraded":"2024-04-25T15:27:20.023903+0000","last_fullsized":"2024-04-25T15:27:20.023903+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:07:00.649972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024089+0000","last_change":"2024-04-25T15:27:18.003668+0000","last_active":"2024-04-25T15:27:20.024089+0000","last_peered":"2024-04-25T15:27:20.024089+0000","last_clean":"2024-04-25T15:27:20.024089+0000","last_became_active":"2024-04-25T15:27:18.003371+0000","last_became_peered":"2024-04-25T15:27:18.003371+0000","last_unstale":"2024-04-25T15:27:20.024089+0000","last_undegraded":"2024-04-25T15:27:20.024089+0000","last_fullsized":"2024-04-25T15:27:20.024089+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:23.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023205+0000","last_change":"2024-04-25T15:27:19.038107+0000","last_active":"2024-04-25T15:27:20.023205+0000","last_peered":"2024-04-25T15:27:20.023205+0000","last_clean":"2024-04-25T15:27:20.023205+0000","last_became_active":"2024-04-25T15:27:19.037700+0000","last_became_peered":"2024-04-25T15:27:19.037700+0000","last_unstale":"2024-04-25T15:27:20.023205+0000","last_undegraded":"2024-04-25T15:27:20.023205+0000","last_fullsized":"2024-04-25T15:27:20.023205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:21:37.658934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082792+0000","last_change":"2024-04-25T15:27:18.009311+0000","last_active":"2024-04-25T15:27:20.082792+0000","last_peered":"2024-04-25T15:27:20.082792+0000","last_clean":"2024-04-25T15:27:20.082792+0000","last_became_active":"2024-04-25T15:27:18.009016+0000","last_became_peered":"2024-04-25T15:27:18.009016+0000","last_unstale":"2024-04-25T15:27:20.082792+0000","last_undegraded":"2024-04-25T15:27:20.082792+0000","last_fullsized":"2024-04-25T15:27:20.082792+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:01.698169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023966+0000","last_change":"2024-04-25T15:27:19.032890+0000","last_active":"2024-04-25T15:27:20.023966+0000","last_peered":"2024-04-25T15:27:20.023966+0000","last_clean":"2024-04-25T15:27:20.023966+0000","last_became_active":"2024-04-25T15:27:19.032575+0000","last_became_peered":"2024-04-25T15:27:19.032575+0000","last_unstale":"2024-04-25T15:27:20.023966+0000","last_undegraded":"2024-04-25T15:27:20.023966+0000","last_fullsized":"2024-04-25T15:27:20.023966+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:34.746570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024058+0000","last_change":"2024-04-25T15:27:18.003793+0000","last_active":"2024-04-25T15:27:20.024058+0000","last_peered":"2024-04-25T15:27:20.024058+0000","last_clean":"2024-04-25T15:27:20.024058+0000","last_became_active":"2024-04-25T15:27:18.003499+0000","last_became_peered":"2024-04-25T15:27:18.003499+0000","last_unstale":"2024-04-25T15:27:20.024058+0000","last_undegraded":"2024-04-25T15:27:20.024058+0000","last_fullsized":"2024-04-25T15:27:20.024058+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:14.847161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023855+0000","last_change":"2024-04-25T15:27:19.032394+0000","last_active":"2024-04-25T15:27:20.023855+0000","last_peered":"2024-04-25T15:27:20.023855+0000","last_clean":"2024-04-25T15:27:20.023855+0000","last_became_active":"2024-04-25T15:27:19.032099+0000","last_became_peered":"2024-04-25T15:27:19.032099+0000","last_unstale":"2024-04-25T15:27:20.023855+0000","last_undegraded":"2024-04-25T15:27:20.023855+0000","last_fullsized":"2024-04-25T15:27:20.023855+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:15.273248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083490+0000","last_change":"2024-04-25T15:27:18.011811+0000","last_active":"2024-04-25T15:27:20.083490+0000","last_peered":"2024-04-25T15:27:20.083490+0000","last_clean":"2024-04-25T15:27:20.083490+0000","last_became_active":"2024-04-25T15:27:18.011453+0000","last_became_peered":"2024-04-25T15:27:18.011453+0000","last_unstale":"2024-04-25T15:27:20.083490+0000","last_undegraded":"2024-04-25T15:27:20.083490+0000","last_fullsized":"2024-04-25T15:27:20.083490+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:13.465091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083346+0000","last_change":"2024-04-25T15:27:19.033329+0000","last_active":"2024-04-25T15:27:20.083346+0000","last_peered":"2024-04-25T15:27:20.083346+0000","last_clean":"2024-04-25T15:27:20.083346+0000","last_became_active":"2024-04-25T15:27:19.032367+0000","last_became_peered":"2024-04-25T15:27:19.032367+0000","last_unstale":"2024-04-25T15:27:20.083346+0000","last_undegraded":"2024-04-25T15:27:20.083346+0000","last_fullsized":"2024-04-25T15:27:20.083346+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:10.301123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024084+0000","last_change":"2024-04-25T15:27:18.009861+0000","last_active":"2024-04-25T15:27:20.024084+0000","last_peered":"2024-04-25T15:27:20.024084+0000","last_clean":"2024-04-25T15:27:20.024084+0000","last_became_active":"2024-04-25T15:27:18.009355+0000","last_became_peered":"2024-04-25T15:27:18.009355+0000","last_unstale":"2024-04-25T15:27:20.024084+0000","last_undegraded":"2024-04-25T15:27:20.024084+0000","last_fullsized":"2024-04-25T15:27:20.024084+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:50.388996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082869+0000","last_change":"2024-04-25T15:27:19.022101+0000","last_active":"2024-04-25T15:27:20.082869+0000","last_peered":"2024-04-25T15:27:20.082869+0000","last_clean":"2024-04-25T15:27:20.082869+0000","last_became_active":"2024-04-25T15:27:19.021731+0000","last_became_peered":"2024-04-25T15:27:19.021731+0000","last_unstale":"2024-04-25T15:27:20.082869+0000","last_undegraded":"2024-04-25T15:27:20.082869+0000","last_fullsized":"2024-04-25T15:27:20.082869+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:24.649736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023628+0000","last_change":"2024-04-25T15:27:18.001183+0000","last_active":"2024-04-25T15:27:20.023628+0000","last_peered":"2024-04-25T15:27:20.023628+0000","last_clean":"2024-04-25T15:27:20.023628+0000","last_became_active":"2024-04-25T15:27:18.000877+0000","last_became_peered":"2024-04-25T15:27:18.000877+0000","last_unstale":"2024-04-25T15:27:20.023628+0000","last_undegraded":"2024-04-25T15:27:20.023628+0000","last_fullsized":"2024-04-25T15:27:20.023628+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:12.823849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030226+0000","last_change":"2024-04-25T15:27:19.034638+0000","last_active":"2024-04-25T15:27:20.030226+0000","last_peered":"2024-04-25T15:27:20.030226+0000","last_clean":"2024-04-25T15:27:20.030226+0000","last_became_active":"2024-04-25T15:27:19.034221+0000","last_became_peered":"2024-04-25T15:27:19.034221+0000","last_unstale":"2024-04-25T15:27:20.030226+0000","last_undegraded":"2024-04-25T15:27:20.030226+0000","last_fullsized":"2024-04-25T15:27:20.030226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:45.250349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082795+0000","last_change":"2024-04-25T15:27:18.003146+0000","last_active":"2024-04-25T15:27:20.082795+0000","last_peered":"2024-04-25T15:27:20.082795+0000","last_clean":"2024-04-25T15:27:20.082795+0000","last_became_active":"2024-04-25T15:27:18.002822+0000","last_became_peered":"2024-04-25T15:27:18.002822+0000","last_unstale":"2024-04-25T15:27:20.082795+0000","last_undegraded":"2024-04-25T15:27:20.082795+0000","last_fullsized":"2024-04-25T15:27:20.082795+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:37.081350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082862+0000","last_change":"2024-04-25T15:27:19.031440+0000","last_active":"2024-04-25T15:27:20.082862+0000","last_peered":"2024-04-25T15:27:20.082862+0000","last_clean":"2024-04-25T15:27:20.082862+0000","last_became_active":"2024-04-25T15:27:19.031125+0000","last_became_peered":"2024-04-25T15:27:19.031125+0000","last_unstale":"2024-04-25T15:27:20.082862+0000","last_undegraded":"2024-04-25T15:27:20.082862+0000","last_fullsized":"2024-04-25T15:27:20.082862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:23.499821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023695+0000","last_change":"2024-04-25T15:27:18.007776+0000","last_active":"2024-04-25T15:27:20.023695+0000","last_peered":"2024-04-25T15:27:20.023695+0000","last_clean":"2024-04-25T15:27:20.023695+0000","last_became_active":"2024-04-25T15:27:18.007441+0000","last_became_peered":"2024-04-25T15:27:18.007441+0000","last_unstale":"2024-04-25T15:27:20.023695+0000","last_undegraded":"2024-04-25T15:27:20.023695+0000","last_fullsized":"2024-04-25T15:27:20.023695+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:00.422138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024210+0000","last_change":"2024-04-25T15:27:19.038218+0000","last_active":"2024-04-25T15:27:20.024210+0000","last_peered":"2024-04-25T15:27:20.024210+0000","last_clean":"2024-04-25T15:27:20.024210+0000","last_became_active":"2024-04-25T15:27:19.037503+0000","last_became_peered":"2024-04-25T15:27:19.037503+0000","last_unstale":"2024-04-25T15:27:20.024210+0000","last_undegraded":"2024-04-25T15:27:20.024210+0000","last_fullsized":"2024-04-25T15:27:20.024210+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:19.801669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083714+0000","last_change":"2024-04-25T15:27:18.015029+0000","last_active":"2024-04-25T15:27:20.083714+0000","last_peered":"2024-04-25T15:27:20.083714+0000","last_clean":"2024-04-25T15:27:20.083714+0000","last_became_active":"2024-04-25T15:27:18.014722+0000","last_became_peered":"2024-04-25T15:27:18.014722+0000","last_unstale":"2024-04-25T15:27:20.083714+0000","last_undegraded":"2024-04-25T15:27:20.083714+0000","last_fullsized":"2024-04-25T15:27:20.083714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:13.465091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083337+0000","last_change":"2024-04-25T15:27:19.026141+0000","last_active":"2024-04-25T15:27:20.083337+0000","last_peered":"2024-04-25T15:27:20.083337+0000","last_clean":"2024-04-25T15:27:20.083337+0000","last_became_active":"2024-04-25T15:27:19.025846+0000","last_became_peered":"2024-04-25T15:27:19.025846+0000","last_unstale":"2024-04-25T15:27:20.083337+0000","last_undegraded":"2024-04-25T15:27:20.083337+0000","last_fullsized":"2024-04-25T15:27:20.083337+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:02.704289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082933+0000","last_change":"2024-04-25T15:27:18.017025+0000","last_active":"2024-04-25T15:27:20.082933+0000","last_peered":"2024-04-25T15:27:20.082933+0000","last_clean":"2024-04-25T15:27:20.082933+0000","last_became_active":"2024-04-25T15:27:18.016684+0000","last_became_peered":"2024-04-25T15:27:18.016684+0000","last_unstale":"2024-04-25T15:27:20.082933+0000","last_undegraded":"2024-04-25T15:27:20.082933+0000","last_fullsized":"2024-04-25T15:27:20.082933+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:44:37.817333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023557+0000","last_change":"2024-04-25T15:27:19.036838+0000","last_active":"2024-04-25T15:27:20.023557+0000","last_peered":"2024-04-25T15:27:20.023557+0000","last_clean":"2024-04-25T15:27:20.023557+0000","last_became_active":"2024-04-25T15:27:19.036528+0000","last_became_peered":"2024-04-25T15:27:19.036528+0000","last_unstale":"2024-04-25T15:27:20.023557+0000","last_undegraded":"2024-04-25T15:27:20.023557+0000","last_fullsized":"2024-04-25T15:27:20.023557+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:35.973311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083917+0000","last_change":"2024-04-25T15:27:18.014280+0000","last_active":"2024-04-25T15:27:20.083917+0000","last_peered":"2024-04-25T15:27:20.083917+0000","last_clean":"2024-04-25T15:27:20.083917+0000","last_became_active":"2024-04-25T15:27:18.013962+0000","last_became_peered":"2024-04-25T15:27:18.013962+0000","last_unstale":"2024-04-25T15:27:20.083917+0000","last_undegraded":"2024-04-25T15:27:20.083917+0000","last_fullsized":"2024-04-25T15:27:20.083917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:18.795548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083635+0000","last_change":"2024-04-25T15:27:19.038942+0000","last_active":"2024-04-25T15:27:20.083635+0000","last_peered":"2024-04-25T15:27:20.083635+0000","last_clean":"2024-04-25T15:27:20.083635+0000","last_became_active":"2024-04-25T15:27:19.038307+0000","last_became_peered":"2024-04-25T15:27:19.038307+0000","last_unstale":"2024-04-25T15:27:20.083635+0000","last_undegraded":"2024-04-25T15:27:20.083635+0000","last_fullsized":"2024-04-25T15:27:20.083635+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:09.754561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024327+0000","last_change":"2024-04-25T15:27:18.013232+0000","last_active":"2024-04-25T15:27:20.024327+0000","last_peered":"2024-04-25T15:27:20.024327+0000","last_clean":"2024-04-25T15:27:20.024327+0000","last_became_active":"2024-04-25T15:27:18.012829+0000","last_became_peered":"2024-04-25T15:27:18.012829+0000","last_unstale":"2024-04-25T15:27:20.024327+0000","last_undegraded":"2024-04-25T15:27:20.024327+0000","last_fullsized":"2024-04-25T15:27:20.024327+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:44.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082851+0000","last_change":"2024-04-25T15:27:19.036282+0000","last_active":"2024-04-25T15:27:20.082851+0000","last_peered":"2024-04-25T15:27:20.082851+0000","last_clean":"2024-04-25T15:27:20.082851+0000","last_became_active":"2024-04-25T15:27:19.035915+0000","last_became_peered":"2024-04-25T15:27:19.035915+0000","last_unstale":"2024-04-25T15:27:20.082851+0000","last_undegraded":"2024-04-25T15:27:20.082851+0000","last_fullsized":"2024-04-25T15:27:20.082851+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:01:38.596800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082996+0000","last_change":"2024-04-25T15:27:18.016988+0000","last_active":"2024-04-25T15:27:20.082996+0000","last_peered":"2024-04-25T15:27:20.082996+0000","last_clean":"2024-04-25T15:27:20.082996+0000","last_became_active":"2024-04-25T15:27:18.016655+0000","last_became_peered":"2024-04-25T15:27:18.016655+0000","last_unstale":"2024-04-25T15:27:20.082996+0000","last_undegraded":"2024-04-25T15:27:20.082996+0000","last_fullsized":"2024-04-25T15:27:20.082996+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:15.678763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083853+0000","last_change":"2024-04-25T15:27:19.029623+0000","last_active":"2024-04-25T15:27:20.083853+0000","last_peered":"2024-04-25T15:27:20.083853+0000","last_clean":"2024-04-25T15:27:20.083853+0000","last_became_active":"2024-04-25T15:27:19.028907+0000","last_became_peered":"2024-04-25T15:27:19.028907+0000","last_unstale":"2024-04-25T15:27:20.083853+0000","last_undegraded":"2024-04-25T15:27:20.083853+0000","last_fullsized":"2024-04-25T15:27:20.083853+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:03.756237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030039+0000","last_change":"2024-04-25T15:27:18.010566+0000","last_active":"2024-04-25T15:27:20.030039+0000","last_peered":"2024-04-25T15:27:20.030039+0000","last_clean":"2024-04-25T15:27:20.030039+0000","last_became_active":"2024-04-25T15:27:18.010256+0000","last_became_peered":"2024-04-25T15:27:18.010256+0000","last_unstale":"2024-04-25T15:27:20.030039+0000","last_undegraded":"2024-04-25T15:27:20.030039+0000","last_fullsized":"2024-04-25T15:27:20.030039+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:41.088283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023714+0000","last_change":"2024-04-25T15:27:19.028681+0000","last_active":"2024-04-25T15:27:20.023714+0000","last_peered":"2024-04-25T15:27:20.023714+0000","last_clean":"2024-04-25T15:27:20.023714+0000","last_became_active":"2024-04-25T15:27:19.028359+0000","last_became_peered":"2024-04-25T15:27:19.028359+0000","last_unstale":"2024-04-25T15:27:20.023714+0000","last_undegraded":"2024-04-25T15:27:20.023714+0000","last_fullsized":"2024-04-25T15:27:20.023714+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:24.132206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082909+0000","last_change":"2024-04-25T15:27:18.012486+0000","last_active":"2024-04-25T15:27:20.082909+0000","last_peered":"2024-04-25T15:27:20.082909+0000","last_clean":"2024-04-25T15:27:20.082909+0000","last_became_active":"2024-04-25T15:27:18.012119+0000","last_became_peered":"2024-04-25T15:27:18.012119+0000","last_unstale":"2024-04-25T15:27:20.082909+0000","last_undegraded":"2024-04-25T15:27:20.082909+0000","last_fullsized":"2024-04-25T15:27:20.082909+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:44.704232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083909+0000","last_change":"2024-04-25T15:27:19.023008+0000","last_active":"2024-04-25T15:27:20.083909+0000","last_peered":"2024-04-25T15:27:20.083909+0000","last_clean":"2024-04-25T15:27:20.083909+0000","last_became_active":"2024-04-25T15:27:19.022711+0000","last_became_peered":"2024-04-25T15:27:19.022711+0000","last_unstale":"2024-04-25T15:27:20.083909+0000","last_undegraded":"2024-04-25T15:27:20.083909+0000","last_fullsized":"2024-04-25T15:27:20.083909+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:25.758868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082999+0000","last_change":"2024-04-25T15:27:18.011783+0000","last_active":"2024-04-25T15:27:20.082999+0000","last_peered":"2024-04-25T15:27:20.082999+0000","last_clean":"2024-04-25T15:27:20.082999+0000","last_became_active":"2024-04-25T15:27:18.011453+0000","last_became_peered":"2024-04-25T15:27:18.011453+0000","last_unstale":"2024-04-25T15:27:20.082999+0000","last_undegraded":"2024-04-25T15:27:20.082999+0000","last_fullsized":"2024-04-25T15:27:20.082999+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:44.704232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083818+0000","last_change":"2024-04-25T15:27:19.030175+0000","last_active":"2024-04-25T15:27:20.083818+0000","last_peered":"2024-04-25T15:27:20.083818+0000","last_clean":"2024-04-25T15:27:20.083818+0000","last_became_active":"2024-04-25T15:27:19.029863+0000","last_became_peered":"2024-04-25T15:27:19.029863+0000","last_unstale":"2024-04-25T15:27:20.083818+0000","last_undegraded":"2024-04-25T15:27:20.083818+0000","last_fullsized":"2024-04-25T15:27:20.083818+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:06.086869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083209+0000","last_change":"2024-04-25T15:27:18.014965+0000","last_active":"2024-04-25T15:27:20.083209+0000","last_peered":"2024-04-25T15:27:20.083209+0000","last_clean":"2024-04-25T15:27:20.083209+0000","last_became_active":"2024-04-25T15:27:18.014656+0000","last_became_peered":"2024-04-25T15:27:18.014656+0000","last_unstale":"2024-04-25T15:27:20.083209+0000","last_undegraded":"2024-04-25T15:27:20.083209+0000","last_fullsized":"2024-04-25T15:27:20.083209+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:33.346649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083161+0000","last_change":"2024-04-25T15:27:19.036421+0000","last_active":"2024-04-25T15:27:20.083161+0000","last_peered":"2024-04-25T15:27:20.083161+0000","last_clean":"2024-04-25T15:27:20.083161+0000","last_became_active":"2024-04-25T15:27:19.036115+0000","last_became_peered":"2024-04-25T15:27:19.036115+0000","last_unstale":"2024-04-25T15:27:20.083161+0000","last_undegraded":"2024-04-25T15:27:20.083161+0000","last_fullsized":"2024-04-25T15:27:20.083161+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:16.684883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023668+0000","last_change":"2024-04-25T15:27:17.999592+0000","last_active":"2024-04-25T15:27:20.023668+0000","last_peered":"2024-04-25T15:27:20.023668+0000","last_clean":"2024-04-25T15:27:20.023668+0000","last_became_active":"2024-04-25T15:27:17.999276+0000","last_became_peered":"2024-04-25T15:27:17.999276+0000","last_unstale":"2024-04-25T15:27:20.023668+0000","last_undegraded":"2024-04-25T15:27:20.023668+0000","last_fullsized":"2024-04-25T15:27:20.023668+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:56.712684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083405+0000","last_change":"2024-04-25T15:27:19.030191+0000","last_active":"2024-04-25T15:27:20.083405+0000","last_peered":"2024-04-25T15:27:20.083405+0000","last_clean":"2024-04-25T15:27:20.083405+0000","last_became_active":"2024-04-25T15:27:19.029873+0000","last_became_peered":"2024-04-25T15:27:19.029873+0000","last_unstale":"2024-04-25T15:27:20.083405+0000","last_undegraded":"2024-04-25T15:27:20.083405+0000","last_fullsized":"2024-04-25T15:27:20.083405+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:35:23.290538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083365+0000","last_change":"2024-04-25T15:27:18.008455+0000","last_active":"2024-04-25T15:27:20.083365+0000","last_peered":"2024-04-25T15:27:20.083365+0000","last_clean":"2024-04-25T15:27:20.083365+0000","last_became_active":"2024-04-25T15:27:18.008154+0000","last_became_peered":"2024-04-25T15:27:18.008154+0000","last_unstale":"2024-04-25T15:27:20.083365+0000","last_undegraded":"2024-04-25T15:27:20.083365+0000","last_fullsized":"2024-04-25T15:27:20.083365+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:24:48.152920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083888+0000","last_change":"2024-04-25T15:27:19.039608+0000","last_active":"2024-04-25T15:27:20.083888+0000","last_peered":"2024-04-25T15:27:20.083888+0000","last_clean":"2024-04-25T15:27:20.083888+0000","last_became_active":"2024-04-25T15:27:19.039214+0000","last_became_peered":"2024-04-25T15:27:19.039214+0000","last_unstale":"2024-04-25T15:27:20.083888+0000","last_undegraded":"2024-04-25T15:27:20.083888+0000","last_fullsized":"2024-04-25T15:27:20.083888+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:40:57.009304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023784+0000","last_change":"2024-04-25T15:27:18.011867+0000","last_active":"2024-04-25T15:27:20.023784+0000","last_peered":"2024-04-25T15:27:20.023784+0000","last_clean":"2024-04-25T15:27:20.023784+0000","last_became_active":"2024-04-25T15:27:18.011542+0000","last_became_peered":"2024-04-25T15:27:18.011542+0000","last_unstale":"2024-04-25T15:27:20.023784+0000","last_undegraded":"2024-04-25T15:27:20.023784+0000","last_fullsized":"2024-04-25T15:27:20.023784+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:40:17.915392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083502+0000","last_change":"2024-04-25T15:27:19.035715+0000","last_active":"2024-04-25T15:27:20.083502+0000","last_peered":"2024-04-25T15:27:20.083502+0000","last_clean":"2024-04-25T15:27:20.083502+0000","last_became_active":"2024-04-25T15:27:19.035398+0000","last_became_peered":"2024-04-25T15:27:19.035398+0000","last_unstale":"2024-04-25T15:27:20.083502+0000","last_undegraded":"2024-04-25T15:27:20.083502+0000","last_fullsized":"2024-04-25T15:27:20.083502+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:44:38.823454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083271+0000","last_change":"2024-04-25T15:27:18.009083+0000","last_active":"2024-04-25T15:27:20.083271+0000","last_peered":"2024-04-25T15:27:20.083271+0000","last_clean":"2024-04-25T15:27:20.083271+0000","last_became_active":"2024-04-25T15:27:18.007368+0000","last_became_peered":"2024-04-25T15:27:18.007368+0000","last_unstale":"2024-04-25T15:27:20.083271+0000","last_undegraded":"2024-04-25T15:27:20.083271+0000","last_fullsized":"2024-04-25T15:27:20.083271+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:54.376029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023570+0000","last_change":"2024-04-25T15:27:19.029061+0000","last_active":"2024-04-25T15:27:20.023570+0000","last_peered":"2024-04-25T15:27:20.023570+0000","last_clean":"2024-04-25T15:27:20.023570+0000","last_became_active":"2024-04-25T15:27:19.028620+0000","last_became_peered":"2024-04-25T15:27:19.028620+0000","last_unstale":"2024-04-25T15:27:20.023570+0000","last_undegraded":"2024-04-25T15:27:20.023570+0000","last_fullsized":"2024-04-25T15:27:20.023570+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:03.756237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.022990+0000","last_change":"2024-04-25T15:27:18.001334+0000","last_active":"2024-04-25T15:27:20.022990+0000","last_peered":"2024-04-25T15:27:20.022990+0000","last_clean":"2024-04-25T15:27:20.022990+0000","last_became_active":"2024-04-25T15:27:18.000809+0000","last_became_peered":"2024-04-25T15:27:18.000809+0000","last_unstale":"2024-04-25T15:27:20.022990+0000","last_undegraded":"2024-04-25T15:27:20.022990+0000","last_fullsized":"2024-04-25T15:27:20.022990+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:23.803156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023616+0000","last_change":"2024-04-25T15:27:19.024839+0000","last_active":"2024-04-25T15:27:20.023616+0000","last_peered":"2024-04-25T15:27:20.023616+0000","last_clean":"2024-04-25T15:27:20.023616+0000","last_became_active":"2024-04-25T15:27:19.024503+0000","last_became_peered":"2024-04-25T15:27:19.024503+0000","last_unstale":"2024-04-25T15:27:20.023616+0000","last_undegraded":"2024-04-25T15:27:20.023616+0000","last_fullsized":"2024-04-25T15:27:20.023616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:40:57.009304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083374+0000","last_change":"2024-04-25T15:27:18.008733+0000","last_active":"2024-04-25T15:27:20.083374+0000","last_peered":"2024-04-25T15:27:20.083374+0000","last_clean":"2024-04-25T15:27:20.083374+0000","last_became_active":"2024-04-25T15:27:18.008455+0000","last_became_peered":"2024-04-25T15:27:18.008455+0000","last_unstale":"2024-04-25T15:27:20.083374+0000","last_undegraded":"2024-04-25T15:27:20.083374+0000","last_fullsized":"2024-04-25T15:27:20.083374+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:36.454156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083846+0000","last_change":"2024-04-25T15:27:19.032233+0000","last_active":"2024-04-25T15:27:20.083846+0000","last_peered":"2024-04-25T15:27:20.083846+0000","last_clean":"2024-04-25T15:27:20.083846+0000","last_became_active":"2024-04-25T15:27:19.031928+0000","last_became_peered":"2024-04-25T15:27:19.031928+0000","last_unstale":"2024-04-25T15:27:20.083846+0000","last_undegraded":"2024-04-25T15:27:20.083846+0000","last_fullsized":"2024-04-25T15:27:20.083846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.352770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024004+0000","last_change":"2024-04-25T15:27:17.999872+0000","last_active":"2024-04-25T15:27:20.024004+0000","last_peered":"2024-04-25T15:27:20.024004+0000","last_clean":"2024-04-25T15:27:20.024004+0000","last_became_active":"2024-04-25T15:27:17.999165+0000","last_became_peered":"2024-04-25T15:27:17.999165+0000","last_unstale":"2024-04-25T15:27:20.024004+0000","last_undegraded":"2024-04-25T15:27:20.024004+0000","last_fullsized":"2024-04-25T15:27:20.024004+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:41.088283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083324+0000","last_change":"2024-04-25T15:27:19.041103+0000","last_active":"2024-04-25T15:27:20.083324+0000","last_peered":"2024-04-25T15:27:20.083324+0000","last_clean":"2024-04-25T15:27:20.083324+0000","last_became_active":"2024-04-25T15:27:19.040763+0000","last_became_peered":"2024-04-25T15:27:19.040763+0000","last_unstale":"2024-04-25T15:27:20.083324+0000","last_undegraded":"2024-04-25T15:27:20.083324+0000","last_fullsized":"2024-04-25T15:27:20.083324+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:02.704289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.027342+0000","last_change":"2024-04-25T15:27:18.003156+0000","last_active":"2024-04-25T15:27:20.027342+0000","last_peered":"2024-04-25T15:27:20.027342+0000","last_clean":"2024-04-25T15:27:20.027342+0000","last_became_active":"2024-04-25T15:27:18.002809+0000","last_became_peered":"2024-04-25T15:27:18.002809+0000","last_unstale":"2024-04-25T15:27:20.027342+0000","last_undegraded":"2024-04-25T15:27:20.027342+0000","last_fullsized":"2024-04-25T15:27:20.027342+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:26.252582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024318+0000","last_change":"2024-04-25T15:27:19.036019+0000","last_active":"2024-04-25T15:27:20.024318+0000","last_peered":"2024-04-25T15:27:20.024318+0000","last_clean":"2024-04-25T15:27:20.024318+0000","last_became_active":"2024-04-25T15:27:19.035708+0000","last_became_peered":"2024-04-25T15:27:19.035708+0000","last_unstale":"2024-04-25T15:27:20.024318+0000","last_undegraded":"2024-04-25T15:27:20.024318+0000","last_fullsized":"2024-04-25T15:27:20.024318+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:23:36.250857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030174+0000","last_change":"2024-04-25T15:27:18.005250+0000","last_active":"2024-04-25T15:27:20.030174+0000","last_peered":"2024-04-25T15:27:20.030174+0000","last_clean":"2024-04-25T15:27:20.030174+0000","last_became_active":"2024-04-25T15:27:18.004919+0000","last_became_peered":"2024-04-25T15:27:18.004919+0000","last_unstale":"2024-04-25T15:27:20.030174+0000","last_undegraded":"2024-04-25T15:27:20.030174+0000","last_fullsized":"2024-04-25T15:27:20.030174+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:14.898632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023949+0000","last_change":"2024-04-25T15:27:19.031050+0000","last_active":"2024-04-25T15:27:20.023949+0000","last_peered":"2024-04-25T15:27:20.023949+0000","last_clean":"2024-04-25T15:27:20.023949+0000","last_became_active":"2024-04-25T15:27:19.030719+0000","last_became_peered":"2024-04-25T15:27:19.030719+0000","last_unstale":"2024-04-25T15:27:20.023949+0000","last_undegraded":"2024-04-25T15:27:20.023949+0000","last_fullsized":"2024-04-25T15:27:20.023949+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:27.887865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083606+0000","last_change":"2024-04-25T15:27:18.004668+0000","last_active":"2024-04-25T15:27:20.083606+0000","last_peered":"2024-04-25T15:27:20.083606+0000","last_clean":"2024-04-25T15:27:20.083606+0000","last_became_active":"2024-04-25T15:27:18.004303+0000","last_became_peered":"2024-04-25T15:27:18.004303+0000","last_unstale":"2024-04-25T15:27:20.083606+0000","last_undegraded":"2024-04-25T15:27:20.083606+0000","last_fullsized":"2024-04-25T15:27:20.083606+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:18.795548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083506+0000","last_change":"2024-04-25T15:27:19.027620+0000","last_active":"2024-04-25T15:27:20.083506+0000","last_peered":"2024-04-25T15:27:20.083506+0000","last_clean":"2024-04-25T15:27:20.083506+0000","last_became_active":"2024-04-25T15:27:19.027314+0000","last_became_peered":"2024-04-25T15:27:19.027314+0000","last_unstale":"2024-04-25T15:27:20.083506+0000","last_undegraded":"2024-04-25T15:27:20.083506+0000","last_fullsized":"2024-04-25T15:27:20.083506+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:15:23.345125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.022922+0000","last_change":"2024-04-25T15:27:18.003435+0000","last_active":"2024-04-25T15:27:20.022922+0000","last_peered":"2024-04-25T15:27:20.022922+0000","last_clean":"2024-04-25T15:27:20.022922+0000","last_became_active":"2024-04-25T15:27:18.003161+0000","last_became_peered":"2024-04-25T15:27:18.003161+0000","last_unstale":"2024-04-25T15:27:20.022922+0000","last_undegraded":"2024-04-25T15:27:20.022922+0000","last_fullsized":"2024-04-25T15:27:20.022922+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:41.715761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.084016+0000","last_change":"2024-04-25T15:27:19.028529+0000","last_active":"2024-04-25T15:27:20.084016+0000","last_peered":"2024-04-25T15:27:20.084016+0000","last_clean":"2024-04-25T15:27:20.084016+0000","last_became_active":"2024-04-25T15:27:19.028173+0000","last_became_peered":"2024-04-25T15:27:19.028173+0000","last_unstale":"2024-04-25T15:27:20.084016+0000","last_undegraded":"2024-04-25T15:27:20.084016+0000","last_fullsized":"2024-04-25T15:27:20.084016+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:15.273248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024172+0000","last_change":"2024-04-25T15:27:18.013156+0000","last_active":"2024-04-25T15:27:20.024172+0000","last_peered":"2024-04-25T15:27:20.024172+0000","last_clean":"2024-04-25T15:27:20.024172+0000","last_became_active":"2024-04-25T15:27:18.012767+0000","last_became_peered":"2024-04-25T15:27:18.012767+0000","last_unstale":"2024-04-25T15:27:20.024172+0000","last_undegraded":"2024-04-25T15:27:20.024172+0000","last_fullsized":"2024-04-25T15:27:20.024172+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:19:48.629294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.022846+0000","last_change":"2024-04-25T15:27:19.034499+0000","last_active":"2024-04-25T15:27:20.022846+0000","last_peered":"2024-04-25T15:27:20.022846+0000","last_clean":"2024-04-25T15:27:20.022846+0000","last_became_active":"2024-04-25T15:27:19.034148+0000","last_became_peered":"2024-04-25T15:27:19.034148+0000","last_unstale":"2024-04-25T15:27:20.022846+0000","last_undegraded":"2024-04-25T15:27:20.022846+0000","last_fullsized":"2024-04-25T15:27:20.022846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:05.241698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024205+0000","last_change":"2024-04-25T15:27:18.003984+0000","last_active":"2024-04-25T15:27:20.024205+0000","last_peered":"2024-04-25T15:27:20.024205+0000","last_clean":"2024-04-25T15:27:20.024205+0000","last_became_active":"2024-04-25T15:27:18.003671+0000","last_became_peered":"2024-04-25T15:27:18.003671+0000","last_unstale":"2024-04-25T15:27:20.024205+0000","last_undegraded":"2024-04-25T15:27:20.024205+0000","last_fullsized":"2024-04-25T15:27:20.024205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:14.898632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083058+0000","last_change":"2024-04-25T15:27:19.031518+0000","last_active":"2024-04-25T15:27:20.083058+0000","last_peered":"2024-04-25T15:27:20.083058+0000","last_clean":"2024-04-25T15:27:20.083058+0000","last_became_active":"2024-04-25T15:27:19.031195+0000","last_became_peered":"2024-04-25T15:27:19.031195+0000","last_unstale":"2024-04-25T15:27:20.083058+0000","last_undegraded":"2024-04-25T15:27:20.083058+0000","last_fullsized":"2024-04-25T15:27:20.083058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:32.771569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082985+0000","last_change":"2024-04-25T15:27:18.011567+0000","last_active":"2024-04-25T15:27:20.082985+0000","last_peered":"2024-04-25T15:27:20.082985+0000","last_clean":"2024-04-25T15:27:20.082985+0000","last_became_active":"2024-04-25T15:27:18.008482+0000","last_became_peered":"2024-04-25T15:27:18.008482+0000","last_unstale":"2024-04-25T15:27:20.082985+0000","last_undegraded":"2024-04-25T15:27:20.082985+0000","last_fullsized":"2024-04-25T15:27:20.082985+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:35.340259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082772+0000","last_change":"2024-04-25T15:27:19.020034+0000","last_active":"2024-04-25T15:27:20.082772+0000","last_peered":"2024-04-25T15:27:20.082772+0000","last_clean":"2024-04-25T15:27:20.082772+0000","last_became_active":"2024-04-25T15:27:19.019692+0000","last_became_peered":"2024-04-25T15:27:19.019692+0000","last_unstale":"2024-04-25T15:27:20.082772+0000","last_undegraded":"2024-04-25T15:27:20.082772+0000","last_fullsized":"2024-04-25T15:27:20.082772+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:26.783369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030048+0000","last_change":"2024-04-25T15:27:18.004051+0000","last_active":"2024-04-25T15:27:20.030048+0000","last_peered":"2024-04-25T15:27:20.030048+0000","last_clean":"2024-04-25T15:27:20.030048+0000","last_became_active":"2024-04-25T15:27:18.003680+0000","last_became_peered":"2024-04-25T15:27:18.003680+0000","last_unstale":"2024-04-25T15:27:20.030048+0000","last_undegraded":"2024-04-25T15:27:20.030048+0000","last_fullsized":"2024-04-25T15:27:20.030048+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:23.803156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":175,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083931+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:27:20.083931+0000","last_peered":"2024-04-25T15:27:20.083931+0000","last_clean":"2024-04-25T15:27:20.083931+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:27:20.083931+0000","last_undegraded":"2024-04-25T15:27:20.083931+0000","last_fullsized":"2024-04-25T15:27:20.083931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":868843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":671744,"data_stored":278315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":278967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":279927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":679936,"data_stored":278371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":679936,"data_stored":278371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":675840,"data_stored":278405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28272,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343568,"statfs":{"total":96636764160,"available":96607813632,"internally_reserved":0,"allocated":1282048,"data_stored":870877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:27:23.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:27:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.233+0000 7f2d09340640 1 Processor -- start 2024-04-25T15:27:23.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.233+0000 7f2d09340640 1 -- start start 2024-04-25T15:27:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.234+0000 7f2d09340640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2d040b6a00 0x7f2d0415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d09340640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d0406d560 0x7f2d040b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d09340640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d041611e0 con 0x7f2d0415a2f0 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d09340640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2d04160d50 con 0x7f2d0406d560 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d09340640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0415a700 con 0x7f2d040b6a00 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d037fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d0406d560 0x7f2d040b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d037fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d0406d560 0x7f2d040b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37510/0 (socket says 172.21.15.67:37510) 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d037fe640 1 -- 172.21.15.67:0/954873947 learned_addr learned my addr 172.21.15.67:0/954873947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d02ffd640 1 --2- 172.21.15.67:0/954873947 >> v2:172.21.15.176:3300/0 conn(0x7f2d040b6a00 0x7f2d0415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d037fe640 1 -- 172.21.15.67:0/954873947 >> v2:172.21.15.176:3300/0 conn(0x7f2d040b6a00 msgr2=0x7f2d0415d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d037fe640 1 --2- 172.21.15.67:0/954873947 >> v2:172.21.15.176:3300/0 conn(0x7f2d040b6a00 0x7f2d0415d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d037fe640 1 -- 172.21.15.67:0/954873947 >> v1:172.21.15.176:6789/0 conn(0x7f2d0415a2f0 legacy=0x7f2d0406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.235+0000 7f2d037fe640 1 -- 172.21.15.67:0/954873947 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d04160420 con 0x7f2d0406d560 2024-04-25T15:27:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.236+0000 7f2d037fe640 1 --2- 172.21.15.67:0/954873947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0406d560 0x7f2d040b4970 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2d040713e0 tx=0x7f2cf402c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77728684a1f27291 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.236+0000 7f2d01ffb640 1 -- 172.21.15.67:0/954873947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4033610 con 0x7f2d0406d560 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.236+0000 7f2d01ffb640 1 -- 172.21.15.67:0/954873947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf40337b0 con 0x7f2d0406d560 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.236+0000 7f2d01ffb640 1 -- 172.21.15.67:0/954873947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4033b80 con 0x7f2d0406d560 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 -- 172.21.15.67:0/954873947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0406d560 msgr2=0x7f2d040b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 --2- 172.21.15.67:0/954873947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0406d560 0x7f2d040b4970 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2d040713e0 tx=0x7f2cf402c900 comp rx=0 tx=0).stop 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 -- 172.21.15.67:0/954873947 shutdown_connections 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 --2- 172.21.15.67:0/954873947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0406d560 0x7f2d040b4970 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 --2- 172.21.15.67:0/954873947 >> v2:172.21.15.176:3300/0 conn(0x7f2d040b6a00 0x7f2d0415d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 -- 172.21.15.67:0/954873947 >> 172.21.15.67:0/954873947 conn(0x7f2d040d1ae0 msgr2=0x7f2d040d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 -- 172.21.15.67:0/954873947 shutdown_connections 2024-04-25T15:27:23.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.237+0000 7f2d09340640 1 -- 172.21.15.67:0/954873947 wait complete. 2024-04-25T15:27:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.239+0000 7f2d09340640 1 Processor -- start 2024-04-25T15:27:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.239+0000 7f2d09340640 1 -- start start 2024-04-25T15:27:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d09340640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0406d560 0x7f2d0414d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d09340640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d040b6a00 0x7f2d041515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d09340640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0415a2f0 0x7f2d0415e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d09340640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d040ce580 con 0x7f2d040b6a00 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d09340640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d041609e0 con 0x7f2d0415a2f0 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d09340640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d040ce700 con 0x7f2d0406d560 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d040b6a00 0x7f2d041515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d040b6a00 0x7f2d041515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37512/0 (socket says 172.21.15.67:37512) 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 -- 172.21.15.67:0/1862657393 learned_addr learned my addr 172.21.15.67:0/1862657393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d02ffd640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0406d560 0x7f2d0414d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d037fe640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0415a2f0 0x7f2d0415e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 -- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0406d560 msgr2=0x7f2d0414d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0406d560 0x7f2d0414d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 -- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0415a2f0 msgr2=0x7f2d0415e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0415a2f0 0x7f2d0415e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.240+0000 7f2d027fc640 1 -- 172.21.15.67:0/1862657393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d04154fc0 con 0x7f2d040b6a00 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.241+0000 7f2d027fc640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d040b6a00 0x7f2d041515b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00b3f0 tx=0x7f2cec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.241+0000 7f2cfb7fe640 1 -- 172.21.15.67:0/1862657393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec015020 con 0x7f2d040b6a00 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.241+0000 7f2cfb7fe640 1 -- 172.21.15.67:0/1862657393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cec011830 con 0x7f2d040b6a00 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.241+0000 7f2cfb7fe640 1 -- 172.21.15.67:0/1862657393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec011c20 con 0x7f2d040b6a00 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.241+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d041552a0 con 0x7f2d040b6a00 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.241+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d04155860 con 0x7f2d040b6a00 2024-04-25T15:27:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.242+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d040705a0 con 0x7f2d040b6a00 2024-04-25T15:27:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.243+0000 7f2cfb7fe640 1 -- 172.21.15.67:0/1862657393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2cec018020 con 0x7f2d040b6a00 2024-04-25T15:27:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.243+0000 7f2cfb7fe640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ce40744b0 0x7f2ce40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.243+0000 7f2cfb7fe640 1 -- 172.21.15.67:0/1862657393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f2cec094dd0 con 0x7f2d040b6a00 2024-04-25T15:27:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.244+0000 7f2d02ffd640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ce40744b0 0x7f2ce40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.244+0000 7f2d02ffd640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ce40744b0 0x7f2ce40769c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2d04070570 tx=0x7f2cfc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.248+0000 7f2cfb7fe640 1 -- 172.21.15.67:0/1862657393 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2cec099070 con 0x7f2d040b6a00 2024-04-25T15:27:23.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.428+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2d04158830 con 0x7f2ce40744b0 2024-04-25T15:27:23.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.435+0000 7f2cfb7fe640 1 -- 172.21.15.67:0/1862657393 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7f2d04158830 con 0x7f2ce40744b0 2024-04-25T15:27:23.440 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:27:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ce40744b0 msgr2=0x7f2ce40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ce40744b0 0x7f2ce40769c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2d04070570 tx=0x7f2cfc008040 comp rx=0 tx=0).stop 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d040b6a00 msgr2=0x7f2d041515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d040b6a00 0x7f2d041515b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00b3f0 tx=0x7f2cec00b8b0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 shutdown_connections 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ce40744b0 0x7f2ce40769c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0415a2f0 0x7f2d0415e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d040b6a00 0x7f2d041515b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 --2- 172.21.15.67:0/1862657393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0406d560 0x7f2d0414d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 >> 172.21.15.67:0/1862657393 conn(0x7f2d040d1ae0 msgr2=0x7f2d04073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 shutdown_connections 2024-04-25T15:27:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.438+0000 7f2d09340640 1 -- 172.21.15.67:0/1862657393 wait complete. 2024-04-25T15:27:23.451 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-25T15:27:22.822331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222672,"kb_used_data":6472,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752048,"statfs":{"total":773094113280,"available":772866097152,"internally_reserved":0,"allocated":6627328,"data_stored":3412076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.834236"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082561+0000","last_change":"2024-04-25T15:27:19.027989+0000","last_active":"2024-04-25T15:27:20.082561+0000","last_peered":"2024-04-25T15:27:20.082561+0000","last_clean":"2024-04-25T15:27:20.082561+0000","last_became_active":"2024-04-25T15:27:19.027574+0000","last_became_peered":"2024-04-25T15:27:19.027574+0000","last_unstale":"2024-04-25T15:27:20.082561+0000","last_undegraded":"2024-04-25T15:27:20.082561+0000","last_fullsized":"2024-04-25T15:27:20.082561+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:11:19.196334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082807+0000","last_change":"2024-04-25T15:27:19.028322+0000","last_active":"2024-04-25T15:27:20.082807+0000","last_peered":"2024-04-25T15:27:20.082807+0000","last_clean":"2024-04-25T15:27:20.082807+0000","last_became_active":"2024-04-25T15:27:19.028023+0000","last_became_peered":"2024-04-25T15:27:19.028023+0000","last_unstale":"2024-04-25T15:27:20.082807+0000","last_undegraded":"2024-04-25T15:27:20.082807+0000","last_fullsized":"2024-04-25T15:27:20.082807+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:27.073882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083553+0000","last_change":"2024-04-25T15:27:19.028642+0000","last_active":"2024-04-25T15:27:20.083553+0000","last_peered":"2024-04-25T15:27:20.083553+0000","last_clean":"2024-04-25T15:27:20.083553+0000","last_became_active":"2024-04-25T15:27:19.028329+0000","last_became_peered":"2024-04-25T15:27:19.028329+0000","last_unstale":"2024-04-25T15:27:20.083553+0000","last_undegraded":"2024-04-25T15:27:20.083553+0000","last_fullsized":"2024-04-25T15:27:20.083553+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:44:38.823454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083176+0000","last_change":"2024-04-25T15:27:19.033445+0000","last_active":"2024-04-25T15:27:20.083176+0000","last_peered":"2024-04-25T15:27:20.083176+0000","last_clean":"2024-04-25T15:27:20.083176+0000","last_became_active":"2024-04-25T15:27:19.032634+0000","last_became_peered":"2024-04-25T15:27:19.032634+0000","last_unstale":"2024-04-25T15:27:20.083176+0000","last_undegraded":"2024-04-25T15:27:20.083176+0000","last_fullsized":"2024-04-25T15:27:20.083176+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:55.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082495+0000","last_change":"2024-04-25T15:27:19.031298+0000","last_active":"2024-04-25T15:27:20.082495+0000","last_peered":"2024-04-25T15:27:20.082495+0000","last_clean":"2024-04-25T15:27:20.082495+0000","last_became_active":"2024-04-25T15:27:19.030955+0000","last_became_peered":"2024-04-25T15:27:19.030955+0000","last_unstale":"2024-04-25T15:27:20.082495+0000","last_undegraded":"2024-04-25T15:27:20.082495+0000","last_fullsized":"2024-04-25T15:27:20.082495+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:06.145828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023486+0000","last_change":"2024-04-25T15:27:19.030506+0000","last_active":"2024-04-25T15:27:20.023486+0000","last_peered":"2024-04-25T15:27:20.023486+0000","last_clean":"2024-04-25T15:27:20.023486+0000","last_became_active":"2024-04-25T15:27:19.030180+0000","last_became_peered":"2024-04-25T15:27:19.030180+0000","last_unstale":"2024-04-25T15:27:20.023486+0000","last_undegraded":"2024-04-25T15:27:20.023486+0000","last_fullsized":"2024-04-25T15:27:20.023486+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:06.086869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023794+0000","last_change":"2024-04-25T15:27:19.032135+0000","last_active":"2024-04-25T15:27:20.023794+0000","last_peered":"2024-04-25T15:27:20.023794+0000","last_clean":"2024-04-25T15:27:20.023794+0000","last_became_active":"2024-04-25T15:27:19.031822+0000","last_became_peered":"2024-04-25T15:27:19.031822+0000","last_unstale":"2024-04-25T15:27:20.023794+0000","last_undegraded":"2024-04-25T15:27:20.023794+0000","last_fullsized":"2024-04-25T15:27:20.023794+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:37.460277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082591+0000","last_change":"2024-04-25T15:27:19.032917+0000","last_active":"2024-04-25T15:27:20.082591+0000","last_peered":"2024-04-25T15:27:20.082591+0000","last_clean":"2024-04-25T15:27:20.082591+0000","last_became_active":"2024-04-25T15:27:19.030486+0000","last_became_peered":"2024-04-25T15:27:19.030486+0000","last_unstale":"2024-04-25T15:27:20.082591+0000","last_undegraded":"2024-04-25T15:27:20.082591+0000","last_fullsized":"2024-04-25T15:27:20.082591+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:22.565243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024122+0000","last_change":"2024-04-25T15:27:19.029718+0000","last_active":"2024-04-25T15:27:20.024122+0000","last_peered":"2024-04-25T15:27:20.024122+0000","last_clean":"2024-04-25T15:27:20.024122+0000","last_became_active":"2024-04-25T15:27:19.029334+0000","last_became_peered":"2024-04-25T15:27:19.029334+0000","last_unstale":"2024-04-25T15:27:20.024122+0000","last_undegraded":"2024-04-25T15:27:20.024122+0000","last_fullsized":"2024-04-25T15:27:20.024122+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:45.710352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024418+0000","last_change":"2024-04-25T15:27:19.034934+0000","last_active":"2024-04-25T15:27:20.024418+0000","last_peered":"2024-04-25T15:27:20.024418+0000","last_clean":"2024-04-25T15:27:20.024418+0000","last_became_active":"2024-04-25T15:27:19.034556+0000","last_became_peered":"2024-04-25T15:27:19.034556+0000","last_unstale":"2024-04-25T15:27:20.024418+0000","last_undegraded":"2024-04-25T15:27:20.024418+0000","last_fullsized":"2024-04-25T15:27:20.024418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:24.132206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024406+0000","last_change":"2024-04-25T15:27:19.026266+0000","last_active":"2024-04-25T15:27:20.024406+0000","last_peered":"2024-04-25T15:27:20.024406+0000","last_clean":"2024-04-25T15:27:20.024406+0000","last_became_active":"2024-04-25T15:27:19.025958+0000","last_became_peered":"2024-04-25T15:27:19.025958+0000","last_unstale":"2024-04-25T15:27:20.024406+0000","last_undegraded":"2024-04-25T15:27:20.024406+0000","last_fullsized":"2024-04-25T15:27:20.024406+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:52.878059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083353+0000","last_change":"2024-04-25T15:27:19.041293+0000","last_active":"2024-04-25T15:27:20.083353+0000","last_peered":"2024-04-25T15:27:20.083353+0000","last_clean":"2024-04-25T15:27:20.083353+0000","last_became_active":"2024-04-25T15:27:19.040871+0000","last_became_peered":"2024-04-25T15:27:19.040871+0000","last_unstale":"2024-04-25T15:27:20.083353+0000","last_undegraded":"2024-04-25T15:27:20.083353+0000","last_fullsized":"2024-04-25T15:27:20.083353+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:36.346379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030231+0000","last_change":"2024-04-25T15:27:19.031072+0000","last_active":"2024-04-25T15:27:20.030231+0000","last_peered":"2024-04-25T15:27:20.030231+0000","last_clean":"2024-04-25T15:27:20.030231+0000","last_became_active":"2024-04-25T15:27:19.030774+0000","last_became_peered":"2024-04-25T15:27:19.030774+0000","last_unstale":"2024-04-25T15:27:20.030231+0000","last_undegraded":"2024-04-25T15:27:20.030231+0000","last_fullsized":"2024-04-25T15:27:20.030231+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:47.390421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024125+0000","last_change":"2024-04-25T15:27:19.033832+0000","last_active":"2024-04-25T15:27:20.024125+0000","last_peered":"2024-04-25T15:27:20.024125+0000","last_clean":"2024-04-25T15:27:20.024125+0000","last_became_active":"2024-04-25T15:27:19.033527+0000","last_became_peered":"2024-04-25T15:27:19.033527+0000","last_unstale":"2024-04-25T15:27:20.024125+0000","last_undegraded":"2024-04-25T15:27:20.024125+0000","last_fullsized":"2024-04-25T15:27:20.024125+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:14.471211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023923+0000","last_change":"2024-04-25T15:27:19.037925+0000","last_active":"2024-04-25T15:27:20.023923+0000","last_peered":"2024-04-25T15:27:20.023923+0000","last_clean":"2024-04-25T15:27:20.023923+0000","last_became_active":"2024-04-25T15:27:19.037565+0000","last_became_peered":"2024-04-25T15:27:19.037565+0000","last_unstale":"2024-04-25T15:27:20.023923+0000","last_undegraded":"2024-04-25T15:27:20.023923+0000","last_fullsized":"2024-04-25T15:27:20.023923+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:38.087471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023443+0000","last_change":"2024-04-25T15:27:19.028130+0000","last_active":"2024-04-25T15:27:20.023443+0000","last_peered":"2024-04-25T15:27:20.023443+0000","last_clean":"2024-04-25T15:27:20.023443+0000","last_became_active":"2024-04-25T15:27:19.027818+0000","last_became_peered":"2024-04-25T15:27:19.027818+0000","last_unstale":"2024-04-25T15:27:20.023443+0000","last_undegraded":"2024-04-25T15:27:20.023443+0000","last_fullsized":"2024-04-25T15:27:20.023443+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:09.754561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083027+0000","last_change":"2024-04-25T15:27:19.030503+0000","last_active":"2024-04-25T15:27:20.083027+0000","last_peered":"2024-04-25T15:27:20.083027+0000","last_clean":"2024-04-25T15:27:20.083027+0000","last_became_active":"2024-04-25T15:27:19.030160+0000","last_became_peered":"2024-04-25T15:27:19.030160+0000","last_unstale":"2024-04-25T15:27:20.083027+0000","last_undegraded":"2024-04-25T15:27:20.083027+0000","last_fullsized":"2024-04-25T15:27:20.083027+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:06.805733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083501+0000","last_change":"2024-04-25T15:27:19.031522+0000","last_active":"2024-04-25T15:27:20.083501+0000","last_peered":"2024-04-25T15:27:20.083501+0000","last_clean":"2024-04-25T15:27:20.083501+0000","last_became_active":"2024-04-25T15:27:19.031161+0000","last_became_peered":"2024-04-25T15:27:19.031161+0000","last_unstale":"2024-04-25T15:27:20.083501+0000","last_undegraded":"2024-04-25T15:27:20.083501+0000","last_fullsized":"2024-04-25T15:27:20.083501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:36.346379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082809+0000","last_change":"2024-04-25T15:27:19.027966+0000","last_active":"2024-04-25T15:27:20.082809+0000","last_peered":"2024-04-25T15:27:20.082809+0000","last_clean":"2024-04-25T15:27:20.082809+0000","last_became_active":"2024-04-25T15:27:19.027626+0000","last_became_peered":"2024-04-25T15:27:19.027626+0000","last_unstale":"2024-04-25T15:27:20.082809+0000","last_undegraded":"2024-04-25T15:27:20.082809+0000","last_fullsized":"2024-04-25T15:27:20.082809+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:39.565081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083235+0000","last_change":"2024-04-25T15:27:19.021908+0000","last_active":"2024-04-25T15:27:20.083235+0000","last_peered":"2024-04-25T15:27:20.083235+0000","last_clean":"2024-04-25T15:27:20.083235+0000","last_became_active":"2024-04-25T15:27:19.021582+0000","last_became_peered":"2024-04-25T15:27:19.021582+0000","last_unstale":"2024-04-25T15:27:20.083235+0000","last_undegraded":"2024-04-25T15:27:20.083235+0000","last_fullsized":"2024-04-25T15:27:20.083235+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:24.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083632+0000","last_change":"2024-04-25T15:27:19.032408+0000","last_active":"2024-04-25T15:27:20.083632+0000","last_peered":"2024-04-25T15:27:20.083632+0000","last_clean":"2024-04-25T15:27:20.083632+0000","last_became_active":"2024-04-25T15:27:19.032095+0000","last_became_peered":"2024-04-25T15:27:19.032095+0000","last_unstale":"2024-04-25T15:27:20.083632+0000","last_undegraded":"2024-04-25T15:27:20.083632+0000","last_fullsized":"2024-04-25T15:27:20.083632+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.352770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083184+0000","last_change":"2024-04-25T15:27:19.032672+0000","last_active":"2024-04-25T15:27:20.083184+0000","last_peered":"2024-04-25T15:27:20.083184+0000","last_clean":"2024-04-25T15:27:20.083184+0000","last_became_active":"2024-04-25T15:27:19.032402+0000","last_became_peered":"2024-04-25T15:27:19.032402+0000","last_unstale":"2024-04-25T15:27:20.083184+0000","last_undegraded":"2024-04-25T15:27:20.083184+0000","last_fullsized":"2024-04-25T15:27:20.083184+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:02.680672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083985+0000","last_change":"2024-04-25T15:27:19.022381+0000","last_active":"2024-04-25T15:27:20.083985+0000","last_peered":"2024-04-25T15:27:20.083985+0000","last_clean":"2024-04-25T15:27:20.083985+0000","last_became_active":"2024-04-25T15:27:19.020844+0000","last_became_peered":"2024-04-25T15:27:19.020844+0000","last_unstale":"2024-04-25T15:27:20.083985+0000","last_undegraded":"2024-04-25T15:27:20.083985+0000","last_fullsized":"2024-04-25T15:27:20.083985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:25:21.742614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030036+0000","last_change":"2024-04-25T15:27:19.032317+0000","last_active":"2024-04-25T15:27:20.030036+0000","last_peered":"2024-04-25T15:27:20.030036+0000","last_clean":"2024-04-25T15:27:20.030036+0000","last_became_active":"2024-04-25T15:27:19.031999+0000","last_became_peered":"2024-04-25T15:27:19.031999+0000","last_unstale":"2024-04-25T15:27:20.030036+0000","last_undegraded":"2024-04-25T15:27:20.030036+0000","last_fullsized":"2024-04-25T15:27:20.030036+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:51.395116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083171+0000","last_change":"2024-04-25T15:27:19.029264+0000","last_active":"2024-04-25T15:27:20.083171+0000","last_peered":"2024-04-25T15:27:20.083171+0000","last_clean":"2024-04-25T15:27:20.083171+0000","last_became_active":"2024-04-25T15:27:19.028366+0000","last_became_peered":"2024-04-25T15:27:19.028366+0000","last_unstale":"2024-04-25T15:27:20.083171+0000","last_undegraded":"2024-04-25T15:27:20.083171+0000","last_fullsized":"2024-04-25T15:27:20.083171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:54.997889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083846+0000","last_change":"2024-04-25T15:27:19.033075+0000","last_active":"2024-04-25T15:27:20.083846+0000","last_peered":"2024-04-25T15:27:20.083846+0000","last_clean":"2024-04-25T15:27:20.083846+0000","last_became_active":"2024-04-25T15:27:19.032699+0000","last_became_peered":"2024-04-25T15:27:19.032699+0000","last_unstale":"2024-04-25T15:27:20.083846+0000","last_undegraded":"2024-04-25T15:27:20.083846+0000","last_fullsized":"2024-04-25T15:27:20.083846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:09.754561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023732+0000","last_change":"2024-04-25T15:27:19.029775+0000","last_active":"2024-04-25T15:27:20.023732+0000","last_peered":"2024-04-25T15:27:20.023732+0000","last_clean":"2024-04-25T15:27:20.023732+0000","last_became_active":"2024-04-25T15:27:19.029366+0000","last_became_peered":"2024-04-25T15:27:19.029366+0000","last_unstale":"2024-04-25T15:27:20.023732+0000","last_undegraded":"2024-04-25T15:27:20.023732+0000","last_fullsized":"2024-04-25T15:27:20.023732+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:25.758868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082650+0000","last_change":"2024-04-25T15:27:19.031695+0000","last_active":"2024-04-25T15:27:20.082650+0000","last_peered":"2024-04-25T15:27:20.082650+0000","last_clean":"2024-04-25T15:27:20.082650+0000","last_became_active":"2024-04-25T15:27:19.031393+0000","last_became_peered":"2024-04-25T15:27:19.031393+0000","last_unstale":"2024-04-25T15:27:20.082650+0000","last_undegraded":"2024-04-25T15:27:20.082650+0000","last_fullsized":"2024-04-25T15:27:20.082650+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:03.196347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083162+0000","last_change":"2024-04-25T15:27:19.033619+0000","last_active":"2024-04-25T15:27:20.083162+0000","last_peered":"2024-04-25T15:27:20.083162+0000","last_clean":"2024-04-25T15:27:20.083162+0000","last_became_active":"2024-04-25T15:27:19.032907+0000","last_became_peered":"2024-04-25T15:27:19.032907+0000","last_unstale":"2024-04-25T15:27:20.083162+0000","last_undegraded":"2024-04-25T15:27:20.083162+0000","last_fullsized":"2024-04-25T15:27:20.083162+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:30.823727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083143+0000","last_change":"2024-04-25T15:27:19.030726+0000","last_active":"2024-04-25T15:27:20.083143+0000","last_peered":"2024-04-25T15:27:20.083143+0000","last_clean":"2024-04-25T15:27:20.083143+0000","last_became_active":"2024-04-25T15:27:19.030058+0000","last_became_peered":"2024-04-25T15:27:19.030058+0000","last_unstale":"2024-04-25T15:27:20.083143+0000","last_undegraded":"2024-04-25T15:27:20.083143+0000","last_fullsized":"2024-04-25T15:27:20.083143+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:55.382150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082329+0000","last_change":"2024-04-25T15:27:19.030427+0000","last_active":"2024-04-25T15:27:20.082329+0000","last_peered":"2024-04-25T15:27:20.082329+0000","last_clean":"2024-04-25T15:27:20.082329+0000","last_became_active":"2024-04-25T15:27:19.030096+0000","last_became_peered":"2024-04-25T15:27:19.030096+0000","last_unstale":"2024-04-25T15:27:20.082329+0000","last_undegraded":"2024-04-25T15:27:20.082329+0000","last_fullsized":"2024-04-25T15:27:20.082329+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:33.206079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023769+0000","last_change":"2024-04-25T15:27:19.023868+0000","last_active":"2024-04-25T15:27:20.023769+0000","last_peered":"2024-04-25T15:27:20.023769+0000","last_clean":"2024-04-25T15:27:20.023769+0000","last_became_active":"2024-04-25T15:27:19.023303+0000","last_became_peered":"2024-04-25T15:27:19.023303+0000","last_unstale":"2024-04-25T15:27:20.023769+0000","last_undegraded":"2024-04-25T15:27:20.023769+0000","last_fullsized":"2024-04-25T15:27:20.023769+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:25:21.742614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023903+0000","last_change":"2024-04-25T15:27:19.029245+0000","last_active":"2024-04-25T15:27:20.023903+0000","last_peered":"2024-04-25T15:27:20.023903+0000","last_clean":"2024-04-25T15:27:20.023903+0000","last_became_active":"2024-04-25T15:27:19.028883+0000","last_became_peered":"2024-04-25T15:27:19.028883+0000","last_unstale":"2024-04-25T15:27:20.023903+0000","last_undegraded":"2024-04-25T15:27:20.023903+0000","last_fullsized":"2024-04-25T15:27:20.023903+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:07:00.649972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024089+0000","last_change":"2024-04-25T15:27:18.003668+0000","last_active":"2024-04-25T15:27:20.024089+0000","last_peered":"2024-04-25T15:27:20.024089+0000","last_clean":"2024-04-25T15:27:20.024089+0000","last_became_active":"2024-04-25T15:27:18.003371+0000","last_became_peered":"2024-04-25T15:27:18.003371+0000","last_unstale":"2024-04-25T15:27:20.024089+0000","last_undegraded":"2024-04-25T15:27:20.024089+0000","last_fullsized":"2024-04-25T15:27:20.024089+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:23.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023205+0000","last_change":"2024-04-25T15:27:19.038107+0000","last_active":"2024-04-25T15:27:20.023205+0000","last_peered":"2024-04-25T15:27:20.023205+0000","last_clean":"2024-04-25T15:27:20.023205+0000","last_became_active":"2024-04-25T15:27:19.037700+0000","last_became_peered":"2024-04-25T15:27:19.037700+0000","last_unstale":"2024-04-25T15:27:20.023205+0000","last_undegraded":"2024-04-25T15:27:20.023205+0000","last_fullsized":"2024-04-25T15:27:20.023205+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:21:37.658934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082792+0000","last_change":"2024-04-25T15:27:18.009311+0000","last_active":"2024-04-25T15:27:20.082792+0000","last_peered":"2024-04-25T15:27:20.082792+0000","last_clean":"2024-04-25T15:27:20.082792+0000","last_became_active":"2024-04-25T15:27:18.009016+0000","last_became_peered":"2024-04-25T15:27:18.009016+0000","last_unstale":"2024-04-25T15:27:20.082792+0000","last_undegraded":"2024-04-25T15:27:20.082792+0000","last_fullsized":"2024-04-25T15:27:20.082792+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:01.698169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023966+0000","last_change":"2024-04-25T15:27:19.032890+0000","last_active":"2024-04-25T15:27:20.023966+0000","last_peered":"2024-04-25T15:27:20.023966+0000","last_clean":"2024-04-25T15:27:20.023966+0000","last_became_active":"2024-04-25T15:27:19.032575+0000","last_became_peered":"2024-04-25T15:27:19.032575+0000","last_unstale":"2024-04-25T15:27:20.023966+0000","last_undegraded":"2024-04-25T15:27:20.023966+0000","last_fullsized":"2024-04-25T15:27:20.023966+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:34.746570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024058+0000","last_change":"2024-04-25T15:27:18.003793+0000","last_active":"2024-04-25T15:27:20.024058+0000","last_peered":"2024-04-25T15:27:20.024058+0000","last_clean":"2024-04-25T15:27:20.024058+0000","last_became_active":"2024-04-25T15:27:18.003499+0000","last_became_peered":"2024-04-25T15:27:18.003499+0000","last_unstale":"2024-04-25T15:27:20.024058+0000","last_undegraded":"2024-04-25T15:27:20.024058+0000","last_fullsized":"2024-04-25T15:27:20.024058+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:14.847161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023855+0000","last_change":"2024-04-25T15:27:19.032394+0000","last_active":"2024-04-25T15:27:20.023855+0000","last_peered":"2024-04-25T15:27:20.023855+0000","last_clean":"2024-04-25T15:27:20.023855+0000","last_became_active":"2024-04-25T15:27:19.032099+0000","last_became_peered":"2024-04-25T15:27:19.032099+0000","last_unstale":"2024-04-25T15:27:20.023855+0000","last_undegraded":"2024-04-25T15:27:20.023855+0000","last_fullsized":"2024-04-25T15:27:20.023855+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:15.273248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083490+0000","last_change":"2024-04-25T15:27:18.011811+0000","last_active":"2024-04-25T15:27:20.083490+0000","last_peered":"2024-04-25T15:27:20.083490+0000","last_clean":"2024-04-25T15:27:20.083490+0000","last_became_active":"2024-04-25T15:27:18.011453+0000","last_became_peered":"2024-04-25T15:27:18.011453+0000","last_unstale":"2024-04-25T15:27:20.083490+0000","last_undegraded":"2024-04-25T15:27:20.083490+0000","last_fullsized":"2024-04-25T15:27:20.083490+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:13.465091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083346+0000","last_change":"2024-04-25T15:27:19.033329+0000","last_active":"2024-04-25T15:27:20.083346+0000","last_peered":"2024-04-25T15:27:20.083346+0000","last_clean":"2024-04-25T15:27:20.083346+0000","last_became_active":"2024-04-25T15:27:19.032367+0000","last_became_peered":"2024-04-25T15:27:19.032367+0000","last_unstale":"2024-04-25T15:27:20.083346+0000","last_undegraded":"2024-04-25T15:27:20.083346+0000","last_fullsized":"2024-04-25T15:27:20.083346+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:10.301123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024084+0000","last_change":"2024-04-25T15:27:18.009861+0000","last_active":"2024-04-25T15:27:20.024084+0000","last_peered":"2024-04-25T15:27:20.024084+0000","last_clean":"2024-04-25T15:27:20.024084+0000","last_became_active":"2024-04-25T15:27:18.009355+0000","last_became_peered":"2024-04-25T15:27:18.009355+0000","last_unstale":"2024-04-25T15:27:20.024084+0000","last_undegraded":"2024-04-25T15:27:20.024084+0000","last_fullsized":"2024-04-25T15:27:20.024084+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:38:50.388996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082869+0000","last_change":"2024-04-25T15:27:19.022101+0000","last_active":"2024-04-25T15:27:20.082869+0000","last_peered":"2024-04-25T15:27:20.082869+0000","last_clean":"2024-04-25T15:27:20.082869+0000","last_became_active":"2024-04-25T15:27:19.021731+0000","last_became_peered":"2024-04-25T15:27:19.021731+0000","last_unstale":"2024-04-25T15:27:20.082869+0000","last_undegraded":"2024-04-25T15:27:20.082869+0000","last_fullsized":"2024-04-25T15:27:20.082869+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:24.649736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023628+0000","last_change":"2024-04-25T15:27:18.001183+0000","last_active":"2024-04-25T15:27:20.023628+0000","last_peered":"2024-04-25T15:27:20.023628+0000","last_clean":"2024-04-25T15:27:20.023628+0000","last_became_active":"2024-04-25T15:27:18.000877+0000","last_became_peered":"2024-04-25T15:27:18.000877+0000","last_unstale":"2024-04-25T15:27:20.023628+0000","last_undegraded":"2024-04-25T15:27:20.023628+0000","last_fullsized":"2024-04-25T15:27:20.023628+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:12.823849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030226+0000","last_change":"2024-04-25T15:27:19.034638+0000","last_active":"2024-04-25T15:27:20.030226+0000","last_peered":"2024-04-25T15:27:20.030226+0000","last_clean":"2024-04-25T15:27:20.030226+0000","last_became_active":"2024-04-25T15:27:19.034221+0000","last_became_peered":"2024-04-25T15:27:19.034221+0000","last_unstale":"2024-04-25T15:27:20.030226+0000","last_undegraded":"2024-04-25T15:27:20.030226+0000","last_fullsized":"2024-04-25T15:27:20.030226+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:45.250349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082795+0000","last_change":"2024-04-25T15:27:18.003146+0000","last_active":"2024-04-25T15:27:20.082795+0000","last_peered":"2024-04-25T15:27:20.082795+0000","last_clean":"2024-04-25T15:27:20.082795+0000","last_became_active":"2024-04-25T15:27:18.002822+0000","last_became_peered":"2024-04-25T15:27:18.002822+0000","last_unstale":"2024-04-25T15:27:20.082795+0000","last_undegraded":"2024-04-25T15:27:20.082795+0000","last_fullsized":"2024-04-25T15:27:20.082795+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:37.081350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082862+0000","last_change":"2024-04-25T15:27:19.031440+0000","last_active":"2024-04-25T15:27:20.082862+0000","last_peered":"2024-04-25T15:27:20.082862+0000","last_clean":"2024-04-25T15:27:20.082862+0000","last_became_active":"2024-04-25T15:27:19.031125+0000","last_became_peered":"2024-04-25T15:27:19.031125+0000","last_unstale":"2024-04-25T15:27:20.082862+0000","last_undegraded":"2024-04-25T15:27:20.082862+0000","last_fullsized":"2024-04-25T15:27:20.082862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:23.499821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023695+0000","last_change":"2024-04-25T15:27:18.007776+0000","last_active":"2024-04-25T15:27:20.023695+0000","last_peered":"2024-04-25T15:27:20.023695+0000","last_clean":"2024-04-25T15:27:20.023695+0000","last_became_active":"2024-04-25T15:27:18.007441+0000","last_became_peered":"2024-04-25T15:27:18.007441+0000","last_unstale":"2024-04-25T15:27:20.023695+0000","last_undegraded":"2024-04-25T15:27:20.023695+0000","last_fullsized":"2024-04-25T15:27:20.023695+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:00.422138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024210+0000","last_change":"2024-04-25T15:27:19.038218+0000","last_active":"2024-04-25T15:27:20.024210+0000","last_peered":"2024-04-25T15:27:20.024210+0000","last_clean":"2024-04-25T15:27:20.024210+0000","last_became_active":"2024-04-25T15:27:19.037503+0000","last_became_peered":"2024-04-25T15:27:19.037503+0000","last_unstale":"2024-04-25T15:27:20.024210+0000","last_undegraded":"2024-04-25T15:27:20.024210+0000","last_fullsized":"2024-04-25T15:27:20.024210+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:19.801669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083714+0000","last_change":"2024-04-25T15:27:18.015029+0000","last_active":"2024-04-25T15:27:20.083714+0000","last_peered":"2024-04-25T15:27:20.083714+0000","last_clean":"2024-04-25T15:27:20.083714+0000","last_became_active":"2024-04-25T15:27:18.014722+0000","last_became_peered":"2024-04-25T15:27:18.014722+0000","last_unstale":"2024-04-25T15:27:20.083714+0000","last_undegraded":"2024-04-25T15:27:20.083714+0000","last_fullsized":"2024-04-25T15:27:20.083714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:13.465091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083337+0000","last_change":"2024-04-25T15:27:19.026141+0000","last_active":"2024-04-25T15:27:20.083337+0000","last_peered":"2024-04-25T15:27:20.083337+0000","last_clean":"2024-04-25T15:27:20.083337+0000","last_became_active":"2024-04-25T15:27:19.025846+0000","last_became_peered":"2024-04-25T15:27:19.025846+0000","last_unstale":"2024-04-25T15:27:20.083337+0000","last_undegraded":"2024-04-25T15:27:20.083337+0000","last_fullsized":"2024-04-25T15:27:20.083337+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:02.704289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082933+0000","last_change":"2024-04-25T15:27:18.017025+0000","last_active":"2024-04-25T15:27:20.082933+0000","last_peered":"2024-04-25T15:27:20.082933+0000","last_clean":"2024-04-25T15:27:20.082933+0000","last_became_active":"2024-04-25T15:27:18.016684+0000","last_became_peered":"2024-04-25T15:27:18.016684+0000","last_unstale":"2024-04-25T15:27:20.082933+0000","last_undegraded":"2024-04-25T15:27:20.082933+0000","last_fullsized":"2024-04-25T15:27:20.082933+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:44:37.817333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023557+0000","last_change":"2024-04-25T15:27:19.036838+0000","last_active":"2024-04-25T15:27:20.023557+0000","last_peered":"2024-04-25T15:27:20.023557+0000","last_clean":"2024-04-25T15:27:20.023557+0000","last_became_active":"2024-04-25T15:27:19.036528+0000","last_became_peered":"2024-04-25T15:27:19.036528+0000","last_unstale":"2024-04-25T15:27:20.023557+0000","last_undegraded":"2024-04-25T15:27:20.023557+0000","last_fullsized":"2024-04-25T15:27:20.023557+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:35.973311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083917+0000","last_change":"2024-04-25T15:27:18.014280+0000","last_active":"2024-04-25T15:27:20.083917+0000","last_peered":"2024-04-25T15:27:20.083917+0000","last_clean":"2024-04-25T15:27:20.083917+0000","last_became_active":"2024-04-25T15:27:18.013962+0000","last_became_peered":"2024-04-25T15:27:18.013962+0000","last_unstale":"2024-04-25T15:27:20.083917+0000","last_undegraded":"2024-04-25T15:27:20.083917+0000","last_fullsized":"2024-04-25T15:27:20.083917+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:18.795548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083635+0000","last_change":"2024-04-25T15:27:19.038942+0000","last_active":"2024-04-25T15:27:20.083635+0000","last_peered":"2024-04-25T15:27:20.083635+0000","last_clean":"2024-04-25T15:27:20.083635+0000","last_became_active":"2024-04-25T15:27:19.038307+0000","last_became_peered":"2024-04-25T15:27:19.038307+0000","last_unstale":"2024-04-25T15:27:20.083635+0000","last_undegraded":"2024-04-25T15:27:20.083635+0000","last_fullsized":"2024-04-25T15:27:20.083635+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:09.754561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":20,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024327+0000","last_change":"2024-04-25T15:27:18.013232+0000","last_active":"2024-04-25T15:27:20.024327+0000","last_peered":"2024-04-25T15:27:20.024327+0000","last_clean":"2024-04-25T15:27:20.024327+0000","last_became_active":"2024-04-25T15:27:18.012829+0000","last_became_peered":"2024-04-25T15:27:18.012829+0000","last_unstale":"2024-04-25T15:27:20.024327+0000","last_undegraded":"2024-04-25T15:27:20.024327+0000","last_fullsized":"2024-04-25T15:27:20.024327+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:44.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082851+0000","last_change":"2024-04-25T15:27:19.036282+0000","last_active":"2024-04-25T15:27:20.082851+0000","last_peered":"2024-04-25T15:27:20.082851+0000","last_clean":"2024-04-25T15:27:20.082851+0000","last_became_active":"2024-04-25T15:27:19.035915+0000","last_became_peered":"2024-04-25T15:27:19.035915+0000","last_unstale":"2024-04-25T15:27:20.082851+0000","last_undegraded":"2024-04-25T15:27:20.082851+0000","last_fullsized":"2024-04-25T15:27:20.082851+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:01:38.596800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082996+0000","last_change":"2024-04-25T15:27:18.016988+0000","last_active":"2024-04-25T15:27:20.082996+0000","last_peered":"2024-04-25T15:27:20.082996+0000","last_clean":"2024-04-25T15:27:20.082996+0000","last_became_active":"2024-04-25T15:27:18.016655+0000","last_became_peered":"2024-04-25T15:27:18.016655+0000","last_unstale":"2024-04-25T15:27:20.082996+0000","last_undegraded":"2024-04-25T15:27:20.082996+0000","last_fullsized":"2024-04-25T15:27:20.082996+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:15.678763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083853+0000","last_change":"2024-04-25T15:27:19.029623+0000","last_active":"2024-04-25T15:27:20.083853+0000","last_peered":"2024-04-25T15:27:20.083853+0000","last_clean":"2024-04-25T15:27:20.083853+0000","last_became_active":"2024-04-25T15:27:19.028907+0000","last_became_peered":"2024-04-25T15:27:19.028907+0000","last_unstale":"2024-04-25T15:27:20.083853+0000","last_undegraded":"2024-04-25T15:27:20.083853+0000","last_fullsized":"2024-04-25T15:27:20.083853+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:03.756237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030039+0000","last_change":"2024-04-25T15:27:18.010566+0000","last_active":"2024-04-25T15:27:20.030039+0000","last_peered":"2024-04-25T15:27:20.030039+0000","last_clean":"2024-04-25T15:27:20.030039+0000","last_became_active":"2024-04-25T15:27:18.010256+0000","last_became_peered":"2024-04-25T15:27:18.010256+0000","last_unstale":"2024-04-25T15:27:20.030039+0000","last_undegraded":"2024-04-25T15:27:20.030039+0000","last_fullsized":"2024-04-25T15:27:20.030039+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:41.088283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023714+0000","last_change":"2024-04-25T15:27:19.028681+0000","last_active":"2024-04-25T15:27:20.023714+0000","last_peered":"2024-04-25T15:27:20.023714+0000","last_clean":"2024-04-25T15:27:20.023714+0000","last_became_active":"2024-04-25T15:27:19.028359+0000","last_became_peered":"2024-04-25T15:27:19.028359+0000","last_unstale":"2024-04-25T15:27:20.023714+0000","last_undegraded":"2024-04-25T15:27:20.023714+0000","last_fullsized":"2024-04-25T15:27:20.023714+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:24.132206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082909+0000","last_change":"2024-04-25T15:27:18.012486+0000","last_active":"2024-04-25T15:27:20.082909+0000","last_peered":"2024-04-25T15:27:20.082909+0000","last_clean":"2024-04-25T15:27:20.082909+0000","last_became_active":"2024-04-25T15:27:18.012119+0000","last_became_peered":"2024-04-25T15:27:18.012119+0000","last_unstale":"2024-04-25T15:27:20.082909+0000","last_undegraded":"2024-04-25T15:27:20.082909+0000","last_fullsized":"2024-04-25T15:27:20.082909+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:44.704232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083909+0000","last_change":"2024-04-25T15:27:19.023008+0000","last_active":"2024-04-25T15:27:20.083909+0000","last_peered":"2024-04-25T15:27:20.083909+0000","last_clean":"2024-04-25T15:27:20.083909+0000","last_became_active":"2024-04-25T15:27:19.022711+0000","last_became_peered":"2024-04-25T15:27:19.022711+0000","last_unstale":"2024-04-25T15:27:20.083909+0000","last_undegraded":"2024-04-25T15:27:20.083909+0000","last_fullsized":"2024-04-25T15:27:20.083909+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:25.758868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":22,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082999+0000","last_change":"2024-04-25T15:27:18.011783+0000","last_active":"2024-04-25T15:27:20.082999+0000","last_peered":"2024-04-25T15:27:20.082999+0000","last_clean":"2024-04-25T15:27:20.082999+0000","last_became_active":"2024-04-25T15:27:18.011453+0000","last_became_peered":"2024-04-25T15:27:18.011453+0000","last_unstale":"2024-04-25T15:27:20.082999+0000","last_undegraded":"2024-04-25T15:27:20.082999+0000","last_fullsized":"2024-04-25T15:27:20.082999+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:44.704232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083818+0000","last_change":"2024-04-25T15:27:19.030175+0000","last_active":"2024-04-25T15:27:20.083818+0000","last_peered":"2024-04-25T15:27:20.083818+0000","last_clean":"2024-04-25T15:27:20.083818+0000","last_became_active":"2024-04-25T15:27:19.029863+0000","last_became_peered":"2024-04-25T15:27:19.029863+0000","last_unstale":"2024-04-25T15:27:20.083818+0000","last_undegraded":"2024-04-25T15:27:20.083818+0000","last_fullsized":"2024-04-25T15:27:20.083818+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:06.086869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083209+0000","last_change":"2024-04-25T15:27:18.014965+0000","last_active":"2024-04-25T15:27:20.083209+0000","last_peered":"2024-04-25T15:27:20.083209+0000","last_clean":"2024-04-25T15:27:20.083209+0000","last_became_active":"2024-04-25T15:27:18.014656+0000","last_became_peered":"2024-04-25T15:27:18.014656+0000","last_unstale":"2024-04-25T15:27:20.083209+0000","last_undegraded":"2024-04-25T15:27:20.083209+0000","last_fullsized":"2024-04-25T15:27:20.083209+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:33.346649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083161+0000","last_change":"2024-04-25T15:27:19.036421+0000","last_active":"2024-04-25T15:27:20.083161+0000","last_peered":"2024-04-25T15:27:20.083161+0000","last_clean":"2024-04-25T15:27:20.083161+0000","last_became_active":"2024-04-25T15:27:19.036115+0000","last_became_peered":"2024-04-25T15:27:19.036115+0000","last_unstale":"2024-04-25T15:27:20.083161+0000","last_undegraded":"2024-04-25T15:27:20.083161+0000","last_fullsized":"2024-04-25T15:27:20.083161+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:16.684883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023668+0000","last_change":"2024-04-25T15:27:17.999592+0000","last_active":"2024-04-25T15:27:20.023668+0000","last_peered":"2024-04-25T15:27:20.023668+0000","last_clean":"2024-04-25T15:27:20.023668+0000","last_became_active":"2024-04-25T15:27:17.999276+0000","last_became_peered":"2024-04-25T15:27:17.999276+0000","last_unstale":"2024-04-25T15:27:20.023668+0000","last_undegraded":"2024-04-25T15:27:20.023668+0000","last_fullsized":"2024-04-25T15:27:20.023668+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:56.712684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083405+0000","last_change":"2024-04-25T15:27:19.030191+0000","last_active":"2024-04-25T15:27:20.083405+0000","last_peered":"2024-04-25T15:27:20.083405+0000","last_clean":"2024-04-25T15:27:20.083405+0000","last_became_active":"2024-04-25T15:27:19.029873+0000","last_became_peered":"2024-04-25T15:27:19.029873+0000","last_unstale":"2024-04-25T15:27:20.083405+0000","last_undegraded":"2024-04-25T15:27:20.083405+0000","last_fullsized":"2024-04-25T15:27:20.083405+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:35:23.290538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083365+0000","last_change":"2024-04-25T15:27:18.008455+0000","last_active":"2024-04-25T15:27:20.083365+0000","last_peered":"2024-04-25T15:27:20.083365+0000","last_clean":"2024-04-25T15:27:20.083365+0000","last_became_active":"2024-04-25T15:27:18.008154+0000","last_became_peered":"2024-04-25T15:27:18.008154+0000","last_unstale":"2024-04-25T15:27:20.083365+0000","last_undegraded":"2024-04-25T15:27:20.083365+0000","last_fullsized":"2024-04-25T15:27:20.083365+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:24:48.152920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083888+0000","last_change":"2024-04-25T15:27:19.039608+0000","last_active":"2024-04-25T15:27:20.083888+0000","last_peered":"2024-04-25T15:27:20.083888+0000","last_clean":"2024-04-25T15:27:20.083888+0000","last_became_active":"2024-04-25T15:27:19.039214+0000","last_became_peered":"2024-04-25T15:27:19.039214+0000","last_unstale":"2024-04-25T15:27:20.083888+0000","last_undegraded":"2024-04-25T15:27:20.083888+0000","last_fullsized":"2024-04-25T15:27:20.083888+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:40:57.009304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023784+0000","last_change":"2024-04-25T15:27:18.011867+0000","last_active":"2024-04-25T15:27:20.023784+0000","last_peered":"2024-04-25T15:27:20.023784+0000","last_clean":"2024-04-25T15:27:20.023784+0000","last_became_active":"2024-04-25T15:27:18.011542+0000","last_became_peered":"2024-04-25T15:27:18.011542+0000","last_unstale":"2024-04-25T15:27:20.023784+0000","last_undegraded":"2024-04-25T15:27:20.023784+0000","last_fullsized":"2024-04-25T15:27:20.023784+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:40:17.915392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083502+0000","last_change":"2024-04-25T15:27:19.035715+0000","last_active":"2024-04-25T15:27:20.083502+0000","last_peered":"2024-04-25T15:27:20.083502+0000","last_clean":"2024-04-25T15:27:20.083502+0000","last_became_active":"2024-04-25T15:27:19.035398+0000","last_became_peered":"2024-04-25T15:27:19.035398+0000","last_unstale":"2024-04-25T15:27:20.083502+0000","last_undegraded":"2024-04-25T15:27:20.083502+0000","last_fullsized":"2024-04-25T15:27:20.083502+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:44:38.823454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083271+0000","last_change":"2024-04-25T15:27:18.009083+0000","last_active":"2024-04-25T15:27:20.083271+0000","last_peered":"2024-04-25T15:27:20.083271+0000","last_clean":"2024-04-25T15:27:20.083271+0000","last_became_active":"2024-04-25T15:27:18.007368+0000","last_became_peered":"2024-04-25T15:27:18.007368+0000","last_unstale":"2024-04-25T15:27:20.083271+0000","last_undegraded":"2024-04-25T15:27:20.083271+0000","last_fullsized":"2024-04-25T15:27:20.083271+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:54.376029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023570+0000","last_change":"2024-04-25T15:27:19.029061+0000","last_active":"2024-04-25T15:27:20.023570+0000","last_peered":"2024-04-25T15:27:20.023570+0000","last_clean":"2024-04-25T15:27:20.023570+0000","last_became_active":"2024-04-25T15:27:19.028620+0000","last_became_peered":"2024-04-25T15:27:19.028620+0000","last_unstale":"2024-04-25T15:27:20.023570+0000","last_undegraded":"2024-04-25T15:27:20.023570+0000","last_fullsized":"2024-04-25T15:27:20.023570+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:03.756237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.022990+0000","last_change":"2024-04-25T15:27:18.001334+0000","last_active":"2024-04-25T15:27:20.022990+0000","last_peered":"2024-04-25T15:27:20.022990+0000","last_clean":"2024-04-25T15:27:20.022990+0000","last_became_active":"2024-04-25T15:27:18.000809+0000","last_became_peered":"2024-04-25T15:27:18.000809+0000","last_unstale":"2024-04-25T15:27:20.022990+0000","last_undegraded":"2024-04-25T15:27:20.022990+0000","last_fullsized":"2024-04-25T15:27:20.022990+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:23.803156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023616+0000","last_change":"2024-04-25T15:27:19.024839+0000","last_active":"2024-04-25T15:27:20.023616+0000","last_peered":"2024-04-25T15:27:20.023616+0000","last_clean":"2024-04-25T15:27:20.023616+0000","last_became_active":"2024-04-25T15:27:19.024503+0000","last_became_peered":"2024-04-25T15:27:19.024503+0000","last_unstale":"2024-04-25T15:27:20.023616+0000","last_undegraded":"2024-04-25T15:27:20.023616+0000","last_fullsized":"2024-04-25T15:27:20.023616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:40:57.009304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083374+0000","last_change":"2024-04-25T15:27:18.008733+0000","last_active":"2024-04-25T15:27:20.083374+0000","last_peered":"2024-04-25T15:27:20.083374+0000","last_clean":"2024-04-25T15:27:20.083374+0000","last_became_active":"2024-04-25T15:27:18.008455+0000","last_became_peered":"2024-04-25T15:27:18.008455+0000","last_unstale":"2024-04-25T15:27:20.083374+0000","last_undegraded":"2024-04-25T15:27:20.083374+0000","last_fullsized":"2024-04-25T15:27:20.083374+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:36.454156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083846+0000","last_change":"2024-04-25T15:27:19.032233+0000","last_active":"2024-04-25T15:27:20.083846+0000","last_peered":"2024-04-25T15:27:20.083846+0000","last_clean":"2024-04-25T15:27:20.083846+0000","last_became_active":"2024-04-25T15:27:19.031928+0000","last_became_peered":"2024-04-25T15:27:19.031928+0000","last_unstale":"2024-04-25T15:27:20.083846+0000","last_undegraded":"2024-04-25T15:27:20.083846+0000","last_fullsized":"2024-04-25T15:27:20.083846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:17:34.352770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024004+0000","last_change":"2024-04-25T15:27:17.999872+0000","last_active":"2024-04-25T15:27:20.024004+0000","last_peered":"2024-04-25T15:27:20.024004+0000","last_clean":"2024-04-25T15:27:20.024004+0000","last_became_active":"2024-04-25T15:27:17.999165+0000","last_became_peered":"2024-04-25T15:27:17.999165+0000","last_unstale":"2024-04-25T15:27:20.024004+0000","last_undegraded":"2024-04-25T15:27:20.024004+0000","last_fullsized":"2024-04-25T15:27:20.024004+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:41.088283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083324+0000","last_change":"2024-04-25T15:27:19.041103+0000","last_active":"2024-04-25T15:27:20.083324+0000","last_peered":"2024-04-25T15:27:20.083324+0000","last_clean":"2024-04-25T15:27:20.083324+0000","last_became_active":"2024-04-25T15:27:19.040763+0000","last_became_peered":"2024-04-25T15:27:19.040763+0000","last_unstale":"2024-04-25T15:27:20.083324+0000","last_undegraded":"2024-04-25T15:27:20.083324+0000","last_fullsized":"2024-04-25T15:27:20.083324+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:02.704289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.027342+0000","last_change":"2024-04-25T15:27:18.003156+0000","last_active":"2024-04-25T15:27:20.027342+0000","last_peered":"2024-04-25T15:27:20.027342+0000","last_clean":"2024-04-25T15:27:20.027342+0000","last_became_active":"2024-04-25T15:27:18.002809+0000","last_became_peered":"2024-04-25T15:27:18.002809+0000","last_unstale":"2024-04-25T15:27:20.027342+0000","last_undegraded":"2024-04-25T15:27:20.027342+0000","last_fullsized":"2024-04-25T15:27:20.027342+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:26.252582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024318+0000","last_change":"2024-04-25T15:27:19.036019+0000","last_active":"2024-04-25T15:27:20.024318+0000","last_peered":"2024-04-25T15:27:20.024318+0000","last_clean":"2024-04-25T15:27:20.024318+0000","last_became_active":"2024-04-25T15:27:19.035708+0000","last_became_peered":"2024-04-25T15:27:19.035708+0000","last_unstale":"2024-04-25T15:27:20.024318+0000","last_undegraded":"2024-04-25T15:27:20.024318+0000","last_fullsized":"2024-04-25T15:27:20.024318+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:23:36.250857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030174+0000","last_change":"2024-04-25T15:27:18.005250+0000","last_active":"2024-04-25T15:27:20.030174+0000","last_peered":"2024-04-25T15:27:20.030174+0000","last_clean":"2024-04-25T15:27:20.030174+0000","last_became_active":"2024-04-25T15:27:18.004919+0000","last_became_peered":"2024-04-25T15:27:18.004919+0000","last_unstale":"2024-04-25T15:27:20.030174+0000","last_undegraded":"2024-04-25T15:27:20.030174+0000","last_fullsized":"2024-04-25T15:27:20.030174+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:14.898632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.023949+0000","last_change":"2024-04-25T15:27:19.031050+0000","last_active":"2024-04-25T15:27:20.023949+0000","last_peered":"2024-04-25T15:27:20.023949+0000","last_clean":"2024-04-25T15:27:20.023949+0000","last_became_active":"2024-04-25T15:27:19.030719+0000","last_became_peered":"2024-04-25T15:27:19.030719+0000","last_unstale":"2024-04-25T15:27:20.023949+0000","last_undegraded":"2024-04-25T15:27:20.023949+0000","last_fullsized":"2024-04-25T15:27:20.023949+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:27.887865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083606+0000","last_change":"2024-04-25T15:27:18.004668+0000","last_active":"2024-04-25T15:27:20.083606+0000","last_peered":"2024-04-25T15:27:20.083606+0000","last_clean":"2024-04-25T15:27:20.083606+0000","last_became_active":"2024-04-25T15:27:18.004303+0000","last_became_peered":"2024-04-25T15:27:18.004303+0000","last_unstale":"2024-04-25T15:27:20.083606+0000","last_undegraded":"2024-04-25T15:27:20.083606+0000","last_fullsized":"2024-04-25T15:27:20.083606+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:18.795548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083506+0000","last_change":"2024-04-25T15:27:19.027620+0000","last_active":"2024-04-25T15:27:20.083506+0000","last_peered":"2024-04-25T15:27:20.083506+0000","last_clean":"2024-04-25T15:27:20.083506+0000","last_became_active":"2024-04-25T15:27:19.027314+0000","last_became_peered":"2024-04-25T15:27:19.027314+0000","last_unstale":"2024-04-25T15:27:20.083506+0000","last_undegraded":"2024-04-25T15:27:20.083506+0000","last_fullsized":"2024-04-25T15:27:20.083506+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:15:23.345125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.022922+0000","last_change":"2024-04-25T15:27:18.003435+0000","last_active":"2024-04-25T15:27:20.022922+0000","last_peered":"2024-04-25T15:27:20.022922+0000","last_clean":"2024-04-25T15:27:20.022922+0000","last_became_active":"2024-04-25T15:27:18.003161+0000","last_became_peered":"2024-04-25T15:27:18.003161+0000","last_unstale":"2024-04-25T15:27:20.022922+0000","last_undegraded":"2024-04-25T15:27:20.022922+0000","last_fullsized":"2024-04-25T15:27:20.022922+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:41.715761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.084016+0000","last_change":"2024-04-25T15:27:19.028529+0000","last_active":"2024-04-25T15:27:20.084016+0000","last_peered":"2024-04-25T15:27:20.084016+0000","last_clean":"2024-04-25T15:27:20.084016+0000","last_became_active":"2024-04-25T15:27:19.028173+0000","last_became_peered":"2024-04-25T15:27:19.028173+0000","last_unstale":"2024-04-25T15:27:20.084016+0000","last_undegraded":"2024-04-25T15:27:20.084016+0000","last_fullsized":"2024-04-25T15:27:20.084016+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:15.273248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024172+0000","last_change":"2024-04-25T15:27:18.013156+0000","last_active":"2024-04-25T15:27:20.024172+0000","last_peered":"2024-04-25T15:27:20.024172+0000","last_clean":"2024-04-25T15:27:20.024172+0000","last_became_active":"2024-04-25T15:27:18.012767+0000","last_became_peered":"2024-04-25T15:27:18.012767+0000","last_unstale":"2024-04-25T15:27:20.024172+0000","last_undegraded":"2024-04-25T15:27:20.024172+0000","last_fullsized":"2024-04-25T15:27:20.024172+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:19:48.629294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.022846+0000","last_change":"2024-04-25T15:27:19.034499+0000","last_active":"2024-04-25T15:27:20.022846+0000","last_peered":"2024-04-25T15:27:20.022846+0000","last_clean":"2024-04-25T15:27:20.022846+0000","last_became_active":"2024-04-25T15:27:19.034148+0000","last_became_peered":"2024-04-25T15:27:19.034148+0000","last_unstale":"2024-04-25T15:27:20.022846+0000","last_undegraded":"2024-04-25T15:27:20.022846+0000","last_fullsized":"2024-04-25T15:27:20.022846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:05.241698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.024205+0000","last_change":"2024-04-25T15:27:18.003984+0000","last_active":"2024-04-25T15:27:20.024205+0000","last_peered":"2024-04-25T15:27:20.024205+0000","last_clean":"2024-04-25T15:27:20.024205+0000","last_became_active":"2024-04-25T15:27:18.003671+0000","last_became_peered":"2024-04-25T15:27:18.003671+0000","last_unstale":"2024-04-25T15:27:20.024205+0000","last_undegraded":"2024-04-25T15:27:20.024205+0000","last_fullsized":"2024-04-25T15:27:20.024205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:14.898632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083058+0000","last_change":"2024-04-25T15:27:19.031518+0000","last_active":"2024-04-25T15:27:20.083058+0000","last_peered":"2024-04-25T15:27:20.083058+0000","last_clean":"2024-04-25T15:27:20.083058+0000","last_became_active":"2024-04-25T15:27:19.031195+0000","last_became_peered":"2024-04-25T15:27:19.031195+0000","last_unstale":"2024-04-25T15:27:20.083058+0000","last_undegraded":"2024-04-25T15:27:20.083058+0000","last_fullsized":"2024-04-25T15:27:20.083058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:32.771569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082985+0000","last_change":"2024-04-25T15:27:18.011567+0000","last_active":"2024-04-25T15:27:20.082985+0000","last_peered":"2024-04-25T15:27:20.082985+0000","last_clean":"2024-04-25T15:27:20.082985+0000","last_became_active":"2024-04-25T15:27:18.008482+0000","last_became_peered":"2024-04-25T15:27:18.008482+0000","last_unstale":"2024-04-25T15:27:20.082985+0000","last_undegraded":"2024-04-25T15:27:20.082985+0000","last_fullsized":"2024-04-25T15:27:20.082985+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:35.340259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.082772+0000","last_change":"2024-04-25T15:27:19.020034+0000","last_active":"2024-04-25T15:27:20.082772+0000","last_peered":"2024-04-25T15:27:20.082772+0000","last_clean":"2024-04-25T15:27:20.082772+0000","last_became_active":"2024-04-25T15:27:19.019692+0000","last_became_peered":"2024-04-25T15:27:19.019692+0000","last_unstale":"2024-04-25T15:27:20.082772+0000","last_undegraded":"2024-04-25T15:27:20.082772+0000","last_fullsized":"2024-04-25T15:27:20.082772+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:17.986934+0000","last_clean_scrub_stamp":"2024-04-25T15:27:17.986934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:26.783369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.030048+0000","last_change":"2024-04-25T15:27:18.004051+0000","last_active":"2024-04-25T15:27:20.030048+0000","last_peered":"2024-04-25T15:27:20.030048+0000","last_clean":"2024-04-25T15:27:20.030048+0000","last_became_active":"2024-04-25T15:27:18.003680+0000","last_became_peered":"2024-04-25T15:27:18.003680+0000","last_unstale":"2024-04-25T15:27:20.030048+0000","last_undegraded":"2024-04-25T15:27:20.030048+0000","last_fullsized":"2024-04-25T15:27:20.030048+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:16.980814+0000","last_clean_scrub_stamp":"2024-04-25T15:27:16.980814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:23.803156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":175,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T15:27:20.083931+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:27:20.083931+0000","last_peered":"2024-04-25T15:27:20.083931+0000","last_clean":"2024-04-25T15:27:20.083931+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:27:20.083931+0000","last_undegraded":"2024-04-25T15:27:20.083931+0000","last_fullsized":"2024-04-25T15:27:20.083931+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":868843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":671744,"data_stored":278315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":278967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":279927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":679936,"data_stored":278371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":679936,"data_stored":278371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":675840,"data_stored":278405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28272,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343568,"statfs":{"total":96636764160,"available":96607813632,"internally_reserved":0,"allocated":1282048,"data_stored":870877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:27:23.456 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:27:23.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:27:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.609+0000 7f8e76cf2640 1 Processor -- start 2024-04-25T15:27:23.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.610+0000 7f8e76cf2640 1 -- start start 2024-04-25T15:27:23.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e76cf2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e700d0e00 0x7f8e700d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e76cf2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8e7006c5c0 0x7f8e7006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e76cf2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e7006cf10 con 0x7f8e700b7810 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e76cf2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8e700d5a60 con 0x7f8e700d0e00 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e76cf2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8e700d1850 con 0x7f8e7006c5c0 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e700d0e00 0x7f8e700d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e700d0e00 0x7f8e700d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37514/0 (socket says 172.21.15.67:37514) 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 -- 172.21.15.67:0/1234899355 learned_addr learned my addr 172.21.15.67:0/1234899355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 -- 172.21.15.67:0/1234899355 >> v2:172.21.15.176:3300/0 conn(0x7f8e7006c5c0 msgr2=0x7f8e7006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e6ffff640 1 --2- 172.21.15.67:0/1234899355 >> v2:172.21.15.176:3300/0 conn(0x7f8e7006c5c0 0x7f8e7006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 --2- 172.21.15.67:0/1234899355 >> v2:172.21.15.176:3300/0 conn(0x7f8e7006c5c0 0x7f8e7006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 -- 172.21.15.67:0/1234899355 >> v1:172.21.15.176:6789/0 conn(0x7f8e700b7810 legacy=0x7f8e700b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:23.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 -- 172.21.15.67:0/1234899355 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e70160de0 con 0x7f8e700d0e00 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e74a8e640 1 --2- 172.21.15.67:0/1234899355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e700d1210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8e700d5be0 tx=0x7f8e6002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dc6c15bb073dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e6f7fe640 1 -- 172.21.15.67:0/1234899355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60037070 con 0x7f8e700d0e00 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e6f7fe640 1 -- 172.21.15.67:0/1234899355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e60032440 con 0x7f8e700d0e00 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.612+0000 7f8e6f7fe640 1 -- 172.21.15.67:0/1234899355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60032810 con 0x7f8e700d0e00 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 -- 172.21.15.67:0/1234899355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 msgr2=0x7f8e700d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/1234899355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e700d1210 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8e700d5be0 tx=0x7f8e6002a180 comp rx=0 tx=0).stop 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 -- 172.21.15.67:0/1234899355 shutdown_connections 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/1234899355 >> v2:172.21.15.176:3300/0 conn(0x7f8e7006c5c0 0x7f8e7006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/1234899355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e700d1210 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 -- 172.21.15.67:0/1234899355 >> 172.21.15.67:0/1234899355 conn(0x7f8e700d5d90 msgr2=0x7f8e700d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 -- 172.21.15.67:0/1234899355 shutdown_connections 2024-04-25T15:27:23.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.613+0000 7f8e76cf2640 1 -- 172.21.15.67:0/1234899355 wait complete. 2024-04-25T15:27:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.615+0000 7f8e76cf2640 1 Processor -- start 2024-04-25T15:27:23.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.615+0000 7f8e76cf2640 1 -- start start 2024-04-25T15:27:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.616+0000 7f8e76cf2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e7006c5c0 0x7f8e700ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.616+0000 7f8e76cf2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e700b7810 0x7f8e7015dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.616+0000 7f8e76cf2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e7015b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.616+0000 7f8e76cf2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e70160ad0 con 0x7f8e700d0e00 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.616+0000 7f8e76cf2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701613a0 con 0x7f8e700b7810 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.616+0000 7f8e76cf2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e700b5320 con 0x7f8e7006c5c0 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.616+0000 7f8e7528f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e7015b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e7015b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37518/0 (socket says 172.21.15.67:37518) 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 -- 172.21.15.67:0/4121977945 learned_addr learned my addr 172.21.15.67:0/4121977945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e6ffff640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e700b7810 0x7f8e7015dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e74a8e640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e7006c5c0 0x7f8e700ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 -- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e7006c5c0 msgr2=0x7f8e700ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e7006c5c0 0x7f8e700ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 -- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e700b7810 msgr2=0x7f8e7015dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e700b7810 0x7f8e7015dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 -- 172.21.15.67:0/4121977945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7015bd50 con 0x7f8e700d0e00 2024-04-25T15:27:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e7528f640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e7015b170 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800b970 tx=0x7f8e6800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e6d7fa640 1 -- 172.21.15.67:0/4121977945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68019070 con 0x7f8e700d0e00 2024-04-25T15:27:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e6d7fa640 1 -- 172.21.15.67:0/4121977945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e6800cb40 con 0x7f8e700d0e00 2024-04-25T15:27:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7015c030 con 0x7f8e700d0e00 2024-04-25T15:27:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e6d7fa640 1 -- 172.21.15.67:0/4121977945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68014440 con 0x7f8e700d0e00 2024-04-25T15:27:23.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.617+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e700c9b70 con 0x7f8e700d0e00 2024-04-25T15:27:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.618+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e70050740 con 0x7f8e700d0e00 2024-04-25T15:27:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.619+0000 7f8e6d7fa640 1 -- 172.21.15.67:0/4121977945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8e6801c070 con 0x7f8e700d0e00 2024-04-25T15:27:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.620+0000 7f8e6d7fa640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e5c0744b0 0x7f8e5c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.620+0000 7f8e6d7fa640 1 -- 172.21.15.67:0/4121977945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f8e68094e00 con 0x7f8e700d0e00 2024-04-25T15:27:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.620+0000 7f8e74a8e640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e5c0744b0 0x7f8e5c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:23.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.620+0000 7f8e74a8e640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e5c0744b0 0x7f8e5c0769c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e60034000 tx=0x7f8e6002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:23.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.624+0000 7f8e6d7fa640 1 -- 172.21.15.67:0/4121977945 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e6805ef40 con 0x7f8e700d0e00 2024-04-25T15:27:23.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.880+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8e7015bae0 con 0x7f8e700d0e00 2024-04-25T15:27:23.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.883+0000 7f8e6d7fa640 1 -- 172.21.15.67:0/4121977945 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f8e68063240 con 0x7f8e700d0e00 2024-04-25T15:27:23.885 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:27:23.885 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:27:23.885 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-25T15:27:23.885 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-25T15:27:23.885 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:27:23.885 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-25T15:27:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.884+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e5c0744b0 msgr2=0x7f8e5c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.884+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e5c0744b0 0x7f8e5c0769c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e60034000 tx=0x7f8e6002e000 comp rx=0 tx=0).stop 2024-04-25T15:27:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 msgr2=0x7f8e7015b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e7015b170 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800b970 tx=0x7f8e6800be30 comp rx=0 tx=0).stop 2024-04-25T15:27:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 shutdown_connections 2024-04-25T15:27:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e5c0744b0 0x7f8e5c0769c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e700d0e00 0x7f8e7015b170 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e700b7810 0x7f8e7015dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 --2- 172.21.15.67:0/4121977945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e7006c5c0 0x7f8e700ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:23.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 >> 172.21.15.67:0/4121977945 conn(0x7f8e700d5d90 msgr2=0x7f8e701601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:23.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 shutdown_connections 2024-04-25T15:27:23.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:23.885+0000 7f8e76cf2640 1 -- 172.21.15.67:0/4121977945 wait complete. 2024-04-25T15:27:23.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:23.984 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-25T15:27:23.982+0000 7f234815b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:27:24.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.008+0000 7f57677a8640 1 Processor -- start 2024-04-25T15:27:24.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.008+0000 7f57677a8640 1 -- start start 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f57677a8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f57600d0e00 0x7f57600d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f57677a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f576006c5c0 0x7f576006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f57677a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f576006cf10 con 0x7f57600b7810 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f57677a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f57600d5a60 con 0x7f576006c5c0 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f57677a8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f57600d1850 con 0x7f57600d0e00 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f576006c5c0 0x7f576006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f576006c5c0 0x7f576006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37530/0 (socket says 172.21.15.67:37530) 2024-04-25T15:27:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 -- 172.21.15.67:0/3479222915 learned_addr learned my addr 172.21.15.67:0/3479222915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5765544640 1 --2- 172.21.15.67:0/3479222915 >> v2:172.21.15.176:3300/0 conn(0x7f57600d0e00 0x7f57600d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 -- 172.21.15.67:0/3479222915 >> v2:172.21.15.176:3300/0 conn(0x7f57600d0e00 msgr2=0x7f57600d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 --2- 172.21.15.67:0/3479222915 >> v2:172.21.15.176:3300/0 conn(0x7f57600d0e00 0x7f57600d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 -- 172.21.15.67:0/3479222915 >> v1:172.21.15.176:6789/0 conn(0x7f57600b7810 legacy=0x7f57600b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 -- 172.21.15.67:0/3479222915 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5760160de0 con 0x7f576006c5c0 2024-04-25T15:27:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f5764d43640 1 --2- 172.21.15.67:0/3479222915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f576006c9d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f57600d5be0 tx=0x7f575402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e80a140a3b44753 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f575ffff640 1 -- 172.21.15.67:0/3479222915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754037070 con 0x7f576006c5c0 2024-04-25T15:27:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f575ffff640 1 -- 172.21.15.67:0/3479222915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5754032440 con 0x7f576006c5c0 2024-04-25T15:27:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.011+0000 7f575ffff640 1 -- 172.21.15.67:0/3479222915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754032810 con 0x7f576006c5c0 2024-04-25T15:27:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 -- 172.21.15.67:0/3479222915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 msgr2=0x7f576006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 --2- 172.21.15.67:0/3479222915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f576006c9d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f57600d5be0 tx=0x7f575402a180 comp rx=0 tx=0).stop 2024-04-25T15:27:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 -- 172.21.15.67:0/3479222915 shutdown_connections 2024-04-25T15:27:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 --2- 172.21.15.67:0/3479222915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f576006c9d0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 --2- 172.21.15.67:0/3479222915 >> v2:172.21.15.176:3300/0 conn(0x7f57600d0e00 0x7f57600d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 -- 172.21.15.67:0/3479222915 >> 172.21.15.67:0/3479222915 conn(0x7f57600d5d90 msgr2=0x7f57600d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 -- 172.21.15.67:0/3479222915 shutdown_connections 2024-04-25T15:27:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.012+0000 7f57677a8640 1 -- 172.21.15.67:0/3479222915 wait complete. 2024-04-25T15:27:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.014+0000 7f57677a8640 1 Processor -- start 2024-04-25T15:27:24.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.014+0000 7f57677a8640 1 -- start start 2024-04-25T15:27:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f57677a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f57600ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f57677a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57600b7810 0x7f576014d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f57677a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57600d0e00 0x7f57601fa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f57677a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5760160ad0 con 0x7f576006c5c0 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f57677a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f57601613a0 con 0x7f57600b7810 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f57677a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f57600b5320 con 0x7f57600d0e00 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f57600ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f57600ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37538/0 (socket says 172.21.15.67:37538) 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 -- 172.21.15.67:0/2225423970 learned_addr learned my addr 172.21.15.67:0/2225423970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5764d43640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57600b7810 0x7f576014d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765d45640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57600d0e00 0x7f57601fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 -- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57600d0e00 msgr2=0x7f57601fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57600d0e00 0x7f57601fa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 -- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57600b7810 msgr2=0x7f576014d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57600b7810 0x7f576014d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5765544640 1 -- 172.21.15.67:0/2225423970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57601fb270 con 0x7f576006c5c0 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.015+0000 7f5764d43640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57600b7810 0x7f576014d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.016+0000 7f5765544640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f57600ca2b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f575800d5f0 tx=0x7f575800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.016+0000 7f575dffb640 1 -- 172.21.15.67:0/2225423970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57580125f0 con 0x7f576006c5c0 2024-04-25T15:27:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.016+0000 7f575dffb640 1 -- 172.21.15.67:0/2225423970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5758019070 con 0x7f576006c5c0 2024-04-25T15:27:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.016+0000 7f575dffb640 1 -- 172.21.15.67:0/2225423970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57580129d0 con 0x7f576006c5c0 2024-04-25T15:27:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.016+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57600c97d0 con 0x7f576006c5c0 2024-04-25T15:27:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.016+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57600c9d90 con 0x7f576006c5c0 2024-04-25T15:27:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.017+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5760050740 con 0x7f576006c5c0 2024-04-25T15:27:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.017+0000 7f575dffb640 1 -- 172.21.15.67:0/2225423970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5758012b70 con 0x7f576006c5c0 2024-04-25T15:27:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.018+0000 7f575dffb640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f574c0744e0 0x7f574c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.018+0000 7f575dffb640 1 -- 172.21.15.67:0/2225423970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5758095660 con 0x7f576006c5c0 2024-04-25T15:27:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.018+0000 7f5764d43640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f574c0744e0 0x7f574c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.019+0000 7f5764d43640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f574c0744e0 0x7f574c0769f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5754034000 tx=0x7f575402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.023+0000 7f575dffb640 1 -- 172.21.15.67:0/2225423970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f575805fdf0 con 0x7f576006c5c0 2024-04-25T15:27:24.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.247+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57600d61a0 con 0x7f576006c5c0 2024-04-25T15:27:24.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.248+0000 7f575dffb640 1 -- 172.21.15.67:0/2225423970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5181 (secure 0 0 0) 0x7f5758066e20 con 0x7f576006c5c0 2024-04-25T15:27:24.249 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:24.249 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/610105566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":610105566},{"type":"v1","addr":"172.21.15.67:6835","nonce":610105566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/947236976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":947236976},{"type":"v1","addr":"172.21.15.176:6837","nonce":947236976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3211936625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3211936625},{"type":"v1","addr":"172.21.15.176:6836","nonce":3211936625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:18.413498+0000","modified":"2024-04-25T15:27:21.411023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5880":{"gid":5880,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3185771646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3185771646},{"type":"v1","addr":"172.21.15.67:6833","nonce":3185771646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5880,"qdb_cluster":[5880]},"id":4}]} 2024-04-25T15:27:24.249 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-25T15:27:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f574c0744e0 msgr2=0x7f574c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f574c0744e0 0x7f574c0769f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5754034000 tx=0x7f575402e000 comp rx=0 tx=0).stop 2024-04-25T15:27:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 msgr2=0x7f57600ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f57600ca2b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f575800d5f0 tx=0x7f575800dab0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 shutdown_connections 2024-04-25T15:27:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f574c0744e0 0x7f574c0769f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57600d0e00 0x7f57601fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57600b7810 0x7f576014d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 --2- 172.21.15.67:0/2225423970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f576006c5c0 0x7f57600ca2b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 >> 172.21.15.67:0/2225423970 conn(0x7f57600d5d90 msgr2=0x7f5760162120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 shutdown_connections 2024-04-25T15:27:24.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.250+0000 7f57677a8640 1 -- 172.21.15.67:0/2225423970 wait complete. 2024-04-25T15:27:24.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:27:18.413498+0000', 'modified': '2024-04-25T15:27:21.411023+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5880}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5880': {'gid': 5880, 'name': 'a', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6833/3185771646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3185771646}, {'type': 'v1', 'addr': '172.21.15.67:6833', 'nonce': 3185771646}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5880, 'qdb_cluster': [5880]} 2024-04-25T15:27:24.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:27:24.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T15:27:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.781+0000 7fb362667640 1 Processor -- start 2024-04-25T15:27:24.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.781+0000 7fb362667640 1 -- start start 2024-04-25T15:27:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb362667640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb35c0b8290 0x7fb35c160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb362667640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35c1615f0 con 0x7fb35c15df80 2024-04-25T15:27:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb362667640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35c164020 con 0x7fb35c164b00 2024-04-25T15:27:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb362667640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb35c164670 con 0x7fb35c0b8290 2024-04-25T15:27:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb360c04640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb35c15df80 0x7fb35c160dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56186/0 (socket says 172.21.15.67:56186) 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb360c04640 1 -- 172.21.15.67:0/1476155924 learned_addr learned my addr 172.21.15.67:0/1476155924 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/1476155924 >> v2:172.21.15.176:3300/0 conn(0x7fb35c0b8290 0x7fb35c160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4223345116 0 0) 0x7fb35c1615f0 con 0x7fb35c15df80 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb344003520 con 0x7fb35c15df80 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4103861017 0 0) 0x7fb35c164020 con 0x7fb35c164b00 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb35c1615f0 con 0x7fb35c164b00 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1753148082 0 0) 0x7fb344003520 con 0x7fb35c15df80 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 >> v2:172.21.15.176:3300/0 conn(0x7fb35c0b8290 msgr2=0x7fb35c160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 --2- 172.21.15.67:0/1476155924 >> v2:172.21.15.176:3300/0 conn(0x7fb35c0b8290 0x7fb35c160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 >> v1:172.21.15.176:6789/0 conn(0x7fb35c164b00 legacy=0x7fb35c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.783+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c164300 con 0x7fb35c15df80 2024-04-25T15:27:24.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fb3540019f0 con 0x7fb35c15df80 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/1476155924 >> v2:172.21.15.176:3300/0 conn(0x7fb35c0b8290 0x7fb35c160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 >> v1:172.21.15.67:6789/0 conn(0x7fb35c15df80 legacy=0x7fb35c160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb344003b40 0x7fb344003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344004800 0x7fb344004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3440051c0 0x7fb34400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c164670 con 0x7fb344003b40 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb34400c160 con 0x7fb3440051c0 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb34400c360 con 0x7fb344004800 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35bfff640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb344003b40 0x7fb344003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344004800 0x7fb344004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb360c04640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3440051c0 0x7fb34400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35bfff640 1 -- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344004800 msgr2=0x7fb344004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35bfff640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344004800 0x7fb344004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35bfff640 1 -- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3440051c0 msgr2=0x7fb34400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35bfff640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3440051c0 0x7fb34400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb35bfff640 1 -- 172.21.15.67:0/1476155924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb344003520 con 0x7fb344003b40 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb360c04640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3440051c0 0x7fb34400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.784+0000 7fb360c04640 1 -- 172.21.15.67:0/1476155924 reap_dead start 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.785+0000 7fb35bfff640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb344003b40 0x7fb344003fb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb3500018d0 tx=0x7fb344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d011a5202c9c812a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.785+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3500014f0 con 0x7fb344003b40 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.785+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3500052d0 con 0x7fb344003b40 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.785+0000 7fb35affd640 1 -- 172.21.15.67:0/1476155924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35002ac70 con 0x7fb344003b40 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.786+0000 7fb362667640 1 -- 172.21.15.67:0/1476155924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb344003b40 msgr2=0x7fb344003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.786+0000 7fb362667640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb344003b40 0x7fb344003fb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb3500018d0 tx=0x7fb344005830 comp rx=0 tx=0).stop 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.786+0000 7fb362667640 1 -- 172.21.15.67:0/1476155924 shutdown_connections 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.786+0000 7fb362667640 1 --2- 172.21.15.67:0/1476155924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb344003b40 0x7fb344003fb0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.786+0000 7fb362667640 1 -- 172.21.15.67:0/1476155924 >> 172.21.15.67:0/1476155924 conn(0x7fb35c0d2bc0 msgr2=0x7fb35c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.786+0000 7fb362667640 1 -- 172.21.15.67:0/1476155924 shutdown_connections 2024-04-25T15:27:24.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.786+0000 7fb362667640 1 -- 172.21.15.67:0/1476155924 wait complete. 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.787+0000 7fb362667640 1 Processor -- start 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.788+0000 7fb362667640 1 -- start start 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb362667640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344003b40 0x7fb35c14e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb362667640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35c0cf590 0x7fb35c0ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb362667640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb35c14e960 0x7fb35c14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb362667640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c0b7f10 con 0x7fb35c0cf590 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb362667640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c0b7d90 con 0x7fb35c14e960 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb362667640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c0b8090 con 0x7fb344003b40 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35c0cf590 0x7fb35c0ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35c0cf590 0x7fb35c0ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37550/0 (socket says 172.21.15.67:37550) 2024-04-25T15:27:24.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 -- 172.21.15.67:0/542171856 learned_addr learned my addr 172.21.15.67:0/542171856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35bfff640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344003b40 0x7fb35c14e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb360c04640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb35c14e960 0x7fb35c14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 -- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344003b40 msgr2=0x7fb35c14e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344003b40 0x7fb35c14e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 -- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb35c14e960 msgr2=0x7fb35c14ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb35c14e960 0x7fb35c14ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.789+0000 7fb35b7fe640 1 -- 172.21.15.67:0/542171856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c14e660 con 0x7fb35c0cf590 2024-04-25T15:27:24.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.790+0000 7fb35bfff640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344003b40 0x7fb35c14e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.790+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35c0cf590 0x7fb35c0ad0e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011d00 tx=0x7fb34c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.790+0000 7fb358ff9640 1 -- 172.21.15.67:0/542171856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00fd20 con 0x7fb35c0cf590 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.790+0000 7fb358ff9640 1 -- 172.21.15.67:0/542171856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c004020 con 0x7fb35c0cf590 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.790+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35c0ce4f0 con 0x7fb35c0cf590 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.790+0000 7fb358ff9640 1 -- 172.21.15.67:0/542171856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004410 con 0x7fb35c0cf590 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.790+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35c0ceab0 con 0x7fb35c0cf590 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.792+0000 7fb358ff9640 1 -- 172.21.15.67:0/542171856 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb34c012040 con 0x7fb35c0cf590 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.792+0000 7fb358ff9640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3480744b0 0x7fb3480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.792+0000 7fb358ff9640 1 -- 172.21.15.67:0/542171856 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fb34c095830 con 0x7fb35c0cf590 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.792+0000 7fb35bfff640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3480744b0 0x7fb3480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.793+0000 7fb35bfff640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3480744b0 0x7fb3480769c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb350032000 tx=0x7fb350009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.793+0000 7fb3695f8740 1 Processor -- start 2024-04-25T15:27:24.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.794+0000 7fb3695f8740 1 -- start start 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca748270 0x5641cab14600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641cab10fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641caaea260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb3695f8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5641caa4e740 con 0x5641ca71f3b0 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x5641ca9ea3a0 con 0x5641ca748270 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x5641caa4a700 con 0x5641ca727590 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641cab10fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641cab10fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37566/0 (socket says 172.21.15.67:37566) 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 -- 172.21.15.67:0/4094651261 learned_addr learned my addr 172.21.15.67:0/4094651261 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35bfff640 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca748270 0x5641cab14600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb360c04640 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641caaea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 -- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 msgr2=0x5641caaea260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641caaea260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 -- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca748270 msgr2=0x5641cab14600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca748270 0x5641cab14600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 -- 172.21.15.67:0/4094651261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641caa06750 con 0x5641ca71f3b0 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb360c04640 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641caaea260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.795+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641cab10fa0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb34c05beb0 tx=0x7fb34c05f030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6148e02e6a001f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb341ffb640 1 -- 172.21.15.67:0/4094651261 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c022050 con 0x5641ca71f3b0 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb341ffb640 1 -- 172.21.15.67:0/4094651261 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c095040 con 0x5641ca71f3b0 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb341ffb640 1 -- 172.21.15.67:0/4094651261 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0684d0 con 0x5641ca71f3b0 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 -- 172.21.15.67:0/4094651261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 msgr2=0x5641cab10fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641cab10fa0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb34c05beb0 tx=0x7fb34c05f030 comp rx=0 tx=0).stop 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 -- 172.21.15.67:0/4094651261 shutdown_connections 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641caaea260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641cab10fa0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 --2- 172.21.15.67:0/4094651261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca748270 0x5641cab14600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 -- 172.21.15.67:0/4094651261 >> 172.21.15.67:0/4094651261 conn(0x5641ca73c3c0 msgr2=0x5641caae99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 -- 172.21.15.67:0/4094651261 shutdown_connections 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.796+0000 7fb3695f8740 1 -- 172.21.15.67:0/4094651261 wait complete. 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.797+0000 7fb3695f8740 1 -- start start 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.798+0000 7fb3695f8740 1 Processor -- start 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca71f3b0 0x5641cab2ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab2eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb3695f8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5641ca9d5fe0 con 0x5641ca748270 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x5641cab2d6b0 con 0x5641ca727590 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x5641cab2d900 con 0x5641ca71f3b0 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb3695f8740 10 client.0 _ll_register_callbacks cb 0x5641ca9ec150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37574/0 (socket says 172.21.15.67:37574) 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 -- 172.21.15.67:0/118040396 learned_addr learned my addr 172.21.15.67:0/118040396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb35bfff640 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca71f3b0 0x5641cab2ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab2eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 -- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca71f3b0 msgr2=0x5641cab2ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca71f3b0 0x5641cab2ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 -- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 msgr2=0x5641cab2eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab2eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.799+0000 7fb360c04640 1 -- 172.21.15.67:0/118040396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641ca7602d0 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb360c04640 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2f3e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b8840 tx=0x7fb354007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b6c3ed9717841a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb33affd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb33affd640 1 -- 172.21.15.67:0/118040396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354011720 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb33affd640 1 -- 172.21.15.67:0/118040396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fb3540118c0 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb34c095040 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb33affd640 1 -- 172.21.15.67:0/118040396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb354022050 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 10 client.6021 fetch_fsmap learned FSMap version 32 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) -- 0x5641ca82a6b0 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb33affd640 1 -- 172.21.15.67:0/118040396 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c095040 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb33affd640 1 -- 172.21.15.67:0/118040396 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 3494+0+0 (secure 0 0 0) 0x7fb354025020 con 0x5641ca748270 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 10 client.6021 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T15:27:24.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 10 client.6021 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 20 client.6021 populate_metadata read hostname 'smithi067' 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641ca706e40 0x5641cab0e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 4 client.6021 mds_command: new command op to 5880 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 --> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641caa5c350 con 0x5641ca706e40 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.800+0000 7fb35bfff640 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641ca706e40 0x5641cab0e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.801+0000 7fb35bfff640 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641ca706e40 0x5641cab0e520 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f662cf67c8bd6ac server_cookie=6a8891b92e0d7fff in_seq=0 out_seq=0 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.801+0000 7fb33affd640 10 client.6021 ms_handle_connect on v2:172.21.15.67:6832/3185771646 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.801+0000 7fb33affd640 1 -- 172.21.15.67:0/118040396 <== mds.0 v2:172.21.15.67:6832/3185771646 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5641caa5c350 con 0x5641ca706e40 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.801+0000 7fb33affd640 10 client.6021 handle_command_reply: tid=0 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.801+0000 7fb3695f8740 1 client.6021 shutdown 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 msgr2=0x5641cab2f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2f3e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0b8840 tx=0x7fb354007860 comp rx=0 tx=0).stop 2024-04-25T15:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 shutdown_connections 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641ca706e40 0x5641cab0e520 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2f3e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab2eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb33affd640 0 client.6021 ms_handle_reset on v2:172.21.15.67:6832/3185771646 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 --2- 172.21.15.67:0/118040396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca71f3b0 0x5641cab2ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 >> 172.21.15.67:0/118040396 conn(0x5641ca73c3c0 msgr2=0x5641caae8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 shutdown_connections 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 1 -- 172.21.15.67:0/118040396 wait complete. 2024-04-25T15:27:24.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.802+0000 7fb3695f8740 20 client.6021 trim_cache size 0 max 16384 2024-04-25T15:27:24.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.821+0000 7fb3695f8740 1 Processor -- start 2024-04-25T15:27:24.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.821+0000 7fb3695f8740 1 -- start start 2024-04-25T15:27:24.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641caaeae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab17980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca748270 0x5641cab17ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb3695f8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c202080 con 0x5641ca71f3b0 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c0cbec0 con 0x5641ca727590 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c201b80 con 0x5641ca748270 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb35bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641caaeae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb35bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641caaeae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37582/0 (socket says 172.21.15.67:37582) 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb35bfff640 1 -- 172.21.15.67:0/1875042720 learned_addr learned my addr 172.21.15.67:0/1875042720 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.822+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab17980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb360c04640 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca748270 0x5641cab17ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb35bfff640 1 -- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca748270 msgr2=0x5641cab17ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb35bfff640 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca748270 0x5641cab17ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb35bfff640 1 -- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 msgr2=0x5641cab17980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb35bfff640 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab17980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb35bfff640 1 -- 172.21.15.67:0/1875042720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641cab2d420 con 0x5641ca71f3b0 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb360c04640 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca748270 0x5641cab17ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb35bfff640 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641caaeae00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb35002a700 tx=0x7fb35002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744bc7fce54005ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb3417fa640 1 -- 172.21.15.67:0/1875042720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350055070 con 0x5641ca71f3b0 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb3417fa640 1 -- 172.21.15.67:0/1875042720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb350006ca0 con 0x5641ca71f3b0 2024-04-25T15:27:24.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.823+0000 7fb3417fa640 1 -- 172.21.15.67:0/1875042720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350037040 con 0x5641ca71f3b0 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 -- 172.21.15.67:0/1875042720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 msgr2=0x5641caaeae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641caaeae00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb35002a700 tx=0x7fb35002e000 comp rx=0 tx=0).stop 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 -- 172.21.15.67:0/1875042720 shutdown_connections 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca748270 0x5641cab17ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca727590 0x5641cab17980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 --2- 172.21.15.67:0/1875042720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca71f3b0 0x5641caaeae00 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 -- 172.21.15.67:0/1875042720 >> 172.21.15.67:0/1875042720 conn(0x5641ca73c3c0 msgr2=0x5641caae4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 -- 172.21.15.67:0/1875042720 shutdown_connections 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.824+0000 7fb3695f8740 1 -- 172.21.15.67:0/1875042720 wait complete. 2024-04-25T15:27:24.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.825+0000 7fb3695f8740 1 -- start start 2024-04-25T15:27:24.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.825+0000 7fb3695f8740 1 Processor -- start 2024-04-25T15:27:24.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca71f3b0 0x5641caba94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641cab40370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb3695f8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb3695f8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5641cab17650 con 0x5641ca748270 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x5641cab2d2a0 con 0x5641ca71f3b0 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb3695f8740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x5641cab159f0 con 0x5641ca727590 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb3695f8740 10 client.0 _ll_register_callbacks cb 0x7fb3540020a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb360c04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb360c04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37596/0 (socket says 172.21.15.67:37596) 2024-04-25T15:27:24.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb360c04640 1 -- 172.21.15.67:0/2140132744 learned_addr learned my addr 172.21.15.67:0/2140132744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb35b7fe640 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641cab40370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.826+0000 7fb35bfff640 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca71f3b0 0x5641caba94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb360c04640 1 -- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 msgr2=0x5641cab40370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb360c04640 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641cab40370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb360c04640 1 -- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca71f3b0 msgr2=0x5641caba94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb360c04640 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca71f3b0 0x5641caba94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb360c04640 1 -- 172.21.15.67:0/2140132744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641cab167f0 con 0x5641ca748270 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb360c04640 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2ecd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb3540013d0 tx=0x7fb35c1645e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efaf1ec7e56305ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb33affd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb33affd640 1 -- 172.21.15.67:0/2140132744 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35402a030 con 0x5641ca748270 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb33affd640 1 -- 172.21.15.67:0/2140132744 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fb354010040 con 0x5641ca748270 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb350006ca0 con 0x5641ca748270 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb33affd640 1 -- 172.21.15.67:0/2140132744 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb354035050 con 0x5641ca748270 2024-04-25T15:27:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb3695f8740 10 client.6027 fetch_fsmap learned FSMap version 32 2024-04-25T15:27:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.827+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) -- 0x5641cab14bb0 con 0x5641ca748270 2024-04-25T15:27:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb33affd640 1 -- 172.21.15.67:0/2140132744 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb350006ca0 con 0x5641ca748270 2024-04-25T15:27:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb33affd640 1 -- 172.21.15.67:0/2140132744 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 3494+0+0 (secure 0 0 0) 0x7fb354038020 con 0x5641ca748270 2024-04-25T15:27:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb3695f8740 10 client.6027 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T15:27:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb3695f8740 10 client.6027 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:27:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb3695f8740 20 client.6027 populate_metadata read hostname 'smithi067' 2024-04-25T15:27:24.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb3695f8740 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641caae3300 0x5641caae3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb3695f8740 4 client.6027 mds_command: new command op to 5880 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 --> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] -- command(tid 0: {"prefix": "status"}) -- 0x5641cab16f10 con 0x5641caae3300 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb35bfff640 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641caae3300 0x5641caae3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb33affd640 10 client.6027 ms_handle_connect on v2:172.21.15.67:6832/3185771646 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb35bfff640 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641caae3300 0x5641caae3710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ef7a0ab2acea2ae server_cookie=5d354e5638804eb6 in_seq=0 out_seq=0 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.828+0000 7fb33affd640 1 -- 172.21.15.67:0/2140132744 <== mds.0 v2:172.21.15.67:6832/3185771646 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5641cab16f10 con 0x5641caae3300 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb33affd640 10 client.6027 handle_command_reply: tid=0 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 client.6027 shutdown 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 msgr2=0x5641cab2ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2ecd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb3540013d0 tx=0x7fb35c1645e0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 shutdown_connections 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:6832/3185771646,v1:172.21.15.67:6833/3185771646] conn(0x5641caae3300 0x5641caae3710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5641ca748270 0x5641cab2ecd0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5641ca727590 0x5641cab40370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 --2- 172.21.15.67:0/2140132744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5641ca71f3b0 0x5641caba94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb33affd640 0 client.6027 ms_handle_reset on v2:172.21.15.67:6832/3185771646 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 >> 172.21.15.67:0/2140132744 conn(0x5641ca73c3c0 msgr2=0x5641caaeae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 shutdown_connections 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 1 -- 172.21.15.67:0/2140132744 wait complete. 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb3695f8740 20 client.6027 trim_cache size 0 max 16384 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b43d0acb-252a-4f7a-bf14-34d789b94b95", 2024-04-25T15:27:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5880, 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 5.8082675950000002, 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 32, 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 56, 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 56, 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 11.792989196000001 2024-04-25T15:27:24.839 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:27:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3480744b0 msgr2=0x7fb3480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3480744b0 0x7fb3480769c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb350032000 tx=0x7fb350009030 comp rx=0 tx=0).stop 2024-04-25T15:27:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35c0cf590 msgr2=0x7fb35c0ad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35c0cf590 0x7fb35c0ad0e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb34c011d00 tx=0x7fb34c00f610 comp rx=0 tx=0).stop 2024-04-25T15:27:24.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 shutdown_connections 2024-04-25T15:27:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3480744b0 0x7fb3480769c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb35c14e960 0x7fb35c14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb35c0cf590 0x7fb35c0ad0e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 --2- 172.21.15.67:0/542171856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb344003b40 0x7fb35c14e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.829+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 >> 172.21.15.67:0/542171856 conn(0x7fb35c0d2bc0 msgr2=0x7fb35c15efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.830+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 shutdown_connections 2024-04-25T15:27:24.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.830+0000 7fb362667640 1 -- 172.21.15.67:0/542171856 wait complete. 2024-04-25T15:27:24.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:27:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.953+0000 7f4f7774e640 1 Processor -- start 2024-04-25T15:27:24.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.953+0000 7f4f7774e640 1 -- start start 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.954+0000 7f4f7774e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f70161e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.954+0000 7f4f7774e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4f701623c0 0x7f4f701627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.954+0000 7f4f7774e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f700cfc30 con 0x7f4f700ce760 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.954+0000 7f4f7774e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4f700d1ff0 con 0x7f4f701623c0 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.954+0000 7f4f7774e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f700d2890 con 0x7f4f700b91d0 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.954+0000 7f4f75ceb640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4f700ce760 0x7f4f700ceb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56194/0 (socket says 172.21.15.67:56194) 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.954+0000 7f4f75ceb640 1 -- 172.21.15.67:0/400123589 learned_addr learned my addr 172.21.15.67:0/400123589 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f754ea640 1 --2- 172.21.15.67:0/400123589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f70161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f74ce9640 1 --2- 172.21.15.67:0/400123589 >> v2:172.21.15.176:3300/0 conn(0x7f4f701623c0 0x7f4f701627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f6ffff640 1 -- 172.21.15.67:0/400123589 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931904073 0 0) 0x7f4f700cfc30 con 0x7f4f700ce760 2024-04-25T15:27:24.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f6ffff640 1 -- 172.21.15.67:0/400123589 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f58003520 con 0x7f4f700ce760 2024-04-25T15:27:24.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f754ea640 1 -- 172.21.15.67:0/400123589 >> v2:172.21.15.176:3300/0 conn(0x7f4f701623c0 msgr2=0x7f4f701627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:24.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f754ea640 1 --2- 172.21.15.67:0/400123589 >> v2:172.21.15.176:3300/0 conn(0x7f4f701623c0 0x7f4f701627d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f754ea640 1 -- 172.21.15.67:0/400123589 >> v1:172.21.15.67:6789/0 conn(0x7f4f700ce760 legacy=0x7f4f700ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.955+0000 7f4f754ea640 1 -- 172.21.15.67:0/400123589 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f700d2240 con 0x7f4f700b91d0 2024-04-25T15:27:24.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.956+0000 7f4f754ea640 1 --2- 172.21.15.67:0/400123589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f70161e80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4f680097a0 tx=0x7f4f6802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6731223d78bb105f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.956+0000 7f4f6ffff640 1 -- 172.21.15.67:0/400123589 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f68037070 con 0x7f4f700b91d0 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.956+0000 7f4f6ffff640 1 -- 172.21.15.67:0/400123589 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f68032440 con 0x7f4f700b91d0 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.956+0000 7f4f6ffff640 1 -- 172.21.15.67:0/400123589 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f680327f0 con 0x7f4f700b91d0 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 -- 172.21.15.67:0/400123589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 msgr2=0x7f4f70161e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 --2- 172.21.15.67:0/400123589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f70161e80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4f680097a0 tx=0x7f4f6802a180 comp rx=0 tx=0).stop 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 -- 172.21.15.67:0/400123589 shutdown_connections 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 --2- 172.21.15.67:0/400123589 >> v2:172.21.15.176:3300/0 conn(0x7f4f701623c0 0x7f4f701627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 --2- 172.21.15.67:0/400123589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f70161e80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 -- 172.21.15.67:0/400123589 >> 172.21.15.67:0/400123589 conn(0x7f4f700d3aa0 msgr2=0x7f4f700d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 -- 172.21.15.67:0/400123589 shutdown_connections 2024-04-25T15:27:24.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.957+0000 7f4f7774e640 1 -- 172.21.15.67:0/400123589 wait complete. 2024-04-25T15:27:24.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.958+0000 7f4f7774e640 1 Processor -- start 2024-04-25T15:27:24.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.958+0000 7f4f7774e640 1 -- start start 2024-04-25T15:27:24.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f7774e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f700b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f7774e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700ce760 0x7f4f701e8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f7774e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f701623c0 0x7f4f7015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f7774e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f700b2320 con 0x7f4f700ce760 2024-04-25T15:27:24.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f7774e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70162d10 con 0x7f4f701623c0 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f7774e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f700b24a0 con 0x7f4f700b91d0 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700ce760 0x7f4f701e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700ce760 0x7f4f701e8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37602/0 (socket says 172.21.15.67:37602) 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 -- 172.21.15.67:0/3877718838 learned_addr learned my addr 172.21.15.67:0/3877718838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f754ea640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f700b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f75ceb640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f701623c0 0x7f4f7015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 -- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 msgr2=0x7f4f700b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f700b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 -- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f701623c0 msgr2=0x7f4f7015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f701623c0 0x7f4f7015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:24.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 -- 172.21.15.67:0/3877718838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f701e8e50 con 0x7f4f700ce760 2024-04-25T15:27:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.960+0000 7f4f74ce9640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700ce760 0x7f4f701e8640 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400e840 tx=0x7f4f6400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.961+0000 7f4f6dffb640 1 -- 172.21.15.67:0/3877718838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f64017070 con 0x7f4f700ce760 2024-04-25T15:27:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.961+0000 7f4f6dffb640 1 -- 172.21.15.67:0/3877718838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f6400c6d0 con 0x7f4f700ce760 2024-04-25T15:27:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.961+0000 7f4f6dffb640 1 -- 172.21.15.67:0/3877718838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400cac0 con 0x7f4f700ce760 2024-04-25T15:27:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.961+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f701e8b80 con 0x7f4f700ce760 2024-04-25T15:27:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.961+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f700c7950 con 0x7f4f700ce760 2024-04-25T15:27:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.962+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f70050740 con 0x7f4f700ce760 2024-04-25T15:27:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.963+0000 7f4f6dffb640 1 -- 172.21.15.67:0/3877718838 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4f6401a070 con 0x7f4f700ce760 2024-04-25T15:27:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.963+0000 7f4f6dffb640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f5c0744e0 0x7f4f5c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.963+0000 7f4f6dffb640 1 -- 172.21.15.67:0/3877718838 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f4f64094a70 con 0x7f4f700ce760 2024-04-25T15:27:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.963+0000 7f4f754ea640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f5c0744e0 0x7f4f5c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.964+0000 7f4f754ea640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f5c0744e0 0x7f4f5c0769f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4f68034000 tx=0x7f4f6802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:24.968+0000 7f4f6dffb640 1 -- 172.21.15.67:0/3877718838 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f701e8e50 con 0x7f4f700ce760 2024-04-25T15:27:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.194+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4f700b5700 con 0x7f4f700ce760 2024-04-25T15:27:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.195+0000 7f4f6dffb640 1 -- 172.21.15.67:0/3877718838 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) ==== 53+0+73 (secure 0 0 0) 0x7f4f6405ec40 con 0x7f4f700ce760 2024-04-25T15:27:25.196 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f5c0744e0 msgr2=0x7f4f5c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f5c0744e0 0x7f4f5c0769f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4f68034000 tx=0x7f4f6802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700ce760 msgr2=0x7f4f701e8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700ce760 0x7f4f701e8640 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400e840 tx=0x7f4f6400ed00 comp rx=0 tx=0).stop 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 shutdown_connections 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f5c0744e0 0x7f4f5c0769f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f701623c0 0x7f4f7015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700ce760 0x7f4f701e8640 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 --2- 172.21.15.67:0/3877718838 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f700b91d0 0x7f4f700b4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.196+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 >> 172.21.15.67:0/3877718838 conn(0x7f4f700d3aa0 msgr2=0x7f4f70072920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.197+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 shutdown_connections 2024-04-25T15:27:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25.197+0000 7f4f7774e640 1 -- 172.21.15.67:0/3877718838 wait complete. 2024-04-25T15:27:25.208 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:27:25.208 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:27:25.209 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:27:25.209 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:27:25.209 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:25.209 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:27:25.209 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:27:25.209 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:27:25.209 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:27:25.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:25.210 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:25.226 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42654sec preferred_lft 42654sec 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:25.227 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7482:42ff:fe30:5ffc/64 scope link 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::885:bff:fe04:f6cb/64 scope link 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ca:af:51:1f:d5:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8af:51ff:fe1f:d5b0/64 scope link 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:25.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:25.228 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:27:25.290 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:27:25.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:25.291 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:27:25.347 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:27:25.347 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:27:25.348 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:27:25.348 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:27:25.348 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:27:25.348 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:27:25.348 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:27:25.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:27:25.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:27:25.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:27:25.348 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:27:25.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:27:25.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:27:25.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:25.474 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:27:25.489 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:27:25.490 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:27:25.490 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:27:25.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:25.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:27:25.553 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:27:25.554 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:25.608 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:27:25.609 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:27:25.609 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:25.663 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T15:27:25.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:25.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:25.758 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:25.758 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:27:25.766 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:25.766 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:27:25.766 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:27:25.766 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:27:25.766 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:27:25.767 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T15:27:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:27:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:27:25.767 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:27:25.768 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:27:25.769 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:27:25.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:25.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:25.793 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:27:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:27:25.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:27:25.895 INFO:teuthology.orchestra.run.smithi067.stdout:client6036 2024-04-25T15:27:25.896 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:25.896 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:25.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:27:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.064+0000 7fbcb1253640 1 Processor -- start 2024-04-25T15:27:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.064+0000 7fbcb1253640 1 -- start start 2024-04-25T15:27:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.065+0000 7fbcb1253640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbcac15df80 0x7fbcac160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.065+0000 7fbcb1253640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcac1615f0 con 0x7fbcac0b8290 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.065+0000 7fbcb1253640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcac164020 con 0x7fbcac164b00 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.065+0000 7fbcb1253640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbcac164670 con 0x7fbcac15df80 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbcaa59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcac0b8290 0x7fbcac160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56198/0 (socket says 172.21.15.67:56198) 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbcaa59c640 1 -- 172.21.15.67:0/2469597711 learned_addr learned my addr 172.21.15.67:0/2469597711 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbcab59e640 1 --2- 172.21.15.67:0/2469597711 >> v2:172.21.15.176:3300/0 conn(0x7fbcac15df80 0x7fbcac160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3404707934 0 0) 0x7fbcac1615f0 con 0x7fbcac0b8290 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc94003520 con 0x7fbcac0b8290 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3511741777 0 0) 0x7fbc94003520 con 0x7fbcac0b8290 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 >> v2:172.21.15.176:3300/0 conn(0x7fbcac15df80 msgr2=0x7fbcac160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 --2- 172.21.15.67:0/2469597711 >> v2:172.21.15.176:3300/0 conn(0x7fbcac15df80 0x7fbcac160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 >> v1:172.21.15.176:6789/0 conn(0x7fbcac164b00 legacy=0x7fbcac0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac164300 con 0x7fbcac0b8290 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fbca0001670 con 0x7fbcac0b8290 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 >> v1:172.21.15.67:6789/0 conn(0x7fbcac0b8290 legacy=0x7fbcac160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc940039b0 0x7fbc94003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94004800 0x7fbc94004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc94005270 0x7fbc9400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac164670 con 0x7fbc94005270 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9400c190 con 0x7fbc94004800 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9400c390 con 0x7fbc940039b0 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbcab59e640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc94005270 0x7fbc9400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbcaa59c640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94004800 0x7fbc94004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.066+0000 7fbcaad9d640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc940039b0 0x7fbc94003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.067+0000 7fbcab59e640 1 -- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc940039b0 msgr2=0x7fbc94003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.067+0000 7fbcab59e640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc940039b0 0x7fbc94003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.067+0000 7fbcab59e640 1 -- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94004800 msgr2=0x7fbc94004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.067+0000 7fbcab59e640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94004800 0x7fbc94004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.067+0000 7fbcab59e640 1 -- 172.21.15.67:0/2469597711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc94003520 con 0x7fbc94005270 2024-04-25T15:27:26.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.067+0000 7fbcab59e640 1 -- 172.21.15.67:0/2469597711 reap_dead start 2024-04-25T15:27:26.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.067+0000 7fbcab59e640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc94005270 0x7fbc9400b9c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009f90 tx=0x7fbc94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4724881c42693ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:26.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c009c10 con 0x7fbc94005270 2024-04-25T15:27:26.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9c031440 con 0x7fbc94005270 2024-04-25T15:27:26.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbca9d9b640 1 -- 172.21.15.67:0/2469597711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c031810 con 0x7fbc94005270 2024-04-25T15:27:26.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbcb1253640 1 -- 172.21.15.67:0/2469597711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc94005270 msgr2=0x7fbc9400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:26.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbcb1253640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc94005270 0x7fbc9400b9c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c009f90 tx=0x7fbc94005830 comp rx=0 tx=0).stop 2024-04-25T15:27:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbcb1253640 1 -- 172.21.15.67:0/2469597711 shutdown_connections 2024-04-25T15:27:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbcb1253640 1 --2- 172.21.15.67:0/2469597711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc94005270 0x7fbc9400b9c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbcb1253640 1 -- 172.21.15.67:0/2469597711 >> 172.21.15.67:0/2469597711 conn(0x7fbcac0d2bc0 msgr2=0x7fbcac0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbcb1253640 1 -- 172.21.15.67:0/2469597711 shutdown_connections 2024-04-25T15:27:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.068+0000 7fbcb1253640 1 -- 172.21.15.67:0/2469597711 wait complete. 2024-04-25T15:27:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.070+0000 7fbcb1253640 1 Processor -- start 2024-04-25T15:27:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.070+0000 7fbcb1253640 1 -- start start 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcb1253640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94005270 0x7fbcac158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcb1253640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcac156a50 0x7fbcac0c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcb1253640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcac155eb0 0x7fbcac1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcb1253640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac0b7f10 con 0x7fbcac155eb0 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcb1253640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbcac0b7d90 con 0x7fbc94005270 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcb1253640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbcac0b8090 con 0x7fbcac156a50 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcac155eb0 0x7fbcac1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcac155eb0 0x7fbcac1562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37622/0 (socket says 172.21.15.67:37622) 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 -- 172.21.15.67:0/3914936349 learned_addr learned my addr 172.21.15.67:0/3914936349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcaad9d640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94005270 0x7fbcac158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcaa59c640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcac156a50 0x7fbcac0c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 -- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcac156a50 msgr2=0x7fbcac0c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcac156a50 0x7fbcac0c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 -- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94005270 msgr2=0x7fbcac158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94005270 0x7fbcac158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 -- 172.21.15.67:0/3914936349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcac156800 con 0x7fbcac155eb0 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.072+0000 7fbcab59e640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcac155eb0 0x7fbcac1562c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00e230 tx=0x7fbc9c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.073+0000 7fbc937fe640 1 -- 172.21.15.67:0/3914936349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c009c10 con 0x7fbcac155eb0 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.073+0000 7fbc937fe640 1 -- 172.21.15.67:0/3914936349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9c043070 con 0x7fbcac155eb0 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.073+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcac1543c0 con 0x7fbcac155eb0 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.073+0000 7fbc937fe640 1 -- 172.21.15.67:0/3914936349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c03c8b0 con 0x7fbcac155eb0 2024-04-25T15:27:26.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.073+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcac155b00 con 0x7fbcac155eb0 2024-04-25T15:27:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.074+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcac050740 con 0x7fbcac155eb0 2024-04-25T15:27:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.074+0000 7fbc937fe640 1 -- 172.21.15.67:0/3914936349 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbc9c005d10 con 0x7fbcac155eb0 2024-04-25T15:27:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.075+0000 7fbc937fe640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc980744e0 0x7fbc980769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.075+0000 7fbc937fe640 1 -- 172.21.15.67:0/3914936349 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fbc9c0b5e40 con 0x7fbcac155eb0 2024-04-25T15:27:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.075+0000 7fbcaad9d640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc980744e0 0x7fbc980769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:26.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.076+0000 7fbcaad9d640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc980744e0 0x7fbc980769f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbca4000c00 tx=0x7fbca4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:26.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.080+0000 7fbc937fe640 1 -- 172.21.15.67:0/3914936349 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbcac156800 con 0x7fbcac155eb0 2024-04-25T15:27:26.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.308+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbcac0d69e0 con 0x7fbcac155eb0 2024-04-25T15:27:26.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.309+0000 7fbc937fe640 1 -- 172.21.15.67:0/3914936349 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) ==== 53+0+73 (secure 0 0 0) 0x7fbc9c0805d0 con 0x7fbcac155eb0 2024-04-25T15:27:26.310 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.310+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc980744e0 msgr2=0x7fbc980769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.310+0000 7fbcb1253640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc980744e0 0x7fbc980769f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbca4000c00 tx=0x7fbca4009040 comp rx=0 tx=0).stop 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.310+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcac155eb0 msgr2=0x7fbcac1562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.310+0000 7fbcb1253640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcac155eb0 0x7fbcac1562c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00e230 tx=0x7fbc9c009750 comp rx=0 tx=0).stop 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 shutdown_connections 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc980744e0 0x7fbc980769f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcac155eb0 0x7fbcac1562c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcac156a50 0x7fbcac0c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 --2- 172.21.15.67:0/3914936349 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc94005270 0x7fbcac158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 >> 172.21.15.67:0/3914936349 conn(0x7fbcac0d2bc0 msgr2=0x7fbcac0d5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 shutdown_connections 2024-04-25T15:27:26.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:26.311+0000 7fbcb1253640 1 -- 172.21.15.67:0/3914936349 wait complete. 2024-04-25T15:27:26.322 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:27:26.323 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:27:26.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:26.323 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:27:26.342 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:27:26.342 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:27:26.342 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:26.343 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42642sec preferred_lft 42642sec 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:27:26.344 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::248e:2bff:fe69:7f63/64 scope link 2024-04-25T15:27:26.345 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:26.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:26.345 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:27:26.408 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:27:26.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:26.410 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:27:26.467 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:27:26.467 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:27:26.468 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:27:26.468 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:27:26.468 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:27:26.468 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:27:26.468 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:27:26.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:27:26.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:27:26.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:27:26.468 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:27:26.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:27:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:27:26.590 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:27:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:27:26.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:26.596 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:27:26.611 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:27:26.611 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:27:26.611 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:27:26.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:26.612 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:27:26.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:27:26.675 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:27:26.730 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:27:26.730 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:27:26.730 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:27:26.784 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T15:27:26.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:26.785 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:26.876 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:26.877 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:27:26.889 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:26.889 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:27:26.889 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:27:26.890 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:27:26.890 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:27:26.890 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:27:26.890 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:27:26.890 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:27:26.890 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:27:26.890 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:27:26.891 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:27:26.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:26.892 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:27:26.914 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:27:27.002 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:27:27 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:27:27.004 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:27:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:27:27.009 INFO:teuthology.orchestra.run.smithi176.stdout:client6045 2024-04-25T15:27:27.010 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:27.010 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:27.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.025 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-25T15:27:27.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.048 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:27:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-04-25T15:27:27.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.107 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10K ./subdir) 2024-04-25T15:27:27.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.166 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.224 INFO:teuthology.orchestra.run.smithi067.stdout:10240 2024-04-25T15:27:27.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.224 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 100Ki ./subdir) 2024-04-25T15:27:27.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.284 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.340 INFO:teuthology.orchestra.run.smithi067.stdout:102400 2024-04-25T15:27:27.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.340 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10M ./subdir) 2024-04-25T15:27:27.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.400 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.457 INFO:teuthology.orchestra.run.smithi067.stdout:10485760 2024-04-25T15:27:27.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.458 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 100Mi ./subdir) 2024-04-25T15:27:27.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.517 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.573 INFO:teuthology.orchestra.run.smithi067.stdout:104857600 2024-04-25T15:27:27.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.573 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 2G ./subdir) 2024-04-25T15:27:27.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.632 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.689 INFO:teuthology.orchestra.run.smithi067.stdout:2147483648 2024-04-25T15:27:27.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.689 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 4Gi ./subdir) 2024-04-25T15:27:27.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.747 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.804 INFO:teuthology.orchestra.run.smithi067.stdout:4294967296 2024-04-25T15:27:27.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.804 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1T ./subdir) 2024-04-25T15:27:27.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.863 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:27.920 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-04-25T15:27:27.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.920 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 2Ti ./subdir) 2024-04-25T15:27:27.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:27.979 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-04-25T15:27:28.034 INFO:teuthology.orchestra.run.smithi067.stdout:2199023255552 2024-04-25T15:27:28.034 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:27:28.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:27:28.113 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:27:28.156 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:27:28.157 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:27:28.157 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:28.157 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:28.172 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:28.172 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:28.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.379+0000 7fcfd1261640 1 Processor -- start 2024-04-25T15:27:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.379+0000 7fcfd1261640 1 -- start start 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.380+0000 7fcfd1261640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.380+0000 7fcfd1261640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcfcc0b2dc0 0x7fcfcc0b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.380+0000 7fcfd1261640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfcc06d870 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.380+0000 7fcfd1261640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcfcc06e460 con 0x7fcfcc0b2dc0 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.380+0000 7fcfd1261640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcfcc06db10 con 0x7fcfcc0d4570 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcaffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcb7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcfcc0b2dc0 0x7fcfcc0b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfca7fc640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fcfcc06e8f0 0x7fcfcc0d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:41012/0 (socket says 172.21.15.67:41012) 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcaffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc0b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60932/0 (socket says 172.21.15.67:60932) 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcb7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcfcc0b2dc0 0x7fcfcc0b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:36140/0 (socket says 172.21.15.67:36140) 2024-04-25T15:27:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfca7fc640 1 -- 172.21.15.67:0/1679329069 learned_addr learned my addr 172.21.15.67:0/1679329069 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcaffd640 1 -- 172.21.15.67:0/1679329069 >> v2:172.21.15.176:3300/0 conn(0x7fcfcc0b2dc0 msgr2=0x7fcfcc0b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcaffd640 1 --2- 172.21.15.67:0/1679329069 >> v2:172.21.15.176:3300/0 conn(0x7fcfcc0b2dc0 0x7fcfcc0b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcaffd640 1 -- 172.21.15.67:0/1679329069 >> v1:172.21.15.176:6789/0 conn(0x7fcfcc06e8f0 legacy=0x7fcfcc0d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.381+0000 7fcfcaffd640 1 -- 172.21.15.67:0/1679329069 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc06e0f0 con 0x7fcfcc0d4570 2024-04-25T15:27:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.382+0000 7fcfcaffd640 1 --2- 172.21.15.67:0/1679329069 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc0b4830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc06e5e0 tx=0x7fcfbc02a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb55583d72e05ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.382+0000 7fcfc9ffb640 1 -- 172.21.15.67:0/1679329069 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc037070 con 0x7fcfcc0d4570 2024-04-25T15:27:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.382+0000 7fcfc9ffb640 1 -- 172.21.15.67:0/1679329069 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfbc02ae20 con 0x7fcfcc0d4570 2024-04-25T15:27:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.382+0000 7fcfc9ffb640 1 -- 172.21.15.67:0/1679329069 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc032660 con 0x7fcfcc0d4570 2024-04-25T15:27:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 -- 172.21.15.67:0/1679329069 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 msgr2=0x7fcfcc0b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 --2- 172.21.15.67:0/1679329069 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc0b4830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc06e5e0 tx=0x7fcfbc02a0d0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 -- 172.21.15.67:0/1679329069 shutdown_connections 2024-04-25T15:27:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 --2- 172.21.15.67:0/1679329069 >> v2:172.21.15.176:3300/0 conn(0x7fcfcc0b2dc0 0x7fcfcc0b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 --2- 172.21.15.67:0/1679329069 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc0b4830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 -- 172.21.15.67:0/1679329069 >> 172.21.15.67:0/1679329069 conn(0x7fcfcc0d3a50 msgr2=0x7fcfcc0d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 -- 172.21.15.67:0/1679329069 shutdown_connections 2024-04-25T15:27:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.383+0000 7fcfd1261640 1 -- 172.21.15.67:0/1679329069 wait complete. 2024-04-25T15:27:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.385+0000 7fcfd1261640 1 Processor -- start 2024-04-25T15:27:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.385+0000 7fcfd1261640 1 -- start start 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.386+0000 7fcfd1261640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfcc06e8f0 0x7fcfcc0b7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.386+0000 7fcfd1261640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfcc0b2dc0 0x7fcfcc0a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.386+0000 7fcfd1261640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc160d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.386+0000 7fcfd1261640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc06ed00 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.386+0000 7fcfd1261640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc0d0fb0 con 0x7fcfcc0b2dc0 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.386+0000 7fcfd1261640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcfcc0b3710 con 0x7fcfcc0d4570 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfcc06e8f0 0x7fcfcc0b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfcc06e8f0 0x7fcfcc0b7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49916/0 (socket says 172.21.15.67:49916) 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 -- 172.21.15.67:0/4217642708 learned_addr learned my addr 172.21.15.67:0/4217642708 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfca7fc640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfcc0b2dc0 0x7fcfcc0a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcb7fe640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 -- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 msgr2=0x7fcfcc160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc160d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 -- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfcc0b2dc0 msgr2=0x7fcfcc0a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfcc0b2dc0 0x7fcfcc0a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 -- 172.21.15.67:0/4217642708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc0a1090 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfcaffd640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfcc06e8f0 0x7fcfcc0b7590 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009880 tx=0x7fcfbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfb37fe640 1 -- 172.21.15.67:0/4217642708 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc041020 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfb37fe640 1 -- 172.21.15.67:0/4217642708 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfbc03c670 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfb37fe640 1 -- 172.21.15.67:0/4217642708 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc037040 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.387+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfcc161240 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.388+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfcc0b6ab0 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.388+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfcc050740 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.389+0000 7fcfb37fe640 1 -- 172.21.15.67:0/4217642708 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcfbc044020 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.390+0000 7fcfb37fe640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcfb80744b0 0x7fcfb80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.390+0000 7fcfb37fe640 1 -- 172.21.15.67:0/4217642708 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fcfbc0b57c0 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.390+0000 7fcfca7fc640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcfb80744b0 0x7fcfb80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.391+0000 7fcfca7fc640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcfb80744b0 0x7fcfb80769c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0d2ec0 tx=0x7fcfc0007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.395+0000 7fcfb37fe640 1 -- 172.21.15.67:0/4217642708 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfbc04b030 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.581+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcfcc0d1220 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.582+0000 7fcfb37fe640 1 -- 172.21.15.67:0/4217642708 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16629 (secure 0 0 0) 0x7fcfbc02ae20 con 0x7fcfcc06e8f0 2024-04-25T15:27:28.583 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:28.583 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:20.019287+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcfb80744b0 msgr2=0x7fcfb80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcfb80744b0 0x7fcfb80769c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0d2ec0 tx=0x7fcfc0007e80 comp rx=0 tx=0).stop 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfcc06e8f0 msgr2=0x7fcfcc0b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfcc06e8f0 0x7fcfcc0b7590 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009880 tx=0x7fcfbc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 shutdown_connections 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcfb80744b0 0x7fcfb80769c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfcc0d4570 0x7fcfcc160d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfcc0b2dc0 0x7fcfcc0a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 --2- 172.21.15.67:0/4217642708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfcc06e8f0 0x7fcfcc0b7590 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 >> 172.21.15.67:0/4217642708 conn(0x7fcfcc0d3a50 msgr2=0x7fcfcc0cf740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:28.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 shutdown_connections 2024-04-25T15:27:28.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.584+0000 7fcfd1261640 1 -- 172.21.15.67:0/4217642708 wait complete. 2024-04-25T15:27:28.596 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:27:16.061447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:27:17.338750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:28.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:28.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.711+0000 7f9f46492640 1 Processor -- start 2024-04-25T15:27:28.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.711+0000 7f9f46492640 1 -- start start 2024-04-25T15:27:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.712+0000 7f9f46492640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9f40164ad0 0x7f9f400b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.712+0000 7f9f46492640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f40163f30 0x7f9f40164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f46492640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f40164880 con 0x7f9f400d1f30 2024-04-25T15:27:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f46492640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9f40164ee0 con 0x7f9f40163f30 2024-04-25T15:27:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f46492640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9f400b31e0 con 0x7f9f40164ad0 2024-04-25T15:27:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f44a2f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9f400d1f30 0x7f9f400d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49616/0 (socket says 172.21.15.67:49616) 2024-04-25T15:27:28.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f3f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f40163f30 0x7f9f40164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f44a2f640 1 -- 172.21.15.67:0/2837514588 learned_addr learned my addr 172.21.15.67:0/2837514588 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:28.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f3ffff640 1 --2- 172.21.15.67:0/2837514588 >> v2:172.21.15.176:3300/0 conn(0x7f9f40164ad0 0x7f9f400b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f3f7fe640 1 -- 172.21.15.67:0/2837514588 >> v2:172.21.15.176:3300/0 conn(0x7f9f40164ad0 msgr2=0x7f9f400b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:28.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f3f7fe640 1 --2- 172.21.15.67:0/2837514588 >> v2:172.21.15.176:3300/0 conn(0x7f9f40164ad0 0x7f9f400b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f3f7fe640 1 -- 172.21.15.67:0/2837514588 >> v1:172.21.15.67:6789/0 conn(0x7f9f400d1f30 legacy=0x7f9f400d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f3effd640 1 -- 172.21.15.67:0/2837514588 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 879505834 0 0) 0x7f9f38001490 con 0x7f9f400d1f30 2024-04-25T15:27:28.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.713+0000 7f9f3f7fe640 1 -- 172.21.15.67:0/2837514588 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f400d3170 con 0x7f9f40163f30 2024-04-25T15:27:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.714+0000 7f9f3f7fe640 1 --2- 172.21.15.67:0/2837514588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40163f30 0x7f9f40164340 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9f40074530 tx=0x7f9f3002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4e7b4dcbce833 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:28.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.714+0000 7f9f3effd640 1 -- 172.21.15.67:0/2837514588 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f30037070 con 0x7f9f40163f30 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.714+0000 7f9f3effd640 1 -- 172.21.15.67:0/2837514588 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f30032440 con 0x7f9f40163f30 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.714+0000 7f9f3effd640 1 -- 172.21.15.67:0/2837514588 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f30032810 con 0x7f9f40163f30 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 -- 172.21.15.67:0/2837514588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40163f30 msgr2=0x7f9f40164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 --2- 172.21.15.67:0/2837514588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40163f30 0x7f9f40164340 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9f40074530 tx=0x7f9f3002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 -- 172.21.15.67:0/2837514588 shutdown_connections 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 --2- 172.21.15.67:0/2837514588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40163f30 0x7f9f40164340 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 --2- 172.21.15.67:0/2837514588 >> v2:172.21.15.176:3300/0 conn(0x7f9f40164ad0 0x7f9f400b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 -- 172.21.15.67:0/2837514588 >> 172.21.15.67:0/2837514588 conn(0x7f9f400d3aa0 msgr2=0x7f9f400d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 -- 172.21.15.67:0/2837514588 shutdown_connections 2024-04-25T15:27:28.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.715+0000 7f9f46492640 1 -- 172.21.15.67:0/2837514588 wait complete. 2024-04-25T15:27:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.717+0000 7f9f46492640 1 Processor -- start 2024-04-25T15:27:28.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.717+0000 7f9f46492640 1 -- start start 2024-04-25T15:27:28.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f46492640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f400d1f30 0x7f9f400b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f46492640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f40163f30 0x7f9f401ea770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f46492640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40164ad0 0x7f9f401fa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f46492640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4006d9b0 con 0x7f9f40164ad0 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f46492640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4006d6d0 con 0x7f9f40163f30 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f46492640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9f400b9430 con 0x7f9f400d1f30 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f44a2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40164ad0 0x7f9f401fa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f44a2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40164ad0 0x7f9f401fa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49930/0 (socket says 172.21.15.67:49930) 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f44a2f640 1 -- 172.21.15.67:0/401984075 learned_addr learned my addr 172.21.15.67:0/401984075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f3f7fe640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f40163f30 0x7f9f401ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f3ffff640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f400d1f30 0x7f9f400b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f44a2f640 1 -- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f400d1f30 msgr2=0x7f9f400b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.718+0000 7f9f44a2f640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f400d1f30 0x7f9f400b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f44a2f640 1 -- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f40163f30 msgr2=0x7f9f401ea770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f44a2f640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f40163f30 0x7f9f401ea770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f44a2f640 1 -- 172.21.15.67:0/401984075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4015d710 con 0x7f9f40164ad0 2024-04-25T15:27:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f44a2f640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40164ad0 0x7f9f401fa0e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800c700 tx=0x7f9f3800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f3cff9640 1 -- 172.21.15.67:0/401984075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f38018070 con 0x7f9f40164ad0 2024-04-25T15:27:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f3cff9640 1 -- 172.21.15.67:0/401984075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3800ce10 con 0x7f9f40164ad0 2024-04-25T15:27:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f40163200 con 0x7f9f40164ad0 2024-04-25T15:27:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f3cff9640 1 -- 172.21.15.67:0/401984075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800a800 con 0x7f9f40164ad0 2024-04-25T15:27:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.719+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f4015dc90 con 0x7f9f40164ad0 2024-04-25T15:27:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.720+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f400b6210 con 0x7f9f40164ad0 2024-04-25T15:27:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.721+0000 7f9f3cff9640 1 -- 172.21.15.67:0/401984075 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9f3801b070 con 0x7f9f40164ad0 2024-04-25T15:27:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.721+0000 7f9f3cff9640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f2c0744b0 0x7f9f2c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.721+0000 7f9f3cff9640 1 -- 172.21.15.67:0/401984075 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f9f380946c0 con 0x7f9f40164ad0 2024-04-25T15:27:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.721+0000 7f9f3ffff640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f2c0744b0 0x7f9f2c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.722+0000 7f9f3ffff640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f2c0744b0 0x7f9f2c0769c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9f400746e0 tx=0x7f9f34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.726+0000 7f9f3cff9640 1 -- 172.21.15.67:0/401984075 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f40164880 con 0x7f9f40164ad0 2024-04-25T15:27:28.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.911+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9f400b64c0 con 0x7f9f40164ad0 2024-04-25T15:27:28.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.912+0000 7f9f3cff9640 1 -- 172.21.15.67:0/401984075 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f9f3805eee0 con 0x7f9f40164ad0 2024-04-25T15:27:28.912 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.913+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f2c0744b0 msgr2=0x7f9f2c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.913+0000 7f9f46492640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f2c0744b0 0x7f9f2c0769c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9f400746e0 tx=0x7f9f34008040 comp rx=0 tx=0).stop 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.913+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40164ad0 msgr2=0x7f9f401fa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.913+0000 7f9f46492640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40164ad0 0x7f9f401fa0e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800c700 tx=0x7f9f3800cbc0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 shutdown_connections 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9f2c0744b0 0x7f9f2c0769c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f40164ad0 0x7f9f401fa0e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f40163f30 0x7f9f401ea770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 --2- 172.21.15.67:0/401984075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f400d1f30 0x7f9f400b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 >> 172.21.15.67:0/401984075 conn(0x7f9f400d3aa0 msgr2=0x7f9f4015ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 shutdown_connections 2024-04-25T15:27:28.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:28.914+0000 7f9f46492640 1 -- 172.21.15.67:0/401984075 wait complete. 2024-04-25T15:27:28.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:27:29.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.040+0000 7fcbece9a640 1 Processor -- start 2024-04-25T15:27:29.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.040+0000 7fcbece9a640 1 -- start start 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbece9a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbe80b6900 0x7fcbe80b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbece9a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe80ce670 con 0x7fcbe80d20c0 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbece9a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe808d9a0 con 0x7fcbe80b4e70 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbece9a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe808d770 con 0x7fcbe80b6900 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbe6d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcbe80d20c0 0x7fcbe8161df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49618/0 (socket says 172.21.15.67:49618) 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbe659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbe80b6900 0x7fcbe80b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbe6d9d640 1 -- 172.21.15.67:0/152960978 learned_addr learned my addr 172.21.15.67:0/152960978 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbe659c640 1 -- 172.21.15.67:0/152960978 >> v1:172.21.15.176:6789/0 conn(0x7fcbe80b4e70 legacy=0x7fcbe80d19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbe659c640 1 -- 172.21.15.67:0/152960978 >> v1:172.21.15.67:6789/0 conn(0x7fcbe80d20c0 legacy=0x7fcbe8161df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.042+0000 7fcbe659c640 1 -- 172.21.15.67:0/152960978 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe808dc60 con 0x7fcbe80b6900 2024-04-25T15:27:29.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.043+0000 7fcbe659c640 1 --2- 172.21.15.67:0/152960978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b6900 0x7fcbe80b8ab0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00097a0 tx=0x7fcbe002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d725d8b2926897 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.043+0000 7fcbe559a640 1 -- 172.21.15.67:0/152960978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0037070 con 0x7fcbe80b6900 2024-04-25T15:27:29.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.043+0000 7fcbe559a640 1 -- 172.21.15.67:0/152960978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe0032440 con 0x7fcbe80b6900 2024-04-25T15:27:29.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.043+0000 7fcbe559a640 1 -- 172.21.15.67:0/152960978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0032810 con 0x7fcbe80b6900 2024-04-25T15:27:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.044+0000 7fcbece9a640 1 -- 172.21.15.67:0/152960978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b6900 msgr2=0x7fcbe80b8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.044+0000 7fcbece9a640 1 --2- 172.21.15.67:0/152960978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b6900 0x7fcbe80b8ab0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00097a0 tx=0x7fcbe002a230 comp rx=0 tx=0).stop 2024-04-25T15:27:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.044+0000 7fcbece9a640 1 -- 172.21.15.67:0/152960978 shutdown_connections 2024-04-25T15:27:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.044+0000 7fcbece9a640 1 --2- 172.21.15.67:0/152960978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b6900 0x7fcbe80b8ab0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.044+0000 7fcbece9a640 1 -- 172.21.15.67:0/152960978 >> 172.21.15.67:0/152960978 conn(0x7fcbe80d0530 msgr2=0x7fcbe80d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.044+0000 7fcbece9a640 1 -- 172.21.15.67:0/152960978 shutdown_connections 2024-04-25T15:27:29.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.044+0000 7fcbece9a640 1 -- 172.21.15.67:0/152960978 wait complete. 2024-04-25T15:27:29.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.046+0000 7fcbece9a640 1 Processor -- start 2024-04-25T15:27:29.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.046+0000 7fcbece9a640 1 -- start start 2024-04-25T15:27:29.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbece9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b4e70 0x7fcbe80cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbece9a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe80b6900 0x7fcbe80cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbece9a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe80d20c0 0x7fcbe80cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbece9a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe80b5280 con 0x7fcbe80b4e70 2024-04-25T15:27:29.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbece9a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe80ce980 con 0x7fcbe80d20c0 2024-04-25T15:27:29.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbece9a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe80b5400 con 0x7fcbe80b6900 2024-04-25T15:27:29.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbe659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b4e70 0x7fcbe80cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbe659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b4e70 0x7fcbe80cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49942/0 (socket says 172.21.15.67:49942) 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbe659c640 1 -- 172.21.15.67:0/1189162084 learned_addr learned my addr 172.21.15.67:0/1189162084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbe6d9d640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe80d20c0 0x7fcbe80cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.047+0000 7fcbe5d9b640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe80b6900 0x7fcbe80cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbe659c640 1 -- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe80b6900 msgr2=0x7fcbe80cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbe659c640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe80b6900 0x7fcbe80cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbe659c640 1 -- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe80d20c0 msgr2=0x7fcbe80cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbe659c640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe80d20c0 0x7fcbe80cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbe659c640 1 -- 172.21.15.67:0/1189162084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe815aa90 con 0x7fcbe80b4e70 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbe659c640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b4e70 0x7fcbe80cd490 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0035040 tx=0x7fcbe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbd6ffd640 1 -- 172.21.15.67:0/1189162084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0037040 con 0x7fcbe80b4e70 2024-04-25T15:27:29.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbd6ffd640 1 -- 172.21.15.67:0/1189162084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe0030770 con 0x7fcbe80b4e70 2024-04-25T15:27:29.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe815ad70 con 0x7fcbe80b4e70 2024-04-25T15:27:29.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbd6ffd640 1 -- 172.21.15.67:0/1189162084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0030b40 con 0x7fcbe80b4e70 2024-04-25T15:27:29.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.048+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe815b330 con 0x7fcbe80b4e70 2024-04-25T15:27:29.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.049+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe80b98f0 con 0x7fcbe80b4e70 2024-04-25T15:27:29.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.050+0000 7fcbd6ffd640 1 -- 172.21.15.67:0/1189162084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcbe0044020 con 0x7fcbe80b4e70 2024-04-25T15:27:29.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.050+0000 7fcbd6ffd640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbc80744b0 0x7fcbc80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.050+0000 7fcbd6ffd640 1 -- 172.21.15.67:0/1189162084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fcbe00b5810 con 0x7fcbe80b4e70 2024-04-25T15:27:29.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.050+0000 7fcbe5d9b640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbc80744b0 0x7fcbc80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.051+0000 7fcbe5d9b640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbc80744b0 0x7fcbc80769c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc007fd0 tx=0x7fcbdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.055+0000 7fcbd6ffd640 1 -- 172.21.15.67:0/1189162084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcbe80b98f0 con 0x7fcbe80b4e70 2024-04-25T15:27:29.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:27:29.064 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:27:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.241+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcbe808d070 con 0x7fcbe80b4e70 2024-04-25T15:27:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.241+0000 7fcbd6ffd640 1 -- 172.21.15.67:0/1189162084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7fcbe808d070 con 0x7fcbe80b4e70 2024-04-25T15:27:29.243 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:27:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbc80744b0 msgr2=0x7fcbc80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbc80744b0 0x7fcbc80769c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc007fd0 tx=0x7fcbdc009040 comp rx=0 tx=0).stop 2024-04-25T15:27:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b4e70 msgr2=0x7fcbe80cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b4e70 0x7fcbe80cd490 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0035040 tx=0x7fcbe00046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 shutdown_connections 2024-04-25T15:27:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbc80744b0 0x7fcbc80769c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe80d20c0 0x7fcbe80cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe80b6900 0x7fcbe80cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 --2- 172.21.15.67:0/1189162084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe80b4e70 0x7fcbe80cd490 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 >> 172.21.15.67:0/1189162084 conn(0x7fcbe80d0530 msgr2=0x7fcbe8071530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 shutdown_connections 2024-04-25T15:27:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.243+0000 7fcbece9a640 1 -- 172.21.15.67:0/1189162084 wait complete. 2024-04-25T15:27:29.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:27:29.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.370+0000 7f7ffa014640 1 Processor -- start 2024-04-25T15:27:29.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.370+0000 7f7ffa014640 1 -- start start 2024-04-25T15:27:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ffa014640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7ff406ecd0 0x7f7ff40b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ffa014640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ff4160640 0x7f7ff40b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ffa014640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff40b3f30 con 0x7f7ff40d0190 2024-04-25T15:27:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ffa014640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7ff40b23e0 con 0x7f7ff4160640 2024-04-25T15:27:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ffa014640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7ff40b4210 con 0x7f7ff406ecd0 2024-04-25T15:27:29.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ff4160640 0x7f7ff40b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7ff40d0190 0x7f7ff40d1a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49634/0 (socket says 172.21.15.67:49634) 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff37fe640 1 -- 172.21.15.67:0/2188932910 learned_addr learned my addr 172.21.15.67:0/2188932910 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff2ffd640 1 --2- 172.21.15.67:0/2188932910 >> v2:172.21.15.176:3300/0 conn(0x7f7ff406ecd0 0x7f7ff40b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff3fff640 1 -- 172.21.15.67:0/2188932910 >> v2:172.21.15.176:3300/0 conn(0x7f7ff406ecd0 msgr2=0x7f7ff40b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff3fff640 1 --2- 172.21.15.67:0/2188932910 >> v2:172.21.15.176:3300/0 conn(0x7f7ff406ecd0 0x7f7ff40b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff3fff640 1 -- 172.21.15.67:0/2188932910 >> v1:172.21.15.67:6789/0 conn(0x7f7ff40d0190 legacy=0x7f7ff40d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff27fc640 1 -- 172.21.15.67:0/2188932910 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729903014 0 0) 0x7f7ff40b3f30 con 0x7f7ff40d0190 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff3fff640 1 -- 172.21.15.67:0/2188932910 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff40b2630 con 0x7f7ff4160640 2024-04-25T15:27:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.372+0000 7f7ff2ffd640 1 --2- 172.21.15.67:0/2188932910 >> v2:172.21.15.176:3300/0 conn(0x7f7ff406ecd0 0x7f7ff40b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.373+0000 7f7ff3fff640 1 --2- 172.21.15.67:0/2188932910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff4160640 0x7f7ff40b7d20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40b2560 tx=0x7f7fec02a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601053921f7d88fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.373+0000 7f7ff27fc640 1 -- 172.21.15.67:0/2188932910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec007740 con 0x7f7ff4160640 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.373+0000 7f7ff27fc640 1 -- 172.21.15.67:0/2188932910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fec0078e0 con 0x7f7ff4160640 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.373+0000 7f7ff27fc640 1 -- 172.21.15.67:0/2188932910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fec007cb0 con 0x7f7ff4160640 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 -- 172.21.15.67:0/2188932910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff4160640 msgr2=0x7f7ff40b7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2188932910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff4160640 0x7f7ff40b7d20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40b2560 tx=0x7f7fec02a540 comp rx=0 tx=0).stop 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 -- 172.21.15.67:0/2188932910 shutdown_connections 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2188932910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff4160640 0x7f7ff40b7d20 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2188932910 >> v2:172.21.15.176:3300/0 conn(0x7f7ff406ecd0 0x7f7ff40b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 -- 172.21.15.67:0/2188932910 >> 172.21.15.67:0/2188932910 conn(0x7f7ff409b7d0 msgr2=0x7f7ff409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 -- 172.21.15.67:0/2188932910 shutdown_connections 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.374+0000 7f7ffa014640 1 -- 172.21.15.67:0/2188932910 wait complete. 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.376+0000 7f7ffa014640 1 Processor -- start 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.376+0000 7f7ffa014640 1 -- start start 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ffa014640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff406ecd0 0x7f7ff40cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ffa014640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ff40d0190 0x7f7ff414d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ffa014640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ff4160640 0x7f7ff414db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ffa014640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff40b3cd0 con 0x7f7ff406ecd0 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ffa014640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff40b3b50 con 0x7f7ff40d0190 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ffa014640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7ff40b2e30 con 0x7f7ff4160640 2024-04-25T15:27:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ff37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff406ecd0 0x7f7ff40cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ff37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff406ecd0 0x7f7ff40cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49956/0 (socket says 172.21.15.67:49956) 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.377+0000 7f7ff37fe640 1 -- 172.21.15.67:0/2459700199 learned_addr learned my addr 172.21.15.67:0/2459700199 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff3fff640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ff4160640 0x7f7ff414db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff2ffd640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ff40d0190 0x7f7ff414d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff37fe640 1 -- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ff4160640 msgr2=0x7f7ff414db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff37fe640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ff4160640 0x7f7ff414db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff37fe640 1 -- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ff40d0190 msgr2=0x7f7ff414d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff37fe640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ff40d0190 0x7f7ff414d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff37fe640 1 -- 172.21.15.67:0/2459700199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff40c8dd0 con 0x7f7ff406ecd0 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff3fff640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ff4160640 0x7f7ff414db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ff37fe640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff406ecd0 0x7f7ff40cc270 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400c410 tx=0x7f7fe400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7fdbfff640 1 -- 172.21.15.67:0/2459700199 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe40074e0 con 0x7f7ff406ecd0 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7fdbfff640 1 -- 172.21.15.67:0/2459700199 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe40047e0 con 0x7f7ff406ecd0 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff40c90b0 con 0x7f7ff406ecd0 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7fdbfff640 1 -- 172.21.15.67:0/2459700199 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4004bd0 con 0x7f7ff406ecd0 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.378+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff40b0e90 con 0x7f7ff406ecd0 2024-04-25T15:27:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.379+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ff4050740 con 0x7f7ff406ecd0 2024-04-25T15:27:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.380+0000 7f7fdbfff640 1 -- 172.21.15.67:0/2459700199 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7fe4011070 con 0x7f7ff406ecd0 2024-04-25T15:27:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.380+0000 7f7fdbfff640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7fe00744e0 0x7f7fe00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.381+0000 7f7fdbfff640 1 -- 172.21.15.67:0/2459700199 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f7fe4094c60 con 0x7f7ff406ecd0 2024-04-25T15:27:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.381+0000 7f7ff2ffd640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7fe00744e0 0x7f7fe00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.381+0000 7f7ff2ffd640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7fe00744e0 0x7f7fe00769f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7ff415d680 tx=0x7f7fe8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.385+0000 7f7fdbfff640 1 -- 172.21.15.67:0/2459700199 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fe405f510 con 0x7f7ff406ecd0 2024-04-25T15:27:29.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.572+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7ff40cf560 con 0x7f7ff406ecd0 2024-04-25T15:27:29.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.573+0000 7f7fdbfff640 1 -- 172.21.15.67:0/2459700199 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f7fe405ee10 con 0x7f7ff406ecd0 2024-04-25T15:27:29.574 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7fe00744e0 msgr2=0x7f7fe00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7fe00744e0 0x7f7fe00769f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7ff415d680 tx=0x7f7fe8008040 comp rx=0 tx=0).stop 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff406ecd0 msgr2=0x7f7ff40cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff406ecd0 0x7f7ff40cc270 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400c410 tx=0x7f7fe400c8d0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 shutdown_connections 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7fe00744e0 0x7f7fe00769f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ff4160640 0x7f7ff414db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ff40d0190 0x7f7ff414d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 --2- 172.21.15.67:0/2459700199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ff406ecd0 0x7f7ff40cc270 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.574+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 >> 172.21.15.67:0/2459700199 conn(0x7f7ff409b7d0 msgr2=0x7f7ff40cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:29.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.575+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 shutdown_connections 2024-04-25T15:27:29.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.575+0000 7f7ffa014640 1 -- 172.21.15.67:0/2459700199 wait complete. 2024-04-25T15:27:29.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:27:29.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.701+0000 7fdd33067640 1 Processor -- start 2024-04-25T15:27:29.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.701+0000 7fdd33067640 1 -- start start 2024-04-25T15:27:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.702+0000 7fdd33067640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd2c0d0e00 0x7fdd2c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.702+0000 7fdd33067640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c06c5c0 0x7fdd2c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.702+0000 7fdd33067640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd2c06cf10 con 0x7fdd2c0b7810 2024-04-25T15:27:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.702+0000 7fdd33067640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdd2c0d5a60 con 0x7fdd2c0d0e00 2024-04-25T15:27:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.702+0000 7fdd33067640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd2c0d1850 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd30e03640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd2c0d0e00 0x7fdd2c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd2bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c06c5c0 0x7fdd2c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd31604640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fdd2c0b7810 0x7fdd2c0b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:41022/0 (socket says 172.21.15.67:41022) 2024-04-25T15:27:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd31604640 1 -- 172.21.15.67:0/3508503419 learned_addr learned my addr 172.21.15.67:0/3508503419 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd2b7fe640 1 -- 172.21.15.67:0/3508503419 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658968073 0 0) 0x7fdd24001490 con 0x7fdd2c0b7810 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd2b7fe640 1 -- 172.21.15.67:0/3508503419 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd1c003520 con 0x7fdd2c0b7810 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd30e03640 1 -- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c06c5c0 msgr2=0x7fdd2c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd30e03640 1 --2- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c06c5c0 0x7fdd2c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd30e03640 1 -- 172.21.15.67:0/3508503419 >> v1:172.21.15.176:6789/0 conn(0x7fdd2c0b7810 legacy=0x7fdd2c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd30e03640 1 -- 172.21.15.67:0/3508503419 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2c160de0 con 0x7fdd2c0d0e00 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.703+0000 7fdd2bfff640 1 --2- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c06c5c0 0x7fdd2c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.704+0000 7fdd30e03640 1 --2- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0d0e00 0x7fdd2c0d1210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdd200097a0 tx=0x7fdd2002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c00c18cdb4367b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.704+0000 7fdd2b7fe640 1 -- 172.21.15.67:0/3508503419 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20037070 con 0x7fdd2c0d0e00 2024-04-25T15:27:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.704+0000 7fdd2b7fe640 1 -- 172.21.15.67:0/3508503419 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd20032440 con 0x7fdd2c0d0e00 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.704+0000 7fdd2b7fe640 1 -- 172.21.15.67:0/3508503419 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd200327f0 con 0x7fdd2c0d0e00 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 -- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0d0e00 msgr2=0x7fdd2c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 --2- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0d0e00 0x7fdd2c0d1210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdd200097a0 tx=0x7fdd2002a180 comp rx=0 tx=0).stop 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 -- 172.21.15.67:0/3508503419 shutdown_connections 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 --2- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c06c5c0 0x7fdd2c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 --2- 172.21.15.67:0/3508503419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0d0e00 0x7fdd2c0d1210 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 -- 172.21.15.67:0/3508503419 >> 172.21.15.67:0/3508503419 conn(0x7fdd2c0d5d90 msgr2=0x7fdd2c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 -- 172.21.15.67:0/3508503419 shutdown_connections 2024-04-25T15:27:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.705+0000 7fdd33067640 1 -- 172.21.15.67:0/3508503419 wait complete. 2024-04-25T15:27:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.707+0000 7fdd33067640 1 Processor -- start 2024-04-25T15:27:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.707+0000 7fdd33067640 1 -- start start 2024-04-25T15:27:29.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd33067640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd2c06c5c0 0x7fdd2c0c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd33067640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0b7810 0x7fdd2c149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd33067640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d0e00 0x7fdd2c0cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd33067640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c160ad0 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd33067640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c1613a0 con 0x7fdd2c0b7810 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd33067640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd2c0b5320 con 0x7fdd2c0d0e00 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd30e03640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd2c06c5c0 0x7fdd2c0c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.708+0000 7fdd30e03640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd2c06c5c0 0x7fdd2c0c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49970/0 (socket says 172.21.15.67:49970) 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd30e03640 1 -- 172.21.15.67:0/2151329434 learned_addr learned my addr 172.21.15.67:0/2151329434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd31604640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d0e00 0x7fdd2c0cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd2bfff640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0b7810 0x7fdd2c149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd30e03640 1 -- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d0e00 msgr2=0x7fdd2c0cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd30e03640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d0e00 0x7fdd2c0cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd30e03640 1 -- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0b7810 msgr2=0x7fdd2c149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd30e03640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0b7810 0x7fdd2c149c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd30e03640 1 -- 172.21.15.67:0/2151329434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2c0cd830 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd2bfff640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0b7810 0x7fdd2c149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd31604640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d0e00 0x7fdd2c0cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:29.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd30e03640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd2c06c5c0 0x7fdd2c0c9830 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdd200098c0 tx=0x7fdd200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd297fa640 1 -- 172.21.15.67:0/2151329434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20037040 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd297fa640 1 -- 172.21.15.67:0/2151329434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd200044c0 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2c0cdb10 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd297fa640 1 -- 172.21.15.67:0/2151329434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd200325f0 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.709+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd2c0ce0d0 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.710+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2c074820 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.711+0000 7fdd297fa640 1 -- 172.21.15.67:0/2151329434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdd20044020 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.712+0000 7fdd297fa640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd100744b0 0x7fdd100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:29.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.712+0000 7fdd297fa640 1 -- 172.21.15.67:0/2151329434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fdd200b5400 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.712+0000 7fdd2bfff640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd100744b0 0x7fdd100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:29.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.713+0000 7fdd2bfff640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd100744b0 0x7fdd100769c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c1fa450 tx=0x7fdd18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:29.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.717+0000 7fdd297fa640 1 -- 172.21.15.67:0/2151329434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd2c074820 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.916+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdd2c050740 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.916+0000 7fdd297fa640 1 -- 172.21.15.67:0/2151329434 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7fdd2007fc30 con 0x7fdd2c06c5c0 2024-04-25T15:27:29.917 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:27:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd100744b0 msgr2=0x7fdd100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd100744b0 0x7fdd100769c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c1fa450 tx=0x7fdd18008040 comp rx=0 tx=0).stop 2024-04-25T15:27:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd2c06c5c0 msgr2=0x7fdd2c0c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd2c06c5c0 0x7fdd2c0c9830 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdd200098c0 tx=0x7fdd200046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 shutdown_connections 2024-04-25T15:27:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdd100744b0 0x7fdd100769c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd2c0d0e00 0x7fdd2c0cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd2c0b7810 0x7fdd2c149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 --2- 172.21.15.67:0/2151329434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd2c06c5c0 0x7fdd2c0c9830 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 >> 172.21.15.67:0/2151329434 conn(0x7fdd2c0d5d90 msgr2=0x7fdd2c161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 shutdown_connections 2024-04-25T15:27:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:29.918+0000 7fdd33067640 1 -- 172.21.15.67:0/2151329434 wait complete. 2024-04-25T15:27:29.930 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T15:27:29.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:29.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:29.958 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:27:29.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:29.958 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:27:30.013 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:27:30.014 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:30.014 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:30.033 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:30.033 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:30.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:30.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.201+0000 7f16de2e0640 1 Processor -- start 2024-04-25T15:27:30.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.201+0000 7f16de2e0640 1 -- start start 2024-04-25T15:27:30.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16de2e0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f16d806ccd0 0x7f16d8164040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16de2e0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d8164db0 con 0x7f16d8164580 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16de2e0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d8160d30 con 0x7f16d815abd0 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16de2e0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f16d8161380 con 0x7f16d806ccd0 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16dc87d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f16d8164580 0x7f16d80d4100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49640/0 (socket says 172.21.15.67:49640) 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16dc87d640 1 -- 172.21.15.67:0/2979232609 learned_addr learned my addr 172.21.15.67:0/2979232609 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231280478 0 0) 0x7f16d8164db0 con 0x7f16d8164580 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d77fe640 1 --2- 172.21.15.67:0/2979232609 >> v2:172.21.15.176:3300/0 conn(0x7f16d806ccd0 0x7f16d8164040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16c0003520 con 0x7f16d8164580 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3079221031 0 0) 0x7f16c0003520 con 0x7f16d8164580 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 >> v2:172.21.15.176:3300/0 conn(0x7f16d806ccd0 msgr2=0x7f16d8164040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 --2- 172.21.15.67:0/2979232609 >> v2:172.21.15.176:3300/0 conn(0x7f16d806ccd0 0x7f16d8164040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 >> v1:172.21.15.176:6789/0 conn(0x7f16d815abd0 legacy=0x7f16d806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d8161010 con 0x7f16d8164580 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f16d00019f0 con 0x7f16d8164580 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 >> v1:172.21.15.67:6789/0 conn(0x7f16d8164580 legacy=0x7f16d80d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.203+0000 7f16d6ffd640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16c00039b0 0x7f16c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d6ffd640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0004800 0x7f16c0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d6ffd640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0005150 0x7f16c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8161380 con 0x7f16c0004800 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f16c000c0c0 con 0x7f16c0005150 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f16c000c2c0 con 0x7f16c00039b0 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d77fe640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0004800 0x7f16c0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d7fff640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16c00039b0 0x7f16c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16dc87d640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0005150 0x7f16c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d77fe640 1 -- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16c00039b0 msgr2=0x7f16c0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d77fe640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16c00039b0 0x7f16c0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d77fe640 1 -- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0005150 msgr2=0x7f16c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:30.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d77fe640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0005150 0x7f16c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16d77fe640 1 -- 172.21.15.67:0/2979232609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c0003520 con 0x7f16c0004800 2024-04-25T15:27:30.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.204+0000 7f16dc87d640 1 -- 172.21.15.67:0/2979232609 reap_dead start 2024-04-25T15:27:30.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.205+0000 7f16d77fe640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0004800 0x7f16c0004c10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009ac0 tx=0x7f16c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0a8f73e2782eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.205+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc036070 con 0x7f16c0004800 2024-04-25T15:27:30.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.205+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16cc031400 con 0x7f16c0004800 2024-04-25T15:27:30.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.205+0000 7f16d6ffd640 1 -- 172.21.15.67:0/2979232609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc009690 con 0x7f16c0004800 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.206+0000 7f16de2e0640 1 -- 172.21.15.67:0/2979232609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0004800 msgr2=0x7f16c0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.206+0000 7f16de2e0640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0004800 0x7f16c0004c10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009ac0 tx=0x7f16c0005830 comp rx=0 tx=0).stop 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.206+0000 7f16de2e0640 1 -- 172.21.15.67:0/2979232609 shutdown_connections 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.206+0000 7f16de2e0640 1 --2- 172.21.15.67:0/2979232609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16c0004800 0x7f16c0004c10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.206+0000 7f16de2e0640 1 -- 172.21.15.67:0/2979232609 >> 172.21.15.67:0/2979232609 conn(0x7f16d80d5df0 msgr2=0x7f16d80d6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.206+0000 7f16de2e0640 1 -- 172.21.15.67:0/2979232609 shutdown_connections 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.206+0000 7f16de2e0640 1 -- 172.21.15.67:0/2979232609 wait complete. 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.207+0000 7f16de2e0640 1 Processor -- start 2024-04-25T15:27:30.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.208+0000 7f16de2e0640 1 -- start start 2024-04-25T15:27:30.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16de2e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0004800 0x7f16d80cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16de2e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ce850 0x7f16d81544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16de2e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16d80cec60 0x7f16d81549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16de2e0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8161810 con 0x7f16d80ce850 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16de2e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8164990 con 0x7f16c0004800 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16de2e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f16d80b7e50 con 0x7f16d80cec60 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ce850 0x7f16d81544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ce850 0x7f16d81544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49982/0 (socket says 172.21.15.67:49982) 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 -- 172.21.15.67:0/275215487 learned_addr learned my addr 172.21.15.67:0/275215487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d7fff640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0004800 0x7f16d80cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16dc87d640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16d80cec60 0x7f16d81549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 -- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16d80cec60 msgr2=0x7f16d81549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16d80cec60 0x7f16d81549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 -- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0004800 msgr2=0x7f16d80cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0004800 0x7f16d80cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d77fe640 1 -- 172.21.15.67:0/275215487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80cf330 con 0x7f16d80ce850 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.209+0000 7f16d7fff640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0004800 0x7f16d80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.210+0000 7f16d77fe640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ce850 0x7f16d81544b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009be0 tx=0x7f16cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.210+0000 7f16d4ff9640 1 -- 172.21.15.67:0/275215487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc03f020 con 0x7f16d80ce850 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.210+0000 7f16d4ff9640 1 -- 172.21.15.67:0/275215487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16cc037040 con 0x7f16d80ce850 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.210+0000 7f16d4ff9640 1 -- 172.21.15.67:0/275215487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc036070 con 0x7f16d80ce850 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.210+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d80cf610 con 0x7f16d80ce850 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.210+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d8157780 con 0x7f16d80ce850 2024-04-25T15:27:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.211+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d80cf7e0 con 0x7f16d80ce850 2024-04-25T15:27:30.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.212+0000 7f16d4ff9640 1 -- 172.21.15.67:0/275215487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f16cc042020 con 0x7f16d80ce850 2024-04-25T15:27:30.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.212+0000 7f16d4ff9640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f16c40744e0 0x7f16c40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.212+0000 7f16d4ff9640 1 -- 172.21.15.67:0/275215487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f16cc0b53e0 con 0x7f16d80ce850 2024-04-25T15:27:30.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.212+0000 7f16d7fff640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f16c40744e0 0x7f16c40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.213+0000 7f16d7fff640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f16c40744e0 0x7f16c40769f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f16c8003700 tx=0x7f16c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.217+0000 7f16d4ff9640 1 -- 172.21.15.67:0/275215487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16d80cf7e0 con 0x7f16d80ce850 2024-04-25T15:27:30.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.407+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16d80cf0c0 con 0x7f16d80ce850 2024-04-25T15:27:30.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.408+0000 7f16d4ff9640 1 -- 172.21.15.67:0/275215487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16629 (secure 0 0 0) 0x7f16cc049030 con 0x7f16d80ce850 2024-04-25T15:27:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:30.409 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:20.019287+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f16c40744e0 msgr2=0x7f16c40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f16c40744e0 0x7f16c40769f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f16c8003700 tx=0x7f16c8009040 comp rx=0 tx=0).stop 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ce850 msgr2=0x7f16d81544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ce850 0x7f16d81544b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009be0 tx=0x7f16cc004730 comp rx=0 tx=0).stop 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 shutdown_connections 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f16c40744e0 0x7f16c40769f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16d80cec60 0x7f16d81549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16d80ce850 0x7f16d81544b0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 --2- 172.21.15.67:0/275215487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16c0004800 0x7f16d80cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 >> 172.21.15.67:0/275215487 conn(0x7f16d80d5df0 msgr2=0x7f16d81631c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:30.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 shutdown_connections 2024-04-25T15:27:30.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.410+0000 7f16de2e0640 1 -- 172.21.15.67:0/275215487 wait complete. 2024-04-25T15:27:30.423 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:27:16.061447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:27:17.338750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:30.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.537+0000 7f478273f640 1 Processor -- start 2024-04-25T15:27:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.537+0000 7f478273f640 1 -- start start 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f478273f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f477c164bd0 0x7f477c160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f478273f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f477c0cffa0 con 0x7f477c1608a0 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f478273f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f477c0d0280 con 0x7f477c0b77d0 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f478273f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f477c0b7be0 con 0x7f477c164bd0 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f477b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f477c164bd0 0x7f477c160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f4780cdc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f477c1608a0 0x7f477c0b2f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49646/0 (socket says 172.21.15.67:49646) 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f4780cdc640 1 -- 172.21.15.67:0/1610777607 learned_addr learned my addr 172.21.15.67:0/1610777607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f477b7fe640 1 --2- 172.21.15.67:0/1610777607 >> v2:172.21.15.67:3300/0 conn(0x7f477c164bd0 0x7f477c160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49998/0 (socket says 172.21.15.67:49998) 2024-04-25T15:27:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f477b7fe640 1 -- 172.21.15.67:0/1610777607 >> v1:172.21.15.176:6789/0 conn(0x7f477c0b77d0 legacy=0x7f477c0d0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f477b7fe640 1 -- 172.21.15.67:0/1610777607 >> v1:172.21.15.67:6789/0 conn(0x7f477c1608a0 legacy=0x7f477c0b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f477affd640 1 -- 172.21.15.67:0/1610777607 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1907545620 0 0) 0x7f477c0d0280 con 0x7f477c0b77d0 2024-04-25T15:27:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.539+0000 7f477b7fe640 1 -- 172.21.15.67:0/1610777607 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c0d0560 con 0x7f477c164bd0 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.540+0000 7f477b7fe640 1 --2- 172.21.15.67:0/1610777607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c160360 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f476c0097a0 tx=0x7f476c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd06c0b918b78c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.540+0000 7f477affd640 1 -- 172.21.15.67:0/1610777607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c037070 con 0x7f477c164bd0 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.540+0000 7f477affd640 1 -- 172.21.15.67:0/1610777607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476c02ae50 con 0x7f477c164bd0 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.540+0000 7f477affd640 1 -- 172.21.15.67:0/1610777607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c032660 con 0x7f477c164bd0 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.541+0000 7f478273f640 1 -- 172.21.15.67:0/1610777607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 msgr2=0x7f477c160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.541+0000 7f478273f640 1 --2- 172.21.15.67:0/1610777607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c160360 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f476c0097a0 tx=0x7f476c02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.541+0000 7f478273f640 1 -- 172.21.15.67:0/1610777607 shutdown_connections 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.541+0000 7f478273f640 1 --2- 172.21.15.67:0/1610777607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c160360 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.541+0000 7f478273f640 1 -- 172.21.15.67:0/1610777607 >> 172.21.15.67:0/1610777607 conn(0x7f477c09b7d0 msgr2=0x7f477c0b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.541+0000 7f478273f640 1 -- 172.21.15.67:0/1610777607 shutdown_connections 2024-04-25T15:27:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.541+0000 7f478273f640 1 -- 172.21.15.67:0/1610777607 wait complete. 2024-04-25T15:27:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.543+0000 7f478273f640 1 Processor -- start 2024-04-25T15:27:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.543+0000 7f478273f640 1 -- start start 2024-04-25T15:27:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f478273f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f477c0b77d0 0x7f477c14e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f478273f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f478273f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f477c0cddf0 0x7f477c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f478273f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f477c0b3650 con 0x7f477c164bd0 2024-04-25T15:27:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f478273f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f477c0d08d0 con 0x7f477c0b77d0 2024-04-25T15:27:30.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f478273f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f477c0b37d0 con 0x7f477c0cddf0 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f477b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f477b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c0cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50010/0 (socket says 172.21.15.67:50010) 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f477b7fe640 1 -- 172.21.15.67:0/404044060 learned_addr learned my addr 172.21.15.67:0/404044060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f477bfff640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f477c0b77d0 0x7f477c14e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.544+0000 7f4780cdc640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f477c0cddf0 0x7f477c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f477b7fe640 1 -- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f477c0cddf0 msgr2=0x7f477c0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f477b7fe640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f477c0cddf0 0x7f477c0b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f477b7fe640 1 -- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f477c0b77d0 msgr2=0x7f477c14e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f477b7fe640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f477c0b77d0 0x7f477c14e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f477b7fe640 1 -- 172.21.15.67:0/404044060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f477c0cf680 con 0x7f477c164bd0 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f477b7fe640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c0cd8b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f476c035040 tx=0x7f476c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f4778ff9640 1 -- 172.21.15.67:0/404044060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c037040 con 0x7f477c164bd0 2024-04-25T15:27:30.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f4778ff9640 1 -- 172.21.15.67:0/404044060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476c004030 con 0x7f477c164bd0 2024-04-25T15:27:30.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f477c0ce5e0 con 0x7f477c164bd0 2024-04-25T15:27:30.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f4778ff9640 1 -- 172.21.15.67:0/404044060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c004460 con 0x7f477c164bd0 2024-04-25T15:27:30.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.545+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f477c0ceba0 con 0x7f477c164bd0 2024-04-25T15:27:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.546+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f477c0b8160 con 0x7f477c164bd0 2024-04-25T15:27:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.547+0000 7f4778ff9640 1 -- 172.21.15.67:0/404044060 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f476c03a040 con 0x7f477c164bd0 2024-04-25T15:27:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.547+0000 7f4778ff9640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47680744e0 0x7f47680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.547+0000 7f4778ff9640 1 -- 172.21.15.67:0/404044060 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f476c0b5460 con 0x7f477c164bd0 2024-04-25T15:27:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.548+0000 7f477bfff640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47680744e0 0x7f47680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.548+0000 7f477bfff640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47680744e0 0x7f47680769f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4770000f60 tx=0x7f4770009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.552+0000 7f4778ff9640 1 -- 172.21.15.67:0/404044060 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f477c0b8160 con 0x7f477c164bd0 2024-04-25T15:27:30.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.739+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f477c0d2e30 con 0x7f477c164bd0 2024-04-25T15:27:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.739+0000 7f4778ff9640 1 -- 172.21.15.67:0/404044060 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f476c042020 con 0x7f477c164bd0 2024-04-25T15:27:30.740 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:30.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47680744e0 msgr2=0x7f47680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47680744e0 0x7f47680769f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4770000f60 tx=0x7f4770009040 comp rx=0 tx=0).stop 2024-04-25T15:27:30.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 msgr2=0x7f477c0cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c0cd8b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f476c035040 tx=0x7f476c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 shutdown_connections 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f47680744e0 0x7f47680769f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f477c0cddf0 0x7f477c0b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f477c164bd0 0x7f477c0cd8b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 --2- 172.21.15.67:0/404044060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f477c0b77d0 0x7f477c14e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 >> 172.21.15.67:0/404044060 conn(0x7f477c09b7d0 msgr2=0x7f477c0d2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 shutdown_connections 2024-04-25T15:27:30.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.741+0000 7f478273f640 1 -- 172.21.15.67:0/404044060 wait complete. 2024-04-25T15:27:30.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:27:30.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.868+0000 7fe8a7172640 1 Processor -- start 2024-04-25T15:27:30.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.868+0000 7fe8a7172640 1 -- start start 2024-04-25T15:27:30.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe8a7172640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8a006ccd0 0x7fe8a00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe8a7172640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00d4a50 0x7fe8a00d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe8a7172640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a00d1850 con 0x7fe8a00d0e00 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe8a7172640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe8a0160de0 con 0x7fe8a006ccd0 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe8a7172640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a01610c0 con 0x7fe8a00d4a50 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe89ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8a006ccd0 0x7fe8a00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe89ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8a006ccd0 0x7fe8a00d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50024/0 (socket says 172.21.15.67:50024) 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe89ffff640 1 -- 172.21.15.67:0/727673175 learned_addr learned my addr 172.21.15.67:0/727673175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe8a4f0e640 1 --1- 172.21.15.67:0/727673175 >> v1:172.21.15.67:6789/0 conn(0x7fe8a00d0e00 0x7fe8a006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49658/0 (socket says 172.21.15.67:49658) 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe8a570f640 1 --2- 172.21.15.67:0/727673175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00d4a50 0x7fe8a00d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe89ffff640 1 -- 172.21.15.67:0/727673175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00d4a50 msgr2=0x7fe8a00d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe89ffff640 1 --2- 172.21.15.67:0/727673175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00d4a50 0x7fe8a00d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.869+0000 7fe89ffff640 1 -- 172.21.15.67:0/727673175 >> v1:172.21.15.67:6789/0 conn(0x7fe8a00d0e00 legacy=0x7fe8a006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.870+0000 7fe89f7fe640 1 -- 172.21.15.67:0/727673175 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569595304 0 0) 0x7fe8a00d1850 con 0x7fe8a00d0e00 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.870+0000 7fe89ffff640 1 -- 172.21.15.67:0/727673175 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a01613a0 con 0x7fe8a006ccd0 2024-04-25T15:27:30.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.870+0000 7fe8a570f640 1 --2- 172.21.15.67:0/727673175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00d4a50 0x7fe8a00d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.870+0000 7fe89ffff640 1 --2- 172.21.15.67:0/727673175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a006ccd0 0x7fe8a00d1210 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe894009870 tx=0x7fe89402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3f703916c6aa62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.870+0000 7fe89f7fe640 1 -- 172.21.15.67:0/727673175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894037070 con 0x7fe8a006ccd0 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.870+0000 7fe89f7fe640 1 -- 172.21.15.67:0/727673175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe894032440 con 0x7fe8a006ccd0 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.870+0000 7fe89f7fe640 1 -- 172.21.15.67:0/727673175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894032810 con 0x7fe8a006ccd0 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 -- 172.21.15.67:0/727673175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a006ccd0 msgr2=0x7fe8a00d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 --2- 172.21.15.67:0/727673175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a006ccd0 0x7fe8a00d1210 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe894009870 tx=0x7fe89402a270 comp rx=0 tx=0).stop 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 -- 172.21.15.67:0/727673175 shutdown_connections 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 --2- 172.21.15.67:0/727673175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00d4a50 0x7fe8a00d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 --2- 172.21.15.67:0/727673175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a006ccd0 0x7fe8a00d1210 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 -- 172.21.15.67:0/727673175 >> 172.21.15.67:0/727673175 conn(0x7fe8a00d5d90 msgr2=0x7fe8a00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 -- 172.21.15.67:0/727673175 shutdown_connections 2024-04-25T15:27:30.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.871+0000 7fe8a7172640 1 -- 172.21.15.67:0/727673175 wait complete. 2024-04-25T15:27:30.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.873+0000 7fe8a7172640 1 Processor -- start 2024-04-25T15:27:30.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.873+0000 7fe8a7172640 1 -- start start 2024-04-25T15:27:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.874+0000 7fe8a7172640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a00d0e00 0x7fe8a015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.874+0000 7fe8a7172640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a00d4a50 0x7fe8a00cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.874+0000 7fe8a7172640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00bb020 0x7fe8a00bb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.874+0000 7fe8a7172640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0164e00 con 0x7fe8a00d4a50 2024-04-25T15:27:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.874+0000 7fe8a7172640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a00b5320 con 0x7fe8a00d0e00 2024-04-25T15:27:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.874+0000 7fe8a7172640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a0164f80 con 0x7fe8a00bb020 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.874+0000 7fe89ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a00d4a50 0x7fe8a00cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a00d4a50 0x7fe8a00cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50032/0 (socket says 172.21.15.67:50032) 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 -- 172.21.15.67:0/151464584 learned_addr learned my addr 172.21.15.67:0/151464584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe8a570f640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00bb020 0x7fe8a00bb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe8a4f0e640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a00d0e00 0x7fe8a015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 -- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00bb020 msgr2=0x7fe8a00bb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00bb020 0x7fe8a00bb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 -- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a00d0e00 msgr2=0x7fe8a015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a00d0e00 0x7fe8a015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 -- 172.21.15.67:0/151464584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a015ba30 con 0x7fe8a00d4a50 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89ffff640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a00d4a50 0x7fe8a00cbf90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe8940090b0 tx=0x7fe8940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89d7fa640 1 -- 172.21.15.67:0/151464584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894037040 con 0x7fe8a00d4a50 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89d7fa640 1 -- 172.21.15.67:0/151464584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe894004230 con 0x7fe8a00d4a50 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a00bb970 con 0x7fe8a00d4a50 2024-04-25T15:27:30.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe89d7fa640 1 -- 172.21.15.67:0/151464584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89403c670 con 0x7fe8a00d4a50 2024-04-25T15:27:30.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.875+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a00cabe0 con 0x7fe8a00d4a50 2024-04-25T15:27:30.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.876+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a0050740 con 0x7fe8a00d4a50 2024-04-25T15:27:30.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.877+0000 7fe89d7fa640 1 -- 172.21.15.67:0/151464584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe89403a070 con 0x7fe8a00d4a50 2024-04-25T15:27:30.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.878+0000 7fe89d7fa640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe88c0744b0 0x7fe88c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:30.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.878+0000 7fe89d7fa640 1 -- 172.21.15.67:0/151464584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe8940b5870 con 0x7fe8a00d4a50 2024-04-25T15:27:30.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.878+0000 7fe8a4f0e640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe88c0744b0 0x7fe88c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:30.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.878+0000 7fe8a4f0e640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe88c0744b0 0x7fe88c0769c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe8980012e0 tx=0x7fe898001630 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:30.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:30.882+0000 7fe89d7fa640 1 -- 172.21.15.67:0/151464584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8a00bb970 con 0x7fe8a00d4a50 2024-04-25T15:27:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.069+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe8a00d5440 con 0x7fe8a00d4a50 2024-04-25T15:27:31.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.069+0000 7fe89d7fa640 1 -- 172.21.15.67:0/151464584 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7fe8a015ba30 con 0x7fe8a00d4a50 2024-04-25T15:27:31.070 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:27:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe88c0744b0 msgr2=0x7fe88c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe88c0744b0 0x7fe88c0769c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe8980012e0 tx=0x7fe898001630 comp rx=0 tx=0).stop 2024-04-25T15:27:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a00d4a50 msgr2=0x7fe8a00cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a00d4a50 0x7fe8a00cbf90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe8940090b0 tx=0x7fe8940046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 shutdown_connections 2024-04-25T15:27:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe88c0744b0 0x7fe88c0769c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8a00bb020 0x7fe8a00bb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8a00d4a50 0x7fe8a00cbf90 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 --2- 172.21.15.67:0/151464584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8a00d0e00 0x7fe8a015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 >> 172.21.15.67:0/151464584 conn(0x7fe8a00d5d90 msgr2=0x7fe8a0163990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 shutdown_connections 2024-04-25T15:27:31.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.071+0000 7fe8a7172640 1 -- 172.21.15.67:0/151464584 wait complete. 2024-04-25T15:27:31.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:27:31.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.197+0000 7fd60c451640 1 Processor -- start 2024-04-25T15:27:31.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.197+0000 7fd60c451640 1 -- start start 2024-04-25T15:27:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60c451640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd6040b6900 0x7fd6040b8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60c451640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6040b4e70 0x7fd6040d19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60c451640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6040ce670 con 0x7fd6040d1ef0 2024-04-25T15:27:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60c451640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd60408d770 con 0x7fd6040b4e70 2024-04-25T15:27:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60c451640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd6040ce980 con 0x7fd6040b6900 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd6099ec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6040b4e70 0x7fd6040d19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60a9ee640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6040d1ef0 0x7fd6040cee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49674/0 (socket says 172.21.15.67:49674) 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60a9ee640 1 -- 172.21.15.67:0/861185322 learned_addr learned my addr 172.21.15.67:0/861185322 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd6099ec640 1 -- 172.21.15.67:0/861185322 >> v2:172.21.15.176:3300/0 conn(0x7fd6040b6900 msgr2=0x7fd6040b8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd60a1ed640 1 --2- 172.21.15.67:0/861185322 >> v2:172.21.15.176:3300/0 conn(0x7fd6040b6900 0x7fd6040b8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd6099ec640 1 --2- 172.21.15.67:0/861185322 >> v2:172.21.15.176:3300/0 conn(0x7fd6040b6900 0x7fd6040b8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd6091eb640 1 -- 172.21.15.67:0/861185322 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1282236857 0 0) 0x7fd5f4001490 con 0x7fd6040d1ef0 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd6099ec640 1 -- 172.21.15.67:0/861185322 >> v1:172.21.15.67:6789/0 conn(0x7fd6040d1ef0 legacy=0x7fd6040cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.199+0000 7fd6099ec640 1 -- 172.21.15.67:0/861185322 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60408d9a0 con 0x7fd6040b4e70 2024-04-25T15:27:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.200+0000 7fd6099ec640 1 --2- 172.21.15.67:0/861185322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b4e70 0x7fd6040d19b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd6040b65d0 tx=0x7fd5f802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f93565545e7536 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.200+0000 7fd6091eb640 1 -- 172.21.15.67:0/861185322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8037070 con 0x7fd6040b4e70 2024-04-25T15:27:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.200+0000 7fd6091eb640 1 -- 172.21.15.67:0/861185322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f8032440 con 0x7fd6040b4e70 2024-04-25T15:27:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.200+0000 7fd6091eb640 1 -- 172.21.15.67:0/861185322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8032810 con 0x7fd6040b4e70 2024-04-25T15:27:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 -- 172.21.15.67:0/861185322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b4e70 msgr2=0x7fd6040d19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 --2- 172.21.15.67:0/861185322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b4e70 0x7fd6040d19b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd6040b65d0 tx=0x7fd5f802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 -- 172.21.15.67:0/861185322 shutdown_connections 2024-04-25T15:27:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 --2- 172.21.15.67:0/861185322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b4e70 0x7fd6040d19b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 --2- 172.21.15.67:0/861185322 >> v2:172.21.15.176:3300/0 conn(0x7fd6040b6900 0x7fd6040b8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 -- 172.21.15.67:0/861185322 >> 172.21.15.67:0/861185322 conn(0x7fd6040d0530 msgr2=0x7fd6040d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 -- 172.21.15.67:0/861185322 shutdown_connections 2024-04-25T15:27:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.201+0000 7fd60c451640 1 -- 172.21.15.67:0/861185322 wait complete. 2024-04-25T15:27:31.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.203+0000 7fd60c451640 1 Processor -- start 2024-04-25T15:27:31.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.203+0000 7fd60c451640 1 -- start start 2024-04-25T15:27:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60c451640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6040b4e70 0x7fd60415d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60c451640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b6900 0x7fd604153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60c451640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6040d1ef0 0x7fd60415b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60c451640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6040b5280 con 0x7fd6040b6900 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60c451640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd60408df40 con 0x7fd6040d1ef0 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60c451640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd6040b5400 con 0x7fd6040b4e70 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd6099ec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b6900 0x7fd604153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd6099ec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b6900 0x7fd604153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50054/0 (socket says 172.21.15.67:50054) 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd6099ec640 1 -- 172.21.15.67:0/307928142 learned_addr learned my addr 172.21.15.67:0/307928142 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60a9ee640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6040d1ef0 0x7fd60415b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.204+0000 7fd60a1ed640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6040b4e70 0x7fd60415d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd6099ec640 1 -- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6040b4e70 msgr2=0x7fd60415d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd6099ec640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6040b4e70 0x7fd60415d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd6099ec640 1 -- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6040d1ef0 msgr2=0x7fd60415b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd6099ec640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6040d1ef0 0x7fd60415b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd6099ec640 1 -- 172.21.15.67:0/307928142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6040c8c80 con 0x7fd6040b6900 2024-04-25T15:27:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd60a1ed640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6040b4e70 0x7fd60415d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd6099ec640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b6900 0x7fd604153160 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8000c00 tx=0x7fd5f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd602ffd640 1 -- 172.21.15.67:0/307928142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8037050 con 0x7fd6040b6900 2024-04-25T15:27:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd602ffd640 1 -- 172.21.15.67:0/307928142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f8038040 con 0x7fd6040b6900 2024-04-25T15:27:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6040c8f60 con 0x7fd6040b6900 2024-04-25T15:27:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd602ffd640 1 -- 172.21.15.67:0/307928142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8004450 con 0x7fd6040b6900 2024-04-25T15:27:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.205+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6040c9520 con 0x7fd6040b6900 2024-04-25T15:27:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.206+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd60415da90 con 0x7fd6040b6900 2024-04-25T15:27:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.207+0000 7fd602ffd640 1 -- 172.21.15.67:0/307928142 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd5f8030980 con 0x7fd6040b6900 2024-04-25T15:27:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.207+0000 7fd602ffd640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5f00744b0 0x7fd5f00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.208+0000 7fd602ffd640 1 -- 172.21.15.67:0/307928142 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fd5f80b5d60 con 0x7fd6040b6900 2024-04-25T15:27:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.208+0000 7fd60a1ed640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5f00744b0 0x7fd5f00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.208+0000 7fd60a1ed640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5f00744b0 0x7fd5f00769c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc006fd0 tx=0x7fd5fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.212+0000 7fd602ffd640 1 -- 172.21.15.67:0/307928142 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd5f8030c70 con 0x7fd6040b6900 2024-04-25T15:27:31.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.395+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd604050740 con 0x7fd6040b6900 2024-04-25T15:27:31.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.396+0000 7fd602ffd640 1 -- 172.21.15.67:0/307928142 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7fd5f8080580 con 0x7fd6040b6900 2024-04-25T15:27:31.397 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5f00744b0 msgr2=0x7fd5f00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5f00744b0 0x7fd5f00769c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc006fd0 tx=0x7fd5fc008040 comp rx=0 tx=0).stop 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b6900 msgr2=0x7fd604153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b6900 0x7fd604153160 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8000c00 tx=0x7fd5f80046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 shutdown_connections 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd5f00744b0 0x7fd5f00769c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6040d1ef0 0x7fd60415b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6040b6900 0x7fd604153160 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 --2- 172.21.15.67:0/307928142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6040b4e70 0x7fd60415d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 >> 172.21.15.67:0/307928142 conn(0x7fd6040d0530 msgr2=0x7fd6040b6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 shutdown_connections 2024-04-25T15:27:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.398+0000 7fd60c451640 1 -- 172.21.15.67:0/307928142 wait complete. 2024-04-25T15:27:31.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:27:31.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.522+0000 7f872835e640 1 Processor -- start 2024-04-25T15:27:31.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.522+0000 7f872835e640 1 -- start start 2024-04-25T15:27:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.523+0000 7f872835e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f87200b16a0 0x7f87201691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.523+0000 7f872835e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87200cfc30 con 0x7f87200b91d0 2024-04-25T15:27:31.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.523+0000 7f872835e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87200d2240 con 0x7f87200cf770 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.523+0000 7f872835e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f87200d1ff0 con 0x7f87200b16a0 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.523+0000 7f87260fa640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f87200b91d0 0x7f8720161e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49684/0 (socket says 172.21.15.67:49684) 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.523+0000 7f87260fa640 1 -- 172.21.15.67:0/3221908814 learned_addr learned my addr 172.21.15.67:0/3221908814 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.523+0000 7f87268fb640 1 --2- 172.21.15.67:0/3221908814 >> v2:172.21.15.176:3300/0 conn(0x7f87200b16a0 0x7f87201691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.524+0000 7f87250f8640 1 -- 172.21.15.67:0/3221908814 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867855185 0 0) 0x7f87200cfc30 con 0x7f87200b91d0 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.524+0000 7f87250f8640 1 -- 172.21.15.67:0/3221908814 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f870c003520 con 0x7f87200b91d0 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.524+0000 7f87268fb640 1 -- 172.21.15.67:0/3221908814 >> v1:172.21.15.176:6789/0 conn(0x7f87200cf770 legacy=0x7f8720162590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.524+0000 7f87268fb640 1 -- 172.21.15.67:0/3221908814 >> v1:172.21.15.67:6789/0 conn(0x7f87200b91d0 legacy=0x7f8720161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.524+0000 7f87268fb640 1 -- 172.21.15.67:0/3221908814 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87200d2520 con 0x7f87200b16a0 2024-04-25T15:27:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.525+0000 7f87268fb640 1 --2- 172.21.15.67:0/3221908814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87200b16a0 0x7f87201691b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f871c0097a0 tx=0x7f871c02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29529bc6b7984890 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:31.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.525+0000 7f87250f8640 1 -- 172.21.15.67:0/3221908814 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c037070 con 0x7f87200b16a0 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.525+0000 7f87250f8640 1 -- 172.21.15.67:0/3221908814 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871c032440 con 0x7f87200b16a0 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.525+0000 7f87250f8640 1 -- 172.21.15.67:0/3221908814 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0327f0 con 0x7f87200b16a0 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.525+0000 7f872835e640 1 -- 172.21.15.67:0/3221908814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87200b16a0 msgr2=0x7f87201691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.525+0000 7f872835e640 1 --2- 172.21.15.67:0/3221908814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87200b16a0 0x7f87201691b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f871c0097a0 tx=0x7f871c02a230 comp rx=0 tx=0).stop 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.526+0000 7f872835e640 1 -- 172.21.15.67:0/3221908814 shutdown_connections 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.526+0000 7f872835e640 1 --2- 172.21.15.67:0/3221908814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87200b16a0 0x7f87201691b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.526+0000 7f872835e640 1 -- 172.21.15.67:0/3221908814 >> 172.21.15.67:0/3221908814 conn(0x7f87200d3aa0 msgr2=0x7f87200d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.526+0000 7f872835e640 1 -- 172.21.15.67:0/3221908814 shutdown_connections 2024-04-25T15:27:31.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.526+0000 7f872835e640 1 -- 172.21.15.67:0/3221908814 wait complete. 2024-04-25T15:27:31.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.527+0000 7f872835e640 1 Processor -- start 2024-04-25T15:27:31.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.527+0000 7f872835e640 1 -- start start 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f872835e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87200b91d0 0x7f87200c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f872835e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87201f3290 0x7f8720160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f872835e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87201f36a0 0x7f8720161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f872835e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f87200b2320 con 0x7f87201f3290 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f872835e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f87200d2890 con 0x7f87201f36a0 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f872835e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f87200b24a0 con 0x7f87200b91d0 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87201f3290 0x7f8720160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87201f3290 0x7f8720160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50066/0 (socket says 172.21.15.67:50066) 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 -- 172.21.15.67:0/3371457508 learned_addr learned my addr 172.21.15.67:0/3371457508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87268fb640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87201f36a0 0x7f8720161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87260fa640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87200b91d0 0x7f87200c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 -- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87200b91d0 msgr2=0x7f87200c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87200b91d0 0x7f87200c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 -- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87201f36a0 msgr2=0x7f8720161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87201f36a0 0x7f8720161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 -- 172.21.15.67:0/3371457508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87200ccb30 con 0x7f87201f3290 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87268fb640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87201f36a0 0x7f8720161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f87258f9640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87201f3290 0x7f8720160af0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f871000c3c0 tx=0x7f871000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f8716ffd640 1 -- 172.21.15.67:0/3371457508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87100052d0 con 0x7f87201f3290 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.529+0000 7f8716ffd640 1 -- 172.21.15.67:0/3371457508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8710005490 con 0x7f87201f3290 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.530+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87200cce10 con 0x7f87201f3290 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.530+0000 7f8716ffd640 1 -- 172.21.15.67:0/3371457508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710007260 con 0x7f87201f3290 2024-04-25T15:27:31.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.530+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87200cd3d0 con 0x7f87201f3290 2024-04-25T15:27:31.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.531+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8720050740 con 0x7f87201f3290 2024-04-25T15:27:31.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.531+0000 7f8716ffd640 1 -- 172.21.15.67:0/3371457508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8710007400 con 0x7f87201f3290 2024-04-25T15:27:31.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.532+0000 7f8716ffd640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f87040744e0 0x7f87040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:31.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.532+0000 7f8716ffd640 1 -- 172.21.15.67:0/3371457508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f87100946e0 con 0x7f87201f3290 2024-04-25T15:27:31.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.532+0000 7f87260fa640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f87040744e0 0x7f87040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:31.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.532+0000 7f87260fa640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f87040744e0 0x7f87040769f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8718001560 tx=0x7f8718009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:31.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.536+0000 7f8716ffd640 1 -- 172.21.15.67:0/3371457508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f871005ee70 con 0x7f87201f3290 2024-04-25T15:27:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.732+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f87200c9370 con 0x7f87201f3290 2024-04-25T15:27:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.733+0000 7f8716ffd640 1 -- 172.21.15.67:0/3371457508 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f871005e770 con 0x7f87201f3290 2024-04-25T15:27:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f87040744e0 msgr2=0x7f87040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f87040744e0 0x7f87040769f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8718001560 tx=0x7f8718009040 comp rx=0 tx=0).stop 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87201f3290 msgr2=0x7f8720160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87201f3290 0x7f8720160af0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f871000c3c0 tx=0x7f871000c880 comp rx=0 tx=0).stop 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 shutdown_connections 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f87040744e0 0x7f87040769f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87201f36a0 0x7f8720161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87201f3290 0x7f8720160af0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 --2- 172.21.15.67:0/3371457508 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87200b91d0 0x7f87200c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 >> 172.21.15.67:0/3371457508 conn(0x7f87200d3aa0 msgr2=0x7f87200b8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 shutdown_connections 2024-04-25T15:27:31.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:31.735+0000 7f872835e640 1 -- 172.21.15.67:0/3371457508 wait complete. 2024-04-25T15:27:31.747 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:27:31.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:31.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-25T15:27:31.779 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:27:31.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:31.780 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:27:31.837 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:27:31.837 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:31.838 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:31.853 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:27:31.853 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:27:31.854 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:31.854 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:31.894 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:27:31.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:32.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.006+0000 7f1bee7b2640 1 Processor -- start 2024-04-25T15:27:32.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.006+0000 7f1bee7b2640 1 -- start start 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.007+0000 7f1bee7b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b4830 0x7f1be806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.007+0000 7f1bee7b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1be80b2dc0 0x7f1be80b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.007+0000 7f1bee7b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be806d870 con 0x7f1be80d4570 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.007+0000 7f1bee7b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1be80b4c40 con 0x7f1be80b2dc0 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.007+0000 7f1bee7b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1be806e460 con 0x7f1be80b4830 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1becd4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1be80b2dc0 0x7f1be80b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1becd4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1be80b2dc0 0x7f1be80b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50070/0 (socket says 172.21.15.67:50070) 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1becd4f640 1 -- 172.21.15.67:0/3042365858 learned_addr learned my addr 172.21.15.67:0/3042365858 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:32.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1be77fe640 1 --2- 172.21.15.67:0/3042365858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b4830 0x7f1be806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1becd4f640 1 -- 172.21.15.67:0/3042365858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b4830 msgr2=0x7f1be806e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1becd4f640 1 --2- 172.21.15.67:0/3042365858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b4830 0x7f1be806e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1becd4f640 1 -- 172.21.15.67:0/3042365858 >> v1:172.21.15.176:6789/0 conn(0x7f1be80d4570 legacy=0x7f1be80d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1becd4f640 1 -- 172.21.15.67:0/3042365858 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be806db10 con 0x7f1be80b2dc0 2024-04-25T15:27:32.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.008+0000 7f1be77fe640 1 --2- 172.21.15.67:0/3042365858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b4830 0x7f1be806e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.009+0000 7f1becd4f640 1 --2- 172.21.15.67:0/3042365858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b2dc0 0x7f1be80b31d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1be806e5e0 tx=0x7f1be002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4961fc115a50e1c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.009+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3042365858 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0036070 con 0x7f1be80b2dc0 2024-04-25T15:27:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.009+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3042365858 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be0031600 con 0x7f1be80b2dc0 2024-04-25T15:27:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.009+0000 7f1be6ffd640 1 -- 172.21.15.67:0/3042365858 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be00319d0 con 0x7f1be80b2dc0 2024-04-25T15:27:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 -- 172.21.15.67:0/3042365858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b2dc0 msgr2=0x7f1be80b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/3042365858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b2dc0 0x7f1be80b31d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1be806e5e0 tx=0x7f1be002a680 comp rx=0 tx=0).stop 2024-04-25T15:27:32.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 -- 172.21.15.67:0/3042365858 shutdown_connections 2024-04-25T15:27:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/3042365858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b2dc0 0x7f1be80b31d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/3042365858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b4830 0x7f1be806e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 -- 172.21.15.67:0/3042365858 >> 172.21.15.67:0/3042365858 conn(0x7f1be80d3a50 msgr2=0x7f1be80d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 -- 172.21.15.67:0/3042365858 shutdown_connections 2024-04-25T15:27:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.010+0000 7f1bee7b2640 1 -- 172.21.15.67:0/3042365858 wait complete. 2024-04-25T15:27:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.011+0000 7f1bee7b2640 1 Processor -- start 2024-04-25T15:27:32.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.012+0000 7f1bee7b2640 1 -- start start 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1bee7b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b2dc0 0x7f1be80a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1bee7b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b4830 0x7f1be80c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1bee7b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1be80d4570 0x7f1be8159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1bee7b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1be80d0fb0 con 0x7f1be80b4830 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1bee7b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1be80d0bc0 con 0x7f1be80d4570 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1bee7b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1be80b3710 con 0x7f1be80b2dc0 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b4830 0x7f1be80c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b4830 0x7f1be80c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50072/0 (socket says 172.21.15.67:50072) 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 -- 172.21.15.67:0/2796012645 learned_addr learned my addr 172.21.15.67:0/2796012645 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be7fff640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b2dc0 0x7f1be80a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1becd4f640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1be80d4570 0x7f1be8159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 -- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b2dc0 msgr2=0x7f1be80a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b2dc0 0x7f1be80a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 -- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1be80d4570 msgr2=0x7f1be8159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1be80d4570 0x7f1be8159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.013+0000 7f1be77fe640 1 -- 172.21.15.67:0/2796012645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be8163340 con 0x7f1be80b4830 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.014+0000 7f1be77fe640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b4830 0x7f1be80c7d70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800d7d0 tx=0x7f1bd800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.014+0000 7f1be4ff9640 1 -- 172.21.15.67:0/2796012645 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8004030 con 0x7f1be80b4830 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.014+0000 7f1be4ff9640 1 -- 172.21.15.67:0/2796012645 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd80041d0 con 0x7f1be80b4830 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.014+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be80a0370 con 0x7f1be80b4830 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.014+0000 7f1be4ff9640 1 -- 172.21.15.67:0/2796012645 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8009640 con 0x7f1be80b4830 2024-04-25T15:27:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.014+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be80a0930 con 0x7f1be80b4830 2024-04-25T15:27:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.015+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be815a400 con 0x7f1be80b4830 2024-04-25T15:27:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.016+0000 7f1be4ff9640 1 -- 172.21.15.67:0/2796012645 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1bd80097e0 con 0x7f1be80b4830 2024-04-25T15:27:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.016+0000 7f1be4ff9640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bd40744b0 0x7f1bd40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.016+0000 7f1be4ff9640 1 -- 172.21.15.67:0/2796012645 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f1bd8095350 con 0x7f1be80b4830 2024-04-25T15:27:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.016+0000 7f1be7fff640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bd40744b0 0x7f1bd40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.017+0000 7f1be7fff640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bd40744b0 0x7f1bd40769c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc007fd0 tx=0x7f1bdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.021+0000 7f1be4ff9640 1 -- 172.21.15.67:0/2796012645 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1be8163340 con 0x7f1be80b4830 2024-04-25T15:27:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.252+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1be8050740 con 0x7f1be80b4830 2024-04-25T15:27:32.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.252+0000 7f1be4ff9640 1 -- 172.21.15.67:0/2796012645 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5181 (secure 0 0 0) 0x7f1bd805f490 con 0x7f1be80b4830 2024-04-25T15:27:32.253 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:32.253 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/610105566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":610105566},{"type":"v1","addr":"172.21.15.67:6835","nonce":610105566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/947236976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":947236976},{"type":"v1","addr":"172.21.15.176:6837","nonce":947236976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3211936625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3211936625},{"type":"v1","addr":"172.21.15.176:6836","nonce":3211936625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:18.413498+0000","modified":"2024-04-25T15:27:21.411023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5880":{"gid":5880,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3185771646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3185771646},{"type":"v1","addr":"172.21.15.67:6833","nonce":3185771646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5880,"qdb_cluster":[5880]},"id":4}]} 2024-04-25T15:27:32.254 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bd40744b0 msgr2=0x7f1bd40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bd40744b0 0x7f1bd40769c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc007fd0 tx=0x7f1bdc009040 comp rx=0 tx=0).stop 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b4830 msgr2=0x7f1be80c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b4830 0x7f1be80c7d70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1bd800d7d0 tx=0x7f1bd800dc90 comp rx=0 tx=0).stop 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 shutdown_connections 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1bd40744b0 0x7f1bd40769c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1be80d4570 0x7f1be8159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1be80b4830 0x7f1be80c7d70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 --2- 172.21.15.67:0/2796012645 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1be80b2dc0 0x7f1be80a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 >> 172.21.15.67:0/2796012645 conn(0x7f1be80d3a50 msgr2=0x7f1be80cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 shutdown_connections 2024-04-25T15:27:32.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.255+0000 7f1bee7b2640 1 -- 172.21.15.67:0/2796012645 wait complete. 2024-04-25T15:27:32.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.378+0000 7fa2b1e61640 1 Processor -- start 2024-04-25T15:27:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.379+0000 7fa2b1e61640 1 -- start start 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2b1e61640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 0x7fa2ac0d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2b1e61640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa2ac162a10 0x7fa2ac0b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2b1e61640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2ac0cf810 con 0x7fa2ac161ef0 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2b1e61640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa2ac162e20 con 0x7fa2ac162a10 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2b1e61640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2ac0b0690 con 0x7fa2ac0b9270 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2aaffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2ac161ef0 0x7fa2ac162300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49692/0 (socket says 172.21.15.67:49692) 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2aaffd640 1 -- 172.21.15.67:0/2422102900 learned_addr learned my addr 172.21.15.67:0/2422102900 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2abfff640 1 --2- 172.21.15.67:0/2422102900 >> v2:172.21.15.176:3300/0 conn(0x7fa2ac162a10 0x7fa2ac0b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 0x7fa2ac0d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2545475495 0 0) 0x7fa2ac0cf810 con 0x7fa2ac161ef0 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.380+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa294003520 con 0x7fa2ac161ef0 2024-04-25T15:27:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 940480431 0 0) 0x7fa294003520 con 0x7fa2ac161ef0 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 msgr2=0x7fa2ac0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 0x7fa2ac0d15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 >> v2:172.21.15.176:3300/0 conn(0x7fa2ac162a10 msgr2=0x7fa2ac0b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 --2- 172.21.15.67:0/2422102900 >> v2:172.21.15.176:3300/0 conn(0x7fa2ac162a10 0x7fa2ac0b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac0cfab0 con 0x7fa2ac161ef0 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fa2a0001670 con 0x7fa2ac161ef0 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 >> v1:172.21.15.67:6789/0 conn(0x7fa2ac161ef0 legacy=0x7fa2ac162300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2940039b0 0x7fa294003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 0x7fa294004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa294004d40 0x7fa294005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac162e20 con 0x7fa294004d40 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac0b0690 con 0x7fa2940039b0 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa29400baf0 con 0x7fa2ac0b9270 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2940039b0 0x7fa294003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2abfff640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa294004d40 0x7fa294005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 0x7fa294004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2abfff640 1 -- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 msgr2=0x7fa294004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2abfff640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac0b9270 0x7fa294004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2abfff640 1 -- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2940039b0 msgr2=0x7fa294003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2abfff640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2940039b0 0x7fa294003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.381+0000 7fa2abfff640 1 -- 172.21.15.67:0/2422102900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa294003520 con 0x7fa294004d40 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.382+0000 7fa2abfff640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa294004d40 0x7fa294005150 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009fc0 tx=0x7fa294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a163346e6d4e12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.382+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4009c10 con 0x7fa294004d40 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.382+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2a4036070 con 0x7fa294004d40 2024-04-25T15:27:32.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.382+0000 7fa2aa7fc640 1 -- 172.21.15.67:0/2422102900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a402d070 con 0x7fa294004d40 2024-04-25T15:27:32.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.383+0000 7fa2b1e61640 1 -- 172.21.15.67:0/2422102900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa294004d40 msgr2=0x7fa294005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.383+0000 7fa2b1e61640 1 --2- 172.21.15.67:0/2422102900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa294004d40 0x7fa294005150 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009fc0 tx=0x7fa294005830 comp rx=0 tx=0).stop 2024-04-25T15:27:32.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.383+0000 7fa2abfff640 1 -- 172.21.15.67:0/2422102900 reap_dead start 2024-04-25T15:27:32.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.383+0000 7fa2b1e61640 1 -- 172.21.15.67:0/2422102900 shutdown_connections 2024-04-25T15:27:32.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.383+0000 7fa2b1e61640 1 -- 172.21.15.67:0/2422102900 >> 172.21.15.67:0/2422102900 conn(0x7fa2ac0d3aa0 msgr2=0x7fa2ac0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:32.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.383+0000 7fa2b1e61640 1 -- 172.21.15.67:0/2422102900 shutdown_connections 2024-04-25T15:27:32.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.383+0000 7fa2b1e61640 1 -- 172.21.15.67:0/2422102900 wait complete. 2024-04-25T15:27:32.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.385+0000 7fa2b1e61640 1 Processor -- start 2024-04-25T15:27:32.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.385+0000 7fa2b1e61640 1 -- start start 2024-04-25T15:27:32.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2b1e61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15e440 0x7fa2ac0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2b1e61640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2ac152850 0x7fa2ac0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2b1e61640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac152c60 0x7fa2ac0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2b1e61640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac0d1760 con 0x7fa2ac15e440 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2b1e61640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac0d15e0 con 0x7fa2ac152850 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2b1e61640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2ac0d18e0 con 0x7fa2ac152c60 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15e440 0x7fa2ac0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2ab7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15e440 0x7fa2ac0b4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50090/0 (socket says 172.21.15.67:50090) 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/3886826245 learned_addr learned my addr 172.21.15.67:0/3886826245 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2ac152850 0x7fa2ac0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.386+0000 7fa2abfff640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac152c60 0x7fa2ac0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac152c60 msgr2=0x7fa2ac0c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac152c60 0x7fa2ac0c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2ac152850 msgr2=0x7fa2ac0c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2ac152850 0x7fa2ac0c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2ab7fe640 1 -- 172.21.15.67:0/3886826245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac0b4860 con 0x7fa2ac15e440 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2ab7fe640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15e440 0x7fa2ac0b4320 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0119a0 tx=0x7fa29c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa293fff640 1 -- 172.21.15.67:0/3886826245 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c00faf0 con 0x7fa2ac15e440 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa293fff640 1 -- 172.21.15.67:0/3886826245 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa29c00fc90 con 0x7fa2ac15e440 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac0c8730 con 0x7fa2ac15e440 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa293fff640 1 -- 172.21.15.67:0/3886826245 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c014440 con 0x7fa2ac15e440 2024-04-25T15:27:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.387+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ac0c8cf0 con 0x7fa2ac15e440 2024-04-25T15:27:32.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.388+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ac050740 con 0x7fa2ac15e440 2024-04-25T15:27:32.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.389+0000 7fa293fff640 1 -- 172.21.15.67:0/3886826245 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa29c012040 con 0x7fa2ac15e440 2024-04-25T15:27:32.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.389+0000 7fa293fff640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2980744b0 0x7fa2980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.389+0000 7fa293fff640 1 -- 172.21.15.67:0/3886826245 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fa29c0956c0 con 0x7fa2ac15e440 2024-04-25T15:27:32.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.389+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2980744b0 0x7fa2980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.390+0000 7fa2aaffd640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2980744b0 0x7fa2980769c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0b5ae0 tx=0x7fa2a0007970 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.394+0000 7fa293fff640 1 -- 172.21.15.67:0/3886826245 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa29c019030 con 0x7fa2ac15e440 2024-04-25T15:27:32.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.581+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2ac0d4770 con 0x7fa2ac15e440 2024-04-25T15:27:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.582+0000 7fa293fff640 1 -- 172.21.15.67:0/3886826245 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16629 (secure 0 0 0) 0x7fa29c05f890 con 0x7fa2ac15e440 2024-04-25T15:27:32.583 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:32.583 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:20.019287+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2980744b0 msgr2=0x7fa2980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2980744b0 0x7fa2980769c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac0b5ae0 tx=0x7fa2a0007970 comp rx=0 tx=0).stop 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15e440 msgr2=0x7fa2ac0b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15e440 0x7fa2ac0b4320 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0119a0 tx=0x7fa29c011e60 comp rx=0 tx=0).stop 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 shutdown_connections 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa2980744b0 0x7fa2980769c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2ac152c60 0x7fa2ac0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2ac152850 0x7fa2ac0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 --2- 172.21.15.67:0/3886826245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2ac15e440 0x7fa2ac0b4320 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 >> 172.21.15.67:0/3886826245 conn(0x7fa2ac0d3aa0 msgr2=0x7fa2ac0685e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:32.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 shutdown_connections 2024-04-25T15:27:32.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.584+0000 7fa2b1e61640 1 -- 172.21.15.67:0/3886826245 wait complete. 2024-04-25T15:27:32.596 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:27:16.061447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:27:17.338750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:32.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.710+0000 7f63d0292640 1 Processor -- start 2024-04-25T15:27:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.710+0000 7f63d0292640 1 -- start start 2024-04-25T15:27:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63d0292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c80d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63d0292640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63c80b2e60 0x7f63c808d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63d0292640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c80d23f0 con 0x7f63c8164020 2024-04-25T15:27:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63d0292640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f63c80d26d0 con 0x7f63c80b2e60 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63d0292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80d29b0 con 0x7f63c80d1e20 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63cd82d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63c80b2e60 0x7f63c808d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63ce82f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f63c8164020 0x7f63c8164430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49706/0 (socket says 172.21.15.67:49706) 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63cd82d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63c80b2e60 0x7f63c808d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50100/0 (socket says 172.21.15.67:50100) 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63ce82f640 1 -- 172.21.15.67:0/2722457878 learned_addr learned my addr 172.21.15.67:0/2722457878 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63ce02e640 1 --2- 172.21.15.67:0/2722457878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c80d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63cd82d640 1 -- 172.21.15.67:0/2722457878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 msgr2=0x7f63c80d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63cd82d640 1 --2- 172.21.15.67:0/2722457878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c80d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63cd82d640 1 -- 172.21.15.67:0/2722457878 >> v1:172.21.15.67:6789/0 conn(0x7f63c8164020 legacy=0x7f63c8164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63cd82d640 1 -- 172.21.15.67:0/2722457878 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c80d2c90 con 0x7f63c80b2e60 2024-04-25T15:27:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.712+0000 7f63cd02c640 1 -- 172.21.15.67:0/2722457878 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2810928152 0 0) 0x7f63c80d23f0 con 0x7f63c8164020 2024-04-25T15:27:32.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.713+0000 7f63cd82d640 1 --2- 172.21.15.67:0/2722457878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c80b2e60 0x7f63c808d770 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f63b80097a0 tx=0x7f63b802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5ff6b863012f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.713+0000 7f63cd02c640 1 -- 172.21.15.67:0/2722457878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8037070 con 0x7f63c80b2e60 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.713+0000 7f63cd02c640 1 -- 172.21.15.67:0/2722457878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b8032440 con 0x7f63c80b2e60 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.713+0000 7f63cd02c640 1 -- 172.21.15.67:0/2722457878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8032810 con 0x7f63c80b2e60 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 -- 172.21.15.67:0/2722457878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c80b2e60 msgr2=0x7f63c808d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 --2- 172.21.15.67:0/2722457878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c80b2e60 0x7f63c808d770 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f63b80097a0 tx=0x7f63b802a210 comp rx=0 tx=0).stop 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 -- 172.21.15.67:0/2722457878 shutdown_connections 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 --2- 172.21.15.67:0/2722457878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c80b2e60 0x7f63c808d770 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 --2- 172.21.15.67:0/2722457878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c80d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 -- 172.21.15.67:0/2722457878 >> 172.21.15.67:0/2722457878 conn(0x7f63c80d0530 msgr2=0x7f63c80d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 -- 172.21.15.67:0/2722457878 shutdown_connections 2024-04-25T15:27:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.714+0000 7f63d0292640 1 -- 172.21.15.67:0/2722457878 wait complete. 2024-04-25T15:27:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.716+0000 7f63d0292640 1 Processor -- start 2024-04-25T15:27:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.716+0000 7f63d0292640 1 -- start start 2024-04-25T15:27:32.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63d0292640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63c80b2e60 0x7f63c81fa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63d0292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63d0292640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8164020 0x7f63c80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63d0292640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f63c80ce070 con 0x7f63c8164020 2024-04-25T15:27:32.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63d0292640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f63c808dcb0 con 0x7f63c80b2e60 2024-04-25T15:27:32.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63d0292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f63c80ce1f0 con 0x7f63c80d1e20 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63ce82f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8164020 0x7f63c80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63ce82f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8164020 0x7f63c80cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50116/0 (socket says 172.21.15.67:50116) 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63ce82f640 1 -- 172.21.15.67:0/1100678231 learned_addr learned my addr 172.21.15.67:0/1100678231 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63ce02e640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63c80b2e60 0x7f63c81fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.717+0000 7f63cd82d640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63ce82f640 1 -- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 msgr2=0x7f63c815dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63ce82f640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c815dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63ce82f640 1 -- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63c80b2e60 msgr2=0x7f63c81fa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63ce82f640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63c80b2e60 0x7f63c81fa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63ce82f640 1 -- 172.21.15.67:0/1100678231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c80b8b30 con 0x7f63c8164020 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63ce82f640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8164020 0x7f63c80cf050 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c7b0 tx=0x7f63c000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63beffd640 1 -- 172.21.15.67:0/1100678231 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0018070 con 0x7f63c8164020 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63beffd640 1 -- 172.21.15.67:0/1100678231 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c000a640 con 0x7f63c8164020 2024-04-25T15:27:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c81f6960 con 0x7f63c8164020 2024-04-25T15:27:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63beffd640 1 -- 172.21.15.67:0/1100678231 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c000ce60 con 0x7f63c8164020 2024-04-25T15:27:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.718+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c80b77a0 con 0x7f63c8164020 2024-04-25T15:27:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.719+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c8163d60 con 0x7f63c8164020 2024-04-25T15:27:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.720+0000 7f63beffd640 1 -- 172.21.15.67:0/1100678231 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f63c001b070 con 0x7f63c8164020 2024-04-25T15:27:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.720+0000 7f63beffd640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f63ac0744e0 0x7f63ac0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.720+0000 7f63beffd640 1 -- 172.21.15.67:0/1100678231 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f63c0094660 con 0x7f63c8164020 2024-04-25T15:27:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.720+0000 7f63ce02e640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f63ac0744e0 0x7f63ac0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.721+0000 7f63ce02e640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f63ac0744e0 0x7f63ac0769f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009aa0 tx=0x7f63c4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.725+0000 7f63beffd640 1 -- 172.21.15.67:0/1100678231 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f63c8163d60 con 0x7f63c8164020 2024-04-25T15:27:32.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.912+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f63c80b88c0 con 0x7f63c8164020 2024-04-25T15:27:32.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.912+0000 7f63beffd640 1 -- 172.21.15.67:0/1100678231 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) ==== 93+0+10 (secure 0 0 0) 0x7f63c0065e20 con 0x7f63c8164020 2024-04-25T15:27:32.913 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:32.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f63ac0744e0 msgr2=0x7f63ac0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f63ac0744e0 0x7f63ac0769f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009aa0 tx=0x7f63c4009190 comp rx=0 tx=0).stop 2024-04-25T15:27:32.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8164020 msgr2=0x7f63c80cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8164020 0x7f63c80cf050 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c7b0 tx=0x7f63c000cc70 comp rx=0 tx=0).stop 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 shutdown_connections 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f63ac0744e0 0x7f63ac0769f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63c8164020 0x7f63c80cf050 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f63c80d1e20 0x7f63c815dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 --2- 172.21.15.67:0/1100678231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63c80b2e60 0x7f63c81fa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 >> 172.21.15.67:0/1100678231 conn(0x7f63c80d0530 msgr2=0x7f63c80713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 shutdown_connections 2024-04-25T15:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:32.914+0000 7f63d0292640 1 -- 172.21.15.67:0/1100678231 wait complete. 2024-04-25T15:27:32.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:27:33.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.040+0000 7f4ea78d9640 1 Processor -- start 2024-04-25T15:27:33.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.040+0000 7f4ea78d9640 1 -- start start 2024-04-25T15:27:33.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea78d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea00d0e00 0x7f4ea00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea78d9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ea006c5c0 0x7f4ea006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea78d9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ea006cf10 con 0x7f4ea00b7810 2024-04-25T15:27:33.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea78d9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4ea00d5a60 con 0x7f4ea006c5c0 2024-04-25T15:27:33.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea78d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea00d1850 con 0x7f4ea00d0e00 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea4e74640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ea006c5c0 0x7f4ea006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea4e74640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ea006c5c0 0x7f4ea006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50128/0 (socket says 172.21.15.67:50128) 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea5e76640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ea00b7810 0x7f4ea00b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49720/0 (socket says 172.21.15.67:49720) 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea4e74640 1 -- 172.21.15.67:0/2284715961 learned_addr learned my addr 172.21.15.67:0/2284715961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea5675640 1 --2- 172.21.15.67:0/2284715961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea00d0e00 0x7f4ea00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea4e74640 1 -- 172.21.15.67:0/2284715961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea00d0e00 msgr2=0x7f4ea00d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea4e74640 1 --2- 172.21.15.67:0/2284715961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea00d0e00 0x7f4ea00d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea4e74640 1 -- 172.21.15.67:0/2284715961 >> v1:172.21.15.67:6789/0 conn(0x7f4ea00b7810 legacy=0x7f4ea00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4ea4e74640 1 -- 172.21.15.67:0/2284715961 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea0160de0 con 0x7f4ea006c5c0 2024-04-25T15:27:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.042+0000 7f4e97fff640 1 -- 172.21.15.67:0/2284715961 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870921590 0 0) 0x7f4e98001490 con 0x7f4ea00b7810 2024-04-25T15:27:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.043+0000 7f4ea4e74640 1 --2- 172.21.15.67:0/2284715961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea006c5c0 0x7f4ea006c9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009870 tx=0x7f4e9002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e10b4073d1b3bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.043+0000 7f4e97fff640 1 -- 172.21.15.67:0/2284715961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90037070 con 0x7f4ea006c5c0 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.043+0000 7f4e97fff640 1 -- 172.21.15.67:0/2284715961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e90032440 con 0x7f4ea006c5c0 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.043+0000 7f4e97fff640 1 -- 172.21.15.67:0/2284715961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90032810 con 0x7f4ea006c5c0 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 -- 172.21.15.67:0/2284715961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea006c5c0 msgr2=0x7f4ea006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/2284715961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea006c5c0 0x7f4ea006c9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009870 tx=0x7f4e9002a270 comp rx=0 tx=0).stop 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 -- 172.21.15.67:0/2284715961 shutdown_connections 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/2284715961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea006c5c0 0x7f4ea006c9d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/2284715961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea00d0e00 0x7f4ea00d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 -- 172.21.15.67:0/2284715961 >> 172.21.15.67:0/2284715961 conn(0x7f4ea00d5d90 msgr2=0x7f4ea00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 -- 172.21.15.67:0/2284715961 shutdown_connections 2024-04-25T15:27:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.044+0000 7f4ea78d9640 1 -- 172.21.15.67:0/2284715961 wait complete. 2024-04-25T15:27:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.046+0000 7f4ea78d9640 1 Processor -- start 2024-04-25T15:27:33.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.046+0000 7f4ea78d9640 1 -- start start 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea78d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea006c5c0 0x7f4ea0156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea78d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00b7810 0x7f4ea00caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea78d9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ea00d0e00 0x7f4ea00baf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea78d9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea0160ad0 con 0x7f4ea00b7810 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea78d9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea01613a0 con 0x7f4ea00d0e00 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea78d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea00b5320 con 0x7f4ea006c5c0 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00b7810 0x7f4ea00caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00b7810 0x7f4ea00caf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50132/0 (socket says 172.21.15.67:50132) 2024-04-25T15:27:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 -- 172.21.15.67:0/648342576 learned_addr learned my addr 172.21.15.67:0/648342576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea5e76640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ea00d0e00 0x7f4ea00baf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 -- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea006c5c0 msgr2=0x7f4ea0156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea5675640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea006c5c0 0x7f4ea0156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea006c5c0 0x7f4ea0156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 -- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ea00d0e00 msgr2=0x7f4ea00baf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ea00d0e00 0x7f4ea00baf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.047+0000 7f4ea4e74640 1 -- 172.21.15.67:0/648342576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea00c99d0 con 0x7f4ea00b7810 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.048+0000 7f4ea4e74640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00b7810 0x7f4ea00caf20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4e900090b0 tx=0x7f4e900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.048+0000 7f4e95ffb640 1 -- 172.21.15.67:0/648342576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90037040 con 0x7f4ea00b7810 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.048+0000 7f4e95ffb640 1 -- 172.21.15.67:0/648342576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e90004030 con 0x7f4ea00b7810 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.048+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea00c9cb0 con 0x7f4ea00b7810 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.048+0000 7f4e95ffb640 1 -- 172.21.15.67:0/648342576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90004440 con 0x7f4ea00b7810 2024-04-25T15:27:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.048+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea00ca270 con 0x7f4ea00b7810 2024-04-25T15:27:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.049+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ea0050740 con 0x7f4ea00b7810 2024-04-25T15:27:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.050+0000 7f4e95ffb640 1 -- 172.21.15.67:0/648342576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4e9003a070 con 0x7f4ea00b7810 2024-04-25T15:27:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.050+0000 7f4e95ffb640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e840744b0 0x7f4e840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.050+0000 7f4e95ffb640 1 -- 172.21.15.67:0/648342576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f4e900b5850 con 0x7f4ea00b7810 2024-04-25T15:27:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.050+0000 7f4ea5675640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e840744b0 0x7f4e840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.051+0000 7f4ea5675640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e840744b0 0x7f4e840769c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c006fd0 tx=0x7f4e9c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.055+0000 7f4e95ffb640 1 -- 172.21.15.67:0/648342576 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e90041020 con 0x7f4ea00b7810 2024-04-25T15:27:33.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.240+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4ea00d4320 con 0x7f4ea00b7810 2024-04-25T15:27:33.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.240+0000 7f4e95ffb640 1 -- 172.21.15.67:0/648342576 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) ==== 104+0+11 (secure 0 0 0) 0x7f4e90080070 con 0x7f4ea00b7810 2024-04-25T15:27:33.241 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:27:33.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e840744b0 msgr2=0x7f4e840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e840744b0 0x7f4e840769c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c006fd0 tx=0x7f4e9c008040 comp rx=0 tx=0).stop 2024-04-25T15:27:33.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00b7810 msgr2=0x7f4ea00caf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00b7810 0x7f4ea00caf20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4e900090b0 tx=0x7f4e900046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 shutdown_connections 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4e840744b0 0x7f4e840769c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ea00d0e00 0x7f4ea00baf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ea00b7810 0x7f4ea00caf20 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 --2- 172.21.15.67:0/648342576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ea006c5c0 0x7f4ea0156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.242+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 >> 172.21.15.67:0/648342576 conn(0x7f4ea00d5d90 msgr2=0x7f4ea00b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.243+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 shutdown_connections 2024-04-25T15:27:33.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.243+0000 7f4ea78d9640 1 -- 172.21.15.67:0/648342576 wait complete. 2024-04-25T15:27:33.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:27:33.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.370+0000 7f58f6405640 1 Processor -- start 2024-04-25T15:27:33.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.370+0000 7f58f6405640 1 -- start start 2024-04-25T15:27:33.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58f6405640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58f0164ad0 0x7f58f00b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58f6405640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f0163f30 0x7f58f0164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58f6405640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f58f00d1f30 0x7f58f00d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58f6405640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f58f0164ee0 con 0x7f58f0164ad0 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58f6405640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f58f00b31e0 con 0x7f58f00d1f30 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58f6405640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58f0164880 con 0x7f58f0163f30 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58effff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58f0164ad0 0x7f58f00b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58effff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58f0164ad0 0x7f58f00b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50138/0 (socket says 172.21.15.67:50138) 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58effff640 1 -- 172.21.15.67:0/1148122820 learned_addr learned my addr 172.21.15.67:0/1148122820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58ef7fe640 1 --2- 172.21.15.67:0/1148122820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f0163f30 0x7f58f0164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.371+0000 7f58f49a2640 1 --2- 172.21.15.67:0/1148122820 >> v2:172.21.15.176:3300/0 conn(0x7f58f00d1f30 0x7f58f00d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58effff640 1 -- 172.21.15.67:0/1148122820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f0163f30 msgr2=0x7f58f0164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58effff640 1 --2- 172.21.15.67:0/1148122820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f0163f30 0x7f58f0164340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58effff640 1 -- 172.21.15.67:0/1148122820 >> v2:172.21.15.176:3300/0 conn(0x7f58f00d1f30 msgr2=0x7f58f00d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58effff640 1 --2- 172.21.15.67:0/1148122820 >> v2:172.21.15.176:3300/0 conn(0x7f58f00d1f30 0x7f58f00d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58effff640 1 -- 172.21.15.67:0/1148122820 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f00d3170 con 0x7f58f0164ad0 2024-04-25T15:27:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58f49a2640 1 --2- 172.21.15.67:0/1148122820 >> v2:172.21.15.176:3300/0 conn(0x7f58f00d1f30 0x7f58f00d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58effff640 1 --2- 172.21.15.67:0/1148122820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0164ad0 0x7f58f00b2ca0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f58f00744e0 tx=0x7f58e002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e82794a35fc6d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58eeffd640 1 -- 172.21.15.67:0/1148122820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e0037070 con 0x7f58f0164ad0 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58eeffd640 1 -- 172.21.15.67:0/1148122820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58e0032440 con 0x7f58f0164ad0 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.372+0000 7f58eeffd640 1 -- 172.21.15.67:0/1148122820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e0032810 con 0x7f58f0164ad0 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 -- 172.21.15.67:0/1148122820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0164ad0 msgr2=0x7f58f00b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 --2- 172.21.15.67:0/1148122820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0164ad0 0x7f58f00b2ca0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f58f00744e0 tx=0x7f58e002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 -- 172.21.15.67:0/1148122820 shutdown_connections 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 --2- 172.21.15.67:0/1148122820 >> v2:172.21.15.176:3300/0 conn(0x7f58f00d1f30 0x7f58f00d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 --2- 172.21.15.67:0/1148122820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f0163f30 0x7f58f0164340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 --2- 172.21.15.67:0/1148122820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0164ad0 0x7f58f00b2ca0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 -- 172.21.15.67:0/1148122820 >> 172.21.15.67:0/1148122820 conn(0x7f58f00d3aa0 msgr2=0x7f58f00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 -- 172.21.15.67:0/1148122820 shutdown_connections 2024-04-25T15:27:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.373+0000 7f58f6405640 1 -- 172.21.15.67:0/1148122820 wait complete. 2024-04-25T15:27:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.375+0000 7f58f6405640 1 Processor -- start 2024-04-25T15:27:33.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.375+0000 7f58f6405640 1 -- start start 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f6405640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f00d1f30 0x7f58f015eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f6405640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0163f30 0x7f58f00c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f6405640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58f0164ad0 0x7f58f00c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f6405640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58f006d9b0 con 0x7f58f0163f30 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f6405640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f58f006d6d0 con 0x7f58f0164ad0 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f6405640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58f00b9430 con 0x7f58f00d1f30 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0163f30 0x7f58f00c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0163f30 0x7f58f00c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50144/0 (socket says 172.21.15.67:50144) 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 -- 172.21.15.67:0/3275175080 learned_addr learned my addr 172.21.15.67:0/3275175080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58effff640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f00d1f30 0x7f58f015eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f49a2640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58f0164ad0 0x7f58f00c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 -- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f00d1f30 msgr2=0x7f58f015eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f00d1f30 0x7f58f015eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 -- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58f0164ad0 msgr2=0x7f58f00c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58f0164ad0 0x7f58f00c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 -- 172.21.15.67:0/3275175080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f01eb4c0 con 0x7f58f0163f30 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f49a2640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58f0164ad0 0x7f58f00c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ef7fe640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0163f30 0x7f58f00c8f60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f58e400e8a0 tx=0x7f58e400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ecff9640 1 -- 172.21.15.67:0/3275175080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e4017070 con 0x7f58f0163f30 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ecff9640 1 -- 172.21.15.67:0/3275175080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58e400c8e0 con 0x7f58f0163f30 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58ecff9640 1 -- 172.21.15.67:0/3275175080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e400ccd0 con 0x7f58f0163f30 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.377+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f0162100 con 0x7f58f0163f30 2024-04-25T15:27:33.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.378+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f01626c0 con 0x7f58f0163f30 2024-04-25T15:27:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.379+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f00d4770 con 0x7f58f0163f30 2024-04-25T15:27:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.379+0000 7f58ecff9640 1 -- 172.21.15.67:0/3275175080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f58e401a070 con 0x7f58f0163f30 2024-04-25T15:27:33.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.380+0000 7f58ecff9640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58dc0744b0 0x7f58dc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.380+0000 7f58ecff9640 1 -- 172.21.15.67:0/3275175080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f58e40952c0 con 0x7f58f0163f30 2024-04-25T15:27:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.380+0000 7f58effff640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58dc0744b0 0x7f58dc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.381+0000 7f58effff640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58dc0744b0 0x7f58dc0769c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f58e0034000 tx=0x7f58e002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.385+0000 7f58ecff9640 1 -- 172.21.15.67:0/3275175080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58e405f520 con 0x7f58f0163f30 2024-04-25T15:27:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.570+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f58f0072dc0 con 0x7f58f0163f30 2024-04-25T15:27:33.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.570+0000 7f58ecff9640 1 -- 172.21.15.67:0/3275175080 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) ==== 100+0+11 (secure 0 0 0) 0x7f58e4063820 con 0x7f58f0163f30 2024-04-25T15:27:33.571 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:27:33.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58dc0744b0 msgr2=0x7f58dc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58dc0744b0 0x7f58dc0769c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f58e0034000 tx=0x7f58e002e000 comp rx=0 tx=0).stop 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0163f30 msgr2=0x7f58f00c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0163f30 0x7f58f00c8f60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f58e400e8a0 tx=0x7f58e400ed60 comp rx=0 tx=0).stop 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 shutdown_connections 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58dc0744b0 0x7f58dc0769c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58f0164ad0 0x7f58f00c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58f0163f30 0x7f58f00c8f60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.572+0000 7f58f6405640 1 --2- 172.21.15.67:0/3275175080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58f00d1f30 0x7f58f015eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.573+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 >> 172.21.15.67:0/3275175080 conn(0x7f58f00d3aa0 msgr2=0x7f58f006e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.573+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 shutdown_connections 2024-04-25T15:27:33.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.573+0000 7f58f6405640 1 -- 172.21.15.67:0/3275175080 wait complete. 2024-04-25T15:27:33.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.696+0000 7f8066dc5640 1 Processor -- start 2024-04-25T15:27:33.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.696+0000 7f8066dc5640 1 -- start start 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f8066dc5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f80600d1210 0x7f80600d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f8066dc5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80600d1850 con 0x7f80600d0e00 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f8066dc5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8060160de0 con 0x7f806006ccd0 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f8066dc5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f80600d1620 con 0x7f80600d1210 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f8064b61640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f80600d0e00 0x7f806006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49732/0 (socket says 172.21.15.67:49732) 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f8064b61640 1 -- 172.21.15.67:0/2660824301 learned_addr learned my addr 172.21.15.67:0/2660824301 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f8065362640 1 --2- 172.21.15.67:0/2660824301 >> v2:172.21.15.176:3300/0 conn(0x7f80600d1210 0x7f80600d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586424734 0 0) 0x7f80600d1850 con 0x7f80600d0e00 2024-04-25T15:27:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.697+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8048003520 con 0x7f80600d0e00 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1900968110 0 0) 0x7f8048003520 con 0x7f80600d0e00 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 >> v2:172.21.15.176:3300/0 conn(0x7f80600d1210 msgr2=0x7f80600d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 --2- 172.21.15.67:0/2660824301 >> v2:172.21.15.176:3300/0 conn(0x7f80600d1210 0x7f80600d5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 >> v1:172.21.15.176:6789/0 conn(0x7f806006ccd0 legacy=0x7f80600d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80601610c0 con 0x7f80600d0e00 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8050001670 con 0x7f80600d0e00 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 >> v1:172.21.15.67:6789/0 conn(0x7f80600d0e00 legacy=0x7f806006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80480039b0 0x7f8048003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8048004800 0x7f8048004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8048005250 0x7f804800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f804800c1b0 con 0x7f8048005250 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f804800c360 con 0x7f8048004800 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f804800c560 con 0x7f80480039b0 2024-04-25T15:27:33.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f8065362640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8048005250 0x7f804800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f805ffff640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8048004800 0x7f8048004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f8064b61640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80480039b0 0x7f8048003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f8065362640 1 -- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80480039b0 msgr2=0x7f8048003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f8065362640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80480039b0 0x7f8048003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f8065362640 1 -- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8048004800 msgr2=0x7f8048004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f8065362640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8048004800 0x7f8048004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.698+0000 7f8065362640 1 -- 172.21.15.67:0/2660824301 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8048003520 con 0x7f8048005250 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.699+0000 7f8065362640 1 -- 172.21.15.67:0/2660824301 reap_dead start 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.699+0000 7f8065362640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8048005250 0x7f804800b9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8058009fc0 tx=0x7f8048005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f78e35bb54b9798 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.699+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058035020 con 0x7f8048005250 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.699+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8058003b90 con 0x7f8048005250 2024-04-25T15:27:33.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.699+0000 7f805f7fe640 1 -- 172.21.15.67:0/2660824301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058009c10 con 0x7f8048005250 2024-04-25T15:27:33.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.700+0000 7f8066dc5640 1 -- 172.21.15.67:0/2660824301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8048005250 msgr2=0x7f804800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.700+0000 7f8066dc5640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8048005250 0x7f804800b9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8058009fc0 tx=0x7f8048005830 comp rx=0 tx=0).stop 2024-04-25T15:27:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.700+0000 7f8066dc5640 1 -- 172.21.15.67:0/2660824301 shutdown_connections 2024-04-25T15:27:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.700+0000 7f8066dc5640 1 --2- 172.21.15.67:0/2660824301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8048005250 0x7f804800b9c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.700+0000 7f8066dc5640 1 -- 172.21.15.67:0/2660824301 >> 172.21.15.67:0/2660824301 conn(0x7f80600d5d90 msgr2=0x7f80600d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.700+0000 7f8066dc5640 1 -- 172.21.15.67:0/2660824301 shutdown_connections 2024-04-25T15:27:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.700+0000 7f8066dc5640 1 -- 172.21.15.67:0/2660824301 wait complete. 2024-04-25T15:27:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.702+0000 7f8066dc5640 1 Processor -- start 2024-04-25T15:27:33.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.702+0000 7f8066dc5640 1 -- start start 2024-04-25T15:27:33.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.703+0000 7f8066dc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8048005250 0x7f806015d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.703+0000 7f8066dc5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f806015dd70 0x7f806014a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.703+0000 7f8066dc5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f806014add0 0x7f806014b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.703+0000 7f8066dc5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80600b5320 con 0x7f806015dd70 2024-04-25T15:27:33.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.703+0000 7f8066dc5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8060160ad0 con 0x7f806014add0 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.703+0000 7f8066dc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8060164e00 con 0x7f8048005250 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.703+0000 7f805ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f806015dd70 0x7f806014a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f806015dd70 0x7f806014a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50160/0 (socket says 172.21.15.67:50160) 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 -- 172.21.15.67:0/1466741142 learned_addr learned my addr 172.21.15.67:0/1466741142 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f8064b61640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8048005250 0x7f806015d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f8065362640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f806014add0 0x7f806014b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 -- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8048005250 msgr2=0x7f806015d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8048005250 0x7f806015d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 -- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f806014add0 msgr2=0x7f806014b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f806014add0 0x7f806014b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 -- 172.21.15.67:0/1466741142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8060157c00 con 0x7f806015dd70 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f8065362640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f806014add0 0x7f806014b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805ffff640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f806015dd70 0x7f806014a890 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f805400f5f0 tx=0x7f805400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805d7fa640 1 -- 172.21.15.67:0/1466741142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80540052e0 con 0x7f806015dd70 2024-04-25T15:27:33.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805d7fa640 1 -- 172.21.15.67:0/1466741142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8054018070 con 0x7f806015dd70 2024-04-25T15:27:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8060157ee0 con 0x7f806015dd70 2024-04-25T15:27:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f805d7fa640 1 -- 172.21.15.67:0/1466741142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054003810 con 0x7f806015dd70 2024-04-25T15:27:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.704+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80601584a0 con 0x7f806015dd70 2024-04-25T15:27:33.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.705+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8060157950 con 0x7f806015dd70 2024-04-25T15:27:33.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.706+0000 7f805d7fa640 1 -- 172.21.15.67:0/1466741142 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f80540039b0 con 0x7f806015dd70 2024-04-25T15:27:33.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.707+0000 7f805d7fa640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f804c0744e0 0x7f804c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:33.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.707+0000 7f805d7fa640 1 -- 172.21.15.67:0/1466741142 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f8054095030 con 0x7f806015dd70 2024-04-25T15:27:33.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.707+0000 7f8064b61640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f804c0744e0 0x7f804c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:33.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.708+0000 7f8064b61640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f804c0744e0 0x7f804c0769f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8050007c10 tx=0x7f8050009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:33.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.711+0000 7f805d7fa640 1 -- 172.21.15.67:0/1466741142 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f805405f7c0 con 0x7f806015dd70 2024-04-25T15:27:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.935+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8060050740 con 0x7f806015dd70 2024-04-25T15:27:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.936+0000 7f805d7fa640 1 -- 172.21.15.67:0/1466741142 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5181 (secure 0 0 0) 0x7f805405f0c0 con 0x7f806015dd70 2024-04-25T15:27:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:33.937 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/610105566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":610105566},{"type":"v1","addr":"172.21.15.67:6835","nonce":610105566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/947236976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":947236976},{"type":"v1","addr":"172.21.15.176:6837","nonce":947236976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3211936625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3211936625},{"type":"v1","addr":"172.21.15.176:6836","nonce":3211936625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:18.413498+0000","modified":"2024-04-25T15:27:21.411023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5880":{"gid":5880,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3185771646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3185771646},{"type":"v1","addr":"172.21.15.67:6833","nonce":3185771646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5880,"qdb_cluster":[5880]},"id":4}]} 2024-04-25T15:27:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f804c0744e0 msgr2=0x7f804c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f804c0744e0 0x7f804c0769f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8050007c10 tx=0x7f8050009040 comp rx=0 tx=0).stop 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f806015dd70 msgr2=0x7f806014a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f806015dd70 0x7f806014a890 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f805400f5f0 tx=0x7f805400fab0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 shutdown_connections 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f804c0744e0 0x7f804c0769f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f806014add0 0x7f806014b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f806015dd70 0x7f806014a890 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 --2- 172.21.15.67:0/1466741142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8048005250 0x7f806015d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 >> 172.21.15.67:0/1466741142 conn(0x7f80600d5d90 msgr2=0x7f806015f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 shutdown_connections 2024-04-25T15:27:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:33.938+0000 7f8066dc5640 1 -- 172.21.15.67:0/1466741142 wait complete. 2024-04-25T15:27:33.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.062+0000 7f9b369f3640 1 Processor -- start 2024-04-25T15:27:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.062+0000 7f9b369f3640 1 -- start start 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b369f3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9b300cf5e0 0x7f9b300cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b369f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b3006c5c0 0x7f9b3006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b369f3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b300cff30 con 0x7f9b301443c0 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b369f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9b300708d0 con 0x7f9b3006c5c0 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b369f3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9b3015f160 con 0x7f9b300cf5e0 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b34f90640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b3006c5c0 0x7f9b3006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b34f90640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9b3006c5c0 0x7f9b3006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50174/0 (socket says 172.21.15.67:50174) 2024-04-25T15:27:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b34f90640 1 -- 172.21.15.67:0/2841437913 learned_addr learned my addr 172.21.15.67:0/2841437913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b2f7fe640 1 --2- 172.21.15.67:0/2841437913 >> v2:172.21.15.176:3300/0 conn(0x7f9b300cf5e0 0x7f9b300cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b34f90640 1 -- 172.21.15.67:0/2841437913 >> v2:172.21.15.176:3300/0 conn(0x7f9b300cf5e0 msgr2=0x7f9b300cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b34f90640 1 --2- 172.21.15.67:0/2841437913 >> v2:172.21.15.176:3300/0 conn(0x7f9b300cf5e0 0x7f9b300cf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b34f90640 1 -- 172.21.15.67:0/2841437913 >> v1:172.21.15.176:6789/0 conn(0x7f9b301443c0 legacy=0x7f9b3006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.064+0000 7f9b34f90640 1 -- 172.21.15.67:0/2841437913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3015f5f0 con 0x7f9b3006c5c0 2024-04-25T15:27:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.065+0000 7f9b34f90640 1 --2- 172.21.15.67:0/2841437913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3006c5c0 0x7f9b3006c9d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9b3006f8f0 tx=0x7f9b2802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a492097a4d2a78f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.065+0000 7f9b2effd640 1 -- 172.21.15.67:0/2841437913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b28036070 con 0x7f9b3006c5c0 2024-04-25T15:27:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.065+0000 7f9b2effd640 1 -- 172.21.15.67:0/2841437913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b28031600 con 0x7f9b3006c5c0 2024-04-25T15:27:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.065+0000 7f9b2effd640 1 -- 172.21.15.67:0/2841437913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b280319d0 con 0x7f9b3006c5c0 2024-04-25T15:27:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 -- 172.21.15.67:0/2841437913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3006c5c0 msgr2=0x7f9b3006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 --2- 172.21.15.67:0/2841437913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3006c5c0 0x7f9b3006c9d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9b3006f8f0 tx=0x7f9b2802a6f0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 -- 172.21.15.67:0/2841437913 shutdown_connections 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 --2- 172.21.15.67:0/2841437913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3006c5c0 0x7f9b3006c9d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 --2- 172.21.15.67:0/2841437913 >> v2:172.21.15.176:3300/0 conn(0x7f9b300cf5e0 0x7f9b300cf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 -- 172.21.15.67:0/2841437913 >> 172.21.15.67:0/2841437913 conn(0x7f9b300d1ae0 msgr2=0x7f9b300d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 -- 172.21.15.67:0/2841437913 shutdown_connections 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.066+0000 7f9b369f3640 1 -- 172.21.15.67:0/2841437913 wait complete. 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.068+0000 7f9b369f3640 1 Processor -- start 2024-04-25T15:27:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.068+0000 7f9b369f3640 1 -- start start 2024-04-25T15:27:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b369f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b3006c5c0 0x7f9b301432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b369f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300cf5e0 0x7f9b300aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b369f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b301443c0 0x7f9b301e9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b369f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3015edf0 con 0x7f9b300cf5e0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b369f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3015eb10 con 0x7f9b301443c0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b369f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b301447d0 con 0x7f9b3006c5c0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300cf5e0 0x7f9b300aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300cf5e0 0x7f9b300aa6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50184/0 (socket says 172.21.15.67:50184) 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.069+0000 7f9b2f7fe640 1 -- 172.21.15.67:0/3039694303 learned_addr learned my addr 172.21.15.67:0/3039694303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b34f90640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b301443c0 0x7f9b301e9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2ffff640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b3006c5c0 0x7f9b301432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2f7fe640 1 -- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b3006c5c0 msgr2=0x7f9b301432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2f7fe640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b3006c5c0 0x7f9b301432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2f7fe640 1 -- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b301443c0 msgr2=0x7f9b301e9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2f7fe640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b301443c0 0x7f9b301e9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2f7fe640 1 -- 172.21.15.67:0/3039694303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b301437f0 con 0x7f9b300cf5e0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2f7fe640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300cf5e0 0x7f9b300aa6b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000d5f0 tx=0x7f9b2000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2cff9640 1 -- 172.21.15.67:0/3039694303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b200125f0 con 0x7f9b300cf5e0 2024-04-25T15:27:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2cff9640 1 -- 172.21.15.67:0/3039694303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b20019070 con 0x7f9b300cf5e0 2024-04-25T15:27:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b30158e50 con 0x7f9b300cf5e0 2024-04-25T15:27:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b2cff9640 1 -- 172.21.15.67:0/3039694303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b200129d0 con 0x7f9b300cf5e0 2024-04-25T15:27:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.070+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b30159410 con 0x7f9b300cf5e0 2024-04-25T15:27:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.072+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b30050740 con 0x7f9b300cf5e0 2024-04-25T15:27:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.072+0000 7f9b2cff9640 1 -- 172.21.15.67:0/3039694303 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9b20012b70 con 0x7f9b300cf5e0 2024-04-25T15:27:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.073+0000 7f9b2cff9640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b1c0744e0 0x7f9b1c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.073+0000 7f9b2cff9640 1 -- 172.21.15.67:0/3039694303 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f9b200954f0 con 0x7f9b300cf5e0 2024-04-25T15:27:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.073+0000 7f9b2ffff640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b1c0744e0 0x7f9b1c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.073+0000 7f9b2ffff640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b1c0744e0 0x7f9b1c0769f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b24006fd0 tx=0x7f9b24009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.077+0000 7f9b2cff9640 1 -- 172.21.15.67:0/3039694303 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b2005f630 con 0x7f9b300cf5e0 2024-04-25T15:27:34.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.262+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b3006f950 con 0x7f9b300cf5e0 2024-04-25T15:27:34.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.263+0000 7f9b2cff9640 1 -- 172.21.15.67:0/3039694303 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16629 (secure 0 0 0) 0x7f9b20063930 con 0x7f9b300cf5e0 2024-04-25T15:27:34.265 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:34.265 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:20.019287+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b1c0744e0 msgr2=0x7f9b1c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b1c0744e0 0x7f9b1c0769f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b24006fd0 tx=0x7f9b24009060 comp rx=0 tx=0).stop 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300cf5e0 msgr2=0x7f9b300aa6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300cf5e0 0x7f9b300aa6b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9b2000d5f0 tx=0x7f9b2000dab0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 shutdown_connections 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b1c0744e0 0x7f9b1c0769f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b301443c0 0x7f9b301e9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b300cf5e0 0x7f9b300aa6b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 --2- 172.21.15.67:0/3039694303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b3006c5c0 0x7f9b301432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.265+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 >> 172.21.15.67:0/3039694303 conn(0x7f9b300d1ae0 msgr2=0x7f9b3015f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.266+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 shutdown_connections 2024-04-25T15:27:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.266+0000 7f9b369f3640 1 -- 172.21.15.67:0/3039694303 wait complete. 2024-04-25T15:27:34.278 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:27:34.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.388+0000 7ff0408d9640 1 Processor -- start 2024-04-25T15:27:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.388+0000 7ff0408d9640 1 -- start start 2024-04-25T15:27:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff0408d9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff03c0d4570 0x7ff03c0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff0408d9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff03c06e8f0 0x7ff03c0d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff0408d9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03c06d870 con 0x7ff03c0d09f0 2024-04-25T15:27:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff0408d9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff03c06e460 con 0x7ff03c06e8f0 2024-04-25T15:27:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff0408d9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff03c06db10 con 0x7ff03c0d4570 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff039d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff03c06e8f0 0x7ff03c0d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff039d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff03c06e8f0 0x7ff03c0d04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50188/0 (socket says 172.21.15.67:50188) 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff039d9b640 1 -- 172.21.15.67:0/2599035864 learned_addr learned my addr 172.21.15.67:0/2599035864 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff03ad9d640 1 --1- 172.21.15.67:0/2599035864 >> v1:172.21.15.67:6789/0 conn(0x7ff03c0d09f0 0x7ff03c0b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49744/0 (socket says 172.21.15.67:49744) 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff03a59c640 1 --2- 172.21.15.67:0/2599035864 >> v2:172.21.15.176:3300/0 conn(0x7ff03c0d4570 0x7ff03c0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff039d9b640 1 -- 172.21.15.67:0/2599035864 >> v2:172.21.15.176:3300/0 conn(0x7ff03c0d4570 msgr2=0x7ff03c0b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff039d9b640 1 --2- 172.21.15.67:0/2599035864 >> v2:172.21.15.176:3300/0 conn(0x7ff03c0d4570 0x7ff03c0b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff039d9b640 1 -- 172.21.15.67:0/2599035864 >> v1:172.21.15.67:6789/0 conn(0x7ff03c0d09f0 legacy=0x7ff03c0b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff039d9b640 1 -- 172.21.15.67:0/2599035864 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c06e0f0 con 0x7ff03c06e8f0 2024-04-25T15:27:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.390+0000 7ff03959a640 1 -- 172.21.15.67:0/2599035864 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054566243 0 0) 0x7ff03c06d870 con 0x7ff03c0d09f0 2024-04-25T15:27:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.391+0000 7ff039d9b640 1 --2- 172.21.15.67:0/2599035864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c0d04b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff03c06e5e0 tx=0x7ff02402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96b9c6a2a298844 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.391+0000 7ff03959a640 1 -- 172.21.15.67:0/2599035864 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024037070 con 0x7ff03c06e8f0 2024-04-25T15:27:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.391+0000 7ff03959a640 1 -- 172.21.15.67:0/2599035864 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff024032440 con 0x7ff03c06e8f0 2024-04-25T15:27:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.391+0000 7ff03959a640 1 -- 172.21.15.67:0/2599035864 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024032810 con 0x7ff03c06e8f0 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 -- 172.21.15.67:0/2599035864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 msgr2=0x7ff03c0d04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 --2- 172.21.15.67:0/2599035864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c0d04b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff03c06e5e0 tx=0x7ff02402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 -- 172.21.15.67:0/2599035864 shutdown_connections 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 --2- 172.21.15.67:0/2599035864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c0d04b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 --2- 172.21.15.67:0/2599035864 >> v2:172.21.15.176:3300/0 conn(0x7ff03c0d4570 0x7ff03c0b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 -- 172.21.15.67:0/2599035864 >> 172.21.15.67:0/2599035864 conn(0x7ff03c0d3a50 msgr2=0x7ff03c0d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 -- 172.21.15.67:0/2599035864 shutdown_connections 2024-04-25T15:27:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.392+0000 7ff0408d9640 1 -- 172.21.15.67:0/2599035864 wait complete. 2024-04-25T15:27:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.394+0000 7ff0408d9640 1 Processor -- start 2024-04-25T15:27:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.394+0000 7ff0408d9640 1 -- start start 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.395+0000 7ff0408d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c15dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff0408d9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c0d09f0 0x7ff03c14e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff0408d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c0d4570 0x7ff03c14ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff0408d9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c0b34d0 con 0x7ff03c06e8f0 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff0408d9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c06ed00 con 0x7ff03c0d09f0 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff0408d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff03c0b3650 con 0x7ff03c0d4570 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c15dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c15dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50202/0 (socket says 172.21.15.67:50202) 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 -- 172.21.15.67:0/1939083068 learned_addr learned my addr 172.21.15.67:0/1939083068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff039d9b640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c0d09f0 0x7ff03c14e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03ad9d640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c0d4570 0x7ff03c14ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 -- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c0d4570 msgr2=0x7ff03c14ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c0d4570 0x7ff03c14ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 -- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c0d09f0 msgr2=0x7ff03c14e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c0d09f0 0x7ff03c14e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 -- 172.21.15.67:0/1939083068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c15e740 con 0x7ff03c06e8f0 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03ad9d640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c0d4570 0x7ff03c14ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff03a59c640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c15dbd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff03400d800 tx=0x7ff03400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff032ffd640 1 -- 172.21.15.67:0/1939083068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034018020 con 0x7ff03c06e8f0 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff032ffd640 1 -- 172.21.15.67:0/1939083068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0340096c0 con 0x7ff03c06e8f0 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03c15ea20 con 0x7ff03c06e8f0 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff032ffd640 1 -- 172.21.15.67:0/1939083068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034009ab0 con 0x7ff03c06e8f0 2024-04-25T15:27:34.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.396+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03c0b0850 con 0x7ff03c06e8f0 2024-04-25T15:27:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.397+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03c0b6430 con 0x7ff03c06e8f0 2024-04-25T15:27:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.398+0000 7ff032ffd640 1 -- 172.21.15.67:0/1939083068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff03401b020 con 0x7ff03c06e8f0 2024-04-25T15:27:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.398+0000 7ff032ffd640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff01c0744b0 0x7ff01c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.399+0000 7ff032ffd640 1 -- 172.21.15.67:0/1939083068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7ff034095890 con 0x7ff03c06e8f0 2024-04-25T15:27:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.399+0000 7ff039d9b640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff01c0744b0 0x7ff01c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.399+0000 7ff039d9b640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff01c0744b0 0x7ff01c0769c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff024034000 tx=0x7ff02402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.403+0000 7ff032ffd640 1 -- 172.21.15.67:0/1939083068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff03c0b6430 con 0x7ff03c06e8f0 2024-04-25T15:27:34.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.626+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff03c050740 con 0x7ff03c06e8f0 2024-04-25T15:27:34.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.627+0000 7ff032ffd640 1 -- 172.21.15.67:0/1939083068 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5181 (secure 0 0 0) 0x7ff034022020 con 0x7ff03c06e8f0 2024-04-25T15:27:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:34.628 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/610105566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":610105566},{"type":"v1","addr":"172.21.15.67:6835","nonce":610105566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/947236976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":947236976},{"type":"v1","addr":"172.21.15.176:6837","nonce":947236976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3211936625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3211936625},{"type":"v1","addr":"172.21.15.176:6836","nonce":3211936625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:18.413498+0000","modified":"2024-04-25T15:27:21.411023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5880":{"gid":5880,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3185771646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3185771646},{"type":"v1","addr":"172.21.15.67:6833","nonce":3185771646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5880,"qdb_cluster":[5880]},"id":4}]} 2024-04-25T15:27:34.628 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff01c0744b0 msgr2=0x7ff01c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff01c0744b0 0x7ff01c0769c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff024034000 tx=0x7ff02402e000 comp rx=0 tx=0).stop 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 msgr2=0x7ff03c15dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c15dbd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff03400d800 tx=0x7ff03400dcc0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 shutdown_connections 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff01c0744b0 0x7ff01c0769c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff03c0d4570 0x7ff03c14ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03c0d09f0 0x7ff03c14e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 --2- 172.21.15.67:0/1939083068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03c06e8f0 0x7ff03c15dbd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 >> 172.21.15.67:0/1939083068 conn(0x7ff03c0d3a50 msgr2=0x7ff03c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:34.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 shutdown_connections 2024-04-25T15:27:34.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.629+0000 7ff0408d9640 1 -- 172.21.15.67:0/1939083068 wait complete. 2024-04-25T15:27:34.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.752+0000 7f5d3b096640 1 Processor -- start 2024-04-25T15:27:34.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.752+0000 7f5d3b096640 1 -- start start 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.753+0000 7f5d3b096640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5d340d4570 0x7f5d340b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.753+0000 7f5d3b096640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d3406e8f0 0x7f5d340d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.753+0000 7f5d3b096640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d3406d870 con 0x7f5d340d09f0 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.753+0000 7f5d3b096640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3406e460 con 0x7f5d3406e8f0 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.753+0000 7f5d3b096640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3406db10 con 0x7f5d340d4570 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d33fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d3406e8f0 0x7f5d340d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d33fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d3406e8f0 0x7f5d340d04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50218/0 (socket says 172.21.15.67:50218) 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d33fff640 1 -- 172.21.15.67:0/4244919041 learned_addr learned my addr 172.21.15.67:0/4244919041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d38e32640 1 --2- 172.21.15.67:0/4244919041 >> v2:172.21.15.176:3300/0 conn(0x7f5d340d4570 0x7f5d340b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d33fff640 1 -- 172.21.15.67:0/4244919041 >> v2:172.21.15.176:3300/0 conn(0x7f5d340d4570 msgr2=0x7f5d340b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d33fff640 1 --2- 172.21.15.67:0/4244919041 >> v2:172.21.15.176:3300/0 conn(0x7f5d340d4570 0x7f5d340b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d33fff640 1 -- 172.21.15.67:0/4244919041 >> v1:172.21.15.176:6789/0 conn(0x7f5d340d09f0 legacy=0x7f5d340b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.754+0000 7f5d33fff640 1 -- 172.21.15.67:0/4244919041 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3406e0f0 con 0x7f5d3406e8f0 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d33fff640 1 --2- 172.21.15.67:0/4244919041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d3406e8f0 0x7f5d340d04b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5d3406e5e0 tx=0x7f5d2002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158379daff5ca206 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d337fe640 1 -- 172.21.15.67:0/4244919041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20037070 con 0x7f5d3406e8f0 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d337fe640 1 -- 172.21.15.67:0/4244919041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d20032440 con 0x7f5d3406e8f0 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d337fe640 1 -- 172.21.15.67:0/4244919041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20032810 con 0x7f5d3406e8f0 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d3b096640 1 -- 172.21.15.67:0/4244919041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d3406e8f0 msgr2=0x7f5d340d04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d3b096640 1 --2- 172.21.15.67:0/4244919041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d3406e8f0 0x7f5d340d04b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5d3406e5e0 tx=0x7f5d2002a180 comp rx=0 tx=0).stop 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d3b096640 1 -- 172.21.15.67:0/4244919041 shutdown_connections 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d3b096640 1 --2- 172.21.15.67:0/4244919041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d3406e8f0 0x7f5d340d04b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d3b096640 1 --2- 172.21.15.67:0/4244919041 >> v2:172.21.15.176:3300/0 conn(0x7f5d340d4570 0x7f5d340b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d3b096640 1 -- 172.21.15.67:0/4244919041 >> 172.21.15.67:0/4244919041 conn(0x7f5d340d3a50 msgr2=0x7f5d340d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.755+0000 7f5d3b096640 1 -- 172.21.15.67:0/4244919041 shutdown_connections 2024-04-25T15:27:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.756+0000 7f5d3b096640 1 -- 172.21.15.67:0/4244919041 wait complete. 2024-04-25T15:27:34.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.757+0000 7f5d3b096640 1 Processor -- start 2024-04-25T15:27:34.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.757+0000 7f5d3b096640 1 -- start start 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d3b096640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3406e8f0 0x7f5d34156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d3b096640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340d09f0 0x7f5d340c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d3b096640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d340d4570 0x7f5d340b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d3b096640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d340b34d0 con 0x7f5d340d09f0 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d3b096640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3406ed00 con 0x7f5d340d4570 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d3b096640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d340b3650 con 0x7f5d3406e8f0 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340d09f0 0x7f5d340c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340d09f0 0x7f5d340c7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50232/0 (socket says 172.21.15.67:50232) 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 -- 172.21.15.67:0/189326353 learned_addr learned my addr 172.21.15.67:0/189326353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d38e32640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3406e8f0 0x7f5d34156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d39633640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d340d4570 0x7f5d340b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 -- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3406e8f0 msgr2=0x7f5d34156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3406e8f0 0x7f5d34156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 -- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d340d4570 msgr2=0x7f5d340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d340d4570 0x7f5d340b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 -- 172.21.15.67:0/189326353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d340a00a0 con 0x7f5d340d09f0 2024-04-25T15:27:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.759+0000 7f5d33fff640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340d09f0 0x7f5d340c7ef0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5d20000c00 tx=0x7f5d200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.760+0000 7f5d317fa640 1 -- 172.21.15.67:0/189326353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20041020 con 0x7f5d340d09f0 2024-04-25T15:27:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.760+0000 7f5d317fa640 1 -- 172.21.15.67:0/189326353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d200306c0 con 0x7f5d340d09f0 2024-04-25T15:27:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.760+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d340a0580 con 0x7f5d340d09f0 2024-04-25T15:27:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.760+0000 7f5d317fa640 1 -- 172.21.15.67:0/189326353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20037040 con 0x7f5d340d09f0 2024-04-25T15:27:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.760+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d340a0b40 con 0x7f5d340d09f0 2024-04-25T15:27:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.761+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d340a02f0 con 0x7f5d340d09f0 2024-04-25T15:27:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.761+0000 7f5d317fa640 1 -- 172.21.15.67:0/189326353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5d20044020 con 0x7f5d340d09f0 2024-04-25T15:27:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.762+0000 7f5d317fa640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d180744b0 0x7f5d180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.762+0000 7f5d317fa640 1 -- 172.21.15.67:0/189326353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f5d200b5990 con 0x7f5d340d09f0 2024-04-25T15:27:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.762+0000 7f5d38e32640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d180744b0 0x7f5d180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.763+0000 7f5d38e32640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d180744b0 0x7f5d180769c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5d28005f80 tx=0x7f5d28008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:34.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.767+0000 7f5d317fa640 1 -- 172.21.15.67:0/189326353 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d2004b030 con 0x7f5d340d09f0 2024-04-25T15:27:34.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.989+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d340d4980 con 0x7f5d340d09f0 2024-04-25T15:27:34.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.990+0000 7f5d317fa640 1 -- 172.21.15.67:0/189326353 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5181 (secure 0 0 0) 0x7f5d340d4980 con 0x7f5d340d09f0 2024-04-25T15:27:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:34.991 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/610105566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":610105566},{"type":"v1","addr":"172.21.15.67:6835","nonce":610105566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/947236976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":947236976},{"type":"v1","addr":"172.21.15.176:6837","nonce":947236976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3211936625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3211936625},{"type":"v1","addr":"172.21.15.176:6836","nonce":3211936625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:18.413498+0000","modified":"2024-04-25T15:27:21.411023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5880},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5880":{"gid":5880,"name":"a","rank":0,"incarnation":29,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6833/3185771646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3185771646},{"type":"v1","addr":"172.21.15.67:6833","nonce":3185771646}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5880,"qdb_cluster":[5880]},"id":4}]} 2024-04-25T15:27:34.991 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 32 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d180744b0 msgr2=0x7f5d180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d180744b0 0x7f5d180769c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5d28005f80 tx=0x7f5d28008040 comp rx=0 tx=0).stop 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340d09f0 msgr2=0x7f5d340c7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340d09f0 0x7f5d340c7ef0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5d20000c00 tx=0x7f5d200046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 shutdown_connections 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5d180744b0 0x7f5d180769c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d340d4570 0x7f5d340b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d340d09f0 0x7f5d340c7ef0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 --2- 172.21.15.67:0/189326353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3406e8f0 0x7f5d34156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 >> 172.21.15.67:0/189326353 conn(0x7f5d340d3a50 msgr2=0x7f5d340712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 shutdown_connections 2024-04-25T15:27:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:34.992+0000 7f5d3b096640 1 -- 172.21.15.67:0/189326353 wait complete. 2024-04-25T15:27:35.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.117+0000 7fc91552e640 1 Processor -- start 2024-04-25T15:27:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.117+0000 7fc91552e640 1 -- start start 2024-04-25T15:27:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.118+0000 7fc91552e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc9100b7750 0x7fc9100d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.118+0000 7fc91552e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 0x7fc910164800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.118+0000 7fc91552e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9100b3af0 con 0x7fc910164d40 2024-04-25T15:27:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.118+0000 7fc91552e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc910164320 con 0x7fc9100b7750 2024-04-25T15:27:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.118+0000 7fc91552e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9100b2f70 con 0x7fc9100cf610 2024-04-25T15:27:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.118+0000 7fc90f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc910164d40 0x7fc91006d7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49756/0 (socket says 172.21.15.67:49756) 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.118+0000 7fc90f7fe640 1 -- 172.21.15.67:0/1330753152 learned_addr learned my addr 172.21.15.67:0/1330753152 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90e7fc640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 0x7fc910164800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90effd640 1 --2- 172.21.15.67:0/1330753152 >> v2:172.21.15.176:3300/0 conn(0x7fc9100b7750 0x7fc9100d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2924136095 0 0) 0x7fc9100b3af0 con 0x7fc910164d40 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8fc003520 con 0x7fc910164d40 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2590755520 0 0) 0x7fc8fc003520 con 0x7fc910164d40 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 msgr2=0x7fc910164800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 0x7fc910164800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 >> v2:172.21.15.176:3300/0 conn(0x7fc9100b7750 msgr2=0x7fc9100d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 --2- 172.21.15.67:0/1330753152 >> v2:172.21.15.176:3300/0 conn(0x7fc9100b7750 0x7fc9100d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9100b3250 con 0x7fc910164d40 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fc9000019f0 con 0x7fc910164d40 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 >> v1:172.21.15.67:6789/0 conn(0x7fc910164d40 legacy=0x7fc91006d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8fc0039b0 0x7fc8fc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 0x7fc8fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc8fc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc910164320 con 0x7fc8fc004d40 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc9100b2f70 con 0x7fc8fc0039b0 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc8fc00bac0 con 0x7fc9100cf610 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90f7fe640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc8fc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90effd640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8fc0039b0 0x7fc8fc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.119+0000 7fc90e7fc640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 0x7fc8fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90f7fe640 1 -- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 msgr2=0x7fc8fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90f7fe640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9100cf610 0x7fc8fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90f7fe640 1 -- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8fc0039b0 msgr2=0x7fc8fc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90f7fe640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8fc0039b0 0x7fc8fc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90f7fe640 1 -- 172.21.15.67:0/1330753152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8fc003520 con 0x7fc8fc004d40 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90f7fe640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc8fc005150 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc90000a870 tx=0x7fc8fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f21d83cf17172df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90000a4c0 con 0x7fc8fc004d40 2024-04-25T15:27:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc90002f070 con 0x7fc8fc004d40 2024-04-25T15:27:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.120+0000 7fc90dffb640 1 -- 172.21.15.67:0/1330753152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90002bce0 con 0x7fc8fc004d40 2024-04-25T15:27:35.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.121+0000 7fc91552e640 1 -- 172.21.15.67:0/1330753152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 msgr2=0x7fc8fc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.121+0000 7fc91552e640 1 --2- 172.21.15.67:0/1330753152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc8fc005150 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc90000a870 tx=0x7fc8fc005830 comp rx=0 tx=0).stop 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.121+0000 7fc90f7fe640 1 -- 172.21.15.67:0/1330753152 reap_dead start 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.121+0000 7fc91552e640 1 -- 172.21.15.67:0/1330753152 shutdown_connections 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.121+0000 7fc91552e640 1 -- 172.21.15.67:0/1330753152 >> 172.21.15.67:0/1330753152 conn(0x7fc91009b7d0 msgr2=0x7fc9100b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.121+0000 7fc91552e640 1 -- 172.21.15.67:0/1330753152 shutdown_connections 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.121+0000 7fc91552e640 1 -- 172.21.15.67:0/1330753152 wait complete. 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.123+0000 7fc91552e640 1 Processor -- start 2024-04-25T15:27:35.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.123+0000 7fc91552e640 1 -- start start 2024-04-25T15:27:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.124+0000 7fc91552e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc91015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.124+0000 7fc91552e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc91015f700 0x7fc910162f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.124+0000 7fc91552e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9101634a0 0x7fc9100aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.124+0000 7fc91552e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9100cfa20 con 0x7fc8fc004d40 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.124+0000 7fc91552e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc9100b3810 con 0x7fc91015f700 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.124+0000 7fc91552e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc91006def0 con 0x7fc9101634a0 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc91015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc91015f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50250/0 (socket says 172.21.15.67:50250) 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 -- 172.21.15.67:0/1434101139 learned_addr learned my addr 172.21.15.67:0/1434101139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90f7fe640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9101634a0 0x7fc9100aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90e7fc640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc91015f700 0x7fc910162f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 -- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9101634a0 msgr2=0x7fc9100aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9101634a0 0x7fc9100aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 -- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc91015f700 msgr2=0x7fc910162f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc91015f700 0x7fc910162f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 -- 172.21.15.67:0/1434101139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9101638b0 con 0x7fc8fc004d40 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc90effd640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc91015f1c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc908011930 tx=0x7fc908011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc9077fe640 1 -- 172.21.15.67:0/1434101139 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90800fa90 con 0x7fc8fc004d40 2024-04-25T15:27:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc9077fe640 1 -- 172.21.15.67:0/1434101139 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc90800fc30 con 0x7fc8fc004d40 2024-04-25T15:27:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc9077fe640 1 -- 172.21.15.67:0/1434101139 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908014440 con 0x7fc8fc004d40 2024-04-25T15:27:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc91015dbd0 con 0x7fc8fc004d40 2024-04-25T15:27:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.125+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc91015e190 con 0x7fc8fc004d40 2024-04-25T15:27:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.126+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc910050740 con 0x7fc8fc004d40 2024-04-25T15:27:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.127+0000 7fc9077fe640 1 -- 172.21.15.67:0/1434101139 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc908012040 con 0x7fc8fc004d40 2024-04-25T15:27:35.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.127+0000 7fc9077fe640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8f00744b0 0x7fc8f00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.127+0000 7fc9077fe640 1 -- 172.21.15.67:0/1434101139 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fc9080958e0 con 0x7fc8fc004d40 2024-04-25T15:27:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.128+0000 7fc90e7fc640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8f00744b0 0x7fc8f00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.128+0000 7fc90e7fc640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8f00744b0 0x7fc8f00769c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc91015a220 tx=0x7fc8f8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.132+0000 7fc9077fe640 1 -- 172.21.15.67:0/1434101139 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc9101638b0 con 0x7fc8fc004d40 2024-04-25T15:27:35.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.318+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc910157d40 con 0x7fc8fc004d40 2024-04-25T15:27:35.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.320+0000 7fc9077fe640 1 -- 172.21.15.67:0/1434101139 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16629 (secure 0 0 0) 0x7fc908019030 con 0x7fc8fc004d40 2024-04-25T15:27:35.320 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:35.321 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:20.019287+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:16.061447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:17.338750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.321+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8f00744b0 msgr2=0x7fc8f00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.321+0000 7fc91552e640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8f00744b0 0x7fc8f00769c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc91015a220 tx=0x7fc8f8008040 comp rx=0 tx=0).stop 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.321+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 msgr2=0x7fc91015f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.321+0000 7fc91552e640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc91015f1c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc908011930 tx=0x7fc908011df0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 shutdown_connections 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc8f00744b0 0x7fc8f00769c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9101634a0 0x7fc9100aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc91015f700 0x7fc910162f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 --2- 172.21.15.67:0/1434101139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8fc004d40 0x7fc91015f1c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 >> 172.21.15.67:0/1434101139 conn(0x7fc91009b7d0 msgr2=0x7fc9100b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 shutdown_connections 2024-04-25T15:27:35.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.322+0000 7fc91552e640 1 -- 172.21.15.67:0/1434101139 wait complete. 2024-04-25T15:27:35.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:27:35.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.446+0000 7fbc8b08c640 1 Processor -- start 2024-04-25T15:27:35.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.446+0000 7fbc8b08c640 1 -- start start 2024-04-25T15:27:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc8b08c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 0x7fbc8406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc8b08c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc8406d870 con 0x7fbc840d4570 2024-04-25T15:27:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc8b08c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc8406db10 con 0x7fbc840b2dc0 2024-04-25T15:27:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc8b08c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbc840b4c40 con 0x7fbc840b4830 2024-04-25T15:27:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc88e28640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbc840d4570 0x7fbc840d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49772/0 (socket says 172.21.15.67:49772) 2024-04-25T15:27:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc88e28640 1 -- 172.21.15.67:0/2353942418 learned_addr learned my addr 172.21.15.67:0/2353942418 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:35.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 725573429 0 0) 0x7fbc8406d870 con 0x7fbc840d4570 2024-04-25T15:27:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc6c003520 con 0x7fbc840d4570 2024-04-25T15:27:35.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.447+0000 7fbc83fff640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 0x7fbc8406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4157423068 0 0) 0x7fbc6c003520 con 0x7fbc840d4570 2024-04-25T15:27:35.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 msgr2=0x7fbc8406e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 0x7fbc8406e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 >> v1:172.21.15.176:6789/0 conn(0x7fbc840b2dc0 legacy=0x7fbc840b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc8406e0f0 con 0x7fbc840d4570 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fbc7c001b70 con 0x7fbc840d4570 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 >> v1:172.21.15.67:6789/0 conn(0x7fbc840d4570 legacy=0x7fbc840d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6c0039e0 0x7fbc6c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 0x7fbc6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c004d40 0x7fbc6c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c00bb50 con 0x7fbc6c0039e0 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6c00bd00 con 0x7fbc6c004d40 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbc6c00bf00 con 0x7fbc840b4830 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc88e28640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6c0039e0 0x7fbc6c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc83fff640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 0x7fbc6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc89629640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c004d40 0x7fbc6c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc88e28640 1 -- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 msgr2=0x7fbc6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc88e28640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc840b4830 0x7fbc6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc88e28640 1 -- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c004d40 msgr2=0x7fbc6c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc88e28640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c004d40 0x7fbc6c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.448+0000 7fbc88e28640 1 -- 172.21.15.67:0/2353942418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6c003520 con 0x7fbc6c0039e0 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.449+0000 7fbc88e28640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6c0039e0 0x7fbc6c003e50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00c540 tx=0x7fbc6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6746c38d1132ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.449+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00c160 con 0x7fbc6c0039e0 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.449+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc7c005d40 con 0x7fbc6c0039e0 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.449+0000 7fbc837fe640 1 -- 172.21.15.67:0/2353942418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c0072b0 con 0x7fbc6c0039e0 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.450+0000 7fbc8b08c640 1 -- 172.21.15.67:0/2353942418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6c0039e0 msgr2=0x7fbc6c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.450+0000 7fbc8b08c640 1 --2- 172.21.15.67:0/2353942418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc6c0039e0 0x7fbc6c003e50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00c540 tx=0x7fbc6c005830 comp rx=0 tx=0).stop 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.450+0000 7fbc89629640 1 -- 172.21.15.67:0/2353942418 reap_dead start 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.450+0000 7fbc8b08c640 1 -- 172.21.15.67:0/2353942418 shutdown_connections 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.450+0000 7fbc8b08c640 1 -- 172.21.15.67:0/2353942418 >> 172.21.15.67:0/2353942418 conn(0x7fbc840d3a50 msgr2=0x7fbc840d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.450+0000 7fbc8b08c640 1 -- 172.21.15.67:0/2353942418 shutdown_connections 2024-04-25T15:27:35.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.450+0000 7fbc8b08c640 1 -- 172.21.15.67:0/2353942418 wait complete. 2024-04-25T15:27:35.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.452+0000 7fbc8b08c640 1 Processor -- start 2024-04-25T15:27:35.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.452+0000 7fbc8b08c640 1 -- start start 2024-04-25T15:27:35.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc8b08c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c0039e0 0x7fbc84153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc8b08c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc841550a0 0x7fbc841554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc8b08c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc84155e80 0x7fbc84156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc8b08c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbc840d0fb0 con 0x7fbc84155e80 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc8b08c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8406e460 con 0x7fbc6c0039e0 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc8b08c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbc840b38e0 con 0x7fbc841550a0 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc89629640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc84155e80 0x7fbc84156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc89629640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc84155e80 0x7fbc84156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50260/0 (socket says 172.21.15.67:50260) 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc89629640 1 -- 172.21.15.67:0/1916002799 learned_addr learned my addr 172.21.15.67:0/1916002799 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc88e28640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c0039e0 0x7fbc84153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.453+0000 7fbc83fff640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc841550a0 0x7fbc841554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc89629640 1 -- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc841550a0 msgr2=0x7fbc841554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc89629640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc841550a0 0x7fbc841554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc89629640 1 -- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c0039e0 msgr2=0x7fbc84153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc89629640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c0039e0 0x7fbc84153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc89629640 1 -- 172.21.15.67:0/1916002799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc841567d0 con 0x7fbc84155e80 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc83fff640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc841550a0 0x7fbc841554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc88e28640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c0039e0 0x7fbc84153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc89629640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc84155e80 0x7fbc84156290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbc8406e3d0 tx=0x7fbc7400f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc817fa640 1 -- 172.21.15.67:0/1916002799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc740053a0 con 0x7fbc84155e80 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc817fa640 1 -- 172.21.15.67:0/1916002799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc74018070 con 0x7fbc84155e80 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc841572e0 con 0x7fbc84155e80 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc817fa640 1 -- 172.21.15.67:0/1916002799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74003830 con 0x7fbc84155e80 2024-04-25T15:27:35.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.454+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc841578a0 con 0x7fbc84155e80 2024-04-25T15:27:35.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.455+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc84050740 con 0x7fbc84155e80 2024-04-25T15:27:35.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.456+0000 7fbc817fa640 1 -- 172.21.15.67:0/1916002799 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbc740039d0 con 0x7fbc84155e80 2024-04-25T15:27:35.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.456+0000 7fbc817fa640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc700744b0 0x7fbc700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.456+0000 7fbc817fa640 1 -- 172.21.15.67:0/1916002799 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fbc74094d30 con 0x7fbc84155e80 2024-04-25T15:27:35.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.456+0000 7fbc88e28640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc700744b0 0x7fbc700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.457+0000 7fbc88e28640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc700744b0 0x7fbc700769c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c033000 tx=0x7fbc7c038040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.461+0000 7fbc817fa640 1 -- 172.21.15.67:0/1916002799 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc841567d0 con 0x7fbc84155e80 2024-04-25T15:27:35.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.683+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbc8406ca80 con 0x7fbc84155e80 2024-04-25T15:27:35.684 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:27:35.684+0000 7f3e02b50640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:27:35.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.685+0000 7fbc817fa640 1 -- 172.21.15.67:0/1916002799 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) ==== 161+0+0 (secure 0 0 0) 0x7fbc7405f550 con 0x7fbc84155e80 2024-04-25T15:27:35.686 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:27:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc700744b0 msgr2=0x7fbc700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc700744b0 0x7fbc700769c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c033000 tx=0x7fbc7c038040 comp rx=0 tx=0).stop 2024-04-25T15:27:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc84155e80 msgr2=0x7fbc84156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc84155e80 0x7fbc84156290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fbc8406e3d0 tx=0x7fbc7400f920 comp rx=0 tx=0).stop 2024-04-25T15:27:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 shutdown_connections 2024-04-25T15:27:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbc700744b0 0x7fbc700769c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc84155e80 0x7fbc84156290 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc841550a0 0x7fbc841554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 --2- 172.21.15.67:0/1916002799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc6c0039e0 0x7fbc84153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.687+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 >> 172.21.15.67:0/1916002799 conn(0x7fbc840d3a50 msgr2=0x7fbc840ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.688+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 shutdown_connections 2024-04-25T15:27:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.688+0000 7fbc8b08c640 1 -- 172.21.15.67:0/1916002799 wait complete. 2024-04-25T15:27:35.692 INFO:tasks.ceph.mds.a.smithi067.stderr: -4519> 2024-04-25T15:27:23.982+0000 7f234815b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:27:35.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:27:35.749 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:27:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.818+0000 7f848c684640 1 Processor -- start 2024-04-25T15:27:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.818+0000 7f848c684640 1 -- start start 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.819+0000 7f848c684640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84840d0e00 0x7f84840d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.819+0000 7f848c684640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f84840d4a50 0x7f84840d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.819+0000 7f848c684640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84840d1850 con 0x7f848406c5c0 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.819+0000 7f848c684640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8484160de0 con 0x7f84840d0e00 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.819+0000 7f848c684640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f84841610c0 con 0x7f84840d4a50 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848a420640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84840d0e00 0x7f84840d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848a420640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84840d0e00 0x7f84840d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50280/0 (socket says 172.21.15.67:50280) 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848a420640 1 -- 172.21.15.67:0/4230259355 learned_addr learned my addr 172.21.15.67:0/4230259355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848ac21640 1 --2- 172.21.15.67:0/4230259355 >> v2:172.21.15.176:3300/0 conn(0x7f84840d4a50 0x7f84840d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848a420640 1 -- 172.21.15.67:0/4230259355 >> v2:172.21.15.176:3300/0 conn(0x7f84840d4a50 msgr2=0x7f84840d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848a420640 1 --2- 172.21.15.67:0/4230259355 >> v2:172.21.15.176:3300/0 conn(0x7f84840d4a50 0x7f84840d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848a420640 1 -- 172.21.15.67:0/4230259355 >> v1:172.21.15.176:6789/0 conn(0x7f848406c5c0 legacy=0x7f848406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.820+0000 7f848a420640 1 -- 172.21.15.67:0/4230259355 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84841613a0 con 0x7f84840d0e00 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.821+0000 7f848a420640 1 --2- 172.21.15.67:0/4230259355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84840d0e00 0x7f84840d1210 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f84840d1750 tx=0x7f847402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd30b603dd72fd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.821+0000 7f848941e640 1 -- 172.21.15.67:0/4230259355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8474037070 con 0x7f84840d0e00 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.821+0000 7f848941e640 1 -- 172.21.15.67:0/4230259355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8474032440 con 0x7f84840d0e00 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.821+0000 7f848941e640 1 -- 172.21.15.67:0/4230259355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8474032810 con 0x7f84840d0e00 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 -- 172.21.15.67:0/4230259355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84840d0e00 msgr2=0x7f84840d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 --2- 172.21.15.67:0/4230259355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84840d0e00 0x7f84840d1210 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f84840d1750 tx=0x7f847402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 -- 172.21.15.67:0/4230259355 shutdown_connections 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 --2- 172.21.15.67:0/4230259355 >> v2:172.21.15.176:3300/0 conn(0x7f84840d4a50 0x7f84840d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 --2- 172.21.15.67:0/4230259355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84840d0e00 0x7f84840d1210 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 -- 172.21.15.67:0/4230259355 >> 172.21.15.67:0/4230259355 conn(0x7f84840d5d90 msgr2=0x7f84840d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 -- 172.21.15.67:0/4230259355 shutdown_connections 2024-04-25T15:27:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.822+0000 7f848c684640 1 -- 172.21.15.67:0/4230259355 wait complete. 2024-04-25T15:27:35.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.823+0000 7f848c684640 1 Processor -- start 2024-04-25T15:27:35.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.824+0000 7f848c684640 1 -- start start 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848c684640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f848406c5c0 0x7f848414f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848c684640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84840d0e00 0x7f848415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848c684640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84840d4a50 0x7f84840ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848c684640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8484164e00 con 0x7f848406c5c0 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848c684640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f84840b5320 con 0x7f84840d0e00 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848c684640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8484164f80 con 0x7f84840d4a50 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f848406c5c0 0x7f848414f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f848406c5c0 0x7f848414f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50292/0 (socket says 172.21.15.67:50292) 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 -- 172.21.15.67:0/3894769319 learned_addr learned my addr 172.21.15.67:0/3894769319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f8489c1f640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84840d0e00 0x7f848415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848ac21640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84840d4a50 0x7f84840ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 -- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84840d4a50 msgr2=0x7f84840ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84840d4a50 0x7f84840ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 -- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84840d0e00 msgr2=0x7f848415ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84840d0e00 0x7f848415ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 -- 172.21.15.67:0/3894769319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848415b7c0 con 0x7f848406c5c0 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f8489c1f640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84840d0e00 0x7f848415ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.825+0000 7f848a420640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f848406c5c0 0x7f848414f2c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f847402a220 tx=0x7f84740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.826+0000 7f847affd640 1 -- 172.21.15.67:0/3894769319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8474041020 con 0x7f848406c5c0 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.826+0000 7f847affd640 1 -- 172.21.15.67:0/3894769319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84740306c0 con 0x7f848406c5c0 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.826+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848415baa0 con 0x7f848406c5c0 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.826+0000 7f847affd640 1 -- 172.21.15.67:0/3894769319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8474037040 con 0x7f848406c5c0 2024-04-25T15:27:35.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.826+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84840c9700 con 0x7f848406c5c0 2024-04-25T15:27:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.827+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8484050740 con 0x7f848406c5c0 2024-04-25T15:27:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.827+0000 7f847affd640 1 -- 172.21.15.67:0/3894769319 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8474044020 con 0x7f848406c5c0 2024-04-25T15:27:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.828+0000 7f847affd640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84680744b0 0x7f84680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.828+0000 7f847affd640 1 -- 172.21.15.67:0/3894769319 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7f84740b5ae0 con 0x7f848406c5c0 2024-04-25T15:27:35.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.828+0000 7f8489c1f640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84680744b0 0x7f84680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:35.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.829+0000 7f8489c1f640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84680744b0 0x7f84680769c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f84841eaf80 tx=0x7f8480009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:35.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:35.832+0000 7f847affd640 1 -- 172.21.15.67:0/3894769319 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f847404b030 con 0x7f848406c5c0 2024-04-25T15:27:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.057+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f84840d4320 con 0x7f848406c5c0 2024-04-25T15:27:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.686+0000 7f847affd640 1 -- 172.21.15.67:0/3894769319 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) ==== 104+0+0 (secure 0 0 0) 0x7f84740802a0 con 0x7f848406c5c0 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84680744b0 msgr2=0x7f84680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84680744b0 0x7f84680769c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f84841eaf80 tx=0x7f8480009040 comp rx=0 tx=0).stop 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f848406c5c0 msgr2=0x7f848414f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f848406c5c0 0x7f848414f2c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f847402a220 tx=0x7f84740046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 shutdown_connections 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84680744b0 0x7f84680769c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84840d4a50 0x7f84840ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84840d0e00 0x7f848415ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 --2- 172.21.15.67:0/3894769319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f848406c5c0 0x7f848414f2c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 >> 172.21.15.67:0/3894769319 conn(0x7f84840d5d90 msgr2=0x7f84841641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:36.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 shutdown_connections 2024-04-25T15:27:36.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.689+0000 7f848c684640 1 -- 172.21.15.67:0/3894769319 wait complete. 2024-04-25T15:27:36.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:27:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.821+0000 7ff4d134d640 1 Processor -- start 2024-04-25T15:27:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.821+0000 7ff4d134d640 1 -- start start 2024-04-25T15:27:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.822+0000 7ff4d134d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff4cc161710 0x7ff4cc15e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.822+0000 7ff4d134d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 0x7ff4cc1643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.822+0000 7ff4d134d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4cc1609b0 con 0x7ff4cc06c5c0 2024-04-25T15:27:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.822+0000 7ff4d134d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff4cc164cd0 con 0x7ff4cc161710 2024-04-25T15:27:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.822+0000 7ff4d134d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc160c30 con 0x7ff4cc163fa0 2024-04-25T15:27:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4ca7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff4cc06c5c0 0x7ff4cc06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49778/0 (socket says 172.21.15.67:49778) 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4ca7fc640 1 -- 172.21.15.67:0/2333429472 learned_addr learned my addr 172.21.15.67:0/2333429472 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4caffd640 1 --2- 172.21.15.67:0/2333429472 >> v2:172.21.15.176:3300/0 conn(0x7ff4cc161710 0x7ff4cc15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467556735 0 0) 0x7ff4b4001110 con 0x7ff4cc06c5c0 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4cb7fe640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 0x7ff4cc1643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4b8003520 con 0x7ff4cc06c5c0 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2820966439 0 0) 0x7ff4b8003520 con 0x7ff4cc06c5c0 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 msgr2=0x7ff4cc1643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 0x7ff4cc1643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 >> v2:172.21.15.176:3300/0 conn(0x7ff4cc161710 msgr2=0x7ff4cc15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 --2- 172.21.15.67:0/2333429472 >> v2:172.21.15.176:3300/0 conn(0x7ff4cc161710 0x7ff4cc15e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc160f10 con 0x7ff4cc06c5c0 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7ff4b4001840 con 0x7ff4cc06c5c0 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 >> v1:172.21.15.67:6789/0 conn(0x7ff4cc06c5c0 legacy=0x7ff4cc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b80039b0 0x7ff4b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 0x7ff4b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b8004d40 0x7ff4b8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc164cd0 con 0x7ff4b80039b0 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc160c30 con 0x7ff4b8004d40 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b800ba40 con 0x7ff4cc163fa0 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.823+0000 7ff4caffd640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b80039b0 0x7ff4b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4ca7fc640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 0x7ff4b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4cb7fe640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b8004d40 0x7ff4b8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4caffd640 1 -- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 msgr2=0x7ff4b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4caffd640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc163fa0 0x7ff4b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4caffd640 1 -- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b8004d40 msgr2=0x7ff4b8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:36.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4caffd640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b8004d40 0x7ff4b8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4caffd640 1 -- 172.21.15.67:0/2333429472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b8003520 con 0x7ff4b80039b0 2024-04-25T15:27:36.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.824+0000 7ff4caffd640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b80039b0 0x7ff4b8003e20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009a90 tx=0x7ff4b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1b028a1e1e67a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:36.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4036070 con 0x7ff4b80039b0 2024-04-25T15:27:36.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c4031400 con 0x7ff4b80039b0 2024-04-25T15:27:36.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4c9ffb640 1 -- 172.21.15.67:0/2333429472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4009690 con 0x7ff4b80039b0 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4d134d640 1 -- 172.21.15.67:0/2333429472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b80039b0 msgr2=0x7ff4b8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4d134d640 1 --2- 172.21.15.67:0/2333429472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4b80039b0 0x7ff4b8003e20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009a90 tx=0x7ff4b8005830 comp rx=0 tx=0).stop 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4cb7fe640 1 -- 172.21.15.67:0/2333429472 reap_dead start 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4d134d640 1 -- 172.21.15.67:0/2333429472 shutdown_connections 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4d134d640 1 -- 172.21.15.67:0/2333429472 >> 172.21.15.67:0/2333429472 conn(0x7ff4cc0d5d90 msgr2=0x7ff4cc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4d134d640 1 -- 172.21.15.67:0/2333429472 shutdown_connections 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.825+0000 7ff4d134d640 1 -- 172.21.15.67:0/2333429472 wait complete. 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.827+0000 7ff4d134d640 1 Processor -- start 2024-04-25T15:27:36.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.827+0000 7ff4d134d640 1 -- start start 2024-04-25T15:27:36.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4d134d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b80039b0 0x7ff4cc151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4d134d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4cc153120 0x7ff4cc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4d134d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc153530 0x7ff4cc0ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4d134d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc164530 con 0x7ff4cc153120 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4d134d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc1643b0 con 0x7ff4b80039b0 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4d134d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc1646b0 con 0x7ff4cc153530 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4cc153120 0x7ff4cc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4cc153120 0x7ff4cc0ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50310/0 (socket says 172.21.15.67:50310) 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 -- 172.21.15.67:0/4070071995 learned_addr learned my addr 172.21.15.67:0/4070071995 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4caffd640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b80039b0 0x7ff4cc151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4cb7fe640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc153530 0x7ff4cc0ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 -- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc153530 msgr2=0x7ff4cc0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc153530 0x7ff4cc0ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 -- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b80039b0 msgr2=0x7ff4cc151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b80039b0 0x7ff4cc151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.829+0000 7ff4ca7fc640 1 -- 172.21.15.67:0/4070071995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc153c90 con 0x7ff4cc153120 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.830+0000 7ff4ca7fc640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4cc153120 0x7ff4cc0ce360 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4007360 tx=0x7ff4b4007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:36.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.830+0000 7ff4c37fe640 1 -- 172.21.15.67:0/4070071995 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400e8d0 con 0x7ff4cc153120 2024-04-25T15:27:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.830+0000 7ff4c37fe640 1 -- 172.21.15.67:0/4070071995 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4018070 con 0x7ff4cc153120 2024-04-25T15:27:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.830+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4cc153f70 con 0x7ff4cc153120 2024-04-25T15:27:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.830+0000 7ff4c37fe640 1 -- 172.21.15.67:0/4070071995 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400ecb0 con 0x7ff4cc153120 2024-04-25T15:27:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.830+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4cc152530 con 0x7ff4cc153120 2024-04-25T15:27:36.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.831+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4cc153a00 con 0x7ff4cc153120 2024-04-25T15:27:36.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.831+0000 7ff4c37fe640 1 -- 172.21.15.67:0/4070071995 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff4b401d460 con 0x7ff4cc153120 2024-04-25T15:27:36.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.832+0000 7ff4c37fe640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4ac0744b0 0x7ff4ac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:36.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.832+0000 7ff4c37fe640 1 -- 172.21.15.67:0/4070071995 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7ff4b40951f0 con 0x7ff4cc153120 2024-04-25T15:27:36.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.832+0000 7ff4caffd640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4ac0744b0 0x7ff4ac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:36.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.833+0000 7ff4caffd640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4ac0744b0 0x7ff4ac0769c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4033000 tx=0x7ff4c4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:36.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:36.837+0000 7ff4c37fe640 1 -- 172.21.15.67:0/4070071995 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4cc1609b0 con 0x7ff4cc153120 2024-04-25T15:27:37.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.021+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff4cc0b6b10 con 0x7ff4cc153120 2024-04-25T15:27:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.693+0000 7ff4c37fe640 1 -- 172.21.15.67:0/4070071995 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) ==== 181+0+0 (secure 0 0 0) 0x7ff4cc0b6b10 con 0x7ff4cc153120 2024-04-25T15:27:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:27:37.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4ac0744b0 msgr2=0x7ff4ac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4ac0744b0 0x7ff4ac0769c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4033000 tx=0x7ff4c4008030 comp rx=0 tx=0).stop 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4cc153120 msgr2=0x7ff4cc0ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4cc153120 0x7ff4cc0ce360 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4007360 tx=0x7ff4b4007820 comp rx=0 tx=0).stop 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 shutdown_connections 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff4ac0744b0 0x7ff4ac0769c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4cc153530 0x7ff4cc0ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4cc153120 0x7ff4cc0ce360 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 --2- 172.21.15.67:0/4070071995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4b80039b0 0x7ff4cc151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 >> 172.21.15.67:0/4070071995 conn(0x7ff4cc0d5d90 msgr2=0x7ff4cc15e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 shutdown_connections 2024-04-25T15:27:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.695+0000 7ff4d134d640 1 -- 172.21.15.67:0/4070071995 wait complete. 2024-04-25T15:27:37.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:27:37.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.832+0000 7f64f8727640 1 Processor -- start 2024-04-25T15:27:37.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.832+0000 7f64f8727640 1 -- start start 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f8727640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f00d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f8727640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f64f0164020 0x7f64f0164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f8727640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64f0164970 con 0x7f64f00d1e20 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f8727640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f64f00d23f0 con 0x7f64f0164020 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f8727640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64f00d26d0 con 0x7f64f00b2e60 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f6cc4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f64f0164020 0x7f64f0164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f5cc2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f00d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f64c3640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f64f00d1e20 0x7f64f008d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:40026/0 (socket says 172.21.15.67:40026) 2024-04-25T15:27:37.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f5cc2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f00d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:39236/0 (socket says 172.21.15.67:39236) 2024-04-25T15:27:37.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f64c3640 1 -- 172.21.15.67:0/3685628649 learned_addr learned my addr 172.21.15.67:0/3685628649 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:37.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.834+0000 7f64f6cc4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f64f0164020 0x7f64f0164430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:34562/0 (socket says 172.21.15.67:34562) 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.835+0000 7f64f6cc4640 1 -- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 msgr2=0x7f64f00d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.835+0000 7f64f6cc4640 1 --2- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f00d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.835+0000 7f64f6cc4640 1 -- 172.21.15.67:0/3685628649 >> v1:172.21.15.176:6789/0 conn(0x7f64f00d1e20 legacy=0x7f64f008d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.835+0000 7f64f6cc4640 1 -- 172.21.15.67:0/3685628649 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f00d29b0 con 0x7f64f0164020 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.835+0000 7f64f5cc2640 1 --2- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f00d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.835+0000 7f64f6cc4640 1 --2- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 0x7f64f0164430 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0097a0 tx=0x7f64ec02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966566535caeb473 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.836+0000 7f64f54c1640 1 -- 172.21.15.67:0/3685628649 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec037070 con 0x7f64f0164020 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.836+0000 7f64f54c1640 1 -- 172.21.15.67:0/3685628649 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64ec032440 con 0x7f64f0164020 2024-04-25T15:27:37.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.836+0000 7f64f54c1640 1 -- 172.21.15.67:0/3685628649 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec032810 con 0x7f64f0164020 2024-04-25T15:27:37.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.836+0000 7f64f8727640 1 -- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 msgr2=0x7f64f0164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:37.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.836+0000 7f64f8727640 1 --2- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 0x7f64f0164430 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0097a0 tx=0x7f64ec02a210 comp rx=0 tx=0).stop 2024-04-25T15:27:37.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.837+0000 7f64f8727640 1 -- 172.21.15.67:0/3685628649 shutdown_connections 2024-04-25T15:27:37.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.837+0000 7f64f8727640 1 --2- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 0x7f64f0164430 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.837+0000 7f64f8727640 1 --2- 172.21.15.67:0/3685628649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f00d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.837+0000 7f64f8727640 1 -- 172.21.15.67:0/3685628649 >> 172.21.15.67:0/3685628649 conn(0x7f64f00d0530 msgr2=0x7f64f00d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:37.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.837+0000 7f64f8727640 1 -- 172.21.15.67:0/3685628649 shutdown_connections 2024-04-25T15:27:37.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.837+0000 7f64f8727640 1 -- 172.21.15.67:0/3685628649 wait complete. 2024-04-25T15:27:37.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.838+0000 7f64f8727640 1 Processor -- start 2024-04-25T15:27:37.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.838+0000 7f64f8727640 1 -- start start 2024-04-25T15:27:37.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f8727640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f01f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f8727640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f00d1e20 0x7f64f00b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f8727640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 0x7f64f015e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f8727640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64f00ce070 con 0x7f64f00d1e20 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f8727640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f64f00d2f70 con 0x7f64f0164020 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f8727640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64f00ce1f0 con 0x7f64f00b2e60 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f00d1e20 0x7f64f00b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f00d1e20 0x7f64f00b4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44654/0 (socket says 172.21.15.67:44654) 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 -- 172.21.15.67:0/242969592 learned_addr learned my addr 172.21.15.67:0/242969592 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f64c3640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f01f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f6cc4640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 0x7f64f015e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 -- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 msgr2=0x7f64f01f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:37.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f01f61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 -- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 msgr2=0x7f64f015e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 0x7f64f015e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 -- 172.21.15.67:0/242969592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f01f6730 con 0x7f64f00d1e20 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.840+0000 7f64f5cc2640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f00d1e20 0x7f64f00b4460 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f64e000e960 tx=0x7f64e000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.841+0000 7f64e6ffd640 1 -- 172.21.15.67:0/242969592 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0017070 con 0x7f64f00d1e20 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.841+0000 7f64e6ffd640 1 -- 172.21.15.67:0/242969592 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e000ca00 con 0x7f64f00d1e20 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.841+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f015cf00 con 0x7f64f00d1e20 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.841+0000 7f64e6ffd640 1 -- 172.21.15.67:0/242969592 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e000cdd0 con 0x7f64f00d1e20 2024-04-25T15:27:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.841+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f015d4c0 con 0x7f64f00d1e20 2024-04-25T15:27:37.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.842+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f0050740 con 0x7f64f00d1e20 2024-04-25T15:27:37.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.842+0000 7f64e6ffd640 1 -- 172.21.15.67:0/242969592 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f64e001a070 con 0x7f64f00d1e20 2024-04-25T15:27:37.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.843+0000 7f64e6ffd640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:37.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.843+0000 7f64e6ffd640 1 -- 172.21.15.67:0/242969592 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5357+0+0 (secure 0 0 0) 0x7f64e0095160 con 0x7f64f00d1e20 2024-04-25T15:27:37.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.843+0000 7f64f64c3640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:37.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.844+0000 7f64f64c3640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f64e80012e0 tx=0x7f64e8001990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:37.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:37.848+0000 7f64e6ffd640 1 -- 172.21.15.67:0/242969592 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64e005f520 con 0x7f64f00d1e20 2024-04-25T15:27:38.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.033+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f64f0071000 con 0x7f64f00d1e20 2024-04-25T15:27:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.694+0000 7f64e6ffd640 1 -- 172.21.15.67:0/242969592 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) ==== 169+0+0 (secure 0 0 0) 0x7f64e0063820 con 0x7f64f00d1e20 2024-04-25T15:27:38.695 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.698+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 msgr2=0x7f64d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.698+0000 7f64f8727640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f64e80012e0 tx=0x7f64e8001990 comp rx=0 tx=0).stop 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.698+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f00d1e20 msgr2=0x7f64f00b4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.698+0000 7f64f8727640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f00d1e20 0x7f64f00b4460 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f64e000e960 tx=0x7f64e000ee20 comp rx=0 tx=0).stop 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 shutdown_connections 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64f0164020 0x7f64f015e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f00d1e20 0x7f64f00b4460 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 --2- 172.21.15.67:0/242969592 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64f00b2e60 0x7f64f01f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 >> 172.21.15.67:0/242969592 conn(0x7f64f00d0530 msgr2=0x7f64f006e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:38.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 shutdown_connections 2024-04-25T15:27:38.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.699+0000 7f64f8727640 1 -- 172.21.15.67:0/242969592 wait complete. 2024-04-25T15:27:38.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:27:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.841+0000 7f27859e3640 1 Processor -- start 2024-04-25T15:27:38.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.841+0000 7f27859e3640 1 -- start start 2024-04-25T15:27:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f27859e3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f278008dec0 0x7f27800b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f27859e3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f278008cf20 0x7f278008d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f27859e3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f278008d870 con 0x7f27800b0c40 2024-04-25T15:27:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f27859e3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f27800b95f0 con 0x7f278008dec0 2024-04-25T15:27:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f27859e3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f27800d21e0 con 0x7f278008cf20 2024-04-25T15:27:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f277effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f278008dec0 0x7f27800b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f277effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f278008dec0 0x7f27800b90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44662/0 (socket says 172.21.15.67:44662) 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f277effd640 1 -- 172.21.15.67:0/11109188 learned_addr learned my addr 172.21.15.67:0/11109188 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.842+0000 7f277f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27800b0c40 0x7f27800b1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49488/0 (socket says 172.21.15.67:49488) 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.843+0000 7f277e7fc640 1 --2- 172.21.15.67:0/11109188 >> v2:172.21.15.176:3300/0 conn(0x7f278008cf20 0x7f278008d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.843+0000 7f277effd640 1 -- 172.21.15.67:0/11109188 >> v2:172.21.15.176:3300/0 conn(0x7f278008cf20 msgr2=0x7f278008d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.843+0000 7f277effd640 1 --2- 172.21.15.67:0/11109188 >> v2:172.21.15.176:3300/0 conn(0x7f278008cf20 0x7f278008d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.843+0000 7f277effd640 1 -- 172.21.15.67:0/11109188 >> v1:172.21.15.67:6789/0 conn(0x7f27800b0c40 legacy=0x7f27800b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.843+0000 7f277effd640 1 -- 172.21.15.67:0/11109188 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27800d1940 con 0x7f278008dec0 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.843+0000 7f277dffb640 1 -- 172.21.15.67:0/11109188 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 428019255 0 0) 0x7f2770001490 con 0x7f27800b0c40 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.843+0000 7f277effd640 1 --2- 172.21.15.67:0/11109188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f27800b90b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f27800d39c0 tx=0x7f277802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0d2f6bdb1664d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f277dffb640 1 -- 172.21.15.67:0/11109188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778037070 con 0x7f278008dec0 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f277dffb640 1 -- 172.21.15.67:0/11109188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2778032440 con 0x7f278008dec0 2024-04-25T15:27:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f277dffb640 1 -- 172.21.15.67:0/11109188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778032810 con 0x7f278008dec0 2024-04-25T15:27:38.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 -- 172.21.15.67:0/11109188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 msgr2=0x7f27800b90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 --2- 172.21.15.67:0/11109188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f27800b90b0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f27800d39c0 tx=0x7f277802a270 comp rx=0 tx=0).stop 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 -- 172.21.15.67:0/11109188 shutdown_connections 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 --2- 172.21.15.67:0/11109188 >> v2:172.21.15.176:3300/0 conn(0x7f278008cf20 0x7f278008d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 --2- 172.21.15.67:0/11109188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f27800b90b0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 -- 172.21.15.67:0/11109188 >> 172.21.15.67:0/11109188 conn(0x7f27800d0510 msgr2=0x7f27800d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 -- 172.21.15.67:0/11109188 shutdown_connections 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.844+0000 7f27859e3640 1 -- 172.21.15.67:0/11109188 wait complete. 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.846+0000 7f27859e3640 1 Processor -- start 2024-04-25T15:27:38.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.846+0000 7f27859e3640 1 -- start start 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f27859e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f278008cf20 0x7f27800c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f27859e3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f2780160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f27859e3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27800b0c40 0x7f2780160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f27859e3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27800b0720 con 0x7f278008dec0 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f27859e3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f27800d1f00 con 0x7f27800b0c40 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f27859e3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27800d15d0 con 0x7f278008cf20 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f2780160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f2780160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44674/0 (socket says 172.21.15.67:44674) 2024-04-25T15:27:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 -- 172.21.15.67:0/4294088314 learned_addr learned my addr 172.21.15.67:0/4294088314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277effd640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f278008cf20 0x7f27800c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277f7fe640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27800b0c40 0x7f2780160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 -- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f278008cf20 msgr2=0x7f27800c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f278008cf20 0x7f27800c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 -- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27800b0c40 msgr2=0x7f2780160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27800b0c40 0x7f2780160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.848+0000 7f277e7fc640 1 -- 172.21.15.67:0/4294088314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27800c74f0 con 0x7f278008dec0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.849+0000 7f277e7fc640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f2780160890 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f276800d5f0 tx=0x7f276800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.849+0000 7f2777fff640 1 -- 172.21.15.67:0/4294088314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27680125f0 con 0x7f278008dec0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.849+0000 7f2777fff640 1 -- 172.21.15.67:0/4294088314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2768019070 con 0x7f278008dec0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.849+0000 7f2777fff640 1 -- 172.21.15.67:0/4294088314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27680129d0 con 0x7f278008dec0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.849+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278006ef10 con 0x7f278008dec0 2024-04-25T15:27:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.849+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27800c7a90 con 0x7f278008dec0 2024-04-25T15:27:38.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.850+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2780050740 con 0x7f278008dec0 2024-04-25T15:27:38.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.850+0000 7f2777fff640 1 -- 172.21.15.67:0/4294088314 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2768012b70 con 0x7f278008dec0 2024-04-25T15:27:38.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.851+0000 7f2777fff640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27600744e0 0x7f27600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.851+0000 7f2777fff640 1 -- 172.21.15.67:0/4294088314 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f2768095000 con 0x7f278008dec0 2024-04-25T15:27:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.851+0000 7f277effd640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27600744e0 0x7f27600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.852+0000 7f277effd640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27600744e0 0x7f27600769f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2778034000 tx=0x7f277802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:38.856+0000 7f2777fff640 1 -- 172.21.15.67:0/4294088314 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f276805f560 con 0x7f278008dec0 2024-04-25T15:27:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.043+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f27800d09c0 con 0x7f278008dec0 2024-04-25T15:27:39.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.047+0000 7f2777fff640 1 -- 172.21.15.67:0/4294088314 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7f2768063860 con 0x7f278008dec0 2024-04-25T15:27:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27600744e0 msgr2=0x7f27600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27600744e0 0x7f27600769f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2778034000 tx=0x7f277802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 msgr2=0x7f2780160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f2780160890 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f276800d5f0 tx=0x7f276800dab0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 shutdown_connections 2024-04-25T15:27:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27600744e0 0x7f27600769f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27800b0c40 0x7f2780160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f278008dec0 0x7f2780160890 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 --2- 172.21.15.67:0/4294088314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f278008cf20 0x7f27800c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 >> 172.21.15.67:0/4294088314 conn(0x7f27800d0510 msgr2=0x7f27800b8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 shutdown_connections 2024-04-25T15:27:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.049+0000 7f27859e3640 1 -- 172.21.15.67:0/4294088314 wait complete. 2024-04-25T15:27:39.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values' 2024-04-25T15:27:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.172+0000 7fdb1d38f640 1 Processor -- start 2024-04-25T15:27:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.172+0000 7fdb1d38f640 1 -- start start 2024-04-25T15:27:39.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.173+0000 7fdb1d38f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb180b6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.173+0000 7fdb1d38f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb181431d0 con 0x7fdb18142730 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.173+0000 7fdb1d38f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb181434b0 con 0x7fdb18070980 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.173+0000 7fdb1d38f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdb180b6090 con 0x7fdb181400a0 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.173+0000 7fdb0ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdb18142730 0x7fdb18070270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49504/0 (socket says 172.21.15.67:49504) 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.173+0000 7fdb0ffff640 1 -- 172.21.15.67:0/788454426 learned_addr learned my addr 172.21.15.67:0/788454426 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439394740 0 0) 0x7fdb181431d0 con 0x7fdb18142730 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb04003520 con 0x7fdb18142730 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb180b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 310418639 0 0) 0x7fdb04003520 con 0x7fdb18142730 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 msgr2=0x7fdb180b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb180b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 >> v1:172.21.15.176:6789/0 conn(0x7fdb18070980 legacy=0x7fdb180b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb180b6370 con 0x7fdb18142730 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb180b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:39.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fdb00001670 con 0x7fdb18142730 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 >> v1:172.21.15.67:6789/0 conn(0x7fdb18142730 legacy=0x7fdb18070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb040039e0 0x7fdb04003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb04004d40 0x7fdb040051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb180b6090 con 0x7fdb040039e0 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0400bb80 con 0x7fdb04004d40 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0400bd80 con 0x7fdb181400a0 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb040039e0 0x7fdb04003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb0ffff640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.174+0000 7fdb177fe640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb04004d40 0x7fdb040051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb16ffd640 1 -- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 msgr2=0x7fdb04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb04004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb16ffd640 1 -- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb04004d40 msgr2=0x7fdb040051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb04004d40 0x7fdb040051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb16ffd640 1 -- 172.21.15.67:0/788454426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb04003520 con 0x7fdb040039e0 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb0ffff640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb181400a0 0x7fdb04004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb040039e0 0x7fdb04003e50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdb18073000 tx=0x7fdb04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea60cb794839f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb08037070 con 0x7fdb040039e0 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.175+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0802aca0 con 0x7fdb040039e0 2024-04-25T15:27:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb167fc640 1 -- 172.21.15.67:0/788454426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0800f040 con 0x7fdb040039e0 2024-04-25T15:27:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb1d38f640 1 -- 172.21.15.67:0/788454426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb040039e0 msgr2=0x7fdb04003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb1d38f640 1 --2- 172.21.15.67:0/788454426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb040039e0 0x7fdb04003e50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdb18073000 tx=0x7fdb04005830 comp rx=0 tx=0).stop 2024-04-25T15:27:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb177fe640 1 -- 172.21.15.67:0/788454426 reap_dead start 2024-04-25T15:27:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb1d38f640 1 -- 172.21.15.67:0/788454426 shutdown_connections 2024-04-25T15:27:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb1d38f640 1 -- 172.21.15.67:0/788454426 >> 172.21.15.67:0/788454426 conn(0x7fdb18073050 msgr2=0x7fdb18073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb1d38f640 1 -- 172.21.15.67:0/788454426 shutdown_connections 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.176+0000 7fdb1d38f640 1 -- 172.21.15.67:0/788454426 wait complete. 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.178+0000 7fdb1d38f640 1 Processor -- start 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.178+0000 7fdb1d38f640 1 -- start start 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.179+0000 7fdb1d38f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb040039e0 0x7fdb18156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.179+0000 7fdb1d38f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb181f6290 0x7fdb181567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.179+0000 7fdb1d38f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb18156ce0 0x7fdb181f9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.179+0000 7fdb1d38f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb18142cc0 con 0x7fdb181f6290 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.179+0000 7fdb1d38f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdb18142b40 con 0x7fdb18156ce0 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.179+0000 7fdb1d38f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdb18142e40 con 0x7fdb040039e0 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb181f6290 0x7fdb181567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb181f6290 0x7fdb181567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44686/0 (socket says 172.21.15.67:44686) 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 -- 172.21.15.67:0/2324907340 learned_addr learned my addr 172.21.15.67:0/2324907340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb040039e0 0x7fdb18156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 -- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb040039e0 msgr2=0x7fdb18156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb177fe640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb18156ce0 0x7fdb181f9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb040039e0 0x7fdb18156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 -- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb18156ce0 msgr2=0x7fdb181f9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb18156ce0 0x7fdb181f9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 -- 172.21.15.67:0/2324907340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb181570f0 con 0x7fdb181f6290 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb177fe640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb18156ce0 0x7fdb181f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0ffff640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb181f6290 0x7fdb181567a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000e7f0 tx=0x7fdb0000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0f7fe640 1 -- 172.21.15.67:0/2324907340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0000c8d0 con 0x7fdb181f6290 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0f7fe640 1 -- 172.21.15.67:0/2324907340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb00018070 con 0x7fdb181f6290 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb18159360 con 0x7fdb181f6290 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb0f7fe640 1 -- 172.21.15.67:0/2324907340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0000cc90 con 0x7fdb181f6290 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.180+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb18159930 con 0x7fdb181f6290 2024-04-25T15:27:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.181+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb18159b00 con 0x7fdb181f6290 2024-04-25T15:27:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.182+0000 7fdb0f7fe640 1 -- 172.21.15.67:0/2324907340 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdb0001d460 con 0x7fdb181f6290 2024-04-25T15:27:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.183+0000 7fdb0f7fe640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdaf80744e0 0x7fdaf80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.183+0000 7fdb0f7fe640 1 -- 172.21.15.67:0/2324907340 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fdb00094e70 con 0x7fdb181f6290 2024-04-25T15:27:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.183+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdaf80744e0 0x7fdaf80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.183+0000 7fdb16ffd640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdaf80744e0 0x7fdaf80769f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdb08034000 tx=0x7fdb0802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.187+0000 7fdb0f7fe640 1 -- 172.21.15.67:0/2324907340 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb181570f0 con 0x7fdb181f6290 2024-04-25T15:27:39.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.453+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]} v 0) -- 0x7fdb18050740 con 0x7fdb181f6290 2024-04-25T15:27:39.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.700+0000 7fdb0f7fe640 1 -- 172.21.15.67:0/2324907340 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]}]=0 v148) ==== 145+0+0 (secure 0 0 0) 0x7fdb0005f900 con 0x7fdb181f6290 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdaf80744e0 msgr2=0x7fdaf80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdaf80744e0 0x7fdaf80769f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fdb08034000 tx=0x7fdb0802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb181f6290 msgr2=0x7fdb181567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb181f6290 0x7fdb181567a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000e7f0 tx=0x7fdb0000ecb0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 shutdown_connections 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdaf80744e0 0x7fdaf80769f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb18156ce0 0x7fdb181f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb181f6290 0x7fdb181567a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 --2- 172.21.15.67:0/2324907340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb040039e0 0x7fdb18156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.702+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 >> 172.21.15.67:0/2324907340 conn(0x7fdb18073050 msgr2=0x7fdb180b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.703+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 shutdown_connections 2024-04-25T15:27:39.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.703+0000 7fdb1d38f640 1 -- 172.21.15.67:0/2324907340 wait complete. 2024-04-25T15:27:39.715 INFO:tasks.cephfs_test_runner:test human-readable values for setting ceph.quota.max_bytes ... ok 2024-04-25T15:27:39.715 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_df (tasks.cephfs.test_quota.TestQuota) That when a client modifies the quota on a directory used 2024-04-25T15:27:39.715 INFO:tasks.cephfs_test_runner:test_remote_update_df (tasks.cephfs.test_quota.TestQuota) 2024-04-25T15:27:39.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.827+0000 7f2d383f3640 1 Processor -- start 2024-04-25T15:27:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.827+0000 7f2d383f3640 1 -- start start 2024-04-25T15:27:39.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d383f3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2d30164b00 0x7f2d300b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d383f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d3015df80 0x7f2d300b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d383f3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d301615f0 con 0x7f2d30160890 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d383f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2d301613b0 con 0x7f2d3015df80 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d383f3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2d30164670 con 0x7f2d30164b00 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3598e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d3015df80 0x7f2d300b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3598e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d3015df80 0x7f2d300b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44692/0 (socket says 172.21.15.67:44692) 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3598e640 1 -- 172.21.15.67:0/2388858961 learned_addr learned my addr 172.21.15.67:0/2388858961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d36990640 1 --1- 172.21.15.67:0/2388858961 >> v1:172.21.15.67:6789/0 conn(0x7f2d30160890 0x7f2d30160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49508/0 (socket says 172.21.15.67:49508) 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3618f640 1 --2- 172.21.15.67:0/2388858961 >> v2:172.21.15.176:3300/0 conn(0x7f2d30164b00 0x7f2d300b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3598e640 1 -- 172.21.15.67:0/2388858961 >> v2:172.21.15.176:3300/0 conn(0x7f2d30164b00 msgr2=0x7f2d300b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3598e640 1 --2- 172.21.15.67:0/2388858961 >> v2:172.21.15.176:3300/0 conn(0x7f2d30164b00 0x7f2d300b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3598e640 1 -- 172.21.15.67:0/2388858961 >> v1:172.21.15.67:6789/0 conn(0x7f2d30160890 legacy=0x7f2d30160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.829+0000 7f2d3598e640 1 -- 172.21.15.67:0/2388858961 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d30164020 con 0x7f2d3015df80 2024-04-25T15:27:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.830+0000 7f2d3598e640 1 --2- 172.21.15.67:0/2388858961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3015df80 0x7f2d300b80c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2d3008e410 tx=0x7f2d2002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f7c0954b2c400e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.830+0000 7f2d3518d640 1 -- 172.21.15.67:0/2388858961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20037070 con 0x7f2d3015df80 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.830+0000 7f2d3518d640 1 -- 172.21.15.67:0/2388858961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d20032440 con 0x7f2d3015df80 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.830+0000 7f2d3518d640 1 -- 172.21.15.67:0/2388858961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20032810 con 0x7f2d3015df80 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 -- 172.21.15.67:0/2388858961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3015df80 msgr2=0x7f2d300b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 --2- 172.21.15.67:0/2388858961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3015df80 0x7f2d300b80c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f2d3008e410 tx=0x7f2d2002a270 comp rx=0 tx=0).stop 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 -- 172.21.15.67:0/2388858961 shutdown_connections 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 --2- 172.21.15.67:0/2388858961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d3015df80 0x7f2d300b80c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 --2- 172.21.15.67:0/2388858961 >> v2:172.21.15.176:3300/0 conn(0x7f2d30164b00 0x7f2d300b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 -- 172.21.15.67:0/2388858961 >> 172.21.15.67:0/2388858961 conn(0x7f2d300d2bc0 msgr2=0x7f2d300d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 -- 172.21.15.67:0/2388858961 shutdown_connections 2024-04-25T15:27:39.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.831+0000 7f2d383f3640 1 -- 172.21.15.67:0/2388858961 wait complete. 2024-04-25T15:27:39.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.833+0000 7f2d383f3640 1 Processor -- start 2024-04-25T15:27:39.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.833+0000 7f2d383f3640 1 -- start start 2024-04-25T15:27:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d383f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d3015df80 0x7f2d301556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d383f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d30160890 0x7f2d301f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d383f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d30164b00 0x7f2d301f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d383f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d300b4a00 con 0x7f2d30160890 2024-04-25T15:27:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d383f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d300b4880 con 0x7f2d3015df80 2024-04-25T15:27:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d383f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d300b4b80 con 0x7f2d30164b00 2024-04-25T15:27:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d3598e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d30160890 0x7f2d301f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d3598e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d30160890 0x7f2d301f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44706/0 (socket says 172.21.15.67:44706) 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.834+0000 7f2d3598e640 1 -- 172.21.15.67:0/845435691 learned_addr learned my addr 172.21.15.67:0/845435691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3618f640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d3015df80 0x7f2d301556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d36990640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d30164b00 0x7f2d301f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3598e640 1 -- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d30164b00 msgr2=0x7f2d301f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3598e640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d30164b00 0x7f2d301f6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3598e640 1 -- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d3015df80 msgr2=0x7f2d301556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3598e640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d3015df80 0x7f2d301556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3598e640 1 -- 172.21.15.67:0/845435691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d301542f0 con 0x7f2d30160890 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3618f640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d3015df80 0x7f2d301556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d36990640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d30164b00 0x7f2d301f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d3598e640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d30160890 0x7f2d301f6a30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2d20000c00 tx=0x7f2d200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d26ffd640 1 -- 172.21.15.67:0/845435691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20037040 con 0x7f2d30160890 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d26ffd640 1 -- 172.21.15.67:0/845435691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d20042070 con 0x7f2d30160890 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d26ffd640 1 -- 172.21.15.67:0/845435691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20030bb0 con 0x7f2d30160890 2024-04-25T15:27:39.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d301545d0 con 0x7f2d30160890 2024-04-25T15:27:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.835+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d30154b90 con 0x7f2d30160890 2024-04-25T15:27:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.836+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d30050740 con 0x7f2d30160890 2024-04-25T15:27:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.837+0000 7f2d26ffd640 1 -- 172.21.15.67:0/845435691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2d20047460 con 0x7f2d30160890 2024-04-25T15:27:39.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.837+0000 7f2d26ffd640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d140744b0 0x7f2d140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:39.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.837+0000 7f2d26ffd640 1 -- 172.21.15.67:0/845435691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f2d200b5b70 con 0x7f2d30160890 2024-04-25T15:27:39.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.838+0000 7f2d3618f640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d140744b0 0x7f2d140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:39.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.838+0000 7f2d3618f640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d140744b0 0x7f2d140769c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c006fd0 tx=0x7f2d2c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:39.842+0000 7f2d26ffd640 1 -- 172.21.15.67:0/845435691 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d20080690 con 0x7f2d30160890 2024-04-25T15:27:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.027+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d30155d90 con 0x7f2d30160890 2024-04-25T15:27:40.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.028+0000 7f2d26ffd640 1 -- 172.21.15.67:0/845435691 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+12984 (secure 0 0 0) 0x7f2d2007ff90 con 0x7f2d30160890 2024-04-25T15:27:40.029 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:40.029 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:38.693964+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6833/3185771646":"2024-04-26T15:27:35.684597+0000","172.21.15.67:6832/3185771646":"2024-04-26T15:27:35.684597+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d140744b0 msgr2=0x7f2d140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d140744b0 0x7f2d140769c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c006fd0 tx=0x7f2d2c008040 comp rx=0 tx=0).stop 2024-04-25T15:27:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d30160890 msgr2=0x7f2d301f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d30160890 0x7f2d301f6a30 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2d20000c00 tx=0x7f2d200046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 shutdown_connections 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d140744b0 0x7f2d140769c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d30164b00 0x7f2d301f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d30160890 0x7f2d301f6a30 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 --2- 172.21.15.67:0/845435691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d3015df80 0x7f2d301556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 >> 172.21.15.67:0/845435691 conn(0x7f2d300d2bc0 msgr2=0x7f2d3015bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 shutdown_connections 2024-04-25T15:27:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.030+0000 7f2d383f3640 1 -- 172.21.15.67:0/845435691 wait complete. 2024-04-25T15:27:40.042 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:40.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:40.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.156+0000 7f5ae1ab1640 1 Processor -- start 2024-04-25T15:27:40.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.156+0000 7f5ae1ab1640 1 -- start start 2024-04-25T15:27:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5ae1ab1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5adc1628e0 0x7f5adc161650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5ae1ab1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0ce3f0 0x7f5adc0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5ae1ab1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5adc0cef70 con 0x7f5adc0b39f0 2024-04-25T15:27:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5ae1ab1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5adc0ced40 con 0x7f5adc1628e0 2024-04-25T15:27:40.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5ae1ab1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5adc0cf250 con 0x7f5adc0ce3f0 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adb7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5adc1628e0 0x7f5adc161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adbfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f5adc0b39f0 0x7f5adc0b3e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:40040/0 (socket says 172.21.15.67:40040) 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adbfff640 1 -- 172.21.15.67:0/4199563951 learned_addr learned my addr 172.21.15.67:0/4199563951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adaffd640 1 --2- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0ce3f0 0x7f5adc0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adb7fe640 1 -- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0ce3f0 msgr2=0x7f5adc0ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adb7fe640 1 --2- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0ce3f0 0x7f5adc0ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adb7fe640 1 -- 172.21.15.67:0/4199563951 >> v1:172.21.15.176:6789/0 conn(0x7f5adc0b39f0 legacy=0x7f5adc0b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adb7fe640 1 -- 172.21.15.67:0/4199563951 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc161e00 con 0x7f5adc1628e0 2024-04-25T15:27:40.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.158+0000 7f5adaffd640 1 --2- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0ce3f0 0x7f5adc0ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:40.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.159+0000 7f5adb7fe640 1 --2- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc1628e0 0x7f5adc161650 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0097a0 tx=0x7f5acc02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1fec18810aff134 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:40.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.159+0000 7f5ada7fc640 1 -- 172.21.15.67:0/4199563951 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc037070 con 0x7f5adc1628e0 2024-04-25T15:27:40.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.159+0000 7f5ada7fc640 1 -- 172.21.15.67:0/4199563951 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5acc02ae50 con 0x7f5adc1628e0 2024-04-25T15:27:40.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.159+0000 7f5ada7fc640 1 -- 172.21.15.67:0/4199563951 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc032660 con 0x7f5adc1628e0 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc1628e0 msgr2=0x7f5adc161650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc1628e0 0x7f5adc161650 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0097a0 tx=0x7f5acc02a100 comp rx=0 tx=0).stop 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/4199563951 shutdown_connections 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0ce3f0 0x7f5adc0ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/4199563951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc1628e0 0x7f5adc161650 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/4199563951 >> 172.21.15.67:0/4199563951 conn(0x7f5adc09b7d0 msgr2=0x7f5adc09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/4199563951 shutdown_connections 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.160+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/4199563951 wait complete. 2024-04-25T15:27:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.162+0000 7f5ae1ab1640 1 Processor -- start 2024-04-25T15:27:40.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.162+0000 7f5ae1ab1640 1 -- start start 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.163+0000 7f5ae1ab1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0b39f0 0x7f5adc15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.163+0000 7f5ae1ab1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc0ce3f0 0x7f5adc148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.163+0000 7f5ae1ab1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1628e0 0x7f5adc1f7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.163+0000 7f5ae1ab1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc0d00c0 con 0x7f5adc1628e0 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.163+0000 7f5ae1ab1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc162450 con 0x7f5adc0ce3f0 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.163+0000 7f5ae1ab1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5adc0d0240 con 0x7f5adc0b39f0 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.163+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1628e0 0x7f5adc1f7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1628e0 0x7f5adc1f7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44720/0 (socket says 172.21.15.67:44720) 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 -- 172.21.15.67:0/540264891 learned_addr learned my addr 172.21.15.67:0/540264891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adaffd640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc0ce3f0 0x7f5adc148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adb7fe640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0b39f0 0x7f5adc15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 -- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0b39f0 msgr2=0x7f5adc15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0b39f0 0x7f5adc15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 -- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc0ce3f0 msgr2=0x7f5adc148590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc0ce3f0 0x7f5adc148590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 -- 172.21.15.67:0/540264891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc15eae0 con 0x7f5adc1628e0 2024-04-25T15:27:40.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5adbfff640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1628e0 0x7f5adc1f7820 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400caa0 tx=0x7f5ad400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:40.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5ac3fff640 1 -- 172.21.15.67:0/540264891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4017070 con 0x7f5adc1628e0 2024-04-25T15:27:40.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5ac3fff640 1 -- 172.21.15.67:0/540264891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad400ac00 con 0x7f5adc1628e0 2024-04-25T15:27:40.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc15eff0 con 0x7f5adc1628e0 2024-04-25T15:27:40.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5ac3fff640 1 -- 172.21.15.67:0/540264891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4012400 con 0x7f5adc1628e0 2024-04-25T15:27:40.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.164+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc15f5b0 con 0x7f5adc1628e0 2024-04-25T15:27:40.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.166+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5adc050740 con 0x7f5adc1628e0 2024-04-25T15:27:40.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.166+0000 7f5ac3fff640 1 -- 172.21.15.67:0/540264891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5ad401a070 con 0x7f5adc1628e0 2024-04-25T15:27:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.166+0000 7f5ac3fff640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5ac80744b0 0x7f5ac80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.167+0000 7f5ac3fff640 1 -- 172.21.15.67:0/540264891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f5ad4094a30 con 0x7f5adc1628e0 2024-04-25T15:27:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.167+0000 7f5adb7fe640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5ac80744b0 0x7f5ac80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.167+0000 7f5adb7fe640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5ac80744b0 0x7f5ac80769c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5acc034000 tx=0x7f5acc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:40.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.171+0000 7f5ac3fff640 1 -- 172.21.15.67:0/540264891 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ad405ef00 con 0x7f5adc1628e0 2024-04-25T15:27:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.355+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5adc15ed10 con 0x7f5adc1628e0 2024-04-25T15:27:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.356+0000 7f5ac3fff640 1 -- 172.21.15.67:0/540264891 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) ==== 93+0+10 (secure 0 0 0) 0x7f5ad4063200 con 0x7f5adc1628e0 2024-04-25T15:27:40.357 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5ac80744b0 msgr2=0x7f5ac80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5ac80744b0 0x7f5ac80769c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5acc034000 tx=0x7f5acc02e000 comp rx=0 tx=0).stop 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1628e0 msgr2=0x7f5adc1f7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1628e0 0x7f5adc1f7820 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400caa0 tx=0x7f5ad400cf60 comp rx=0 tx=0).stop 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 shutdown_connections 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5ac80744b0 0x7f5ac80769c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5adc1628e0 0x7f5adc1f7820 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5adc0ce3f0 0x7f5adc148590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 --2- 172.21.15.67:0/540264891 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5adc0b39f0 0x7f5adc15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 >> 172.21.15.67:0/540264891 conn(0x7f5adc09b7d0 msgr2=0x7f5adc0b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 shutdown_connections 2024-04-25T15:27:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.358+0000 7f5ae1ab1640 1 -- 172.21.15.67:0/540264891 wait complete. 2024-04-25T15:27:40.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df' 2024-04-25T15:27:40.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.481+0000 7f5296b7a640 1 Processor -- start 2024-04-25T15:27:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.481+0000 7f5296b7a640 1 -- start start 2024-04-25T15:27:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5296b7a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52900d0e00 0x7f52900d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5296b7a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f52900d4a50 0x7f52900d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5296b7a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52900d1850 con 0x7f529006c5c0 2024-04-25T15:27:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5296b7a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5290160de0 con 0x7f52900d0e00 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5296b7a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f52901610c0 con 0x7f52900d4a50 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5294916640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52900d0e00 0x7f52900d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5294916640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52900d0e00 0x7f52900d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44730/0 (socket says 172.21.15.67:44730) 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.482+0000 7f5294916640 1 -- 172.21.15.67:0/786467176 learned_addr learned my addr 172.21.15.67:0/786467176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f5295117640 1 --2- 172.21.15.67:0/786467176 >> v2:172.21.15.176:3300/0 conn(0x7f52900d4a50 0x7f52900d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f5294916640 1 -- 172.21.15.67:0/786467176 >> v2:172.21.15.176:3300/0 conn(0x7f52900d4a50 msgr2=0x7f52900d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f5294916640 1 --2- 172.21.15.67:0/786467176 >> v2:172.21.15.176:3300/0 conn(0x7f52900d4a50 0x7f52900d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f5294916640 1 -- 172.21.15.67:0/786467176 >> v1:172.21.15.176:6789/0 conn(0x7f529006c5c0 legacy=0x7f529006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f5294916640 1 -- 172.21.15.67:0/786467176 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52901613a0 con 0x7f52900d0e00 2024-04-25T15:27:40.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f5295117640 1 --2- 172.21.15.67:0/786467176 >> v2:172.21.15.176:3300/0 conn(0x7f52900d4a50 0x7f52900d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f5294916640 1 --2- 172.21.15.67:0/786467176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900d1210 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f52900d1750 tx=0x7f528402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafdf259e06fdf32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.483+0000 7f528f7fe640 1 -- 172.21.15.67:0/786467176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284037070 con 0x7f52900d0e00 2024-04-25T15:27:40.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f528f7fe640 1 -- 172.21.15.67:0/786467176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5284032440 con 0x7f52900d0e00 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f528f7fe640 1 -- 172.21.15.67:0/786467176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284032810 con 0x7f52900d0e00 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 -- 172.21.15.67:0/786467176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 msgr2=0x7f52900d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 --2- 172.21.15.67:0/786467176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900d1210 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f52900d1750 tx=0x7f528402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 -- 172.21.15.67:0/786467176 shutdown_connections 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 --2- 172.21.15.67:0/786467176 >> v2:172.21.15.176:3300/0 conn(0x7f52900d4a50 0x7f52900d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 --2- 172.21.15.67:0/786467176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900d1210 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 -- 172.21.15.67:0/786467176 >> 172.21.15.67:0/786467176 conn(0x7f52900d5d90 msgr2=0x7f52900d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 -- 172.21.15.67:0/786467176 shutdown_connections 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.484+0000 7f5296b7a640 1 -- 172.21.15.67:0/786467176 wait complete. 2024-04-25T15:27:40.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.486+0000 7f5296b7a640 1 Processor -- start 2024-04-25T15:27:40.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.486+0000 7f5296b7a640 1 -- start start 2024-04-25T15:27:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.487+0000 7f5296b7a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f529006c5c0 0x7f52901f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.487+0000 7f5296b7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.487+0000 7f5296b7a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52900d4a50 0x7f52900cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.487+0000 7f5296b7a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5290164e00 con 0x7f52900d0e00 2024-04-25T15:27:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.487+0000 7f5296b7a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f52900b5320 con 0x7f52900d4a50 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.487+0000 7f5296b7a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5290164f80 con 0x7f529006c5c0 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44732/0 (socket says 172.21.15.67:44732) 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 -- 172.21.15.67:0/2923250703 learned_addr learned my addr 172.21.15.67:0/2923250703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f5294916640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f529006c5c0 0x7f52901f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f5295117640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52900d4a50 0x7f52900cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 -- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f529006c5c0 msgr2=0x7f52901f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f529006c5c0 0x7f52901f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 -- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52900d4a50 msgr2=0x7f52900cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52900d4a50 0x7f52900cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 -- 172.21.15.67:0/2923250703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52900ca6b0 con 0x7f52900d0e00 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f5294916640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f529006c5c0 0x7f52901f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528ffff640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900cb470 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f527c00b420 tx=0x7f527c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528d7fa640 1 -- 172.21.15.67:0/2923250703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c011830 con 0x7f52900d0e00 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528d7fa640 1 -- 172.21.15.67:0/2923250703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527c016070 con 0x7f52900d0e00 2024-04-25T15:27:40.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52900ca990 con 0x7f52900d0e00 2024-04-25T15:27:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f528d7fa640 1 -- 172.21.15.67:0/2923250703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c011c10 con 0x7f52900d0e00 2024-04-25T15:27:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.488+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52900caf50 con 0x7f52900d0e00 2024-04-25T15:27:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.489+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52900ca020 con 0x7f52900d0e00 2024-04-25T15:27:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.490+0000 7f528d7fa640 1 -- 172.21.15.67:0/2923250703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f527c01b460 con 0x7f52900d0e00 2024-04-25T15:27:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.490+0000 7f528d7fa640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52740744e0 0x7f52740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:40.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.491+0000 7f528d7fa640 1 -- 172.21.15.67:0/2923250703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f527c0949b0 con 0x7f52900d0e00 2024-04-25T15:27:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.491+0000 7f5294916640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52740744e0 0x7f52740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:40.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.491+0000 7f5294916640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52740744e0 0x7f52740769f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5284034000 tx=0x7f528402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:40.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.495+0000 7f528d7fa640 1 -- 172.21.15.67:0/2923250703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f527c020020 con 0x7f52900d0e00 2024-04-25T15:27:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:40.749+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]} v 0) -- 0x7f52900d0150 con 0x7f52900d0e00 2024-04-25T15:27:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.707+0000 7f528d7fa640 1 -- 172.21.15.67:0/2923250703 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]}]=0 v150) ==== 137+0+0 (secure 0 0 0) 0x7f527c05f4d0 con 0x7f52900d0e00 2024-04-25T15:27:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.710+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52740744e0 msgr2=0x7f52740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.710+0000 7f5296b7a640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52740744e0 0x7f52740769f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5284034000 tx=0x7f528402e000 comp rx=0 tx=0).stop 2024-04-25T15:27:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.710+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 msgr2=0x7f52900cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.710+0000 7f5296b7a640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900cb470 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f527c00b420 tx=0x7f527c00b8e0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 shutdown_connections 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52740744e0 0x7f52740769f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52900d4a50 0x7f52900cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52900d0e00 0x7f52900cb470 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 --2- 172.21.15.67:0/2923250703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f529006c5c0 0x7f52901f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 >> 172.21.15.67:0/2923250703 conn(0x7f52900d5d90 msgr2=0x7f52900b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 shutdown_connections 2024-04-25T15:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.711+0000 7f5296b7a640 1 -- 172.21.15.67:0/2923250703 wait complete. 2024-04-25T15:27:41.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:27:41.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.846+0000 7f581e6d1640 1 Processor -- start 2024-04-25T15:27:41.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.846+0000 7f581e6d1640 1 -- start start 2024-04-25T15:27:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.847+0000 7f581e6d1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 0x7f58180d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.847+0000 7f581e6d1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58180d1830 con 0x7f581806c5c0 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.847+0000 7f581e6d1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5818160dc0 con 0x7f58180d4a30 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.847+0000 7f581e6d1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58181610a0 con 0x7f58180d0de0 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.847+0000 7f58177fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f581806c5c0 0x7f581806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49510/0 (socket says 172.21.15.67:49510) 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.847+0000 7f58177fe640 1 -- 172.21.15.67:0/3663138098 learned_addr learned my addr 172.21.15.67:0/3663138098 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488084072 0 0) 0x7f58180d1830 con 0x7f581806c5c0 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5800003520 con 0x7f581806c5c0 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5817fff640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 0x7f58180d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2488276206 0 0) 0x7f5800003520 con 0x7f581806c5c0 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 msgr2=0x7f58180d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 0x7f58180d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 >> v1:172.21.15.176:6789/0 conn(0x7f58180d4a30 legacy=0x7f58180d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5818161380 con 0x7f581806c5c0 2024-04-25T15:27:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f580c001670 con 0x7f581806c5c0 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 >> v1:172.21.15.67:6789/0 conn(0x7f581806c5c0 legacy=0x7f581806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000039e0 0x7f5800003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5800004800 0x7f5800004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 0x7f5800005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58181610a0 con 0x7f58000039e0 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f580000baf0 con 0x7f5800004800 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f580000bcf0 con 0x7f58180d0de0 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5817fff640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000039e0 0x7f5800003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f58177fe640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5800004800 0x7f5800004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f581cc6e640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 0x7f5800005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5817fff640 1 -- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 msgr2=0x7f5800005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.848+0000 7f5817fff640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58180d0de0 0x7f5800005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.849+0000 7f5817fff640 1 -- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5800004800 msgr2=0x7f5800004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.849+0000 7f5817fff640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5800004800 0x7f5800004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.849+0000 7f5817fff640 1 -- 172.21.15.67:0/3663138098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5800003520 con 0x7f58000039e0 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.849+0000 7f5817fff640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000039e0 0x7f5800003e50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5808009a90 tx=0x7f5800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72f3036f952cd94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.849+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808036070 con 0x7f58000039e0 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.849+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5808031400 con 0x7f58000039e0 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.849+0000 7f5816ffd640 1 -- 172.21.15.67:0/3663138098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808009690 con 0x7f58000039e0 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.850+0000 7f581e6d1640 1 -- 172.21.15.67:0/3663138098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000039e0 msgr2=0x7f5800003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.850+0000 7f581e6d1640 1 --2- 172.21.15.67:0/3663138098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58000039e0 0x7f5800003e50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5808009a90 tx=0x7f5800005830 comp rx=0 tx=0).stop 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.850+0000 7f581cc6e640 1 -- 172.21.15.67:0/3663138098 reap_dead start 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.850+0000 7f581e6d1640 1 -- 172.21.15.67:0/3663138098 shutdown_connections 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.850+0000 7f581e6d1640 1 -- 172.21.15.67:0/3663138098 >> 172.21.15.67:0/3663138098 conn(0x7f58180d5d70 msgr2=0x7f58180d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.850+0000 7f581e6d1640 1 -- 172.21.15.67:0/3663138098 shutdown_connections 2024-04-25T15:27:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.850+0000 7f581e6d1640 1 -- 172.21.15.67:0/3663138098 wait complete. 2024-04-25T15:27:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.852+0000 7f581e6d1640 1 Processor -- start 2024-04-25T15:27:41.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.852+0000 7f581e6d1640 1 -- start start 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.853+0000 7f581e6d1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58000039e0 0x7f58180cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.853+0000 7f581e6d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180ce940 0x7f5818155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.853+0000 7f581e6d1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58180cefb0 0x7f58180cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.853+0000 7f581e6d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5818164de0 con 0x7f58180ce940 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.853+0000 7f581e6d1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f58180b5300 con 0x7f58180cefb0 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.853+0000 7f581e6d1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5818164f60 con 0x7f58000039e0 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.853+0000 7f58177fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180ce940 0x7f5818155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180ce940 0x7f5818155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44762/0 (socket says 172.21.15.67:44762) 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 -- 172.21.15.67:0/3114090461 learned_addr learned my addr 172.21.15.67:0/3114090461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f5817fff640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58000039e0 0x7f58180cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f581cc6e640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58180cefb0 0x7f58180cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 -- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58000039e0 msgr2=0x7f58180cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:41.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58000039e0 0x7f58180cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 -- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58180cefb0 msgr2=0x7f58180cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58180cefb0 0x7f58180cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 -- 172.21.15.67:0/3114090461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58180cf900 con 0x7f58180ce940 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f58177fe640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180ce940 0x7f5818155300 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f580c007310 tx=0x7f580c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f5814ff9640 1 -- 172.21.15.67:0/3114090461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c00f9c0 con 0x7f58180ce940 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f5814ff9640 1 -- 172.21.15.67:0/3114090461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c018070 con 0x7f58180ce940 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58180cc070 con 0x7f58180ce940 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f5814ff9640 1 -- 172.21.15.67:0/3114090461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c00fde0 con 0x7f58180ce940 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.854+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58180cce50 con 0x7f58180ce940 2024-04-25T15:27:41.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.855+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5818050740 con 0x7f58180ce940 2024-04-25T15:27:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.856+0000 7f5814ff9640 1 -- 172.21.15.67:0/3114090461 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f580c01d460 con 0x7f58180ce940 2024-04-25T15:27:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.856+0000 7f5814ff9640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58040744b0 0x7f58040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.857+0000 7f5814ff9640 1 -- 172.21.15.67:0/3114090461 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f580c0950d0 con 0x7f58180ce940 2024-04-25T15:27:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.857+0000 7f5817fff640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58040744b0 0x7f58040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.857+0000 7f5817fff640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58040744b0 0x7f58040769c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5808033000 tx=0x7f5808008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:41.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:41.861+0000 7f5814ff9640 1 -- 172.21.15.67:0/3114090461 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58180cf900 con 0x7f58180ce940 2024-04-25T15:27:42.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.045+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f58180cbb20 con 0x7f58180ce940 2024-04-25T15:27:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.049+0000 7f5814ff9640 1 -- 172.21.15.67:0/3114090461 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f580c05fae0 con 0x7f58180ce940 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58040744b0 msgr2=0x7f58040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58040744b0 0x7f58040769c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5808033000 tx=0x7f5808008030 comp rx=0 tx=0).stop 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180ce940 msgr2=0x7f5818155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180ce940 0x7f5818155300 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f580c007310 tx=0x7f580c0077d0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 shutdown_connections 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58040744b0 0x7f58040769c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58180cefb0 0x7f58180cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58180ce940 0x7f5818155300 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 --2- 172.21.15.67:0/3114090461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58000039e0 0x7f58180cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 >> 172.21.15.67:0/3114090461 conn(0x7f58180d5d70 msgr2=0x7f581806c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 shutdown_connections 2024-04-25T15:27:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.051+0000 7f581e6d1640 1 -- 172.21.15.67:0/3114090461 wait complete. 2024-04-25T15:27:42.063 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:27:42.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:27:42.102 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:27:42.157 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:27:42.158 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:42.158 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:42.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:27:42.173 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:42.173 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:42.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:27:42.190 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:42.190 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:42.228 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:27:42.228 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:27:42.229 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:42.247 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:27:42.247 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:27:42.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.379+0000 7f09f7cbd640 1 Processor -- start 2024-04-25T15:27:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.379+0000 7f09f7cbd640 1 -- start start 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.380+0000 7f09f7cbd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 0x7f09f00b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.380+0000 7f09f7cbd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f006d870 con 0x7f09f00b4830 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.380+0000 7f09f7cbd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f006db10 con 0x7f09f00d4570 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.380+0000 7f09f7cbd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f09f00b4c40 con 0x7f09f006e8f0 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.380+0000 7f09f5258640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f09f00b4830 0x7f09f00b2dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49514/0 (socket says 172.21.15.67:49514) 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.380+0000 7f09f5258640 1 -- 172.21.15.67:0/2354793053 learned_addr learned my addr 172.21.15.67:0/2354793053 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f625a640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 0x7f09f00b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200095814 0 0) 0x7f09f006d870 con 0x7f09f00b4830 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09dc003520 con 0x7f09f00b4830 2024-04-25T15:27:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3115733532 0 0) 0x7f09dc003520 con 0x7f09f00b4830 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 msgr2=0x7f09f00b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 0x7f09f00b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 >> v1:172.21.15.176:6789/0 conn(0x7f09f00d4570 legacy=0x7f09f00d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f006e0f0 con 0x7f09f00b4830 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f09e0001670 con 0x7f09f00b4830 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 >> v1:172.21.15.67:6789/0 conn(0x7f09f00b4830 legacy=0x7f09f00b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09dc0039e0 0x7f09dc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 0x7f09dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09dc004d40 0x7f09dc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc00bb50 con 0x7f09dc004d40 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc00bcd0 con 0x7f09dc0039e0 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f09dc00be80 con 0x7f09f006e8f0 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f625a640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09dc004d40 0x7f09dc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f5a59640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09dc0039e0 0x7f09dc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f5258640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 0x7f09dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f625a640 1 -- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 msgr2=0x7f09dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f625a640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f006e8f0 0x7f09dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f625a640 1 -- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09dc0039e0 msgr2=0x7f09dc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.381+0000 7f09f625a640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09dc0039e0 0x7f09dc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.382+0000 7f09f625a640 1 -- 172.21.15.67:0/2354793053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc003520 con 0x7f09dc004d40 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.382+0000 7f09f625a640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09dc004d40 0x7f09dc005150 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009fc0 tx=0x7f09dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c6c8be5a2f835d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.382+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8009c10 con 0x7f09dc004d40 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.382+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e8036070 con 0x7f09dc004d40 2024-04-25T15:27:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.382+0000 7f09f4a57640 1 -- 172.21.15.67:0/2354793053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8031670 con 0x7f09dc004d40 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.383+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2354793053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09dc004d40 msgr2=0x7f09dc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.383+0000 7f09f7cbd640 1 --2- 172.21.15.67:0/2354793053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09dc004d40 0x7f09dc005150 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009fc0 tx=0x7f09dc005830 comp rx=0 tx=0).stop 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.383+0000 7f09f625a640 1 -- 172.21.15.67:0/2354793053 reap_dead start 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.383+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2354793053 shutdown_connections 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.383+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2354793053 >> 172.21.15.67:0/2354793053 conn(0x7f09f00d3a50 msgr2=0x7f09f00d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.383+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2354793053 shutdown_connections 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.383+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2354793053 wait complete. 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.385+0000 7f09f7cbd640 1 Processor -- start 2024-04-25T15:27:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.385+0000 7f09f7cbd640 1 -- start start 2024-04-25T15:27:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f7cbd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09f014bd70 0x7f09f015e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f7cbd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09f014c180 0x7f09f015f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f7cbd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f015eb60 0x7f09f015fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f7cbd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00d0fb0 con 0x7f09f014bd70 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f7cbd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f09f006e460 con 0x7f09f014c180 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f7cbd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f09f006ed00 con 0x7f09f015eb60 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f5a59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09f014bd70 0x7f09f015e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f5a59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09f014bd70 0x7f09f015e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44772/0 (socket says 172.21.15.67:44772) 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.386+0000 7f09f5a59640 1 -- 172.21.15.67:0/2154957571 learned_addr learned my addr 172.21.15.67:0/2154957571 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5258640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09f014c180 0x7f09f015f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f625a640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f015eb60 0x7f09f015fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5a59640 1 -- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f015eb60 msgr2=0x7f09f015fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5a59640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f015eb60 0x7f09f015fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5a59640 1 -- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09f014c180 msgr2=0x7f09f015f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5a59640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09f014c180 0x7f09f015f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5a59640 1 -- 172.21.15.67:0/2154957571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f015d580 con 0x7f09f014bd70 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5258640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09f014c180 0x7f09f015f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f625a640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f015eb60 0x7f09f015fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f5a59640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09f014bd70 0x7f09f015e620 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00f7f0 tx=0x7f09ec00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09e67fc640 1 -- 172.21.15.67:0/2154957571 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0052e0 con 0x7f09f014bd70 2024-04-25T15:27:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09e67fc640 1 -- 172.21.15.67:0/2154957571 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09ec018070 con 0x7f09f014bd70 2024-04-25T15:27:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09e67fc640 1 -- 172.21.15.67:0/2154957571 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00d910 con 0x7f09f014bd70 2024-04-25T15:27:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f015d860 con 0x7f09f014bd70 2024-04-25T15:27:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.387+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f015de20 con 0x7f09f014bd70 2024-04-25T15:27:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.388+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f0050740 con 0x7f09f014bd70 2024-04-25T15:27:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.389+0000 7f09e67fc640 1 -- 172.21.15.67:0/2154957571 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f09ec00dab0 con 0x7f09f014bd70 2024-04-25T15:27:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.389+0000 7f09e67fc640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f09d40744b0 0x7f09d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.390+0000 7f09e67fc640 1 -- 172.21.15.67:0/2154957571 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f09ec094eb0 con 0x7f09f014bd70 2024-04-25T15:27:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.390+0000 7f09f5258640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f09d40744b0 0x7f09d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.390+0000 7f09f5258640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f09d40744b0 0x7f09d40769c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f09f00cfff0 tx=0x7f09e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.394+0000 7f09e67fc640 1 -- 172.21.15.67:0/2154957571 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09ec05f9d0 con 0x7f09f014bd70 2024-04-25T15:27:42.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.625+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09f00d09a0 con 0x7f09f014bd70 2024-04-25T15:27:42.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.625+0000 7f09e67fc640 1 -- 172.21.15.67:0/2154957571 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+3869 (secure 0 0 0) 0x7f09ec0054a0 con 0x7f09f014bd70 2024-04-25T15:27:42.626 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:42.626 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":34,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/610105566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":610105566},{"type":"v1","addr":"172.21.15.67:6835","nonce":610105566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/947236976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":947236976},{"type":"v1","addr":"172.21.15.176:6837","nonce":947236976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":5644,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/3211936625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":3211936625},{"type":"v1","addr":"172.21.15.176:6836","nonce":3211936625}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6198,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/2322433764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322433764},{"type":"v1","addr":"172.21.15.67:6833","nonce":2322433764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[]} 2024-04-25T15:27:42.627 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 34 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.627+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f09d40744b0 msgr2=0x7f09d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.627+0000 7f09f7cbd640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f09d40744b0 0x7f09d40769c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f09f00cfff0 tx=0x7f09e0009040 comp rx=0 tx=0).stop 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.627+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09f014bd70 msgr2=0x7f09f015e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.627+0000 7f09f7cbd640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09f014bd70 0x7f09f015e620 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00f7f0 tx=0x7f09ec00fcb0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 shutdown_connections 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f09d40744b0 0x7f09d40769c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09f015eb60 0x7f09f015fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09f014c180 0x7f09f015f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 --2- 172.21.15.67:0/2154957571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09f014bd70 0x7f09f015e620 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 >> 172.21.15.67:0/2154957571 conn(0x7f09f00d3a50 msgr2=0x7f09f0162eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 shutdown_connections 2024-04-25T15:27:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.628+0000 7f09f7cbd640 1 -- 172.21.15.67:0/2154957571 wait complete. 2024-04-25T15:27:42.640 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:27:42.640 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:27:42.640 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:27:42.640 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:42.640 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:27:42.640 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:27:42.640 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:27:42.641 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:42.641 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:27:42.641 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:27:42.641 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:27:42.641 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:42.641 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:27:42.641 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:27:42.641 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:27:42.642 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:27:42.651 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:27:42.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:27:42.653 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:27:42.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:27:42.654 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:27:42.654 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:27:42.655 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:27:42.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:27:42.695 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:27:42.695 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:42.696 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:27:42.697 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:42.697 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:27:42.698 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:42.698 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:27:42.698 DEBUG:teuthology.parallel:result is None 2024-04-25T15:27:42.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:27:42.784 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:27:42.784 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:27:42.789 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:27:42.789 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:27:42.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.816+0000 7f231a39d640 1 Processor -- start 2024-04-25T15:27:42.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.816+0000 7f231a39d640 1 -- start start 2024-04-25T15:27:42.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.817+0000 7f231a39d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f231406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.817+0000 7f231a39d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f23140d3030 0x7f231406e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f231a39d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f231406d9d0 con 0x7f231406e9b0 2024-04-25T15:27:42.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f231a39d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f231406d780 con 0x7f23140d3030 2024-04-25T15:27:42.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f231a39d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f23140d0f10 con 0x7f23140b91b0 2024-04-25T15:27:42.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2318b9a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f23140d3030 0x7f231406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f231939b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f231406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2318b9a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f23140d3030 0x7f231406e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:34716/0 (socket says 172.21.15.67:34716) 2024-04-25T15:27:42.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2318b9a640 1 -- 172.21.15.67:0/2094573140 learned_addr learned my addr 172.21.15.67:0/2094573140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:42.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f231939b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f231406dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:39396/0 (socket says 172.21.15.67:39396) 2024-04-25T15:27:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2319b9c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f231406e9b0 0x7f23140b71d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:40088/0 (socket says 172.21.15.67:40088) 2024-04-25T15:27:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2318b9a640 1 -- 172.21.15.67:0/2094573140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 msgr2=0x7f231406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2318b9a640 1 --2- 172.21.15.67:0/2094573140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f231406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2318b9a640 1 -- 172.21.15.67:0/2094573140 >> v1:172.21.15.176:6789/0 conn(0x7f231406e9b0 legacy=0x7f23140b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.818+0000 7f2318b9a640 1 -- 172.21.15.67:0/2094573140 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23140d1160 con 0x7f23140d3030 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.819+0000 7f2318b9a640 1 --2- 172.21.15.67:0/2094573140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23140d3030 0x7f231406e470 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2304009870 tx=0x7f230402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb1fba9173a7c367 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.819+0000 7f230bfff640 1 -- 172.21.15.67:0/2094573140 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2304037070 con 0x7f23140d3030 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.819+0000 7f230bfff640 1 -- 172.21.15.67:0/2094573140 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2304032440 con 0x7f23140d3030 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.819+0000 7f230bfff640 1 -- 172.21.15.67:0/2094573140 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2304032810 con 0x7f23140d3030 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 -- 172.21.15.67:0/2094573140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23140d3030 msgr2=0x7f231406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 --2- 172.21.15.67:0/2094573140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23140d3030 0x7f231406e470 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2304009870 tx=0x7f230402a1c0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 -- 172.21.15.67:0/2094573140 shutdown_connections 2024-04-25T15:27:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 --2- 172.21.15.67:0/2094573140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23140d3030 0x7f231406e470 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 --2- 172.21.15.67:0/2094573140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f231406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 -- 172.21.15.67:0/2094573140 >> 172.21.15.67:0/2094573140 conn(0x7f23140d03f0 msgr2=0x7f23140d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 -- 172.21.15.67:0/2094573140 shutdown_connections 2024-04-25T15:27:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.820+0000 7f231a39d640 1 -- 172.21.15.67:0/2094573140 wait complete. 2024-04-25T15:27:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.822+0000 7f231a39d640 1 Processor -- start 2024-04-25T15:27:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.822+0000 7f231a39d640 1 -- start start 2024-04-25T15:27:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.823+0000 7f231a39d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f231406e9b0 0x7f23141d8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.823+0000 7f231a39d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f23140b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.823+0000 7f231a39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23140d3030 0x7f23140b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.823+0000 7f231a39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f23140b78e0 con 0x7f23140d3030 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.823+0000 7f231a39d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f231409ccc0 con 0x7f231406e9b0 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.823+0000 7f231a39d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f23140b7a60 con 0x7f23140b91b0 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23140d3030 0x7f23140b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23140d3030 0x7f23140b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44804/0 (socket says 172.21.15.67:44804) 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 -- 172.21.15.67:0/781736017 learned_addr learned my addr 172.21.15.67:0/781736017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2318b9a640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f23140b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f231939b640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f231406e9b0 0x7f23141d8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 -- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 msgr2=0x7f23140b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f23140b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 -- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f231406e9b0 msgr2=0x7f23141d8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f231406e9b0 0x7f23141d8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 -- 172.21.15.67:0/781736017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231409e920 con 0x7f23140d3030 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f231939b640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f231406e9b0 0x7f23141d8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2319b9c640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23140d3030 0x7f23140b3b60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f230c00c6f0 tx=0x7f230c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2309ffb640 1 -- 172.21.15.67:0/781736017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c017070 con 0x7f23140d3030 2024-04-25T15:27:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231409ec00 con 0x7f23140d3030 2024-04-25T15:27:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2309ffb640 1 -- 172.21.15.67:0/781736017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230c005da0 con 0x7f23140d3030 2024-04-25T15:27:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.824+0000 7f2309ffb640 1 -- 172.21.15.67:0/781736017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c00a840 con 0x7f23140d3030 2024-04-25T15:27:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.825+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23140b2030 con 0x7f23140d3030 2024-04-25T15:27:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.825+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f231409e670 con 0x7f23140d3030 2024-04-25T15:27:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.826+0000 7f2309ffb640 1 -- 172.21.15.67:0/781736017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f230c01a070 con 0x7f23140d3030 2024-04-25T15:27:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.827+0000 7f2309ffb640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f22f80744b0 0x7f22f80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.827+0000 7f2309ffb640 1 -- 172.21.15.67:0/781736017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f230c0944d0 con 0x7f23140d3030 2024-04-25T15:27:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.827+0000 7f231939b640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f22f80744b0 0x7f22f80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.827+0000 7f231939b640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f22f80744b0 0x7f22f80769c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f23140c7470 tx=0x7f2310009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:42.831+0000 7f2309ffb640 1 -- 172.21.15.67:0/781736017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f230c05e880 con 0x7f23140d3030 2024-04-25T15:27:42.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-04-25T15:27:42.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-25T15:27:42.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-04-25T15:27:43.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.031+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f2314050740 con 0x7f23140d3030 2024-04-25T15:27:43.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.728+0000 7f2309ffb640 1 -- 172.21.15.67:0/781736017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v61) ==== 97+0+0 (secure 0 0 0) 0x7f230c062b80 con 0x7f23140d3030 2024-04-25T15:27:43.729 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f22f80744b0 msgr2=0x7f22f80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f22f80744b0 0x7f22f80769c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f23140c7470 tx=0x7f2310009110 comp rx=0 tx=0).stop 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23140d3030 msgr2=0x7f23140b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23140d3030 0x7f23140b3b60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f230c00c6f0 tx=0x7f230c00cbb0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 shutdown_connections 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f22f80744b0 0x7f22f80769c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23140d3030 0x7f23140b3b60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23140b91b0 0x7f23140b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 --2- 172.21.15.67:0/781736017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f231406e9b0 0x7f23141d8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 >> 172.21.15.67:0/781736017 conn(0x7f23140d03f0 msgr2=0x7f23140d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 shutdown_connections 2024-04-25T15:27:43.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.731+0000 7f231a39d640 1 -- 172.21.15.67:0/781736017 wait complete. 2024-04-25T15:27:43.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:27:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.866+0000 7f1dceb46640 1 Processor -- start 2024-04-25T15:27:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.866+0000 7f1dceb46640 1 -- start start 2024-04-25T15:27:43.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.867+0000 7f1dceb46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc80d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.867+0000 7f1dceb46640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1dc8163130 0x7f1dc80b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.867+0000 7f1dceb46640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dc80b0770 con 0x7f1dc80ce800 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.867+0000 7f1dceb46640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1dc80b10a0 con 0x7f1dc8163130 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.867+0000 7f1dceb46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc80d2100 con 0x7f1dc80d1620 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.867+0000 7f1dcc8e2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1dc80ce800 0x7f1dc8162a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49550/0 (socket says 172.21.15.67:49550) 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.867+0000 7f1dcc8e2640 1 -- 172.21.15.67:0/2801752068 learned_addr learned my addr 172.21.15.67:0/2801752068 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357475516 0 0) 0x7f1dc80b0770 con 0x7f1dc80ce800 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dcd0e3640 1 --2- 172.21.15.67:0/2801752068 >> v2:172.21.15.176:3300/0 conn(0x7f1dc8163130 0x7f1dc80b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc80d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db8003520 con 0x7f1dc80ce800 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2600018490 0 0) 0x7f1db8003520 con 0x7f1dc80ce800 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 msgr2=0x7f1dc80d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc80d1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 >> v2:172.21.15.176:3300/0 conn(0x7f1dc8163130 msgr2=0x7f1dc80b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 --2- 172.21.15.67:0/2801752068 >> v2:172.21.15.176:3300/0 conn(0x7f1dc8163130 0x7f1dc80b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc80b0a50 con 0x7f1dc80ce800 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f1dbc0017f0 con 0x7f1dc80ce800 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 >> v1:172.21.15.67:6789/0 conn(0x7f1dc80ce800 legacy=0x7f1dc8162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1db80039b0 0x7f1db8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1db8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db8004d40 0x7f1db8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc80b10a0 con 0x7f1db8004d40 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc80d2100 con 0x7f1db80039b0 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1db800baf0 con 0x7f1dc80d1620 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dcc8e2640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1db80039b0 0x7f1db8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1db8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.868+0000 7f1dcd0e3640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db8004d40 0x7f1db8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.869+0000 7f1dc7fff640 1 -- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1db80039b0 msgr2=0x7f1db8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.869+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1db80039b0 0x7f1db8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.869+0000 7f1dc7fff640 1 -- 172.21.15.67:0/2801752068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db8004d40 msgr2=0x7f1db8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.869+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db8004d40 0x7f1db8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.869+0000 7f1dc7fff640 1 -- 172.21.15.67:0/2801752068 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db8003520 con 0x7f1dc80d1620 2024-04-25T15:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.869+0000 7f1dcd0e3640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db8004d40 0x7f1db8005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.869+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1db8004800 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009a90 tx=0x7f1db8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5733e5cfc39b3e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4036070 con 0x7f1dc80d1620 2024-04-25T15:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db4031400 con 0x7f1dc80d1620 2024-04-25T15:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dc77fe640 1 -- 172.21.15.67:0/2801752068 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4009690 con 0x7f1dc80d1620 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dceb46640 1 -- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 msgr2=0x7f1db8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dceb46640 1 --2- 172.21.15.67:0/2801752068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1db8004800 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009a90 tx=0x7f1db8005830 comp rx=0 tx=0).stop 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dcd0e3640 1 -- 172.21.15.67:0/2801752068 reap_dead start 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dceb46640 1 -- 172.21.15.67:0/2801752068 shutdown_connections 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.870+0000 7f1dceb46640 1 -- 172.21.15.67:0/2801752068 >> 172.21.15.67:0/2801752068 conn(0x7f1dc80d3a50 msgr2=0x7f1dc80d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.871+0000 7f1dceb46640 1 -- 172.21.15.67:0/2801752068 shutdown_connections 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.871+0000 7f1dceb46640 1 -- 172.21.15.67:0/2801752068 wait complete. 2024-04-25T15:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.872+0000 7f1dceb46640 1 Processor -- start 2024-04-25T15:27:43.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.872+0000 7f1dceb46640 1 -- start start 2024-04-25T15:27:43.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dceb46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc81f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dceb46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc80b4140 0x7f1dc81f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dceb46640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1dc80b2480 0x7f1dc80cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dceb46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc8163540 con 0x7f1dc80b4140 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dceb46640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc80b51c0 con 0x7f1dc80b2480 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dceb46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc80b2c60 con 0x7f1dc80d1620 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc80b4140 0x7f1dc81f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc80b4140 0x7f1dc81f3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44820/0 (socket says 172.21.15.67:44820) 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 -- 172.21.15.67:0/1064480647 learned_addr learned my addr 172.21.15.67:0/1064480647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dcd0e3640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1dc80b2480 0x7f1dc80cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dcc8e2640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc81f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 -- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 msgr2=0x7f1dc81f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc81f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 -- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1dc80b2480 msgr2=0x7f1dc80cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1dc80b2480 0x7f1dc80cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dc7fff640 1 -- 172.21.15.67:0/1064480647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc81f4440 con 0x7f1dc80b4140 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.874+0000 7f1dcc8e2640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc81f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.875+0000 7f1dc7fff640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc80b4140 0x7f1dc81f3f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009a90 tx=0x7f1db4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.875+0000 7f1dc57fa640 1 -- 172.21.15.67:0/1064480647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4040020 con 0x7f1dc80b4140 2024-04-25T15:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.875+0000 7f1dc57fa640 1 -- 172.21.15.67:0/1064480647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db4003940 con 0x7f1dc80b4140 2024-04-25T15:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.875+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc806d6d0 con 0x7f1dc80b4140 2024-04-25T15:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.875+0000 7f1dc57fa640 1 -- 172.21.15.67:0/1064480647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4036040 con 0x7f1dc80b4140 2024-04-25T15:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.875+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc81f7590 con 0x7f1dc80b4140 2024-04-25T15:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.876+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc8050740 con 0x7f1dc80b4140 2024-04-25T15:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.877+0000 7f1dc57fa640 1 -- 172.21.15.67:0/1064480647 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1db4043020 con 0x7f1dc80b4140 2024-04-25T15:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.877+0000 7f1dc57fa640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1dac0744b0 0x7f1dac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.877+0000 7f1dc57fa640 1 -- 172.21.15.67:0/1064480647 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7f1db40b50f0 con 0x7f1dc80b4140 2024-04-25T15:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.877+0000 7f1dcc8e2640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1dac0744b0 0x7f1dac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.878+0000 7f1dcc8e2640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1dac0744b0 0x7f1dac0769c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc0010e0 tx=0x7f1dbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:43.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:43.882+0000 7f1dc57fa640 1 -- 172.21.15.67:0/1064480647 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1db404a070 con 0x7f1dc80b4140 2024-04-25T15:27:44.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.136+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1dc81f7040 con 0x7f1dc80b4140 2024-04-25T15:27:44.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.136+0000 7f1dc57fa640 1 -- 172.21.15.67:0/1064480647 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7f1db407fbe0 con 0x7f1dc80b4140 2024-04-25T15:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-25T15:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg==", 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA==", 2024-04-25T15:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q==", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw==", 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg==", 2024-04-25T15:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw==", 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA==", 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g==", 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA==", 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA==", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ==", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA==", 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q==", 2024-04-25T15:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ==", 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg==", 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-25T15:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA==", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmVIPuGRAAswFIakCxyA7OvvoJIBYUiQ==", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg==", 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag==", 2024-04-25T15:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:27:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1dac0744b0 msgr2=0x7f1dac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1dac0744b0 0x7f1dac0769c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc0010e0 tx=0x7f1dbc009040 comp rx=0 tx=0).stop 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc80b4140 msgr2=0x7f1dc81f3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc80b4140 0x7f1dc81f3f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009a90 tx=0x7f1db4004730 comp rx=0 tx=0).stop 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 shutdown_connections 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1dac0744b0 0x7f1dac0769c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1dc80b2480 0x7f1dc80cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc80b4140 0x7f1dc81f3f00 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 --2- 172.21.15.67:0/1064480647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1dc80d1620 0x7f1dc81f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 >> 172.21.15.67:0/1064480647 conn(0x7f1dc80d3a50 msgr2=0x7f1dc81612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 shutdown_connections 2024-04-25T15:27:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.138+0000 7f1dceb46640 1 -- 172.21.15.67:0/1064480647 wait complete. 2024-04-25T15:27:44.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:44.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.264+0000 7fe30ce61640 1 Processor -- start 2024-04-25T15:27:44.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.264+0000 7fe30ce61640 1 -- start start 2024-04-25T15:27:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.265+0000 7fe30ce61640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d0e00 0x7fe3080d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.265+0000 7fe30ce61640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3080d1850 con 0x7fe30806c5c0 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.265+0000 7fe30ce61640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe308160de0 con 0x7fe3080d4a50 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.265+0000 7fe30ce61640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3081610c0 con 0x7fe3080d0e00 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.265+0000 7fe2fffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe30806c5c0 0x7fe30806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49558/0 (socket says 172.21.15.67:49558) 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.265+0000 7fe2fffff640 1 -- 172.21.15.67:0/4137863119 learned_addr learned my addr 172.21.15.67:0/4137863119 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.265+0000 7fe30659c640 1 --2- 172.21.15.67:0/4137863119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d0e00 0x7fe3080d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.266+0000 7fe305d9b640 1 -- 172.21.15.67:0/4137863119 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036387751 0 0) 0x7fe3080d1850 con 0x7fe30806c5c0 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.266+0000 7fe305d9b640 1 -- 172.21.15.67:0/4137863119 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2f4003520 con 0x7fe30806c5c0 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.266+0000 7fe305d9b640 1 -- 172.21.15.67:0/4137863119 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2276574325 0 0) 0x7fe308160de0 con 0x7fe3080d4a50 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.266+0000 7fe30659c640 1 -- 172.21.15.67:0/4137863119 >> v1:172.21.15.176:6789/0 conn(0x7fe3080d4a50 legacy=0x7fe3080d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.266+0000 7fe30659c640 1 -- 172.21.15.67:0/4137863119 >> v1:172.21.15.67:6789/0 conn(0x7fe30806c5c0 legacy=0x7fe30806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.266+0000 7fe30659c640 1 -- 172.21.15.67:0/4137863119 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3081613a0 con 0x7fe3080d0e00 2024-04-25T15:27:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.267+0000 7fe30659c640 1 --2- 172.21.15.67:0/4137863119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d0e00 0x7fe3080d1210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80097d0 tx=0x7fe2f802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f7dde23c32781d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.267+0000 7fe305d9b640 1 -- 172.21.15.67:0/4137863119 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8037070 con 0x7fe3080d0e00 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.267+0000 7fe305d9b640 1 -- 172.21.15.67:0/4137863119 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f8032440 con 0x7fe3080d0e00 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.267+0000 7fe305d9b640 1 -- 172.21.15.67:0/4137863119 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80327f0 con 0x7fe3080d0e00 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.267+0000 7fe30ce61640 1 -- 172.21.15.67:0/4137863119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d0e00 msgr2=0x7fe3080d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.267+0000 7fe30ce61640 1 --2- 172.21.15.67:0/4137863119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d0e00 0x7fe3080d1210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80097d0 tx=0x7fe2f802a210 comp rx=0 tx=0).stop 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.268+0000 7fe30ce61640 1 -- 172.21.15.67:0/4137863119 shutdown_connections 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.268+0000 7fe30ce61640 1 --2- 172.21.15.67:0/4137863119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d0e00 0x7fe3080d1210 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.268+0000 7fe30ce61640 1 -- 172.21.15.67:0/4137863119 >> 172.21.15.67:0/4137863119 conn(0x7fe3080d5d90 msgr2=0x7fe3080d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.268+0000 7fe30ce61640 1 -- 172.21.15.67:0/4137863119 shutdown_connections 2024-04-25T15:27:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.268+0000 7fe30ce61640 1 -- 172.21.15.67:0/4137863119 wait complete. 2024-04-25T15:27:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.269+0000 7fe30ce61640 1 Processor -- start 2024-04-25T15:27:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.270+0000 7fe30ce61640 1 -- start start 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe30ce61640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe30806c5c0 0x7fe3081ec3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe30ce61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3080d0e00 0x7fe3081f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe30ce61640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d4a50 0x7fe3081f8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe30ce61640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe308164e00 con 0x7fe3080d0e00 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe30ce61640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3080b5320 con 0x7fe30806c5c0 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe30ce61640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe308164f80 con 0x7fe3080d4a50 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3080d0e00 0x7fe3081f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3080d0e00 0x7fe3081f7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44832/0 (socket says 172.21.15.67:44832) 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 -- 172.21.15.67:0/2450826375 learned_addr learned my addr 172.21.15.67:0/2450826375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe30659c640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe30806c5c0 0x7fe3081ec3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe306d9d640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d4a50 0x7fe3081f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 -- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d4a50 msgr2=0x7fe3081f8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d4a50 0x7fe3081f8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 -- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe30806c5c0 msgr2=0x7fe3081ec3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe30806c5c0 0x7fe3081ec3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.271+0000 7fe2fffff640 1 -- 172.21.15.67:0/2450826375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3081eb320 con 0x7fe3080d0e00 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.272+0000 7fe2fffff640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3080d0e00 0x7fe3081f7c00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c3c0 tx=0x7fe2f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.272+0000 7fe2feffd640 1 -- 172.21.15.67:0/2450826375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0017070 con 0x7fe3080d0e00 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.272+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3081eb600 con 0x7fe3080d0e00 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.272+0000 7fe2feffd640 1 -- 172.21.15.67:0/2450826375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f0005ce0 con 0x7fe3080d0e00 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.272+0000 7fe2feffd640 1 -- 172.21.15.67:0/2450826375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f00072f0 con 0x7fe3080d0e00 2024-04-25T15:27:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.272+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3081ebbc0 con 0x7fe3080d0e00 2024-04-25T15:27:44.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.273+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3080d66d0 con 0x7fe3080d0e00 2024-04-25T15:27:44.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.273+0000 7fe2feffd640 1 -- 172.21.15.67:0/2450826375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe2f001a070 con 0x7fe3080d0e00 2024-04-25T15:27:44.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.274+0000 7fe2feffd640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2e80744b0 0x7fe2e80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.274+0000 7fe2feffd640 1 -- 172.21.15.67:0/2450826375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7fe2f0094250 con 0x7fe3080d0e00 2024-04-25T15:27:44.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.274+0000 7fe30659c640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2e80744b0 0x7fe2e80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.275+0000 7fe30659c640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2e80744b0 0x7fe2e80769c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8034000 tx=0x7fe2f802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.279+0000 7fe2feffd640 1 -- 172.21.15.67:0/2450826375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3080d66d0 con 0x7fe3080d0e00 2024-04-25T15:27:44.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.464+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3081eb0b0 con 0x7fe3080d0e00 2024-04-25T15:27:44.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.464+0000 7fe2feffd640 1 -- 172.21.15.67:0/2450826375 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+12855 (secure 0 0 0) 0x7fe2f005e660 con 0x7fe3080d0e00 2024-04-25T15:27:44.465 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:44.465 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":63,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:43.738532+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2e80744b0 msgr2=0x7fe2e80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2e80744b0 0x7fe2e80769c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8034000 tx=0x7fe2f802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3080d0e00 msgr2=0x7fe3081f7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3080d0e00 0x7fe3081f7c00 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c3c0 tx=0x7fe2f000c880 comp rx=0 tx=0).stop 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 shutdown_connections 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2e80744b0 0x7fe2e80769c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3080d4a50 0x7fe3081f8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3080d0e00 0x7fe3081f7c00 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 --2- 172.21.15.67:0/2450826375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe30806c5c0 0x7fe3081ec3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 >> 172.21.15.67:0/2450826375 conn(0x7fe3080d5d90 msgr2=0x7fe3080d4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:44.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 shutdown_connections 2024-04-25T15:27:44.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.467+0000 7fe30ce61640 1 -- 172.21.15.67:0/2450826375 wait complete. 2024-04-25T15:27:44.479 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:27:44.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.595+0000 7f246a1d7640 1 Processor -- start 2024-04-25T15:27:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.595+0000 7f246a1d7640 1 -- start start 2024-04-25T15:27:44.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f246a1d7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2464161710 0x7f246415e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f246a1d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f246406c5c0 0x7f246406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f246a1d7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24641609b0 con 0x7f2464163fa0 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f246a1d7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f246406cf10 con 0x7f2464161710 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f246a1d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2464164cd0 con 0x7f246406c5c0 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f2463fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2464163fa0 0x7f24641643b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49568/0 (socket says 172.21.15.67:49568) 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f2463fff640 1 -- 172.21.15.67:0/3000677364 learned_addr learned my addr 172.21.15.67:0/3000677364 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f24637fe640 1 --2- 172.21.15.67:0/3000677364 >> v2:172.21.15.176:3300/0 conn(0x7f2464161710 0x7f246415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.596+0000 7f2462ffd640 1 --2- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f246406c5c0 0x7f246406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.597+0000 7f24627fc640 1 -- 172.21.15.67:0/3000677364 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 896927784 0 0) 0x7f245c001490 con 0x7f2464163fa0 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.597+0000 7f24627fc640 1 -- 172.21.15.67:0/3000677364 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f244c003520 con 0x7f2464163fa0 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.597+0000 7f24637fe640 1 -- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f246406c5c0 msgr2=0x7f246406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.597+0000 7f24637fe640 1 --2- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f246406c5c0 0x7f246406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.597+0000 7f24637fe640 1 -- 172.21.15.67:0/3000677364 >> v1:172.21.15.67:6789/0 conn(0x7f2464163fa0 legacy=0x7f24641643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.597+0000 7f24637fe640 1 -- 172.21.15.67:0/3000677364 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2464160c30 con 0x7f2464161710 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f24637fe640 1 --2- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464161710 0x7f246415e080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f24540097a0 tx=0x7f245402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2c24b2bf8a3f2dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f24627fc640 1 -- 172.21.15.67:0/3000677364 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454037070 con 0x7f2464161710 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f24627fc640 1 -- 172.21.15.67:0/3000677364 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f245402ae50 con 0x7f2464161710 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f24627fc640 1 -- 172.21.15.67:0/3000677364 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454032660 con 0x7f2464161710 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f246a1d7640 1 -- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464161710 msgr2=0x7f246415e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f246a1d7640 1 --2- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464161710 0x7f246415e080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f24540097a0 tx=0x7f245402a100 comp rx=0 tx=0).stop 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f246a1d7640 1 -- 172.21.15.67:0/3000677364 shutdown_connections 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f246a1d7640 1 --2- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f246406c5c0 0x7f246406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f246a1d7640 1 --2- 172.21.15.67:0/3000677364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464161710 0x7f246415e080 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.598+0000 7f246a1d7640 1 -- 172.21.15.67:0/3000677364 >> 172.21.15.67:0/3000677364 conn(0x7f24640d5d90 msgr2=0x7f24640d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.599+0000 7f246a1d7640 1 -- 172.21.15.67:0/3000677364 shutdown_connections 2024-04-25T15:27:44.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.599+0000 7f246a1d7640 1 -- 172.21.15.67:0/3000677364 wait complete. 2024-04-25T15:27:44.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.600+0000 7f246a1d7640 1 Processor -- start 2024-04-25T15:27:44.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.600+0000 7f246a1d7640 1 -- start start 2024-04-25T15:27:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f246a1d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f246406c5c0 0x7f24640bb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f246a1d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2464161710 0x7f24640baa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f246a1d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464163fa0 0x7f24640ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f246a1d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24640d49f0 con 0x7f246406c5c0 2024-04-25T15:27:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f246a1d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f24641611f0 con 0x7f2464163fa0 2024-04-25T15:27:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f246a1d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f24640d4b70 con 0x7f2464161710 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f246406c5c0 0x7f24640bb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f246406c5c0 0x7f24640bb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44836/0 (socket says 172.21.15.67:44836) 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 -- 172.21.15.67:0/2494832764 learned_addr learned my addr 172.21.15.67:0/2494832764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f2462ffd640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2464161710 0x7f24640baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f2463fff640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464163fa0 0x7f24640ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 -- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2464161710 msgr2=0x7f24640baa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2464161710 0x7f24640baa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 -- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464163fa0 msgr2=0x7f24640ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464163fa0 0x7f24640ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 -- 172.21.15.67:0/2494832764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24640cf200 con 0x7f246406c5c0 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.602+0000 7f24637fe640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f246406c5c0 0x7f24640bb510 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f24540098c0 tx=0x7f24540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.603+0000 7f244bfff640 1 -- 172.21.15.67:0/2494832764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454041030 con 0x7f246406c5c0 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.603+0000 7f244bfff640 1 -- 172.21.15.67:0/2494832764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f245402ac50 con 0x7f246406c5c0 2024-04-25T15:27:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.603+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24640cf4e0 con 0x7f246406c5c0 2024-04-25T15:27:44.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.603+0000 7f244bfff640 1 -- 172.21.15.67:0/2494832764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454037040 con 0x7f246406c5c0 2024-04-25T15:27:44.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.603+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24640cfaa0 con 0x7f246406c5c0 2024-04-25T15:27:44.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.604+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2464050740 con 0x7f246406c5c0 2024-04-25T15:27:44.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.604+0000 7f244bfff640 1 -- 172.21.15.67:0/2494832764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2454044070 con 0x7f246406c5c0 2024-04-25T15:27:44.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.605+0000 7f244bfff640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24500744e0 0x7f24500769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.605+0000 7f244bfff640 1 -- 172.21.15.67:0/2494832764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7f24540b5360 con 0x7f246406c5c0 2024-04-25T15:27:44.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.605+0000 7f2462ffd640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24500744e0 0x7f24500769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.606+0000 7f2462ffd640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24500744e0 0x7f24500769f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2464159780 tx=0x7f2458009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.610+0000 7f244bfff640 1 -- 172.21.15.67:0/2494832764 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f245407f7f0 con 0x7f246406c5c0 2024-04-25T15:27:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.794+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f24640ced40 con 0x7f246406c5c0 2024-04-25T15:27:44.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.795+0000 7f244bfff640 1 -- 172.21.15.67:0/2494832764 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) ==== 93+0+10 (secure 0 0 0) 0x7f2454083af0 con 0x7f246406c5c0 2024-04-25T15:27:44.796 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24500744e0 msgr2=0x7f24500769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24500744e0 0x7f24500769f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2464159780 tx=0x7f2458009190 comp rx=0 tx=0).stop 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f246406c5c0 msgr2=0x7f24640bb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f246406c5c0 0x7f24640bb510 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f24540098c0 tx=0x7f24540046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 shutdown_connections 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f24500744e0 0x7f24500769f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2464163fa0 0x7f24640ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2464161710 0x7f24640baa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 --2- 172.21.15.67:0/2494832764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f246406c5c0 0x7f24640bb510 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 >> 172.21.15.67:0/2494832764 conn(0x7f24640d5d90 msgr2=0x7f24640d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:44.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 shutdown_connections 2024-04-25T15:27:44.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.797+0000 7f246a1d7640 1 -- 172.21.15.67:0/2494832764 wait complete. 2024-04-25T15:27:44.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:27:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.920+0000 7f8d7dafa640 1 Processor -- start 2024-04-25T15:27:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.920+0000 7f8d7dafa640 1 -- start start 2024-04-25T15:27:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d7dafa640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8d7815d960 0x7f8d780b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d7dafa640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d780cf510 con 0x7f8d780d0bb0 2024-04-25T15:27:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d7dafa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d780b5380 con 0x7f8d78153820 2024-04-25T15:27:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d7dafa640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8d780b5660 con 0x7f8d7815d960 2024-04-25T15:27:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d77fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8d780d0bb0 0x7f8d780d0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49576/0 (socket says 172.21.15.67:49576) 2024-04-25T15:27:44.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d77fff640 1 -- 172.21.15.67:0/950491584 learned_addr learned my addr 172.21.15.67:0/950491584 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d777fe640 1 --2- 172.21.15.67:0/950491584 >> v2:172.21.15.176:3300/0 conn(0x7f8d7815d960 0x7f8d780b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d767fc640 1 -- 172.21.15.67:0/950491584 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 607679636 0 0) 0x7f8d780cf510 con 0x7f8d780d0bb0 2024-04-25T15:27:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d767fc640 1 -- 172.21.15.67:0/950491584 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d60003520 con 0x7f8d780d0bb0 2024-04-25T15:27:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d777fe640 1 -- 172.21.15.67:0/950491584 >> v1:172.21.15.176:6789/0 conn(0x7f8d78153820 legacy=0x7f8d780cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d777fe640 1 -- 172.21.15.67:0/950491584 >> v1:172.21.15.67:6789/0 conn(0x7f8d780d0bb0 legacy=0x7f8d780d0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.922+0000 7f8d777fe640 1 -- 172.21.15.67:0/950491584 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d780b5940 con 0x7f8d7815d960 2024-04-25T15:27:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.923+0000 7f8d777fe640 1 --2- 172.21.15.67:0/950491584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d7815d960 0x7f8d780b6140 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8d680097a0 tx=0x7f8d6802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3775d97d56ad9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.923+0000 7f8d767fc640 1 -- 172.21.15.67:0/950491584 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68037070 con 0x7f8d7815d960 2024-04-25T15:27:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.923+0000 7f8d767fc640 1 -- 172.21.15.67:0/950491584 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d68032440 con 0x7f8d7815d960 2024-04-25T15:27:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.923+0000 7f8d767fc640 1 -- 172.21.15.67:0/950491584 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d680327f0 con 0x7f8d7815d960 2024-04-25T15:27:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.924+0000 7f8d7dafa640 1 -- 172.21.15.67:0/950491584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d7815d960 msgr2=0x7f8d780b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.924+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/950491584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d7815d960 0x7f8d780b6140 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8d680097a0 tx=0x7f8d6802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.924+0000 7f8d7dafa640 1 -- 172.21.15.67:0/950491584 shutdown_connections 2024-04-25T15:27:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.924+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/950491584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d7815d960 0x7f8d780b6140 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.924+0000 7f8d7dafa640 1 -- 172.21.15.67:0/950491584 >> 172.21.15.67:0/950491584 conn(0x7f8d78073050 msgr2=0x7f8d78073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.924+0000 7f8d7dafa640 1 -- 172.21.15.67:0/950491584 shutdown_connections 2024-04-25T15:27:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.924+0000 7f8d7dafa640 1 -- 172.21.15.67:0/950491584 wait complete. 2024-04-25T15:27:44.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.926+0000 7f8d7dafa640 1 Processor -- start 2024-04-25T15:27:44.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.926+0000 7f8d7dafa640 1 -- start start 2024-04-25T15:27:44.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.927+0000 7f8d7dafa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d780d0bb0 0x7f8d781f9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.927+0000 7f8d7dafa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d78153820 0x7f8d7814fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.927+0000 7f8d7dafa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d7815d960 0x7f8d7814ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.927+0000 7f8d7dafa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7806d750 con 0x7f8d7815d960 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.927+0000 7f8d7dafa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7806d5d0 con 0x7f8d78153820 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.927+0000 7f8d7dafa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d7806d8d0 con 0x7f8d780d0bb0 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d7815d960 0x7f8d7814ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d7815d960 0x7f8d7814ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44846/0 (socket says 172.21.15.67:44846) 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 -- 172.21.15.67:0/130517289 learned_addr learned my addr 172.21.15.67:0/130517289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d777fe640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d780d0bb0 0x7f8d781f9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d76ffd640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d78153820 0x7f8d7814fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 -- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d780d0bb0 msgr2=0x7f8d781f9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d780d0bb0 0x7f8d781f9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 -- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d78153820 msgr2=0x7f8d7814fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d78153820 0x7f8d7814fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 -- 172.21.15.67:0/130517289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d781461a0 con 0x7f8d7815d960 2024-04-25T15:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d77fff640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d7815d960 0x7f8d7814ff90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000caa0 tx=0x7f8d7000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d5ffff640 1 -- 172.21.15.67:0/130517289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70017070 con 0x7f8d7815d960 2024-04-25T15:27:44.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d5ffff640 1 -- 172.21.15.67:0/130517289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d7000ac00 con 0x7f8d7815d960 2024-04-25T15:27:44.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d78155030 con 0x7f8d7815d960 2024-04-25T15:27:44.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d5ffff640 1 -- 172.21.15.67:0/130517289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70012400 con 0x7f8d7815d960 2024-04-25T15:27:44.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.928+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d781555f0 con 0x7f8d7815d960 2024-04-25T15:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.929+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d78050740 con 0x7f8d7815d960 2024-04-25T15:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.930+0000 7f8d5ffff640 1 -- 172.21.15.67:0/130517289 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8d7001a070 con 0x7f8d7815d960 2024-04-25T15:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.930+0000 7f8d5ffff640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d640744b0 0x7f8d640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.930+0000 7f8d5ffff640 1 -- 172.21.15.67:0/130517289 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7f8d70094df0 con 0x7f8d7815d960 2024-04-25T15:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.931+0000 7f8d777fe640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d640744b0 0x7f8d640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.931+0000 7f8d777fe640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d640744b0 0x7f8d640769c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8d68034000 tx=0x7f8d6802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:44.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:44.935+0000 7f8d5ffff640 1 -- 172.21.15.67:0/130517289 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d7005f850 con 0x7f8d7815d960 2024-04-25T15:27:45.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.120+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f8d78156e90 con 0x7f8d7815d960 2024-04-25T15:27:45.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.121+0000 7f8d5ffff640 1 -- 172.21.15.67:0/130517289 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) ==== 84+0+65 (secure 0 0 0) 0x7f8d7005f150 con 0x7f8d7815d960 2024-04-25T15:27:45.122 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:45.122 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:27:45.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:27:45.122 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:27:45.122 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:27:45.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:27:45.122 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:27:45.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.122+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d640744b0 msgr2=0x7f8d640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.122+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d640744b0 0x7f8d640769c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8d68034000 tx=0x7f8d6802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.122+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d7815d960 msgr2=0x7f8d7814ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.122+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d7815d960 0x7f8d7814ff90 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000caa0 tx=0x7f8d7000cf60 comp rx=0 tx=0).stop 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 shutdown_connections 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8d640744b0 0x7f8d640769c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d7815d960 0x7f8d7814ff90 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d78153820 0x7f8d7814fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 --2- 172.21.15.67:0/130517289 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d780d0bb0 0x7f8d781f9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 >> 172.21.15.67:0/130517289 conn(0x7f8d78073050 msgr2=0x7f8d780b66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 shutdown_connections 2024-04-25T15:27:45.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.123+0000 7f8d7dafa640 1 -- 172.21.15.67:0/130517289 wait complete. 2024-04-25T15:27:45.135 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:27:45.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:27:45.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.250+0000 7f64ef1b2640 1 Processor -- start 2024-04-25T15:27:45.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.250+0000 7f64ef1b2640 1 -- start start 2024-04-25T15:27:45.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ef1b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80d1e20 0x7f64e80d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ef1b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64e80b2e60 0x7f64e808d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ef1b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f64e8164020 0x7f64e8164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ef1b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f64e8164970 con 0x7f64e80b2e60 2024-04-25T15:27:45.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ef1b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f64e80d23f0 con 0x7f64e8164020 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ef1b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64e80d26d0 con 0x7f64e80d1e20 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64e80b2e60 0x7f64e808d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64e80b2e60 0x7f64e808d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44854/0 (socket says 172.21.15.67:44854) 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 -- 172.21.15.67:0/988148354 learned_addr learned my addr 172.21.15.67:0/988148354 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ecf4e640 1 --2- 172.21.15.67:0/988148354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80d1e20 0x7f64e80d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ed74f640 1 --2- 172.21.15.67:0/988148354 >> v2:172.21.15.176:3300/0 conn(0x7f64e8164020 0x7f64e8164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 -- 172.21.15.67:0/988148354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80d1e20 msgr2=0x7f64e80d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 --2- 172.21.15.67:0/988148354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80d1e20 0x7f64e80d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 -- 172.21.15.67:0/988148354 >> v2:172.21.15.176:3300/0 conn(0x7f64e8164020 msgr2=0x7f64e8164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 --2- 172.21.15.67:0/988148354 >> v2:172.21.15.176:3300/0 conn(0x7f64e8164020 0x7f64e8164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64e7fff640 1 -- 172.21.15.67:0/988148354 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e80d29b0 con 0x7f64e80b2e60 2024-04-25T15:27:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.252+0000 7f64ed74f640 1 --2- 172.21.15.67:0/988148354 >> v2:172.21.15.176:3300/0 conn(0x7f64e8164020 0x7f64e8164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.253+0000 7f64e7fff640 1 --2- 172.21.15.67:0/988148354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e80b2e60 0x7f64e808d770 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f64e80b9b10 tx=0x7f64dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e724eacab4f6105 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.253+0000 7f64e77fe640 1 -- 172.21.15.67:0/988148354 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc037070 con 0x7f64e80b2e60 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.253+0000 7f64e77fe640 1 -- 172.21.15.67:0/988148354 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64dc032440 con 0x7f64e80b2e60 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.253+0000 7f64e77fe640 1 -- 172.21.15.67:0/988148354 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc032810 con 0x7f64e80b2e60 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 -- 172.21.15.67:0/988148354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e80b2e60 msgr2=0x7f64e808d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/988148354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e80b2e60 0x7f64e808d770 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f64e80b9b10 tx=0x7f64dc02a270 comp rx=0 tx=0).stop 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 -- 172.21.15.67:0/988148354 shutdown_connections 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/988148354 >> v2:172.21.15.176:3300/0 conn(0x7f64e8164020 0x7f64e8164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/988148354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e80b2e60 0x7f64e808d770 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/988148354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80d1e20 0x7f64e80d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 -- 172.21.15.67:0/988148354 >> 172.21.15.67:0/988148354 conn(0x7f64e80d0530 msgr2=0x7f64e80d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 -- 172.21.15.67:0/988148354 shutdown_connections 2024-04-25T15:27:45.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.254+0000 7f64ef1b2640 1 -- 172.21.15.67:0/988148354 wait complete. 2024-04-25T15:27:45.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.256+0000 7f64ef1b2640 1 Processor -- start 2024-04-25T15:27:45.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.256+0000 7f64ef1b2640 1 -- start start 2024-04-25T15:27:45.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ef1b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80b2e60 0x7f64e80b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ef1b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e80d1e20 0x7f64e8155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ef1b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e8164020 0x7f64e81560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ef1b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64e808dcb0 con 0x7f64e8164020 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ef1b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f64e80d2f70 con 0x7f64e80d1e20 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ef1b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64e80ce070 con 0x7f64e80b2e60 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ed74f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e8164020 0x7f64e81560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ed74f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e8164020 0x7f64e81560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44862/0 (socket says 172.21.15.67:44862) 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.257+0000 7f64ed74f640 1 -- 172.21.15.67:0/4009416009 learned_addr learned my addr 172.21.15.67:0/4009416009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ecf4e640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80b2e60 0x7f64e80b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64e7fff640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e80d1e20 0x7f64e8155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ed74f640 1 -- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80b2e60 msgr2=0x7f64e80b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ed74f640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80b2e60 0x7f64e80b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ed74f640 1 -- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e80d1e20 msgr2=0x7f64e8155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ed74f640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e80d1e20 0x7f64e8155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ed74f640 1 -- 172.21.15.67:0/4009416009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e80cbba0 con 0x7f64e8164020 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64e7fff640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e80d1e20 0x7f64e8155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ed74f640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e8164020 0x7f64e81560a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c5a0 tx=0x7f64d800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64e57fa640 1 -- 172.21.15.67:0/4009416009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d8015070 con 0x7f64e8164020 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64e57fa640 1 -- 172.21.15.67:0/4009416009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64d8004240 con 0x7f64e8164020 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e80cbe80 con 0x7f64e8164020 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64e57fa640 1 -- 172.21.15.67:0/4009416009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d8003610 con 0x7f64e8164020 2024-04-25T15:27:45.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.258+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e80cc440 con 0x7f64e8164020 2024-04-25T15:27:45.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.259+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e81617f0 con 0x7f64e8164020 2024-04-25T15:27:45.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.260+0000 7f64e57fa640 1 -- 172.21.15.67:0/4009416009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f64d80037b0 con 0x7f64e8164020 2024-04-25T15:27:45.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.260+0000 7f64e57fa640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.261+0000 7f64e57fa640 1 -- 172.21.15.67:0/4009416009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 4857+0+0 (secure 0 0 0) 0x7f64d8094cd0 con 0x7f64e8164020 2024-04-25T15:27:45.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.261+0000 7f64ecf4e640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.261+0000 7f64ecf4e640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f64e00099c0 tx=0x7f64e0003990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:45.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.265+0000 7f64e57fa640 1 -- 172.21.15.67:0/4009416009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64d805f720 con 0x7f64e8164020 2024-04-25T15:27:45.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.448+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f64e8050740 con 0x7f64e8164020 2024-04-25T15:27:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.727+0000 7f64e57fa640 1 -- 172.21.15.67:0/4009416009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) ==== 138+0+0 (secure 0 0 0) 0x7f64d805f020 con 0x7f64e8164020 2024-04-25T15:27:45.728 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:27:45.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.730+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 msgr2=0x7f64d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.730+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f64e00099c0 tx=0x7f64e0003990 comp rx=0 tx=0).stop 2024-04-25T15:27:45.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.730+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e8164020 msgr2=0x7f64e81560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.730+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e8164020 0x7f64e81560a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f64d800c5a0 tx=0x7f64d800ca60 comp rx=0 tx=0).stop 2024-04-25T15:27:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.731+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 shutdown_connections 2024-04-25T15:27:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.731+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64d40744b0 0x7f64d40769c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.731+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64e8164020 0x7f64e81560a0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.731+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e80d1e20 0x7f64e8155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.731+0000 7f64ef1b2640 1 --2- 172.21.15.67:0/4009416009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64e80b2e60 0x7f64e80b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.731+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 >> 172.21.15.67:0/4009416009 conn(0x7f64e80d0530 msgr2=0x7f64e80b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:45.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.732+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 shutdown_connections 2024-04-25T15:27:45.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.732+0000 7f64ef1b2640 1 -- 172.21.15.67:0/4009416009 wait complete. 2024-04-25T15:27:45.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:27:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.866+0000 7fe283b1c640 1 Processor -- start 2024-04-25T15:27:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.867+0000 7fe283b1c640 1 -- start start 2024-04-25T15:27:45.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe283b1c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0d1e20 0x7fe27c0d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe283b1c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe27c0b2e60 0x7fe27c08d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe283b1c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe27c164020 0x7fe27c164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe283b1c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe27c164970 con 0x7fe27c0b2e60 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe283b1c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe27c0d23f0 con 0x7fe27c164020 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe283b1c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe27c0d26d0 con 0x7fe27c0d1e20 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe27c0b2e60 0x7fe27c08d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe27c0b2e60 0x7fe27c08d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44872/0 (socket says 172.21.15.67:44872) 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 -- 172.21.15.67:0/1794280675 learned_addr learned my addr 172.21.15.67:0/1794280675 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2820b9640 1 --2- 172.21.15.67:0/1794280675 >> v2:172.21.15.176:3300/0 conn(0x7fe27c164020 0x7fe27c164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2818b8640 1 --2- 172.21.15.67:0/1794280675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0d1e20 0x7fe27c0d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 -- 172.21.15.67:0/1794280675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0d1e20 msgr2=0x7fe27c0d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 --2- 172.21.15.67:0/1794280675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0d1e20 0x7fe27c0d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 -- 172.21.15.67:0/1794280675 >> v2:172.21.15.176:3300/0 conn(0x7fe27c164020 msgr2=0x7fe27c164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 --2- 172.21.15.67:0/1794280675 >> v2:172.21.15.176:3300/0 conn(0x7fe27c164020 0x7fe27c164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2810b7640 1 -- 172.21.15.67:0/1794280675 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27c0d29b0 con 0x7fe27c0b2e60 2024-04-25T15:27:45.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2820b9640 1 --2- 172.21.15.67:0/1794280675 >> v2:172.21.15.176:3300/0 conn(0x7fe27c164020 0x7fe27c164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:45.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.868+0000 7fe2818b8640 1 --2- 172.21.15.67:0/1794280675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0d1e20 0x7fe27c0d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.869+0000 7fe2810b7640 1 --2- 172.21.15.67:0/1794280675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0b2e60 0x7fe27c08d770 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0b9b10 tx=0x7fe27002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4581a29d6de2662 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.869+0000 7fe2808b6640 1 -- 172.21.15.67:0/1794280675 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270037070 con 0x7fe27c0b2e60 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.869+0000 7fe2808b6640 1 -- 172.21.15.67:0/1794280675 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe270032440 con 0x7fe27c0b2e60 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.869+0000 7fe2808b6640 1 -- 172.21.15.67:0/1794280675 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe270032810 con 0x7fe27c0b2e60 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 -- 172.21.15.67:0/1794280675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0b2e60 msgr2=0x7fe27c08d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 --2- 172.21.15.67:0/1794280675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0b2e60 0x7fe27c08d770 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0b9b10 tx=0x7fe27002a270 comp rx=0 tx=0).stop 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 -- 172.21.15.67:0/1794280675 shutdown_connections 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 --2- 172.21.15.67:0/1794280675 >> v2:172.21.15.176:3300/0 conn(0x7fe27c164020 0x7fe27c164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 --2- 172.21.15.67:0/1794280675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c0b2e60 0x7fe27c08d770 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 --2- 172.21.15.67:0/1794280675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0d1e20 0x7fe27c0d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 -- 172.21.15.67:0/1794280675 >> 172.21.15.67:0/1794280675 conn(0x7fe27c0d0530 msgr2=0x7fe27c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 -- 172.21.15.67:0/1794280675 shutdown_connections 2024-04-25T15:27:45.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.870+0000 7fe283b1c640 1 -- 172.21.15.67:0/1794280675 wait complete. 2024-04-25T15:27:45.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.872+0000 7fe283b1c640 1 Processor -- start 2024-04-25T15:27:45.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.872+0000 7fe283b1c640 1 -- start start 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe283b1c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0b2e60 0x7fe27c0b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe283b1c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe27c0d1e20 0x7fe27c1eb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe283b1c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c164020 0x7fe27c1eb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe283b1c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c08dcb0 con 0x7fe27c164020 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe283b1c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c0d2f70 con 0x7fe27c0d1e20 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe283b1c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe27c0ce070 con 0x7fe27c0b2e60 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe2820b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c164020 0x7fe27c1eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe2820b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c164020 0x7fe27c1eb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44880/0 (socket says 172.21.15.67:44880) 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe2820b9640 1 -- 172.21.15.67:0/2068503916 learned_addr learned my addr 172.21.15.67:0/2068503916 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe2818b8640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0b2e60 0x7fe27c0b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.873+0000 7fe2810b7640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe27c0d1e20 0x7fe27c1eb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe2820b9640 1 -- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0b2e60 msgr2=0x7fe27c0b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe2820b9640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0b2e60 0x7fe27c0b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe2820b9640 1 -- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe27c0d1e20 msgr2=0x7fe27c1eb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe2820b9640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe27c0d1e20 0x7fe27c1eb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe2820b9640 1 -- 172.21.15.67:0/2068503916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27c0cb670 con 0x7fe27c164020 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe2810b7640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe27c0d1e20 0x7fe27c1eb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe2820b9640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c164020 0x7fe27c1eb950 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00cd40 tx=0x7fe26c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe27a7fc640 1 -- 172.21.15.67:0/2068503916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c003b90 con 0x7fe27c164020 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe27a7fc640 1 -- 172.21.15.67:0/2068503916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe26c003d50 con 0x7fe27c164020 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27c0b8890 con 0x7fe27c164020 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe27a7fc640 1 -- 172.21.15.67:0/2068503916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c004260 con 0x7fe27c164020 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.874+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe27c0cd4a0 con 0x7fe27c164020 2024-04-25T15:27:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.875+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe27c050740 con 0x7fe27c164020 2024-04-25T15:27:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.876+0000 7fe27a7fc640 1 -- 172.21.15.67:0/2068503916 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe26c004400 con 0x7fe27c164020 2024-04-25T15:27:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.876+0000 7fe27a7fc640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2680744e0 0x7fe2680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.876+0000 7fe27a7fc640 1 -- 172.21.15.67:0/2068503916 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5227+0+0 (secure 0 0 0) 0x7fe26c095840 con 0x7fe27c164020 2024-04-25T15:27:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.877+0000 7fe2818b8640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2680744e0 0x7fe2680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.877+0000 7fe2818b8640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2680744e0 0x7fe2680769f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe27400aac0 tx=0x7fe274008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:45.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:45.881+0000 7fe27a7fc640 1 -- 172.21.15.67:0/2068503916 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe26c014030 con 0x7fe27c164020 2024-04-25T15:27:46.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.068+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe27c0b8ac0 con 0x7fe27c164020 2024-04-25T15:27:46.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.730+0000 7fe27a7fc640 1 -- 172.21.15.67:0/2068503916 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) ==== 170+0+0 (secure 0 0 0) 0x7fe26c05fbf0 con 0x7fe27c164020 2024-04-25T15:27:46.731 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:27:46.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2680744e0 msgr2=0x7fe2680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2680744e0 0x7fe2680769f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe27400aac0 tx=0x7fe274008040 comp rx=0 tx=0).stop 2024-04-25T15:27:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c164020 msgr2=0x7fe27c1eb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c164020 0x7fe27c1eb950 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00cd40 tx=0x7fe26c00a650 comp rx=0 tx=0).stop 2024-04-25T15:27:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 shutdown_connections 2024-04-25T15:27:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe2680744e0 0x7fe2680769f0 secure :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe27400aac0 tx=0x7fe274008040 comp rx=0 tx=0).stop 2024-04-25T15:27:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe27c164020 0x7fe27c1eb950 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe27c0d1e20 0x7fe27c1eb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 --2- 172.21.15.67:0/2068503916 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe27c0b2e60 0x7fe27c0b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.734+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 >> 172.21.15.67:0/2068503916 conn(0x7fe27c0d0530 msgr2=0x7fe27c0d32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.735+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 shutdown_connections 2024-04-25T15:27:46.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.736+0000 7fe283b1c640 1 -- 172.21.15.67:0/2068503916 wait complete. 2024-04-25T15:27:46.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:27:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.882+0000 7fb3942aa640 1 Processor -- start 2024-04-25T15:27:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.882+0000 7fb3942aa640 1 -- start start 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb3942aa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb38c0d1e20 0x7fb38c0d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb3942aa640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb38c0b2e60 0x7fb38c08d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb3942aa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb38c0d23f0 con 0x7fb38c164020 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb3942aa640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb38c0d26d0 con 0x7fb38c0d1e20 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb3942aa640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb38c0d29b0 con 0x7fb38c0b2e60 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb392046640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb38c0d1e20 0x7fb38c0d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb392046640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb38c0d1e20 0x7fb38c0d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44886/0 (socket says 172.21.15.67:44886) 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.883+0000 7fb392046640 1 -- 172.21.15.67:0/4088908387 learned_addr learned my addr 172.21.15.67:0/4088908387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.884+0000 7fb391845640 1 --2- 172.21.15.67:0/4088908387 >> v2:172.21.15.176:3300/0 conn(0x7fb38c0b2e60 0x7fb38c08d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.884+0000 7fb392046640 1 -- 172.21.15.67:0/4088908387 >> v2:172.21.15.176:3300/0 conn(0x7fb38c0b2e60 msgr2=0x7fb38c08d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.884+0000 7fb392046640 1 --2- 172.21.15.67:0/4088908387 >> v2:172.21.15.176:3300/0 conn(0x7fb38c0b2e60 0x7fb38c08d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.884+0000 7fb392046640 1 -- 172.21.15.67:0/4088908387 >> v1:172.21.15.176:6789/0 conn(0x7fb38c164020 legacy=0x7fb38c164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.884+0000 7fb392046640 1 -- 172.21.15.67:0/4088908387 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38c0d2c90 con 0x7fb38c0d1e20 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.884+0000 7fb392046640 1 --2- 172.21.15.67:0/4088908387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c0d1e20 0x7fb38c0d3490 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb38c074650 tx=0x7fb37c02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3d0d0f71588ae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb391044640 1 -- 172.21.15.67:0/4088908387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c037070 con 0x7fb38c0d1e20 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb391044640 1 -- 172.21.15.67:0/4088908387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37c032440 con 0x7fb38c0d1e20 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb391044640 1 -- 172.21.15.67:0/4088908387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c032810 con 0x7fb38c0d1e20 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb3942aa640 1 -- 172.21.15.67:0/4088908387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c0d1e20 msgr2=0x7fb38c0d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb3942aa640 1 --2- 172.21.15.67:0/4088908387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c0d1e20 0x7fb38c0d3490 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb38c074650 tx=0x7fb37c02a2c0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb3942aa640 1 -- 172.21.15.67:0/4088908387 shutdown_connections 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb3942aa640 1 --2- 172.21.15.67:0/4088908387 >> v2:172.21.15.176:3300/0 conn(0x7fb38c0b2e60 0x7fb38c08d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb3942aa640 1 --2- 172.21.15.67:0/4088908387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c0d1e20 0x7fb38c0d3490 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.885+0000 7fb3942aa640 1 -- 172.21.15.67:0/4088908387 >> 172.21.15.67:0/4088908387 conn(0x7fb38c0d0530 msgr2=0x7fb38c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.886+0000 7fb3942aa640 1 -- 172.21.15.67:0/4088908387 shutdown_connections 2024-04-25T15:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.886+0000 7fb3942aa640 1 -- 172.21.15.67:0/4088908387 wait complete. 2024-04-25T15:27:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.887+0000 7fb3942aa640 1 Processor -- start 2024-04-25T15:27:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.887+0000 7fb3942aa640 1 -- start start 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.888+0000 7fb3942aa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb38c0b2e60 0x7fb38c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb3942aa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb38c0d1e20 0x7fb38c0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb3942aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c164020 0x7fb38c161af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb3942aa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c0ce070 con 0x7fb38c164020 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb3942aa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c08dcb0 con 0x7fb38c0d1e20 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb3942aa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb38c0ce1f0 con 0x7fb38c0b2e60 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c164020 0x7fb38c161af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c164020 0x7fb38c161af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44898/0 (socket says 172.21.15.67:44898) 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 -- 172.21.15.67:0/204920320 learned_addr learned my addr 172.21.15.67:0/204920320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb391845640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb38c0d1e20 0x7fb38c0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:46.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392046640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb38c0b2e60 0x7fb38c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 -- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb38c0b2e60 msgr2=0x7fb38c0b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb38c0b2e60 0x7fb38c0b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 -- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb38c0d1e20 msgr2=0x7fb38c0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb38c0d1e20 0x7fb38c0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 -- 172.21.15.67:0/204920320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38c0cd230 con 0x7fb38c164020 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb392847640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c164020 0x7fb38c161af0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb38400ba10 tx=0x7fb38400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb382ffd640 1 -- 172.21.15.67:0/204920320 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb384019070 con 0x7fb38c164020 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb382ffd640 1 -- 172.21.15.67:0/204920320 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38400cb40 con 0x7fb38c164020 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb38c0cb0f0 con 0x7fb38c164020 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb382ffd640 1 -- 172.21.15.67:0/204920320 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb384014440 con 0x7fb38c164020 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.889+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb38c0cb6b0 con 0x7fb38c164020 2024-04-25T15:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.890+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb38c0ccfa0 con 0x7fb38c164020 2024-04-25T15:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.891+0000 7fb382ffd640 1 -- 172.21.15.67:0/204920320 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb38401c070 con 0x7fb38c164020 2024-04-25T15:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.892+0000 7fb382ffd640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3700744e0 0x7fb3700769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.892+0000 7fb382ffd640 1 -- 172.21.15.67:0/204920320 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5609+0+0 (secure 0 0 0) 0x7fb384094c40 con 0x7fb38c164020 2024-04-25T15:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.892+0000 7fb392046640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3700744e0 0x7fb3700769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.892+0000 7fb392046640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3700744e0 0x7fb3700769f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb37c034000 tx=0x7fb37c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:46.897+0000 7fb382ffd640 1 -- 172.21.15.67:0/204920320 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb38405ee70 con 0x7fb38c164020 2024-04-25T15:27:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.122+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb38c050740 con 0x7fb38c164020 2024-04-25T15:27:47.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.750+0000 7fb382ffd640 1 -- 172.21.15.67:0/204920320 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v37) ==== 174+0+0 (secure 0 0 0) 0x7fb384063170 con 0x7fb38c164020 2024-04-25T15:27:47.751 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 10 and data pool 11 2024-04-25T15:27:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.756+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3700744e0 msgr2=0x7fb3700769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.756+0000 7fb3942aa640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3700744e0 0x7fb3700769f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb37c034000 tx=0x7fb37c02e000 comp rx=0 tx=0).stop 2024-04-25T15:27:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.756+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c164020 msgr2=0x7fb38c161af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.756+0000 7fb3942aa640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c164020 0x7fb38c161af0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb38400ba10 tx=0x7fb38400bed0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 shutdown_connections 2024-04-25T15:27:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3700744e0 0x7fb3700769f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb38c164020 0x7fb38c161af0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb38c0d1e20 0x7fb38c0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 --2- 172.21.15.67:0/204920320 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb38c0b2e60 0x7fb38c0b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 >> 172.21.15.67:0/204920320 conn(0x7fb38c0d0530 msgr2=0x7fb38c15ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:47.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 shutdown_connections 2024-04-25T15:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.758+0000 7fb3942aa640 1 -- 172.21.15.67:0/204920320 wait complete. 2024-04-25T15:27:47.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.903+0000 7f6ba2abd640 1 Processor -- start 2024-04-25T15:27:47.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.903+0000 7f6ba2abd640 1 -- start start 2024-04-25T15:27:47.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.904+0000 7f6ba2abd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:47.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.904+0000 7f6ba2abd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.904+0000 7f6ba2abd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9c0d1850 con 0x7f6b9c06c5c0 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.904+0000 7f6ba2abd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6b9c160de0 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.904+0000 7f6ba2abd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c1610c0 con 0x7f6b9c0d0e00 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba105a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba105a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35066/0 (socket says 172.21.15.67:35066) 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba105a640 1 -- 172.21.15.67:0/2557751637 learned_addr learned my addr 172.21.15.67:0/2557751637 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba0859640 1 --2- 172.21.15.67:0/2557751637 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba105a640 1 -- 172.21.15.67:0/2557751637 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 msgr2=0x7f6b9c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba105a640 1 --2- 172.21.15.67:0/2557751637 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba105a640 1 -- 172.21.15.67:0/2557751637 >> v1:172.21.15.176:6789/0 conn(0x7f6b9c06c5c0 legacy=0x7f6b9c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.905+0000 7f6ba105a640 1 -- 172.21.15.67:0/2557751637 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c1613a0 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.906+0000 7f6ba105a640 1 --2- 172.21.15.67:0/2557751637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009e50 tx=0x7f6b9402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bf2fc2a6b7b527 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.906+0000 7f6b937fe640 1 -- 172.21.15.67:0/2557751637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94036070 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.906+0000 7f6b937fe640 1 -- 172.21.15.67:0/2557751637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b94031600 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.906+0000 7f6b937fe640 1 -- 172.21.15.67:0/2557751637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b940319d0 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.906+0000 7f6ba2abd640 1 -- 172.21.15.67:0/2557751637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 msgr2=0x7f6b9c0d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:47.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.906+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/2557751637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009e50 tx=0x7f6b9402a710 comp rx=0 tx=0).stop 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.907+0000 7f6ba2abd640 1 -- 172.21.15.67:0/2557751637 shutdown_connections 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.907+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/2557751637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.907+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/2557751637 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.907+0000 7f6ba2abd640 1 -- 172.21.15.67:0/2557751637 >> 172.21.15.67:0/2557751637 conn(0x7f6b9c0d5d90 msgr2=0x7f6b9c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.907+0000 7f6ba2abd640 1 -- 172.21.15.67:0/2557751637 shutdown_connections 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.907+0000 7f6ba2abd640 1 -- 172.21.15.67:0/2557751637 wait complete. 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.908+0000 7f6ba2abd640 1 Processor -- start 2024-04-25T15:27:47.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.908+0000 7f6ba2abd640 1 -- start start 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba2abd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9c06c5c0 0x7f6b9c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba2abd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba2abd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba2abd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c164e00 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba2abd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c0b5320 con 0x7f6b9c06c5c0 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba2abd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c164f80 con 0x7f6b9c0d0e00 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35070/0 (socket says 172.21.15.67:35070) 2024-04-25T15:27:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 -- 172.21.15.67:0/3084170581 learned_addr learned my addr 172.21.15.67:0/3084170581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba0859640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9c06c5c0 0x7f6b9c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6b93fff640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 -- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 msgr2=0x7f6b9c0b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 -- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9c06c5c0 msgr2=0x7f6b9c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9c06c5c0 0x7f6b9c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 -- 172.21.15.67:0/3084170581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c1fa3a0 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.910+0000 7f6ba105a640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6b94034040 tx=0x7f6b94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.911+0000 7f6b917fa640 1 -- 172.21.15.67:0/3084170581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94041020 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.911+0000 7f6b917fa640 1 -- 172.21.15.67:0/3084170581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b940077a0 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.911+0000 7f6b917fa640 1 -- 172.21.15.67:0/3084170581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94036040 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.911+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c1fa680 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.911+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c1fac40 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.912+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c050740 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.912+0000 7f6b917fa640 1 -- 172.21.15.67:0/3084170581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6b94044020 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.913+0000 7f6b917fa640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b880744b0 0x7f6b880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:47.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.913+0000 7f6b917fa640 1 -- 172.21.15.67:0/3084170581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f6b940b5d20 con 0x7f6b9c0d4a50 2024-04-25T15:27:47.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.913+0000 7f6ba0859640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b880744b0 0x7f6b880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:47.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.914+0000 7f6ba0859640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b880744b0 0x7f6b880769c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009aa0 tx=0x7f6b98009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:47.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:47.917+0000 7f6b917fa640 1 -- 172.21.15.67:0/3084170581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b9404b030 con 0x7f6b9c0d4a50 2024-04-25T15:27:48.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.103+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b9c0d4320 con 0x7f6b9c0d4a50 2024-04-25T15:27:48.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.104+0000 7f6b917fa640 1 -- 172.21.15.67:0/3084170581 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+16645 (secure 0 0 0) 0x7f6b9407fe70 con 0x7f6b9c0d4a50 2024-04-25T15:27:48.106 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:48.106 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:47.123665+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b880744b0 msgr2=0x7f6b880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b880744b0 0x7f6b880769c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009aa0 tx=0x7f6b98009190 comp rx=0 tx=0).stop 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 msgr2=0x7f6b9c0d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6b94034040 tx=0x7f6b94004a40 comp rx=0 tx=0).stop 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 shutdown_connections 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b880744b0 0x7f6b880769c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b9c0d4a50 0x7f6b9c0d4e60 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b9c0d0e00 0x7f6b9c0b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 --2- 172.21.15.67:0/3084170581 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9c06c5c0 0x7f6b9c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 >> 172.21.15.67:0/3084170581 conn(0x7f6b9c0d5d90 msgr2=0x7f6b9c0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 shutdown_connections 2024-04-25T15:27:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.107+0000 7f6ba2abd640 1 -- 172.21.15.67:0/3084170581 wait complete. 2024-04-25T15:27:48.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:48.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.230+0000 7f1d5caf9640 1 Processor -- start 2024-04-25T15:27:48.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.230+0000 7f1d5caf9640 1 -- start start 2024-04-25T15:27:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.231+0000 7f1d5caf9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d58161da0 0x7f1d581621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.231+0000 7f1d5caf9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d5815e6c0 con 0x7f1d5815e2b0 2024-04-25T15:27:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.231+0000 7f1d5caf9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d581626f0 con 0x7f1d5806f840 2024-04-25T15:27:48.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.231+0000 7f1d5caf9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1d5806d020 con 0x7f1d58161da0 2024-04-25T15:27:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.231+0000 7f1d56d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d58161da0 0x7f1d581621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.231+0000 7f1d55d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1d5815e2b0 0x7f1d580af320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55582/0 (socket says 172.21.15.67:55582) 2024-04-25T15:27:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.231+0000 7f1d55d9b640 1 -- 172.21.15.67:0/2619920602 learned_addr learned my addr 172.21.15.67:0/2619920602 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d56d9d640 1 -- 172.21.15.67:0/2619920602 >> v1:172.21.15.176:6789/0 conn(0x7f1d5806f840 legacy=0x7f1d58070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d56d9d640 1 -- 172.21.15.67:0/2619920602 >> v1:172.21.15.67:6789/0 conn(0x7f1d5815e2b0 legacy=0x7f1d580af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d5559a640 1 -- 172.21.15.67:0/2619920602 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062185210 0 0) 0x7f1d4c001110 con 0x7f1d5815e2b0 2024-04-25T15:27:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d56d9d640 1 -- 172.21.15.67:0/2619920602 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d58071370 con 0x7f1d58161da0 2024-04-25T15:27:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d56d9d640 1 --2- 172.21.15.67:0/2619920602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d581621b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009d90 tx=0x7f1d5002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e08f226c28261ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d5559a640 1 -- 172.21.15.67:0/2619920602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50007740 con 0x7f1d58161da0 2024-04-25T15:27:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d5559a640 1 -- 172.21.15.67:0/2619920602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d500078e0 con 0x7f1d58161da0 2024-04-25T15:27:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.232+0000 7f1d5559a640 1 -- 172.21.15.67:0/2619920602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50007cb0 con 0x7f1d58161da0 2024-04-25T15:27:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.233+0000 7f1d5caf9640 1 -- 172.21.15.67:0/2619920602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 msgr2=0x7f1d581621b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.233+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/2619920602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d581621b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009d90 tx=0x7f1d5002a660 comp rx=0 tx=0).stop 2024-04-25T15:27:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.233+0000 7f1d5caf9640 1 -- 172.21.15.67:0/2619920602 shutdown_connections 2024-04-25T15:27:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.233+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/2619920602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d581621b0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.233+0000 7f1d5caf9640 1 -- 172.21.15.67:0/2619920602 >> 172.21.15.67:0/2619920602 conn(0x7f1d580ce760 msgr2=0x7f1d580ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.233+0000 7f1d5caf9640 1 -- 172.21.15.67:0/2619920602 shutdown_connections 2024-04-25T15:27:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.233+0000 7f1d5caf9640 1 -- 172.21.15.67:0/2619920602 wait complete. 2024-04-25T15:27:48.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.235+0000 7f1d5caf9640 1 Processor -- start 2024-04-25T15:27:48.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.235+0000 7f1d5caf9640 1 -- start start 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d5caf9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d5806f840 0x7f1d581fa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d5caf9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d5815e2b0 0x7f1d581faa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d5caf9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d58148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d5caf9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5806ccb0 con 0x7f1d58161da0 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d5caf9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5806c9d0 con 0x7f1d5815e2b0 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d5caf9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1d580afa30 con 0x7f1d5806f840 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d58148850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d58148850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35082/0 (socket says 172.21.15.67:35082) 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 -- 172.21.15.67:0/189847802 learned_addr learned my addr 172.21.15.67:0/189847802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d55d9b640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d5815e2b0 0x7f1d581faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d5659c640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d5806f840 0x7f1d581fa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 -- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d5806f840 msgr2=0x7f1d581fa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d5806f840 0x7f1d581fa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 -- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d5815e2b0 msgr2=0x7f1d581faa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d5815e2b0 0x7f1d581faa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 -- 172.21.15.67:0/189847802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d58149eb0 con 0x7f1d58161da0 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d56d9d640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d58148850 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009d90 tx=0x7f1d50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d46ffd640 1 -- 172.21.15.67:0/189847802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5003f070 con 0x7f1d58161da0 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.237+0000 7f1d46ffd640 1 -- 172.21.15.67:0/189847802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d50031c90 con 0x7f1d58161da0 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.238+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d581498f0 con 0x7f1d58161da0 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.238+0000 7f1d46ffd640 1 -- 172.21.15.67:0/189847802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50043630 con 0x7f1d58161da0 2024-04-25T15:27:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.238+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d580cc3d0 con 0x7f1d58161da0 2024-04-25T15:27:48.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.239+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d580d0380 con 0x7f1d58161da0 2024-04-25T15:27:48.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.239+0000 7f1d46ffd640 1 -- 172.21.15.67:0/189847802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1d5003b410 con 0x7f1d58161da0 2024-04-25T15:27:48.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.240+0000 7f1d46ffd640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d380744e0 0x7f1d380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.240+0000 7f1d46ffd640 1 -- 172.21.15.67:0/189847802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f1d500b6600 con 0x7f1d58161da0 2024-04-25T15:27:48.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.240+0000 7f1d5659c640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d380744e0 0x7f1d380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.241+0000 7f1d5659c640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d380744e0 0x7f1d380769f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1d48000f60 tx=0x7f1d48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:48.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.244+0000 7f1d46ffd640 1 -- 172.21.15.67:0/189847802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d580d0380 con 0x7f1d58161da0 2024-04-25T15:27:48.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.429+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d580b2d10 con 0x7f1d58161da0 2024-04-25T15:27:48.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.430+0000 7f1d46ffd640 1 -- 172.21.15.67:0/189847802 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) ==== 74+0+16645 (secure 0 0 0) 0x7f1d50080920 con 0x7f1d58161da0 2024-04-25T15:27:48.431 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:48.431 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:47.123665+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d380744e0 msgr2=0x7f1d380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d380744e0 0x7f1d380769f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1d48000f60 tx=0x7f1d48009040 comp rx=0 tx=0).stop 2024-04-25T15:27:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 msgr2=0x7f1d58148850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d58148850 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009d90 tx=0x7f1d50004a40 comp rx=0 tx=0).stop 2024-04-25T15:27:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 shutdown_connections 2024-04-25T15:27:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d380744e0 0x7f1d380769f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d58161da0 0x7f1d58148850 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d5815e2b0 0x7f1d581faa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 --2- 172.21.15.67:0/189847802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d5806f840 0x7f1d581fa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 >> 172.21.15.67:0/189847802 conn(0x7f1d580ce760 msgr2=0x7f1d580cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 shutdown_connections 2024-04-25T15:27:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.432+0000 7f1d5caf9640 1 -- 172.21.15.67:0/189847802 wait complete. 2024-04-25T15:27:48.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:27:48.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.558+0000 7f42c080c640 1 Processor -- start 2024-04-25T15:27:48.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.558+0000 7f42c080c640 1 -- start start 2024-04-25T15:27:48.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42c080c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42bc164640 0x7f42bc0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42c080c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0b0a70 0x7f42bc0b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42c080c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42bc0d3170 con 0x7f42bc164ad0 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42c080c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f42bc0d2f40 con 0x7f42bc164640 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42c080c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42bc164ee0 con 0x7f42bc0b0a70 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42bc164640 0x7f42bc0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42bc164640 0x7f42bc0b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35094/0 (socket says 172.21.15.67:35094) 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.559+0000 7f42b3fff640 1 -- 172.21.15.67:0/1552246105 learned_addr learned my addr 172.21.15.67:0/1552246105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.560+0000 7f42bad9d640 1 --2- 172.21.15.67:0/1552246105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0b0a70 0x7f42bc0b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.560+0000 7f42b3fff640 1 -- 172.21.15.67:0/1552246105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0b0a70 msgr2=0x7f42bc0b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.560+0000 7f42b3fff640 1 --2- 172.21.15.67:0/1552246105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0b0a70 0x7f42bc0b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.560+0000 7f42b3fff640 1 -- 172.21.15.67:0/1552246105 >> v1:172.21.15.176:6789/0 conn(0x7f42bc164ad0 legacy=0x7f42bc163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:48.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.560+0000 7f42b3fff640 1 -- 172.21.15.67:0/1552246105 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc0d3450 con 0x7f42bc164640 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.560+0000 7f42b3fff640 1 --2- 172.21.15.67:0/1552246105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc164640 0x7f42bc0b2ca0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009870 tx=0x7f42a402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f821925ceb7b50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42b9d9b640 1 -- 172.21.15.67:0/1552246105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a4037070 con 0x7f42bc164640 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42b9d9b640 1 -- 172.21.15.67:0/1552246105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42a4032440 con 0x7f42bc164640 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42b9d9b640 1 -- 172.21.15.67:0/1552246105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a4032810 con 0x7f42bc164640 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 -- 172.21.15.67:0/1552246105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc164640 msgr2=0x7f42bc0b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 --2- 172.21.15.67:0/1552246105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc164640 0x7f42bc0b2ca0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009870 tx=0x7f42a402a210 comp rx=0 tx=0).stop 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 -- 172.21.15.67:0/1552246105 shutdown_connections 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 --2- 172.21.15.67:0/1552246105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0b0a70 0x7f42bc0b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 --2- 172.21.15.67:0/1552246105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc164640 0x7f42bc0b2ca0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 -- 172.21.15.67:0/1552246105 >> 172.21.15.67:0/1552246105 conn(0x7f42bc0d3aa0 msgr2=0x7f42bc0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 -- 172.21.15.67:0/1552246105 shutdown_connections 2024-04-25T15:27:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.561+0000 7f42c080c640 1 -- 172.21.15.67:0/1552246105 wait complete. 2024-04-25T15:27:48.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.563+0000 7f42c080c640 1 Processor -- start 2024-04-25T15:27:48.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.563+0000 7f42c080c640 1 -- start start 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0b0a70 0x7f42bc158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc164ad0 0x7f42bc0c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc0c96f0 0x7f42bc0c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc06d9b0 con 0x7f42bc0c96f0 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0b31e0 con 0x7f42bc0b0a70 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42bc0b9430 con 0x7f42bc164ad0 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc0c96f0 0x7f42bc0c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc0c96f0 0x7f42bc0c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35104/0 (socket says 172.21.15.67:35104) 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 -- 172.21.15.67:0/1537675319 learned_addr learned my addr 172.21.15.67:0/1537675319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42b3fff640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc164ad0 0x7f42bc0c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42ba59c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0b0a70 0x7f42bc158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 -- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc164ad0 msgr2=0x7f42bc0c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc164ad0 0x7f42bc0c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 -- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0b0a70 msgr2=0x7f42bc158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:48.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0b0a70 0x7f42bc158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 -- 172.21.15.67:0/1537675319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc0ca040 con 0x7f42bc0c96f0 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42b3fff640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc164ad0 0x7f42bc0c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42ba59c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0b0a70 0x7f42bc158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42bad9d640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc0c96f0 0x7f42bc0c9b00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f42b400cc30 tx=0x7f42b400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42b2ffd640 1 -- 172.21.15.67:0/1537675319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b400ce40 con 0x7f42bc0c96f0 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42b2ffd640 1 -- 172.21.15.67:0/1537675319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b4003990 con 0x7f42bc0c96f0 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc162b40 con 0x7f42bc0c96f0 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42b2ffd640 1 -- 172.21.15.67:0/1537675319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4003d40 con 0x7f42bc0c96f0 2024-04-25T15:27:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.565+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc163350 con 0x7f42bc0c96f0 2024-04-25T15:27:48.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.566+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42bc162de0 con 0x7f42bc0c96f0 2024-04-25T15:27:48.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.567+0000 7f42b2ffd640 1 -- 172.21.15.67:0/1537675319 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f42b4003f00 con 0x7f42bc0c96f0 2024-04-25T15:27:48.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.568+0000 7f42b2ffd640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f429c0744b0 0x7f429c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:48.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.568+0000 7f42b2ffd640 1 -- 172.21.15.67:0/1537675319 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5709+0+0 (secure 0 0 0) 0x7f42b4095910 con 0x7f42bc0c96f0 2024-04-25T15:27:48.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.568+0000 7f42ba59c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f429c0744b0 0x7f429c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:48.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.569+0000 7f42ba59c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f429c0744b0 0x7f429c0769c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0746e0 tx=0x7f42ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:48.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.571+0000 7f42b2ffd640 1 -- 172.21.15.67:0/1537675319 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42bc0ca040 con 0x7f42bc0c96f0 2024-04-25T15:27:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:48.796+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f42bc0cfa70 con 0x7f42bc0c96f0 2024-04-25T15:27:49.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-04-25T15:27:49.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-25T15:27:49.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-04-25T15:27:49.727 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-25T15:27:49.727+0000 7f9fe118f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:27:49.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.777+0000 7f42b2ffd640 1 -- 172.21.15.67:0/1537675319 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) ==== 118+0+0 (secure 0 0 0) 0x7f42b4014030 con 0x7f42bc0c96f0 2024-04-25T15:27:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f429c0744b0 msgr2=0x7f429c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f429c0744b0 0x7f429c0769c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0746e0 tx=0x7f42ac008040 comp rx=0 tx=0).stop 2024-04-25T15:27:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc0c96f0 msgr2=0x7f42bc0c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc0c96f0 0x7f42bc0c9b00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f42b400cc30 tx=0x7f42b400a5c0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 shutdown_connections 2024-04-25T15:27:49.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f429c0744b0 0x7f429c0769c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42bc0c96f0 0x7f42bc0c9b00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc164ad0 0x7f42bc0c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 --2- 172.21.15.67:0/1537675319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0b0a70 0x7f42bc158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 >> 172.21.15.67:0/1537675319 conn(0x7f42bc0d3aa0 msgr2=0x7f42bc0687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 shutdown_connections 2024-04-25T15:27:49.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.780+0000 7f42c080c640 1 -- 172.21.15.67:0/1537675319 wait complete. 2024-04-25T15:27:49.795 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:27:49.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:27:49.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.916+0000 7fbce1f57640 1 Processor -- start 2024-04-25T15:27:49.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.916+0000 7fbce1f57640 1 -- start start 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbce1f57640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcdc0d0e00 0x7fbcdc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbce1f57640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbce1f57640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcdc0d1850 con 0x7fbcdc06c5c0 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbce1f57640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbcdc160de0 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbce1f57640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbcdc1610c0 con 0x7fbcdc0d4a50 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcdc0d0e00 0x7fbcdc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdaffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcdc06c5c0 0x7fbcdc06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55596/0 (socket says 172.21.15.67:55596) 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdaffd640 1 -- 172.21.15.67:0/3384570748 learned_addr learned my addr 172.21.15.67:0/3384570748 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdbfff640 1 --2- 172.21.15.67:0/3384570748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdb7fe640 1 -- 172.21.15.67:0/3384570748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 msgr2=0x7fbcdc0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdb7fe640 1 --2- 172.21.15.67:0/3384570748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdb7fe640 1 -- 172.21.15.67:0/3384570748 >> v1:172.21.15.67:6789/0 conn(0x7fbcdc06c5c0 legacy=0x7fbcdc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdb7fe640 1 -- 172.21.15.67:0/3384570748 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc1613a0 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcda7fc640 1 -- 172.21.15.67:0/3384570748 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846172937 0 0) 0x7fbcdc0d1850 con 0x7fbcdc06c5c0 2024-04-25T15:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.918+0000 7fbcdbfff640 1 --2- 172.21.15.67:0/3384570748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.919+0000 7fbcdb7fe640 1 --2- 172.21.15.67:0/3384570748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0d1210 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097a0 tx=0x7fbccc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2050df6cb9ed651 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.919+0000 7fbcda7fc640 1 -- 172.21.15.67:0/3384570748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc037070 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.919+0000 7fbcda7fc640 1 -- 172.21.15.67:0/3384570748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbccc032440 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.919+0000 7fbcda7fc640 1 -- 172.21.15.67:0/3384570748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc032810 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 -- 172.21.15.67:0/3384570748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 msgr2=0x7fbcdc0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 --2- 172.21.15.67:0/3384570748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0d1210 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097a0 tx=0x7fbccc02a180 comp rx=0 tx=0).stop 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 -- 172.21.15.67:0/3384570748 shutdown_connections 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 --2- 172.21.15.67:0/3384570748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 --2- 172.21.15.67:0/3384570748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0d1210 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 -- 172.21.15.67:0/3384570748 >> 172.21.15.67:0/3384570748 conn(0x7fbcdc0d5d90 msgr2=0x7fbcdc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 -- 172.21.15.67:0/3384570748 shutdown_connections 2024-04-25T15:27:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.920+0000 7fbce1f57640 1 -- 172.21.15.67:0/3384570748 wait complete. 2024-04-25T15:27:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.922+0000 7fbce1f57640 1 Processor -- start 2024-04-25T15:27:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.922+0000 7fbce1f57640 1 -- start start 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbce1f57640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcdc06c5c0 0x7fbcdc0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbce1f57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbce1f57640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbce1f57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc164e00 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbce1f57640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc0b5320 con 0x7fbcdc06c5c0 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbce1f57640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbcdc164f80 con 0x7fbcdc0d4a50 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbcdaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbcdaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0ca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35126/0 (socket says 172.21.15.67:35126) 2024-04-25T15:27:49.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbcdaffd640 1 -- 172.21.15.67:0/2722168847 learned_addr learned my addr 172.21.15.67:0/2722168847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbcdbfff640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.923+0000 7fbcdb7fe640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcdc06c5c0 0x7fbcdc0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcdaffd640 1 -- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 msgr2=0x7fbcdc0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcdaffd640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcdaffd640 1 -- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcdc06c5c0 msgr2=0x7fbcdc0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcdaffd640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcdc06c5c0 0x7fbcdc0cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcdaffd640 1 -- 172.21.15.67:0/2722168847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc0cb0c0 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcdb7fe640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcdc06c5c0 0x7fbcdc0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcdaffd640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0ca7c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c460 tx=0x7fbcd000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcc3fff640 1 -- 172.21.15.67:0/2722168847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0018070 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcc3fff640 1 -- 172.21.15.67:0/2722168847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd0007500 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcdc0cb3a0 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbcc3fff640 1 -- 172.21.15.67:0/2722168847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0004a20 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.924+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcdc0cb960 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.925+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcdc050740 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.926+0000 7fbcc3fff640 1 -- 172.21.15.67:0/2722168847 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbcd001b070 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.926+0000 7fbcc3fff640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbcc80744b0 0x7fbcc80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:49.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.926+0000 7fbcc3fff640 1 -- 172.21.15.67:0/2722168847 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fbcd0094450 con 0x7fbcdc0d0e00 2024-04-25T15:27:49.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.926+0000 7fbcdb7fe640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbcc80744b0 0x7fbcc80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:49.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.927+0000 7fbcdb7fe640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbcc80744b0 0x7fbcc80769c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbccc034000 tx=0x7fbccc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:49.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:49.931+0000 7fbcc3fff640 1 -- 172.21.15.67:0/2722168847 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbcd005e590 con 0x7fbcdc0d0e00 2024-04-25T15:27:50.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.153+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fbcdc0cae50 con 0x7fbcdc0d0e00 2024-04-25T15:27:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.784+0000 7fbcc3fff640 1 -- 172.21.15.67:0/2722168847 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v41) ==== 122+0+0 (secure 0 0 0) 0x7fbcd0062890 con 0x7fbcdc0d0e00 2024-04-25T15:27:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbcc80744b0 msgr2=0x7fbcc80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbcc80744b0 0x7fbcc80769c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbccc034000 tx=0x7fbccc02e000 comp rx=0 tx=0).stop 2024-04-25T15:27:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 msgr2=0x7fbcdc0ca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0ca7c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c460 tx=0x7fbcd000c920 comp rx=0 tx=0).stop 2024-04-25T15:27:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 shutdown_connections 2024-04-25T15:27:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbcc80744b0 0x7fbcc80769c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcdc0d4a50 0x7fbcdc0cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcdc0d0e00 0x7fbcdc0ca7c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 --2- 172.21.15.67:0/2722168847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcdc06c5c0 0x7fbcdc0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 >> 172.21.15.67:0/2722168847 conn(0x7fbcdc0d5d90 msgr2=0x7fbcdc1641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 shutdown_connections 2024-04-25T15:27:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.786+0000 7fbce1f57640 1 -- 172.21.15.67:0/2722168847 wait complete. 2024-04-25T15:27:50.799 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:27:50.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:50.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.912+0000 7f68557c0640 1 Processor -- start 2024-04-25T15:27:50.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.912+0000 7f68557c0640 1 -- start start 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f68557c0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68500d0e00 0x7f68500d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f68557c0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f685006c5c0 0x7f685006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f68557c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68500b7810 0x7f68500b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f68557c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f685006cf10 con 0x7f68500b7810 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f68557c0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f68500d1850 con 0x7f685006c5c0 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f68557c0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6850160de0 con 0x7f68500d0e00 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68500b7810 0x7f68500b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68500b7810 0x7f68500b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35140/0 (socket says 172.21.15.67:35140) 2024-04-25T15:27:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 -- 172.21.15.67:0/1557876747 learned_addr learned my addr 172.21.15.67:0/1557876747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684effd640 1 --2- 172.21.15.67:0/1557876747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68500d0e00 0x7f68500d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 -- 172.21.15.67:0/1557876747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68500d0e00 msgr2=0x7f68500d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684e7fc640 1 --2- 172.21.15.67:0/1557876747 >> v2:172.21.15.176:3300/0 conn(0x7f685006c5c0 0x7f685006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 --2- 172.21.15.67:0/1557876747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68500d0e00 0x7f68500d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 -- 172.21.15.67:0/1557876747 >> v2:172.21.15.176:3300/0 conn(0x7f685006c5c0 msgr2=0x7f685006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 --2- 172.21.15.67:0/1557876747 >> v2:172.21.15.176:3300/0 conn(0x7f685006c5c0 0x7f685006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.914+0000 7f684f7fe640 1 -- 172.21.15.67:0/1557876747 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68501610c0 con 0x7f68500b7810 2024-04-25T15:27:50.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.915+0000 7f684f7fe640 1 --2- 172.21.15.67:0/1557876747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500b7c20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f685006f0a0 tx=0x7f684002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26efafa5682b2194 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:50.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.915+0000 7f684dffb640 1 -- 172.21.15.67:0/1557876747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840036070 con 0x7f68500b7810 2024-04-25T15:27:50.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.915+0000 7f684dffb640 1 -- 172.21.15.67:0/1557876747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6840031600 con 0x7f68500b7810 2024-04-25T15:27:50.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.915+0000 7f684dffb640 1 -- 172.21.15.67:0/1557876747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68400319d0 con 0x7f68500b7810 2024-04-25T15:27:50.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 -- 172.21.15.67:0/1557876747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 msgr2=0x7f68500b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 --2- 172.21.15.67:0/1557876747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500b7c20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f685006f0a0 tx=0x7f684002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 -- 172.21.15.67:0/1557876747 shutdown_connections 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 --2- 172.21.15.67:0/1557876747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500b7c20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 --2- 172.21.15.67:0/1557876747 >> v2:172.21.15.176:3300/0 conn(0x7f685006c5c0 0x7f685006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 --2- 172.21.15.67:0/1557876747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68500d0e00 0x7f68500d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 -- 172.21.15.67:0/1557876747 >> 172.21.15.67:0/1557876747 conn(0x7f68500d5d90 msgr2=0x7f68500d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 -- 172.21.15.67:0/1557876747 shutdown_connections 2024-04-25T15:27:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.916+0000 7f68557c0640 1 -- 172.21.15.67:0/1557876747 wait complete. 2024-04-25T15:27:50.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.918+0000 7f68557c0640 1 Processor -- start 2024-04-25T15:27:50.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.918+0000 7f68557c0640 1 -- start start 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f68557c0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f685006c5c0 0x7f68500cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f68557c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f68557c0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68500d0e00 0x7f68501fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f68557c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f68500b5320 con 0x7f68500b7810 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f68557c0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6850160ad0 con 0x7f68500d0e00 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f68557c0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6850164e00 con 0x7f685006c5c0 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f684e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f684e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35142/0 (socket says 172.21.15.67:35142) 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f684e7fc640 1 -- 172.21.15.67:0/3439151991 learned_addr learned my addr 172.21.15.67:0/3439151991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f684effd640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f685006c5c0 0x7f68500cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.919+0000 7f684f7fe640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68500d0e00 0x7f68501fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f684e7fc640 1 -- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f685006c5c0 msgr2=0x7f68500cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f684e7fc640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f685006c5c0 0x7f68500cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f684e7fc640 1 -- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68500d0e00 msgr2=0x7f68501fa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f684e7fc640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68500d0e00 0x7f68501fa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f684e7fc640 1 -- 172.21.15.67:0/3439151991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6850155840 con 0x7f68500b7810 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f684e7fc640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500ca670 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f683800c570 tx=0x7f683800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f68477fe640 1 -- 172.21.15.67:0/3439151991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838015070 con 0x7f68500b7810 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f68477fe640 1 -- 172.21.15.67:0/3439151991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6838011800 con 0x7f68500b7810 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68500cec00 con 0x7f68500b7810 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f68477fe640 1 -- 172.21.15.67:0/3439151991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838011bf0 con 0x7f68500b7810 2024-04-25T15:27:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.920+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68500ceea0 con 0x7f68500b7810 2024-04-25T15:27:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.921+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6850050740 con 0x7f68500b7810 2024-04-25T15:27:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.922+0000 7f68477fe640 1 -- 172.21.15.67:0/3439151991 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6838018070 con 0x7f68500b7810 2024-04-25T15:27:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.922+0000 7f68477fe640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f68300744b0 0x7f68300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.922+0000 7f68477fe640 1 -- 172.21.15.67:0/3439151991 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f6838094f30 con 0x7f68500b7810 2024-04-25T15:27:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.922+0000 7f684effd640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f68300744b0 0x7f68300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:50.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.923+0000 7f684effd640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f68300744b0 0x7f68300769c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f684800aa20 tx=0x7f6848008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:50.927+0000 7f68477fe640 1 -- 172.21.15.67:0/3439151991 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6850155840 con 0x7f68500b7810 2024-04-25T15:27:51.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.153+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68501ea610 con 0x7f68500b7810 2024-04-25T15:27:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.153+0000 7f68477fe640 1 -- 172.21.15.67:0/3439151991 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5189 (secure 0 0 0) 0x7f683805f100 con 0x7f68500b7810 2024-04-25T15:27:51.154 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:51.154 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2076054232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2076054232},{"type":"v1","addr":"172.21.15.176:6836","nonce":2076054232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":5537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3930910548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3930910548},{"type":"v1","addr":"172.21.15.176:6837","nonce":3930910548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2322157634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322157634},{"type":"v1","addr":"172.21.15.67:6834","nonce":2322157634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:47.123622+0000","modified":"2024-04-25T15:27:50.154657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6273":{"gid":6273,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3903876028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3903876028},{"type":"v1","addr":"172.21.15.67:6835","nonce":3903876028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6273,"qdb_cluster":[6273]},"id":5}]} 2024-04-25T15:27:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f68300744b0 msgr2=0x7f68300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f68300744b0 0x7f68300769c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f684800aa20 tx=0x7f6848008040 comp rx=0 tx=0).stop 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 msgr2=0x7f68500ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500ca670 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f683800c570 tx=0x7f683800ca30 comp rx=0 tx=0).stop 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 shutdown_connections 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f68300744b0 0x7f68300769c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68500d0e00 0x7f68501fa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68500b7810 0x7f68500ca670 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 --2- 172.21.15.67:0/3439151991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f685006c5c0 0x7f68500cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 >> 172.21.15.67:0/3439151991 conn(0x7f68500d5d90 msgr2=0x7f68500bb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 shutdown_connections 2024-04-25T15:27:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.156+0000 7f68557c0640 1 -- 172.21.15.67:0/3439151991 wait complete. 2024-04-25T15:27:51.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:27:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.279+0000 7ffa5bb38640 1 Processor -- start 2024-04-25T15:27:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.279+0000 7ffa5bb38640 1 -- start start 2024-04-25T15:27:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa5bb38640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ffa540d0e00 0x7ffa540d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa5bb38640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa5406c5c0 0x7ffa5406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa5bb38640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540b7810 0x7ffa540b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa5bb38640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ffa5406cf10 con 0x7ffa5406c5c0 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa5bb38640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ffa540d1850 con 0x7ffa540d0e00 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa5bb38640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa54160de0 con 0x7ffa540b7810 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa590d3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa5406c5c0 0x7ffa5406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa590d3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa5406c5c0 0x7ffa5406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35148/0 (socket says 172.21.15.67:35148) 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa590d3640 1 -- 172.21.15.67:0/3761447172 learned_addr learned my addr 172.21.15.67:0/3761447172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa5a0d5640 1 --2- 172.21.15.67:0/3761447172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540b7810 0x7ffa540b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.280+0000 7ffa598d4640 1 --2- 172.21.15.67:0/3761447172 >> v2:172.21.15.176:3300/0 conn(0x7ffa540d0e00 0x7ffa540d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa590d3640 1 -- 172.21.15.67:0/3761447172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540b7810 msgr2=0x7ffa540b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa590d3640 1 --2- 172.21.15.67:0/3761447172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540b7810 0x7ffa540b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa590d3640 1 -- 172.21.15.67:0/3761447172 >> v2:172.21.15.176:3300/0 conn(0x7ffa540d0e00 msgr2=0x7ffa540d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa590d3640 1 --2- 172.21.15.67:0/3761447172 >> v2:172.21.15.176:3300/0 conn(0x7ffa540d0e00 0x7ffa540d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa590d3640 1 -- 172.21.15.67:0/3761447172 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa541610c0 con 0x7ffa5406c5c0 2024-04-25T15:27:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa598d4640 1 --2- 172.21.15.67:0/3761447172 >> v2:172.21.15.176:3300/0 conn(0x7ffa540d0e00 0x7ffa540d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa5a0d5640 1 --2- 172.21.15.67:0/3761447172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540b7810 0x7ffa540b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:51.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa590d3640 1 --2- 172.21.15.67:0/3761447172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa5406c9d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ffa540d3e60 tx=0x7ffa4802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9174f466e7574b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.281+0000 7ffa588d2640 1 -- 172.21.15.67:0/3761447172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48037070 con 0x7ffa5406c5c0 2024-04-25T15:27:51.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa588d2640 1 -- 172.21.15.67:0/3761447172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa48032440 con 0x7ffa5406c5c0 2024-04-25T15:27:51.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa588d2640 1 -- 172.21.15.67:0/3761447172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48032810 con 0x7ffa5406c5c0 2024-04-25T15:27:51.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 -- 172.21.15.67:0/3761447172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 msgr2=0x7ffa5406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/3761447172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa5406c9d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7ffa540d3e60 tx=0x7ffa4802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 -- 172.21.15.67:0/3761447172 shutdown_connections 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/3761447172 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540b7810 0x7ffa540b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/3761447172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa5406c9d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/3761447172 >> v2:172.21.15.176:3300/0 conn(0x7ffa540d0e00 0x7ffa540d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 -- 172.21.15.67:0/3761447172 >> 172.21.15.67:0/3761447172 conn(0x7ffa540d5d90 msgr2=0x7ffa540d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 -- 172.21.15.67:0/3761447172 shutdown_connections 2024-04-25T15:27:51.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.282+0000 7ffa5bb38640 1 -- 172.21.15.67:0/3761447172 wait complete. 2024-04-25T15:27:51.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.284+0000 7ffa5bb38640 1 Processor -- start 2024-04-25T15:27:51.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.284+0000 7ffa5bb38640 1 -- start start 2024-04-25T15:27:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.285+0000 7ffa5bb38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa540cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.285+0000 7ffa5bb38640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa540b7810 0x7ffa541f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.285+0000 7ffa5bb38640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540d0e00 0x7ffa541fa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.285+0000 7ffa5bb38640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffa540b5320 con 0x7ffa5406c5c0 2024-04-25T15:27:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.285+0000 7ffa5bb38640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa54160ad0 con 0x7ffa540b7810 2024-04-25T15:27:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.285+0000 7ffa5bb38640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa54164e00 con 0x7ffa540d0e00 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa540cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa540cc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35156/0 (socket says 172.21.15.67:35156) 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 -- 172.21.15.67:0/1932638223 learned_addr learned my addr 172.21.15.67:0/1932638223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa590d3640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa540b7810 0x7ffa541f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa5a0d5640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540d0e00 0x7ffa541fa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 -- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540d0e00 msgr2=0x7ffa541fa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540d0e00 0x7ffa541fa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 -- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa540b7810 msgr2=0x7ffa541f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa540b7810 0x7ffa541f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 -- 172.21.15.67:0/1932638223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa541f7c40 con 0x7ffa5406c5c0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa598d4640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa540cc240 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00d7d0 tx=0x7ffa4c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa527fc640 1 -- 172.21.15.67:0/1932638223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c004030 con 0x7ffa5406c5c0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa527fc640 1 -- 172.21.15.67:0/1932638223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c0041d0 con 0x7ffa5406c5c0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa527fc640 1 -- 172.21.15.67:0/1932638223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0096a0 con 0x7ffa5406c5c0 2024-04-25T15:27:51.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa541584e0 con 0x7ffa5406c5c0 2024-04-25T15:27:51.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.286+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa54158aa0 con 0x7ffa5406c5c0 2024-04-25T15:27:51.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.287+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa54050740 con 0x7ffa5406c5c0 2024-04-25T15:27:51.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.288+0000 7ffa527fc640 1 -- 172.21.15.67:0/1932638223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffa4c009840 con 0x7ffa5406c5c0 2024-04-25T15:27:51.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.288+0000 7ffa527fc640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.289+0000 7ffa527fc640 1 -- 172.21.15.67:0/1932638223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7ffa4c095510 con 0x7ffa5406c5c0 2024-04-25T15:27:51.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.289+0000 7ffa590d3640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.289+0000 7ffa590d3640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffa48034000 tx=0x7ffa4802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.293+0000 7ffa527fc640 1 -- 172.21.15.67:0/1932638223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa541f7c40 con 0x7ffa5406c5c0 2024-04-25T15:27:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.480+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa540d4320 con 0x7ffa5406c5c0 2024-04-25T15:27:51.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.482+0000 7ffa527fc640 1 -- 172.21.15.67:0/1932638223 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16632 (secure 0 0 0) 0x7ffa4c05fd30 con 0x7ffa5406c5c0 2024-04-25T15:27:51.483 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:51.483 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:48.764450+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 msgr2=0x7ffa400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ffa48034000 tx=0x7ffa4802e000 comp rx=0 tx=0).stop 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 msgr2=0x7ffa540cc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa540cc240 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00d7d0 tx=0x7ffa4c00dc90 comp rx=0 tx=0).stop 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 shutdown_connections 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ffa400744b0 0x7ffa400769c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa540d0e00 0x7ffa541fa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa540b7810 0x7ffa541f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 --2- 172.21.15.67:0/1932638223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5406c5c0 0x7ffa540cc240 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 >> 172.21.15.67:0/1932638223 conn(0x7ffa540d5d90 msgr2=0x7ffa540684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 shutdown_connections 2024-04-25T15:27:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.484+0000 7ffa5bb38640 1 -- 172.21.15.67:0/1932638223 wait complete. 2024-04-25T15:27:51.496 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:27:51.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:27:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.608+0000 7f875754d640 1 Processor -- start 2024-04-25T15:27:51.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.608+0000 7f875754d640 1 -- start start 2024-04-25T15:27:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.609+0000 7f875754d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 0x7f87500d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.609+0000 7f875754d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87500d1850 con 0x7f875006ccd0 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.609+0000 7f875754d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8750160de0 con 0x7f87500d0e00 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.609+0000 7f875754d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f87500d1620 con 0x7f87500d1210 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.609+0000 7f8754ae8640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f875006ccd0 0x7f87500d4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55606/0 (socket says 172.21.15.67:55606) 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.609+0000 7f8754ae8640 1 -- 172.21.15.67:0/874580271 learned_addr learned my addr 172.21.15.67:0/874580271 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8755aea640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 0x7f87500d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822703473 0 0) 0x7f87500d1850 con 0x7f875006ccd0 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8738003520 con 0x7f875006ccd0 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2765756151 0 0) 0x7f8738003520 con 0x7f875006ccd0 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 msgr2=0x7f87500d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 0x7f87500d5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 >> v1:172.21.15.176:6789/0 conn(0x7f87500d0e00 legacy=0x7f875006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87501610c0 con 0x7f875006ccd0 2024-04-25T15:27:51.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8744001670 con 0x7f875006ccd0 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 >> v1:172.21.15.67:6789/0 conn(0x7f875006ccd0 legacy=0x7f87500d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87380039e0 0x7f8738003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8738004800 0x7f8738004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 0x7f8738005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f873800bac0 con 0x7f8738004800 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f873800bc70 con 0x7f87380039e0 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f873800be70 con 0x7f87500d1210 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8754ae8640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8738004800 0x7f8738004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f87552e9640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87380039e0 0x7f8738003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8755aea640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 0x7f8738005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8754ae8640 1 -- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 msgr2=0x7f8738005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8754ae8640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87500d1210 0x7f8738005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8754ae8640 1 -- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87380039e0 msgr2=0x7f8738003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8754ae8640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87380039e0 0x7f8738003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.610+0000 7f8754ae8640 1 -- 172.21.15.67:0/874580271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8738003520 con 0x7f8738004800 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.611+0000 7f8754ae8640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8738004800 0x7f8738004c10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f874400c210 tx=0x7f8738005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2324ff8c177729d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.611+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744007cf0 con 0x7f8738004800 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.611+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8744005af0 con 0x7f8738004800 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.611+0000 7f874ffff640 1 -- 172.21.15.67:0/874580271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744007070 con 0x7f8738004800 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.612+0000 7f875754d640 1 -- 172.21.15.67:0/874580271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8738004800 msgr2=0x7f8738004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.612+0000 7f875754d640 1 --2- 172.21.15.67:0/874580271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8738004800 0x7f8738004c10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f874400c210 tx=0x7f8738005830 comp rx=0 tx=0).stop 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.612+0000 7f8755aea640 1 -- 172.21.15.67:0/874580271 reap_dead start 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.612+0000 7f875754d640 1 -- 172.21.15.67:0/874580271 shutdown_connections 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.612+0000 7f875754d640 1 -- 172.21.15.67:0/874580271 >> 172.21.15.67:0/874580271 conn(0x7f87500d5d90 msgr2=0x7f87500d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.612+0000 7f875754d640 1 -- 172.21.15.67:0/874580271 shutdown_connections 2024-04-25T15:27:51.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.612+0000 7f875754d640 1 -- 172.21.15.67:0/874580271 wait complete. 2024-04-25T15:27:51.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.614+0000 7f875754d640 1 Processor -- start 2024-04-25T15:27:51.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.614+0000 7f875754d640 1 -- start start 2024-04-25T15:27:51.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f875754d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8738004800 0x7f8750154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f875754d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87500b56f0 0x7f87500b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f875754d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87500b74f0 0x7f87500b7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f875754d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f87500b5320 con 0x7f87500b56f0 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f875754d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8750160ad0 con 0x7f87500b74f0 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f875754d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8750164e00 con 0x7f8738004800 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f8754ae8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87500b56f0 0x7f87500b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f8754ae8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87500b56f0 0x7f87500b5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35178/0 (socket says 172.21.15.67:35178) 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.615+0000 7f8754ae8640 1 -- 172.21.15.67:0/3599929639 learned_addr learned my addr 172.21.15.67:0/3599929639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f87552e9640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8738004800 0x7f8750154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f8755aea640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87500b74f0 0x7f87500b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f8754ae8640 1 -- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8738004800 msgr2=0x7f8750154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f8754ae8640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8738004800 0x7f8750154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f8754ae8640 1 -- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87500b74f0 msgr2=0x7f87500b7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f8754ae8640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87500b74f0 0x7f87500b7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f8754ae8640 1 -- 172.21.15.67:0/3599929639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87500b7e40 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f87552e9640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8738004800 0x7f8750154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f8754ae8640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87500b56f0 0x7f87500b5b00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f874400c330 tx=0x7f8750161290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f874dffb640 1 -- 172.21.15.67:0/3599929639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744001630 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f874dffb640 1 -- 172.21.15.67:0/3599929639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8744042070 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87500cb1e0 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f874dffb640 1 -- 172.21.15.67:0/3599929639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87440072b0 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.616+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87500b6340 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.618+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8750050740 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.618+0000 7f874dffb640 1 -- 172.21.15.67:0/3599929639 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8744007450 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.618+0000 7f874dffb640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f873c0744e0 0x7f873c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.618+0000 7f874dffb640 1 -- 172.21.15.67:0/3599929639 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f87440b5350 con 0x7f87500b56f0 2024-04-25T15:27:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.619+0000 7f87552e9640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f873c0744e0 0x7f873c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.619+0000 7f87552e9640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f873c0744e0 0x7f873c0769f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f87400055c0 tx=0x7f8740009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.623+0000 7f874dffb640 1 -- 172.21.15.67:0/3599929639 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87500b7e40 con 0x7f87500b56f0 2024-04-25T15:27:51.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.807+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f87500d4320 con 0x7f873c0744e0 2024-04-25T15:27:51.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.813+0000 7f874dffb640 1 -- 172.21.15.67:0/3599929639 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246737 (secure 0 0 0) 0x7f87500d4320 con 0x7f873c0744e0 2024-04-25T15:27:51.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:51.818 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:27:51.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f873c0744e0 msgr2=0x7f873c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f873c0744e0 0x7f873c0769f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f87400055c0 tx=0x7f8740009040 comp rx=0 tx=0).stop 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87500b56f0 msgr2=0x7f87500b5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87500b56f0 0x7f87500b5b00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f874400c330 tx=0x7f8750161290 comp rx=0 tx=0).stop 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 shutdown_connections 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f873c0744e0 0x7f873c0769f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87500b74f0 0x7f87500b7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87500b56f0 0x7f87500b5b00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 --2- 172.21.15.67:0/3599929639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8738004800 0x7f8750154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.816+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 >> 172.21.15.67:0/3599929639 conn(0x7f87500d5d90 msgr2=0x7f8750161970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.817+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 shutdown_connections 2024-04-25T15:27:51.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.817+0000 7f875754d640 1 -- 172.21.15.67:0/3599929639 wait complete. 2024-04-25T15:27:51.829 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-04-25T15:27:50.829478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223456,"kb_used_data":7252,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751264,"statfs":{"total":773094113280,"available":772865294336,"internally_reserved":0,"allocated":7426048,"data_stored":3764331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.099960"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742158+0000","last_change":"2024-04-25T15:27:46.742158+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742158+0000","last_undegraded":"2024-04-25T15:27:46.742158+0000","last_fullsized":"2024-04-25T15:27:46.742158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.746577+0000","last_change":"2024-04-25T15:27:46.746577+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.746577+0000","last_undegraded":"2024-04-25T15:27:46.746577+0000","last_fullsized":"2024-04-25T15:27:46.746577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741554+0000","last_change":"2024-04-25T15:27:46.741554+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741554+0000","last_undegraded":"2024-04-25T15:27:46.741554+0000","last_fullsized":"2024-04-25T15:27:46.741554+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739391+0000","last_change":"2024-04-25T15:27:46.739391+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739391+0000","last_undegraded":"2024-04-25T15:27:46.739391+0000","last_fullsized":"2024-04-25T15:27:46.739391+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741588+0000","last_change":"2024-04-25T15:27:46.741588+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741588+0000","last_undegraded":"2024-04-25T15:27:46.741588+0000","last_fullsized":"2024-04-25T15:27:46.741588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744353+0000","last_change":"2024-04-25T15:27:46.744353+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744353+0000","last_undegraded":"2024-04-25T15:27:46.744353+0000","last_fullsized":"2024-04-25T15:27:46.744353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741489+0000","last_change":"2024-04-25T15:27:46.741489+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741489+0000","last_undegraded":"2024-04-25T15:27:46.741489+0000","last_fullsized":"2024-04-25T15:27:46.741489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741504+0000","last_change":"2024-04-25T15:27:46.741504+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741504+0000","last_undegraded":"2024-04-25T15:27:46.741504+0000","last_fullsized":"2024-04-25T15:27:46.741504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743205+0000","last_change":"2024-04-25T15:27:46.743205+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743205+0000","last_undegraded":"2024-04-25T15:27:46.743205+0000","last_fullsized":"2024-04-25T15:27:46.743205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737954+0000","last_change":"2024-04-25T15:27:46.737954+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737954+0000","last_undegraded":"2024-04-25T15:27:46.737954+0000","last_fullsized":"2024-04-25T15:27:46.737954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743381+0000","last_change":"2024-04-25T15:27:46.743381+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743381+0000","last_undegraded":"2024-04-25T15:27:46.743381+0000","last_fullsized":"2024-04-25T15:27:46.743381+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738618+0000","last_change":"2024-04-25T15:27:46.738618+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738618+0000","last_undegraded":"2024-04-25T15:27:46.738618+0000","last_fullsized":"2024-04-25T15:27:46.738618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734472+0000","last_change":"2024-04-25T15:27:46.734472+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734472+0000","last_undegraded":"2024-04-25T15:27:46.734472+0000","last_fullsized":"2024-04-25T15:27:46.734472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740602+0000","last_change":"2024-04-25T15:27:46.740602+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740602+0000","last_undegraded":"2024-04-25T15:27:46.740602+0000","last_fullsized":"2024-04-25T15:27:46.740602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735281+0000","last_change":"2024-04-25T15:27:46.735281+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735281+0000","last_undegraded":"2024-04-25T15:27:46.735281+0000","last_fullsized":"2024-04-25T15:27:46.735281+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735174+0000","last_change":"2024-04-25T15:27:46.735174+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735174+0000","last_undegraded":"2024-04-25T15:27:46.735174+0000","last_fullsized":"2024-04-25T15:27:46.735174+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742694+0000","last_change":"2024-04-25T15:27:46.742694+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742694+0000","last_undegraded":"2024-04-25T15:27:46.742694+0000","last_fullsized":"2024-04-25T15:27:46.742694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738105+0000","last_change":"2024-04-25T15:27:46.738105+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738105+0000","last_undegraded":"2024-04-25T15:27:46.738105+0000","last_fullsized":"2024-04-25T15:27:46.738105+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742298+0000","last_change":"2024-04-25T15:27:46.742298+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742298+0000","last_undegraded":"2024-04-25T15:27:46.742298+0000","last_fullsized":"2024-04-25T15:27:46.742298+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734487+0000","last_change":"2024-04-25T15:27:46.734487+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734487+0000","last_undegraded":"2024-04-25T15:27:46.734487+0000","last_fullsized":"2024-04-25T15:27:46.734487+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.736254+0000","last_change":"2024-04-25T15:27:46.736254+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.736254+0000","last_undegraded":"2024-04-25T15:27:46.736254+0000","last_fullsized":"2024-04-25T15:27:46.736254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735290+0000","last_change":"2024-04-25T15:27:46.735290+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735290+0000","last_undegraded":"2024-04-25T15:27:46.735290+0000","last_fullsized":"2024-04-25T15:27:46.735290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740685+0000","last_change":"2024-04-25T15:27:46.740685+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740685+0000","last_undegraded":"2024-04-25T15:27:46.740685+0000","last_fullsized":"2024-04-25T15:27:46.740685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740976+0000","last_change":"2024-04-25T15:27:46.740976+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740976+0000","last_undegraded":"2024-04-25T15:27:46.740976+0000","last_fullsized":"2024-04-25T15:27:46.740976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737831+0000","last_change":"2024-04-25T15:27:46.737831+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737831+0000","last_undegraded":"2024-04-25T15:27:46.737831+0000","last_fullsized":"2024-04-25T15:27:46.737831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739762+0000","last_change":"2024-04-25T15:27:46.739762+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739762+0000","last_undegraded":"2024-04-25T15:27:46.739762+0000","last_fullsized":"2024-04-25T15:27:46.739762+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741068+0000","last_change":"2024-04-25T15:27:46.741068+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741068+0000","last_undegraded":"2024-04-25T15:27:46.741068+0000","last_fullsized":"2024-04-25T15:27:46.741068+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.736975+0000","last_change":"2024-04-25T15:27:46.736975+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.736975+0000","last_undegraded":"2024-04-25T15:27:46.736975+0000","last_fullsized":"2024-04-25T15:27:46.736975+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742531+0000","last_change":"2024-04-25T15:27:46.742531+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742531+0000","last_undegraded":"2024-04-25T15:27:46.742531+0000","last_fullsized":"2024-04-25T15:27:46.742531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744160+0000","last_change":"2024-04-25T15:27:46.744160+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744160+0000","last_undegraded":"2024-04-25T15:27:46.744160+0000","last_fullsized":"2024-04-25T15:27:46.744160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740651+0000","last_change":"2024-04-25T15:27:46.740651+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740651+0000","last_undegraded":"2024-04-25T15:27:46.740651+0000","last_fullsized":"2024-04-25T15:27:46.740651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740954+0000","last_change":"2024-04-25T15:27:46.740954+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740954+0000","last_undegraded":"2024-04-25T15:27:46.740954+0000","last_fullsized":"2024-04-25T15:27:46.740954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739785+0000","last_change":"2024-04-25T15:27:46.739785+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739785+0000","last_undegraded":"2024-04-25T15:27:46.739785+0000","last_fullsized":"2024-04-25T15:27:46.739785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760243+0000","last_change":"2024-04-25T15:27:46.756488+0000","last_active":"2024-04-25T15:27:46.760243+0000","last_peered":"2024-04-25T15:27:46.760243+0000","last_clean":"2024-04-25T15:27:46.760243+0000","last_became_active":"2024-04-25T15:27:46.756137+0000","last_became_peered":"2024-04-25T15:27:46.756137+0000","last_unstale":"2024-04-25T15:27:46.760243+0000","last_undegraded":"2024-04-25T15:27:46.760243+0000","last_fullsized":"2024-04-25T15:27:46.760243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:51.235314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742196+0000","last_change":"2024-04-25T15:27:46.742196+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742196+0000","last_undegraded":"2024-04-25T15:27:46.742196+0000","last_fullsized":"2024-04-25T15:27:46.742196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755354+0000","last_change":"2024-04-25T15:27:46.748183+0000","last_active":"2024-04-25T15:27:46.755354+0000","last_peered":"2024-04-25T15:27:46.755354+0000","last_clean":"2024-04-25T15:27:46.755354+0000","last_became_active":"2024-04-25T15:27:46.747854+0000","last_became_peered":"2024-04-25T15:27:46.747854+0000","last_unstale":"2024-04-25T15:27:46.755354+0000","last_undegraded":"2024-04-25T15:27:46.755354+0000","last_fullsized":"2024-04-25T15:27:46.755354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:52.385229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741214+0000","last_change":"2024-04-25T15:27:46.741214+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741214+0000","last_undegraded":"2024-04-25T15:27:46.741214+0000","last_fullsized":"2024-04-25T15:27:46.741214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753483+0000","last_change":"2024-04-25T15:27:46.750527+0000","last_active":"2024-04-25T15:27:46.753483+0000","last_peered":"2024-04-25T15:27:46.753483+0000","last_clean":"2024-04-25T15:27:46.753483+0000","last_became_active":"2024-04-25T15:27:46.749838+0000","last_became_peered":"2024-04-25T15:27:46.749838+0000","last_unstale":"2024-04-25T15:27:46.753483+0000","last_undegraded":"2024-04-25T15:27:46.753483+0000","last_fullsized":"2024-04-25T15:27:46.753483+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:44.420376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742915+0000","last_change":"2024-04-25T15:27:46.742915+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742915+0000","last_undegraded":"2024-04-25T15:27:46.742915+0000","last_fullsized":"2024-04-25T15:27:46.742915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752841+0000","last_change":"2024-04-25T15:27:46.747175+0000","last_active":"2024-04-25T15:27:46.752841+0000","last_peered":"2024-04-25T15:27:46.752841+0000","last_clean":"2024-04-25T15:27:46.752841+0000","last_became_active":"2024-04-25T15:27:46.746878+0000","last_became_peered":"2024-04-25T15:27:46.746878+0000","last_unstale":"2024-04-25T15:27:46.752841+0000","last_undegraded":"2024-04-25T15:27:46.752841+0000","last_fullsized":"2024-04-25T15:27:46.752841+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:42.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738688+0000","last_change":"2024-04-25T15:27:46.738688+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738688+0000","last_undegraded":"2024-04-25T15:27:46.738688+0000","last_fullsized":"2024-04-25T15:27:46.738688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751539+0000","last_change":"2024-04-25T15:27:46.748931+0000","last_active":"2024-04-25T15:27:46.751539+0000","last_peered":"2024-04-25T15:27:46.751539+0000","last_clean":"2024-04-25T15:27:46.751539+0000","last_became_active":"2024-04-25T15:27:46.748400+0000","last_became_peered":"2024-04-25T15:27:46.748400+0000","last_unstale":"2024-04-25T15:27:46.751539+0000","last_undegraded":"2024-04-25T15:27:46.751539+0000","last_fullsized":"2024-04-25T15:27:46.751539+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:32.140220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735434+0000","last_change":"2024-04-25T15:27:46.735434+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735434+0000","last_undegraded":"2024-04-25T15:27:46.735434+0000","last_fullsized":"2024-04-25T15:27:46.735434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.756042+0000","last_change":"2024-04-25T15:27:46.753325+0000","last_active":"2024-04-25T15:27:46.756042+0000","last_peered":"2024-04-25T15:27:46.756042+0000","last_clean":"2024-04-25T15:27:46.756042+0000","last_became_active":"2024-04-25T15:27:46.753006+0000","last_became_peered":"2024-04-25T15:27:46.753006+0000","last_unstale":"2024-04-25T15:27:46.756042+0000","last_undegraded":"2024-04-25T15:27:46.756042+0000","last_fullsized":"2024-04-25T15:27:46.756042+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:16.894533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744277+0000","last_change":"2024-04-25T15:27:46.744277+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744277+0000","last_undegraded":"2024-04-25T15:27:46.744277+0000","last_fullsized":"2024-04-25T15:27:46.744277+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.742751+0000","last_change":"2024-04-25T15:27:46.740274+0000","last_active":"2024-04-25T15:27:46.742751+0000","last_peered":"2024-04-25T15:27:46.742751+0000","last_clean":"2024-04-25T15:27:46.742751+0000","last_became_active":"2024-04-25T15:27:46.739275+0000","last_became_peered":"2024-04-25T15:27:46.739275+0000","last_unstale":"2024-04-25T15:27:46.742751+0000","last_undegraded":"2024-04-25T15:27:46.742751+0000","last_fullsized":"2024-04-25T15:27:46.742751+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:43.588775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734472+0000","last_change":"2024-04-25T15:27:46.734472+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734472+0000","last_undegraded":"2024-04-25T15:27:46.734472+0000","last_fullsized":"2024-04-25T15:27:46.734472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755355+0000","last_change":"2024-04-25T15:27:46.748972+0000","last_active":"2024-04-25T15:27:46.755355+0000","last_peered":"2024-04-25T15:27:46.755355+0000","last_clean":"2024-04-25T15:27:46.755355+0000","last_became_active":"2024-04-25T15:27:46.748630+0000","last_became_peered":"2024-04-25T15:27:46.748630+0000","last_unstale":"2024-04-25T15:27:46.755355+0000","last_undegraded":"2024-04-25T15:27:46.755355+0000","last_fullsized":"2024-04-25T15:27:46.755355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:01.865516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739213+0000","last_change":"2024-04-25T15:27:46.739213+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739213+0000","last_undegraded":"2024-04-25T15:27:46.739213+0000","last_fullsized":"2024-04-25T15:27:46.739213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757922+0000","last_change":"2024-04-25T15:27:46.753409+0000","last_active":"2024-04-25T15:27:46.757922+0000","last_peered":"2024-04-25T15:27:46.757922+0000","last_clean":"2024-04-25T15:27:46.757922+0000","last_became_active":"2024-04-25T15:27:46.753090+0000","last_became_peered":"2024-04-25T15:27:46.753090+0000","last_unstale":"2024-04-25T15:27:46.757922+0000","last_undegraded":"2024-04-25T15:27:46.757922+0000","last_fullsized":"2024-04-25T15:27:46.757922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:23.117643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741658+0000","last_change":"2024-04-25T15:27:46.741658+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741658+0000","last_undegraded":"2024-04-25T15:27:46.741658+0000","last_fullsized":"2024-04-25T15:27:46.741658+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755265+0000","last_change":"2024-04-25T15:27:46.751600+0000","last_active":"2024-04-25T15:27:46.755265+0000","last_peered":"2024-04-25T15:27:46.755265+0000","last_clean":"2024-04-25T15:27:46.755265+0000","last_became_active":"2024-04-25T15:27:46.751255+0000","last_became_peered":"2024-04-25T15:27:46.751255+0000","last_unstale":"2024-04-25T15:27:46.755265+0000","last_undegraded":"2024-04-25T15:27:46.755265+0000","last_fullsized":"2024-04-25T15:27:46.755265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.537162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740683+0000","last_change":"2024-04-25T15:27:46.740683+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740683+0000","last_undegraded":"2024-04-25T15:27:46.740683+0000","last_fullsized":"2024-04-25T15:27:46.740683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752242+0000","last_change":"2024-04-25T15:27:46.750192+0000","last_active":"2024-04-25T15:27:46.752242+0000","last_peered":"2024-04-25T15:27:46.752242+0000","last_clean":"2024-04-25T15:27:46.752242+0000","last_became_active":"2024-04-25T15:27:46.749859+0000","last_became_peered":"2024-04-25T15:27:46.749859+0000","last_unstale":"2024-04-25T15:27:46.752242+0000","last_undegraded":"2024-04-25T15:27:46.752242+0000","last_fullsized":"2024-04-25T15:27:46.752242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.634184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740931+0000","last_change":"2024-04-25T15:27:46.740931+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740931+0000","last_undegraded":"2024-04-25T15:27:46.740931+0000","last_fullsized":"2024-04-25T15:27:46.740931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.754580+0000","last_change":"2024-04-25T15:27:46.751593+0000","last_active":"2024-04-25T15:27:46.754580+0000","last_peered":"2024-04-25T15:27:46.754580+0000","last_clean":"2024-04-25T15:27:46.754580+0000","last_became_active":"2024-04-25T15:27:46.751123+0000","last_became_peered":"2024-04-25T15:27:46.751123+0000","last_unstale":"2024-04-25T15:27:46.754580+0000","last_undegraded":"2024-04-25T15:27:46.754580+0000","last_fullsized":"2024-04-25T15:27:46.754580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:51.896417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740179+0000","last_change":"2024-04-25T15:27:46.740179+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740179+0000","last_undegraded":"2024-04-25T15:27:46.740179+0000","last_fullsized":"2024-04-25T15:27:46.740179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751660+0000","last_change":"2024-04-25T15:27:46.747059+0000","last_active":"2024-04-25T15:27:46.751660+0000","last_peered":"2024-04-25T15:27:46.751660+0000","last_clean":"2024-04-25T15:27:46.751660+0000","last_became_active":"2024-04-25T15:27:46.746774+0000","last_became_peered":"2024-04-25T15:27:46.746774+0000","last_unstale":"2024-04-25T15:27:46.751660+0000","last_undegraded":"2024-04-25T15:27:46.751660+0000","last_fullsized":"2024-04-25T15:27:46.751660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:54.994195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743383+0000","last_change":"2024-04-25T15:27:46.743383+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743383+0000","last_undegraded":"2024-04-25T15:27:46.743383+0000","last_fullsized":"2024-04-25T15:27:46.743383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760439+0000","last_change":"2024-04-25T15:27:46.755998+0000","last_active":"2024-04-25T15:27:46.760439+0000","last_peered":"2024-04-25T15:27:46.760439+0000","last_clean":"2024-04-25T15:27:46.760439+0000","last_became_active":"2024-04-25T15:27:46.755762+0000","last_became_peered":"2024-04-25T15:27:46.755762+0000","last_unstale":"2024-04-25T15:27:46.760439+0000","last_undegraded":"2024-04-25T15:27:46.760439+0000","last_fullsized":"2024-04-25T15:27:46.760439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:43.588775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734455+0000","last_change":"2024-04-25T15:27:46.734455+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734455+0000","last_undegraded":"2024-04-25T15:27:46.734455+0000","last_fullsized":"2024-04-25T15:27:46.734455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755722+0000","last_change":"2024-04-25T15:27:46.753451+0000","last_active":"2024-04-25T15:27:46.755722+0000","last_peered":"2024-04-25T15:27:46.755722+0000","last_clean":"2024-04-25T15:27:46.755722+0000","last_became_active":"2024-04-25T15:27:46.753133+0000","last_became_peered":"2024-04-25T15:27:46.753133+0000","last_unstale":"2024-04-25T15:27:46.755722+0000","last_undegraded":"2024-04-25T15:27:46.755722+0000","last_fullsized":"2024-04-25T15:27:46.755722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:02.482063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743730+0000","last_change":"2024-04-25T15:27:46.743730+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743730+0000","last_undegraded":"2024-04-25T15:27:46.743730+0000","last_fullsized":"2024-04-25T15:27:46.743730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752208+0000","last_change":"2024-04-25T15:27:46.747236+0000","last_active":"2024-04-25T15:27:46.752208+0000","last_peered":"2024-04-25T15:27:46.752208+0000","last_clean":"2024-04-25T15:27:46.752208+0000","last_became_active":"2024-04-25T15:27:46.746401+0000","last_became_peered":"2024-04-25T15:27:46.746401+0000","last_unstale":"2024-04-25T15:27:46.752208+0000","last_undegraded":"2024-04-25T15:27:46.752208+0000","last_fullsized":"2024-04-25T15:27:46.752208+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:25.199824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737357+0000","last_change":"2024-04-25T15:27:46.737357+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737357+0000","last_undegraded":"2024-04-25T15:27:46.737357+0000","last_fullsized":"2024-04-25T15:27:46.737357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760768+0000","last_change":"2024-04-25T15:27:46.759114+0000","last_active":"2024-04-25T15:27:46.760768+0000","last_peered":"2024-04-25T15:27:46.760768+0000","last_clean":"2024-04-25T15:27:46.760768+0000","last_became_active":"2024-04-25T15:27:46.758785+0000","last_became_peered":"2024-04-25T15:27:46.758785+0000","last_unstale":"2024-04-25T15:27:46.760768+0000","last_undegraded":"2024-04-25T15:27:46.760768+0000","last_fullsized":"2024-04-25T15:27:46.760768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:51.896417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740337+0000","last_change":"2024-04-25T15:27:46.740337+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740337+0000","last_undegraded":"2024-04-25T15:27:46.740337+0000","last_fullsized":"2024-04-25T15:27:46.740337+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753377+0000","last_change":"2024-04-25T15:27:46.750313+0000","last_active":"2024-04-25T15:27:46.753377+0000","last_peered":"2024-04-25T15:27:46.753377+0000","last_clean":"2024-04-25T15:27:46.753377+0000","last_became_active":"2024-04-25T15:27:46.749988+0000","last_became_peered":"2024-04-25T15:27:46.749988+0000","last_unstale":"2024-04-25T15:27:46.753377+0000","last_undegraded":"2024-04-25T15:27:46.753377+0000","last_fullsized":"2024-04-25T15:27:46.753377+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:06.332293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737978+0000","last_change":"2024-04-25T15:27:46.737978+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737978+0000","last_undegraded":"2024-04-25T15:27:46.737978+0000","last_fullsized":"2024-04-25T15:27:46.737978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757737+0000","last_change":"2024-04-25T15:27:46.754470+0000","last_active":"2024-04-25T15:27:46.757737+0000","last_peered":"2024-04-25T15:27:46.757737+0000","last_clean":"2024-04-25T15:27:46.757737+0000","last_became_active":"2024-04-25T15:27:46.753975+0000","last_became_peered":"2024-04-25T15:27:46.753975+0000","last_unstale":"2024-04-25T15:27:46.757737+0000","last_undegraded":"2024-04-25T15:27:46.757737+0000","last_fullsized":"2024-04-25T15:27:46.757737+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:41.565463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740377+0000","last_change":"2024-04-25T15:27:46.740377+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740377+0000","last_undegraded":"2024-04-25T15:27:46.740377+0000","last_fullsized":"2024-04-25T15:27:46.740377+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757718+0000","last_change":"2024-04-25T15:27:46.754424+0000","last_active":"2024-04-25T15:27:46.757718+0000","last_peered":"2024-04-25T15:27:46.757718+0000","last_clean":"2024-04-25T15:27:46.757718+0000","last_became_active":"2024-04-25T15:27:46.754102+0000","last_became_peered":"2024-04-25T15:27:46.754102+0000","last_unstale":"2024-04-25T15:27:46.757718+0000","last_undegraded":"2024-04-25T15:27:46.757718+0000","last_fullsized":"2024-04-25T15:27:46.757718+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:54.809375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742041+0000","last_change":"2024-04-25T15:27:46.742041+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742041+0000","last_undegraded":"2024-04-25T15:27:46.742041+0000","last_fullsized":"2024-04-25T15:27:46.742041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760460+0000","last_change":"2024-04-25T15:27:46.756182+0000","last_active":"2024-04-25T15:27:46.760460+0000","last_peered":"2024-04-25T15:27:46.760460+0000","last_clean":"2024-04-25T15:27:46.760460+0000","last_became_active":"2024-04-25T15:27:46.755789+0000","last_became_peered":"2024-04-25T15:27:46.755789+0000","last_unstale":"2024-04-25T15:27:46.760460+0000","last_undegraded":"2024-04-25T15:27:46.760460+0000","last_fullsized":"2024-04-25T15:27:46.760460+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.454298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743258+0000","last_change":"2024-04-25T15:27:46.743258+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743258+0000","last_undegraded":"2024-04-25T15:27:46.743258+0000","last_fullsized":"2024-04-25T15:27:46.743258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753687+0000","last_change":"2024-04-25T15:27:46.751621+0000","last_active":"2024-04-25T15:27:46.753687+0000","last_peered":"2024-04-25T15:27:46.753687+0000","last_clean":"2024-04-25T15:27:46.753687+0000","last_became_active":"2024-04-25T15:27:46.751328+0000","last_became_peered":"2024-04-25T15:27:46.751328+0000","last_unstale":"2024-04-25T15:27:46.753687+0000","last_undegraded":"2024-04-25T15:27:46.753687+0000","last_fullsized":"2024-04-25T15:27:46.753687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:07:28.385465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740778+0000","last_change":"2024-04-25T15:27:46.740778+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740778+0000","last_undegraded":"2024-04-25T15:27:46.740778+0000","last_fullsized":"2024-04-25T15:27:46.740778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755225+0000","last_change":"2024-04-25T15:27:46.748665+0000","last_active":"2024-04-25T15:27:46.755225+0000","last_peered":"2024-04-25T15:27:46.755225+0000","last_clean":"2024-04-25T15:27:46.755225+0000","last_became_active":"2024-04-25T15:27:46.748364+0000","last_became_peered":"2024-04-25T15:27:46.748364+0000","last_unstale":"2024-04-25T15:27:46.755225+0000","last_undegraded":"2024-04-25T15:27:46.755225+0000","last_fullsized":"2024-04-25T15:27:46.755225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:34.541226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734502+0000","last_change":"2024-04-25T15:27:46.734502+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734502+0000","last_undegraded":"2024-04-25T15:27:46.734502+0000","last_fullsized":"2024-04-25T15:27:46.734502+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760752+0000","last_change":"2024-04-25T15:27:46.759084+0000","last_active":"2024-04-25T15:27:46.760752+0000","last_peered":"2024-04-25T15:27:46.760752+0000","last_clean":"2024-04-25T15:27:46.760752+0000","last_became_active":"2024-04-25T15:27:46.758757+0000","last_became_peered":"2024-04-25T15:27:46.758757+0000","last_unstale":"2024-04-25T15:27:46.760752+0000","last_undegraded":"2024-04-25T15:27:46.760752+0000","last_fullsized":"2024-04-25T15:27:46.760752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:54.994195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739729+0000","last_change":"2024-04-25T15:27:46.739729+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739729+0000","last_undegraded":"2024-04-25T15:27:46.739729+0000","last_fullsized":"2024-04-25T15:27:46.739729+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755747+0000","last_change":"2024-04-25T15:27:46.753206+0000","last_active":"2024-04-25T15:27:46.755747+0000","last_peered":"2024-04-25T15:27:46.755747+0000","last_clean":"2024-04-25T15:27:46.755747+0000","last_became_active":"2024-04-25T15:27:46.752883+0000","last_became_peered":"2024-04-25T15:27:46.752883+0000","last_unstale":"2024-04-25T15:27:46.755747+0000","last_undegraded":"2024-04-25T15:27:46.755747+0000","last_fullsized":"2024-04-25T15:27:46.755747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:55.623358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.745243+0000","last_change":"2024-04-25T15:27:46.745243+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.745243+0000","last_undegraded":"2024-04-25T15:27:46.745243+0000","last_fullsized":"2024-04-25T15:27:46.745243+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755764+0000","last_change":"2024-04-25T15:27:46.753592+0000","last_active":"2024-04-25T15:27:46.755764+0000","last_peered":"2024-04-25T15:27:46.755764+0000","last_clean":"2024-04-25T15:27:46.755764+0000","last_became_active":"2024-04-25T15:27:46.753163+0000","last_became_peered":"2024-04-25T15:27:46.753163+0000","last_unstale":"2024-04-25T15:27:46.755764+0000","last_undegraded":"2024-04-25T15:27:46.755764+0000","last_fullsized":"2024-04-25T15:27:46.755764+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:19.025834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742091+0000","last_change":"2024-04-25T15:27:46.742091+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742091+0000","last_undegraded":"2024-04-25T15:27:46.742091+0000","last_fullsized":"2024-04-25T15:27:46.742091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752245+0000","last_change":"2024-04-25T15:27:46.744686+0000","last_active":"2024-04-25T15:27:46.752245+0000","last_peered":"2024-04-25T15:27:46.752245+0000","last_clean":"2024-04-25T15:27:46.752245+0000","last_became_active":"2024-04-25T15:27:46.744321+0000","last_became_peered":"2024-04-25T15:27:46.744321+0000","last_unstale":"2024-04-25T15:27:46.752245+0000","last_undegraded":"2024-04-25T15:27:46.752245+0000","last_fullsized":"2024-04-25T15:27:46.752245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:51.235314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739940+0000","last_change":"2024-04-25T15:27:46.739940+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739940+0000","last_undegraded":"2024-04-25T15:27:46.739940+0000","last_fullsized":"2024-04-25T15:27:46.739940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755161+0000","last_change":"2024-04-25T15:27:46.749552+0000","last_active":"2024-04-25T15:27:46.755161+0000","last_peered":"2024-04-25T15:27:46.755161+0000","last_clean":"2024-04-25T15:27:46.755161+0000","last_became_active":"2024-04-25T15:27:46.749242+0000","last_became_peered":"2024-04-25T15:27:46.749242+0000","last_unstale":"2024-04-25T15:27:46.755161+0000","last_undegraded":"2024-04-25T15:27:46.755161+0000","last_fullsized":"2024-04-25T15:27:46.755161+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:57:28.326266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743195+0000","last_change":"2024-04-25T15:27:46.743195+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743195+0000","last_undegraded":"2024-04-25T15:27:46.743195+0000","last_fullsized":"2024-04-25T15:27:46.743195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751802+0000","last_change":"2024-04-25T15:27:46.746473+0000","last_active":"2024-04-25T15:27:46.751802+0000","last_peered":"2024-04-25T15:27:46.751802+0000","last_clean":"2024-04-25T15:27:46.751802+0000","last_became_active":"2024-04-25T15:27:46.746083+0000","last_became_peered":"2024-04-25T15:27:46.746083+0000","last_unstale":"2024-04-25T15:27:46.751802+0000","last_undegraded":"2024-04-25T15:27:46.751802+0000","last_fullsized":"2024-04-25T15:27:46.751802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.454298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737498+0000","last_change":"2024-04-25T15:27:46.737498+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737498+0000","last_undegraded":"2024-04-25T15:27:46.737498+0000","last_fullsized":"2024-04-25T15:27:46.737498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752198+0000","last_change":"2024-04-25T15:27:46.745918+0000","last_active":"2024-04-25T15:27:46.752198+0000","last_peered":"2024-04-25T15:27:46.752198+0000","last_clean":"2024-04-25T15:27:46.752198+0000","last_became_active":"2024-04-25T15:27:46.745596+0000","last_became_peered":"2024-04-25T15:27:46.745596+0000","last_unstale":"2024-04-25T15:27:46.752198+0000","last_undegraded":"2024-04-25T15:27:46.752198+0000","last_fullsized":"2024-04-25T15:27:46.752198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:37.598034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739515+0000","last_change":"2024-04-25T15:27:46.739515+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739515+0000","last_undegraded":"2024-04-25T15:27:46.739515+0000","last_fullsized":"2024-04-25T15:27:46.739515+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.749424+0000","last_change":"2024-04-25T15:27:46.746186+0000","last_active":"2024-04-25T15:27:46.749424+0000","last_peered":"2024-04-25T15:27:46.749424+0000","last_clean":"2024-04-25T15:27:46.749424+0000","last_became_active":"2024-04-25T15:27:46.745867+0000","last_became_peered":"2024-04-25T15:27:46.745867+0000","last_unstale":"2024-04-25T15:27:46.749424+0000","last_undegraded":"2024-04-25T15:27:46.749424+0000","last_fullsized":"2024-04-25T15:27:46.749424+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.445845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741940+0000","last_change":"2024-04-25T15:27:46.741940+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741940+0000","last_undegraded":"2024-04-25T15:27:46.741940+0000","last_fullsized":"2024-04-25T15:27:46.741940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760410+0000","last_change":"2024-04-25T15:27:46.755483+0000","last_active":"2024-04-25T15:27:46.760410+0000","last_peered":"2024-04-25T15:27:46.760410+0000","last_clean":"2024-04-25T15:27:46.760410+0000","last_became_active":"2024-04-25T15:27:46.755160+0000","last_became_peered":"2024-04-25T15:27:46.755160+0000","last_unstale":"2024-04-25T15:27:46.760410+0000","last_undegraded":"2024-04-25T15:27:46.760410+0000","last_fullsized":"2024-04-25T15:27:46.760410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:40:46.657005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":191,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.743997+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:27:46.743997+0000","last_peered":"2024-04-25T15:27:46.743997+0000","last_clean":"2024-04-25T15:27:46.743997+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:27:46.743997+0000","last_undegraded":"2024-04-25T15:27:46.743997+0000","last_fullsized":"2024-04-25T15:27:46.743997+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343472,"statfs":{"total":96636764160,"available":96607715328,"internally_reserved":0,"allocated":1372160,"data_stored":912533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":328440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28376,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343464,"statfs":{"total":96636764160,"available":96607707136,"internally_reserved":0,"allocated":1372160,"data_stored":912533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:27:51.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:27:51.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.980+0000 7fa787d2e640 1 Processor -- start 2024-04-25T15:27:51.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.981+0000 7fa787d2e640 1 -- start start 2024-04-25T15:27:51.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa787d2e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7800d1780 0x7fa780163ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa787d2e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa7800cc710 0x7fa780164410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa787d2e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7800cce10 con 0x7fa780160510 2024-04-25T15:27:51.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa787d2e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa7800cd0b0 con 0x7fa7800d1780 2024-04-25T15:27:51.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa787d2e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa780164aa0 con 0x7fa7800cc710 2024-04-25T15:27:51.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa7852c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7800d1780 0x7fa780163ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa785aca640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa780160510 0x7fa78006f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55618/0 (socket says 172.21.15.67:55618) 2024-04-25T15:27:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa7852c9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa7800d1780 0x7fa780163ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35194/0 (socket says 172.21.15.67:35194) 2024-04-25T15:27:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa785aca640 1 -- 172.21.15.67:0/1725392899 learned_addr learned my addr 172.21.15.67:0/1725392899 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa7862cb640 1 --2- 172.21.15.67:0/1725392899 >> v2:172.21.15.176:3300/0 conn(0x7fa7800cc710 0x7fa780164410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa7852c9640 1 -- 172.21.15.67:0/1725392899 >> v2:172.21.15.176:3300/0 conn(0x7fa7800cc710 msgr2=0x7fa780164410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa7852c9640 1 --2- 172.21.15.67:0/1725392899 >> v2:172.21.15.176:3300/0 conn(0x7fa7800cc710 0x7fa780164410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa7852c9640 1 -- 172.21.15.67:0/1725392899 >> v1:172.21.15.67:6789/0 conn(0x7fa780160510 legacy=0x7fa78006f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.982+0000 7fa7852c9640 1 -- 172.21.15.67:0/1725392899 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa780164d80 con 0x7fa7800d1780 2024-04-25T15:27:51.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.983+0000 7fa7852c9640 1 --2- 172.21.15.67:0/1725392899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa780163ed0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b3830 tx=0x7fa77002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96393010b022fd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.983+0000 7fa784ac8640 1 -- 172.21.15.67:0/1725392899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa770037070 con 0x7fa7800d1780 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.983+0000 7fa784ac8640 1 -- 172.21.15.67:0/1725392899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa770032440 con 0x7fa7800d1780 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.983+0000 7fa784ac8640 1 -- 172.21.15.67:0/1725392899 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77002ae60 con 0x7fa7800d1780 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 -- 172.21.15.67:0/1725392899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 msgr2=0x7fa780163ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 --2- 172.21.15.67:0/1725392899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa780163ed0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa7800b3830 tx=0x7fa77002a110 comp rx=0 tx=0).stop 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 -- 172.21.15.67:0/1725392899 shutdown_connections 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 --2- 172.21.15.67:0/1725392899 >> v2:172.21.15.176:3300/0 conn(0x7fa7800cc710 0x7fa780164410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 --2- 172.21.15.67:0/1725392899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa780163ed0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 -- 172.21.15.67:0/1725392899 >> 172.21.15.67:0/1725392899 conn(0x7fa7800ce790 msgr2=0x7fa7800ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 -- 172.21.15.67:0/1725392899 shutdown_connections 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.984+0000 7fa787d2e640 1 -- 172.21.15.67:0/1725392899 wait complete. 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.986+0000 7fa787d2e640 1 Processor -- start 2024-04-25T15:27:51.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.986+0000 7fa787d2e640 1 -- start start 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa787d2e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7800cc710 0x7fa7800cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa787d2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa7800a2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa787d2e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa780160510 0x7fa78014c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa787d2e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa7800d14a0 con 0x7fa7800d1780 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa787d2e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa7800d1320 con 0x7fa780160510 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa787d2e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa78006d550 con 0x7fa7800cc710 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa7852c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa7800a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa7852c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa7800a2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35198/0 (socket says 172.21.15.67:35198) 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.987+0000 7fa7852c9640 1 -- 172.21.15.67:0/3829914497 learned_addr learned my addr 172.21.15.67:0/3829914497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa785aca640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7800cc710 0x7fa7800cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7862cb640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa780160510 0x7fa78014c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7852c9640 1 -- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7800cc710 msgr2=0x7fa7800cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7852c9640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7800cc710 0x7fa7800cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7852c9640 1 -- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa780160510 msgr2=0x7fa78014c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7852c9640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa780160510 0x7fa78014c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7852c9640 1 -- 172.21.15.67:0/3829914497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7801507a0 con 0x7fa7800d1780 2024-04-25T15:27:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7852c9640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa7800a2960 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa77002a150 tx=0x7fa7700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7767fc640 1 -- 172.21.15.67:0/3829914497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa770037040 con 0x7fa7800d1780 2024-04-25T15:27:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7767fc640 1 -- 172.21.15.67:0/3829914497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa770030770 con 0x7fa7800d1780 2024-04-25T15:27:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7800c9340 con 0x7fa7800d1780 2024-04-25T15:27:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa7767fc640 1 -- 172.21.15.67:0/3829914497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa770030b40 con 0x7fa7800d1780 2024-04-25T15:27:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.988+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7800c9900 con 0x7fa7800d1780 2024-04-25T15:27:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.989+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa780050740 con 0x7fa7800d1780 2024-04-25T15:27:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.990+0000 7fa7767fc640 1 -- 172.21.15.67:0/3829914497 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa770044020 con 0x7fa7800d1780 2024-04-25T15:27:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.990+0000 7fa7767fc640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7640744b0 0x7fa7640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.991+0000 7fa7767fc640 1 -- 172.21.15.67:0/3829914497 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fa7700b5830 con 0x7fa7800d1780 2024-04-25T15:27:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.991+0000 7fa785aca640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7640744b0 0x7fa7640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.991+0000 7fa785aca640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7640744b0 0x7fa7640769c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa77c000f60 tx=0x7fa77c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:51.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:51.995+0000 7fa7767fc640 1 -- 172.21.15.67:0/3829914497 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa770032e20 con 0x7fa7800d1780 2024-04-25T15:27:52.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.178+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7800cfa90 con 0x7fa7640744b0 2024-04-25T15:27:52.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.185+0000 7fa7767fc640 1 -- 172.21.15.67:0/3829914497 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246737 (secure 0 0 0) 0x7fa7800cfa90 con 0x7fa7640744b0 2024-04-25T15:27:52.188 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:52.189 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7640744b0 msgr2=0x7fa7640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7640744b0 0x7fa7640769c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa77c000f60 tx=0x7fa77c009040 comp rx=0 tx=0).stop 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 msgr2=0x7fa7800a2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa7800a2960 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fa77002a150 tx=0x7fa7700046c0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 shutdown_connections 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa7640744b0 0x7fa7640769c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa780160510 0x7fa78014c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7800d1780 0x7fa7800a2960 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 --2- 172.21.15.67:0/3829914497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7800cc710 0x7fa7800cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.187+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 >> 172.21.15.67:0/3829914497 conn(0x7fa7800ce790 msgr2=0x7fa7800ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.188+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 shutdown_connections 2024-04-25T15:27:52.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.188+0000 7fa787d2e640 1 -- 172.21.15.67:0/3829914497 wait complete. 2024-04-25T15:27:52.200 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-04-25T15:27:50.829478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223456,"kb_used_data":7252,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751264,"statfs":{"total":773094113280,"available":772865294336,"internally_reserved":0,"allocated":7426048,"data_stored":3764331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.099960"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742158+0000","last_change":"2024-04-25T15:27:46.742158+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742158+0000","last_undegraded":"2024-04-25T15:27:46.742158+0000","last_fullsized":"2024-04-25T15:27:46.742158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.746577+0000","last_change":"2024-04-25T15:27:46.746577+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.746577+0000","last_undegraded":"2024-04-25T15:27:46.746577+0000","last_fullsized":"2024-04-25T15:27:46.746577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741554+0000","last_change":"2024-04-25T15:27:46.741554+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741554+0000","last_undegraded":"2024-04-25T15:27:46.741554+0000","last_fullsized":"2024-04-25T15:27:46.741554+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739391+0000","last_change":"2024-04-25T15:27:46.739391+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739391+0000","last_undegraded":"2024-04-25T15:27:46.739391+0000","last_fullsized":"2024-04-25T15:27:46.739391+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741588+0000","last_change":"2024-04-25T15:27:46.741588+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741588+0000","last_undegraded":"2024-04-25T15:27:46.741588+0000","last_fullsized":"2024-04-25T15:27:46.741588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744353+0000","last_change":"2024-04-25T15:27:46.744353+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744353+0000","last_undegraded":"2024-04-25T15:27:46.744353+0000","last_fullsized":"2024-04-25T15:27:46.744353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741489+0000","last_change":"2024-04-25T15:27:46.741489+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741489+0000","last_undegraded":"2024-04-25T15:27:46.741489+0000","last_fullsized":"2024-04-25T15:27:46.741489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741504+0000","last_change":"2024-04-25T15:27:46.741504+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741504+0000","last_undegraded":"2024-04-25T15:27:46.741504+0000","last_fullsized":"2024-04-25T15:27:46.741504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743205+0000","last_change":"2024-04-25T15:27:46.743205+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743205+0000","last_undegraded":"2024-04-25T15:27:46.743205+0000","last_fullsized":"2024-04-25T15:27:46.743205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737954+0000","last_change":"2024-04-25T15:27:46.737954+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737954+0000","last_undegraded":"2024-04-25T15:27:46.737954+0000","last_fullsized":"2024-04-25T15:27:46.737954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743381+0000","last_change":"2024-04-25T15:27:46.743381+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743381+0000","last_undegraded":"2024-04-25T15:27:46.743381+0000","last_fullsized":"2024-04-25T15:27:46.743381+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738618+0000","last_change":"2024-04-25T15:27:46.738618+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738618+0000","last_undegraded":"2024-04-25T15:27:46.738618+0000","last_fullsized":"2024-04-25T15:27:46.738618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734472+0000","last_change":"2024-04-25T15:27:46.734472+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734472+0000","last_undegraded":"2024-04-25T15:27:46.734472+0000","last_fullsized":"2024-04-25T15:27:46.734472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740602+0000","last_change":"2024-04-25T15:27:46.740602+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740602+0000","last_undegraded":"2024-04-25T15:27:46.740602+0000","last_fullsized":"2024-04-25T15:27:46.740602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735281+0000","last_change":"2024-04-25T15:27:46.735281+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735281+0000","last_undegraded":"2024-04-25T15:27:46.735281+0000","last_fullsized":"2024-04-25T15:27:46.735281+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735174+0000","last_change":"2024-04-25T15:27:46.735174+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735174+0000","last_undegraded":"2024-04-25T15:27:46.735174+0000","last_fullsized":"2024-04-25T15:27:46.735174+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742694+0000","last_change":"2024-04-25T15:27:46.742694+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742694+0000","last_undegraded":"2024-04-25T15:27:46.742694+0000","last_fullsized":"2024-04-25T15:27:46.742694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738105+0000","last_change":"2024-04-25T15:27:46.738105+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738105+0000","last_undegraded":"2024-04-25T15:27:46.738105+0000","last_fullsized":"2024-04-25T15:27:46.738105+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742298+0000","last_change":"2024-04-25T15:27:46.742298+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742298+0000","last_undegraded":"2024-04-25T15:27:46.742298+0000","last_fullsized":"2024-04-25T15:27:46.742298+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734487+0000","last_change":"2024-04-25T15:27:46.734487+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734487+0000","last_undegraded":"2024-04-25T15:27:46.734487+0000","last_fullsized":"2024-04-25T15:27:46.734487+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.736254+0000","last_change":"2024-04-25T15:27:46.736254+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.736254+0000","last_undegraded":"2024-04-25T15:27:46.736254+0000","last_fullsized":"2024-04-25T15:27:46.736254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735290+0000","last_change":"2024-04-25T15:27:46.735290+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735290+0000","last_undegraded":"2024-04-25T15:27:46.735290+0000","last_fullsized":"2024-04-25T15:27:46.735290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740685+0000","last_change":"2024-04-25T15:27:46.740685+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740685+0000","last_undegraded":"2024-04-25T15:27:46.740685+0000","last_fullsized":"2024-04-25T15:27:46.740685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740976+0000","last_change":"2024-04-25T15:27:46.740976+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740976+0000","last_undegraded":"2024-04-25T15:27:46.740976+0000","last_fullsized":"2024-04-25T15:27:46.740976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737831+0000","last_change":"2024-04-25T15:27:46.737831+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737831+0000","last_undegraded":"2024-04-25T15:27:46.737831+0000","last_fullsized":"2024-04-25T15:27:46.737831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739762+0000","last_change":"2024-04-25T15:27:46.739762+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739762+0000","last_undegraded":"2024-04-25T15:27:46.739762+0000","last_fullsized":"2024-04-25T15:27:46.739762+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741068+0000","last_change":"2024-04-25T15:27:46.741068+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741068+0000","last_undegraded":"2024-04-25T15:27:46.741068+0000","last_fullsized":"2024-04-25T15:27:46.741068+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.736975+0000","last_change":"2024-04-25T15:27:46.736975+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.736975+0000","last_undegraded":"2024-04-25T15:27:46.736975+0000","last_fullsized":"2024-04-25T15:27:46.736975+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742531+0000","last_change":"2024-04-25T15:27:46.742531+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742531+0000","last_undegraded":"2024-04-25T15:27:46.742531+0000","last_fullsized":"2024-04-25T15:27:46.742531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744160+0000","last_change":"2024-04-25T15:27:46.744160+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744160+0000","last_undegraded":"2024-04-25T15:27:46.744160+0000","last_fullsized":"2024-04-25T15:27:46.744160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740651+0000","last_change":"2024-04-25T15:27:46.740651+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740651+0000","last_undegraded":"2024-04-25T15:27:46.740651+0000","last_fullsized":"2024-04-25T15:27:46.740651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740954+0000","last_change":"2024-04-25T15:27:46.740954+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740954+0000","last_undegraded":"2024-04-25T15:27:46.740954+0000","last_fullsized":"2024-04-25T15:27:46.740954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739785+0000","last_change":"2024-04-25T15:27:46.739785+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739785+0000","last_undegraded":"2024-04-25T15:27:46.739785+0000","last_fullsized":"2024-04-25T15:27:46.739785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760243+0000","last_change":"2024-04-25T15:27:46.756488+0000","last_active":"2024-04-25T15:27:46.760243+0000","last_peered":"2024-04-25T15:27:46.760243+0000","last_clean":"2024-04-25T15:27:46.760243+0000","last_became_active":"2024-04-25T15:27:46.756137+0000","last_became_peered":"2024-04-25T15:27:46.756137+0000","last_unstale":"2024-04-25T15:27:46.760243+0000","last_undegraded":"2024-04-25T15:27:46.760243+0000","last_fullsized":"2024-04-25T15:27:46.760243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:51.235314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742196+0000","last_change":"2024-04-25T15:27:46.742196+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742196+0000","last_undegraded":"2024-04-25T15:27:46.742196+0000","last_fullsized":"2024-04-25T15:27:46.742196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755354+0000","last_change":"2024-04-25T15:27:46.748183+0000","last_active":"2024-04-25T15:27:46.755354+0000","last_peered":"2024-04-25T15:27:46.755354+0000","last_clean":"2024-04-25T15:27:46.755354+0000","last_became_active":"2024-04-25T15:27:46.747854+0000","last_became_peered":"2024-04-25T15:27:46.747854+0000","last_unstale":"2024-04-25T15:27:46.755354+0000","last_undegraded":"2024-04-25T15:27:46.755354+0000","last_fullsized":"2024-04-25T15:27:46.755354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:52.385229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741214+0000","last_change":"2024-04-25T15:27:46.741214+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741214+0000","last_undegraded":"2024-04-25T15:27:46.741214+0000","last_fullsized":"2024-04-25T15:27:46.741214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753483+0000","last_change":"2024-04-25T15:27:46.750527+0000","last_active":"2024-04-25T15:27:46.753483+0000","last_peered":"2024-04-25T15:27:46.753483+0000","last_clean":"2024-04-25T15:27:46.753483+0000","last_became_active":"2024-04-25T15:27:46.749838+0000","last_became_peered":"2024-04-25T15:27:46.749838+0000","last_unstale":"2024-04-25T15:27:46.753483+0000","last_undegraded":"2024-04-25T15:27:46.753483+0000","last_fullsized":"2024-04-25T15:27:46.753483+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:44.420376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742915+0000","last_change":"2024-04-25T15:27:46.742915+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742915+0000","last_undegraded":"2024-04-25T15:27:46.742915+0000","last_fullsized":"2024-04-25T15:27:46.742915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752841+0000","last_change":"2024-04-25T15:27:46.747175+0000","last_active":"2024-04-25T15:27:46.752841+0000","last_peered":"2024-04-25T15:27:46.752841+0000","last_clean":"2024-04-25T15:27:46.752841+0000","last_became_active":"2024-04-25T15:27:46.746878+0000","last_became_peered":"2024-04-25T15:27:46.746878+0000","last_unstale":"2024-04-25T15:27:46.752841+0000","last_undegraded":"2024-04-25T15:27:46.752841+0000","last_fullsized":"2024-04-25T15:27:46.752841+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:42.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738688+0000","last_change":"2024-04-25T15:27:46.738688+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738688+0000","last_undegraded":"2024-04-25T15:27:46.738688+0000","last_fullsized":"2024-04-25T15:27:46.738688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751539+0000","last_change":"2024-04-25T15:27:46.748931+0000","last_active":"2024-04-25T15:27:46.751539+0000","last_peered":"2024-04-25T15:27:46.751539+0000","last_clean":"2024-04-25T15:27:46.751539+0000","last_became_active":"2024-04-25T15:27:46.748400+0000","last_became_peered":"2024-04-25T15:27:46.748400+0000","last_unstale":"2024-04-25T15:27:46.751539+0000","last_undegraded":"2024-04-25T15:27:46.751539+0000","last_fullsized":"2024-04-25T15:27:46.751539+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:32.140220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735434+0000","last_change":"2024-04-25T15:27:46.735434+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735434+0000","last_undegraded":"2024-04-25T15:27:46.735434+0000","last_fullsized":"2024-04-25T15:27:46.735434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.756042+0000","last_change":"2024-04-25T15:27:46.753325+0000","last_active":"2024-04-25T15:27:46.756042+0000","last_peered":"2024-04-25T15:27:46.756042+0000","last_clean":"2024-04-25T15:27:46.756042+0000","last_became_active":"2024-04-25T15:27:46.753006+0000","last_became_peered":"2024-04-25T15:27:46.753006+0000","last_unstale":"2024-04-25T15:27:46.756042+0000","last_undegraded":"2024-04-25T15:27:46.756042+0000","last_fullsized":"2024-04-25T15:27:46.756042+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:16.894533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744277+0000","last_change":"2024-04-25T15:27:46.744277+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744277+0000","last_undegraded":"2024-04-25T15:27:46.744277+0000","last_fullsized":"2024-04-25T15:27:46.744277+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.742751+0000","last_change":"2024-04-25T15:27:46.740274+0000","last_active":"2024-04-25T15:27:46.742751+0000","last_peered":"2024-04-25T15:27:46.742751+0000","last_clean":"2024-04-25T15:27:46.742751+0000","last_became_active":"2024-04-25T15:27:46.739275+0000","last_became_peered":"2024-04-25T15:27:46.739275+0000","last_unstale":"2024-04-25T15:27:46.742751+0000","last_undegraded":"2024-04-25T15:27:46.742751+0000","last_fullsized":"2024-04-25T15:27:46.742751+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:43.588775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734472+0000","last_change":"2024-04-25T15:27:46.734472+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734472+0000","last_undegraded":"2024-04-25T15:27:46.734472+0000","last_fullsized":"2024-04-25T15:27:46.734472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755355+0000","last_change":"2024-04-25T15:27:46.748972+0000","last_active":"2024-04-25T15:27:46.755355+0000","last_peered":"2024-04-25T15:27:46.755355+0000","last_clean":"2024-04-25T15:27:46.755355+0000","last_became_active":"2024-04-25T15:27:46.748630+0000","last_became_peered":"2024-04-25T15:27:46.748630+0000","last_unstale":"2024-04-25T15:27:46.755355+0000","last_undegraded":"2024-04-25T15:27:46.755355+0000","last_fullsized":"2024-04-25T15:27:46.755355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:01.865516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739213+0000","last_change":"2024-04-25T15:27:46.739213+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739213+0000","last_undegraded":"2024-04-25T15:27:46.739213+0000","last_fullsized":"2024-04-25T15:27:46.739213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757922+0000","last_change":"2024-04-25T15:27:46.753409+0000","last_active":"2024-04-25T15:27:46.757922+0000","last_peered":"2024-04-25T15:27:46.757922+0000","last_clean":"2024-04-25T15:27:46.757922+0000","last_became_active":"2024-04-25T15:27:46.753090+0000","last_became_peered":"2024-04-25T15:27:46.753090+0000","last_unstale":"2024-04-25T15:27:46.757922+0000","last_undegraded":"2024-04-25T15:27:46.757922+0000","last_fullsized":"2024-04-25T15:27:46.757922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:23.117643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741658+0000","last_change":"2024-04-25T15:27:46.741658+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741658+0000","last_undegraded":"2024-04-25T15:27:46.741658+0000","last_fullsized":"2024-04-25T15:27:46.741658+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755265+0000","last_change":"2024-04-25T15:27:46.751600+0000","last_active":"2024-04-25T15:27:46.755265+0000","last_peered":"2024-04-25T15:27:46.755265+0000","last_clean":"2024-04-25T15:27:46.755265+0000","last_became_active":"2024-04-25T15:27:46.751255+0000","last_became_peered":"2024-04-25T15:27:46.751255+0000","last_unstale":"2024-04-25T15:27:46.755265+0000","last_undegraded":"2024-04-25T15:27:46.755265+0000","last_fullsized":"2024-04-25T15:27:46.755265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.537162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740683+0000","last_change":"2024-04-25T15:27:46.740683+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740683+0000","last_undegraded":"2024-04-25T15:27:46.740683+0000","last_fullsized":"2024-04-25T15:27:46.740683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752242+0000","last_change":"2024-04-25T15:27:46.750192+0000","last_active":"2024-04-25T15:27:46.752242+0000","last_peered":"2024-04-25T15:27:46.752242+0000","last_clean":"2024-04-25T15:27:46.752242+0000","last_became_active":"2024-04-25T15:27:46.749859+0000","last_became_peered":"2024-04-25T15:27:46.749859+0000","last_unstale":"2024-04-25T15:27:46.752242+0000","last_undegraded":"2024-04-25T15:27:46.752242+0000","last_fullsized":"2024-04-25T15:27:46.752242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.634184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740931+0000","last_change":"2024-04-25T15:27:46.740931+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740931+0000","last_undegraded":"2024-04-25T15:27:46.740931+0000","last_fullsized":"2024-04-25T15:27:46.740931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.754580+0000","last_change":"2024-04-25T15:27:46.751593+0000","last_active":"2024-04-25T15:27:46.754580+0000","last_peered":"2024-04-25T15:27:46.754580+0000","last_clean":"2024-04-25T15:27:46.754580+0000","last_became_active":"2024-04-25T15:27:46.751123+0000","last_became_peered":"2024-04-25T15:27:46.751123+0000","last_unstale":"2024-04-25T15:27:46.754580+0000","last_undegraded":"2024-04-25T15:27:46.754580+0000","last_fullsized":"2024-04-25T15:27:46.754580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:51.896417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740179+0000","last_change":"2024-04-25T15:27:46.740179+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740179+0000","last_undegraded":"2024-04-25T15:27:46.740179+0000","last_fullsized":"2024-04-25T15:27:46.740179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751660+0000","last_change":"2024-04-25T15:27:46.747059+0000","last_active":"2024-04-25T15:27:46.751660+0000","last_peered":"2024-04-25T15:27:46.751660+0000","last_clean":"2024-04-25T15:27:46.751660+0000","last_became_active":"2024-04-25T15:27:46.746774+0000","last_became_peered":"2024-04-25T15:27:46.746774+0000","last_unstale":"2024-04-25T15:27:46.751660+0000","last_undegraded":"2024-04-25T15:27:46.751660+0000","last_fullsized":"2024-04-25T15:27:46.751660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:54.994195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743383+0000","last_change":"2024-04-25T15:27:46.743383+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743383+0000","last_undegraded":"2024-04-25T15:27:46.743383+0000","last_fullsized":"2024-04-25T15:27:46.743383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760439+0000","last_change":"2024-04-25T15:27:46.755998+0000","last_active":"2024-04-25T15:27:46.760439+0000","last_peered":"2024-04-25T15:27:46.760439+0000","last_clean":"2024-04-25T15:27:46.760439+0000","last_became_active":"2024-04-25T15:27:46.755762+0000","last_became_peered":"2024-04-25T15:27:46.755762+0000","last_unstale":"2024-04-25T15:27:46.760439+0000","last_undegraded":"2024-04-25T15:27:46.760439+0000","last_fullsized":"2024-04-25T15:27:46.760439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:43.588775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734455+0000","last_change":"2024-04-25T15:27:46.734455+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734455+0000","last_undegraded":"2024-04-25T15:27:46.734455+0000","last_fullsized":"2024-04-25T15:27:46.734455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755722+0000","last_change":"2024-04-25T15:27:46.753451+0000","last_active":"2024-04-25T15:27:46.755722+0000","last_peered":"2024-04-25T15:27:46.755722+0000","last_clean":"2024-04-25T15:27:46.755722+0000","last_became_active":"2024-04-25T15:27:46.753133+0000","last_became_peered":"2024-04-25T15:27:46.753133+0000","last_unstale":"2024-04-25T15:27:46.755722+0000","last_undegraded":"2024-04-25T15:27:46.755722+0000","last_fullsized":"2024-04-25T15:27:46.755722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:02.482063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743730+0000","last_change":"2024-04-25T15:27:46.743730+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743730+0000","last_undegraded":"2024-04-25T15:27:46.743730+0000","last_fullsized":"2024-04-25T15:27:46.743730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752208+0000","last_change":"2024-04-25T15:27:46.747236+0000","last_active":"2024-04-25T15:27:46.752208+0000","last_peered":"2024-04-25T15:27:46.752208+0000","last_clean":"2024-04-25T15:27:46.752208+0000","last_became_active":"2024-04-25T15:27:46.746401+0000","last_became_peered":"2024-04-25T15:27:46.746401+0000","last_unstale":"2024-04-25T15:27:46.752208+0000","last_undegraded":"2024-04-25T15:27:46.752208+0000","last_fullsized":"2024-04-25T15:27:46.752208+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:25.199824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737357+0000","last_change":"2024-04-25T15:27:46.737357+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737357+0000","last_undegraded":"2024-04-25T15:27:46.737357+0000","last_fullsized":"2024-04-25T15:27:46.737357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760768+0000","last_change":"2024-04-25T15:27:46.759114+0000","last_active":"2024-04-25T15:27:46.760768+0000","last_peered":"2024-04-25T15:27:46.760768+0000","last_clean":"2024-04-25T15:27:46.760768+0000","last_became_active":"2024-04-25T15:27:46.758785+0000","last_became_peered":"2024-04-25T15:27:46.758785+0000","last_unstale":"2024-04-25T15:27:46.760768+0000","last_undegraded":"2024-04-25T15:27:46.760768+0000","last_fullsized":"2024-04-25T15:27:46.760768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:51.896417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740337+0000","last_change":"2024-04-25T15:27:46.740337+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740337+0000","last_undegraded":"2024-04-25T15:27:46.740337+0000","last_fullsized":"2024-04-25T15:27:46.740337+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753377+0000","last_change":"2024-04-25T15:27:46.750313+0000","last_active":"2024-04-25T15:27:46.753377+0000","last_peered":"2024-04-25T15:27:46.753377+0000","last_clean":"2024-04-25T15:27:46.753377+0000","last_became_active":"2024-04-25T15:27:46.749988+0000","last_became_peered":"2024-04-25T15:27:46.749988+0000","last_unstale":"2024-04-25T15:27:46.753377+0000","last_undegraded":"2024-04-25T15:27:46.753377+0000","last_fullsized":"2024-04-25T15:27:46.753377+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:06.332293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737978+0000","last_change":"2024-04-25T15:27:46.737978+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737978+0000","last_undegraded":"2024-04-25T15:27:46.737978+0000","last_fullsized":"2024-04-25T15:27:46.737978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757737+0000","last_change":"2024-04-25T15:27:46.754470+0000","last_active":"2024-04-25T15:27:46.757737+0000","last_peered":"2024-04-25T15:27:46.757737+0000","last_clean":"2024-04-25T15:27:46.757737+0000","last_became_active":"2024-04-25T15:27:46.753975+0000","last_became_peered":"2024-04-25T15:27:46.753975+0000","last_unstale":"2024-04-25T15:27:46.757737+0000","last_undegraded":"2024-04-25T15:27:46.757737+0000","last_fullsized":"2024-04-25T15:27:46.757737+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:41.565463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740377+0000","last_change":"2024-04-25T15:27:46.740377+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740377+0000","last_undegraded":"2024-04-25T15:27:46.740377+0000","last_fullsized":"2024-04-25T15:27:46.740377+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757718+0000","last_change":"2024-04-25T15:27:46.754424+0000","last_active":"2024-04-25T15:27:46.757718+0000","last_peered":"2024-04-25T15:27:46.757718+0000","last_clean":"2024-04-25T15:27:46.757718+0000","last_became_active":"2024-04-25T15:27:46.754102+0000","last_became_peered":"2024-04-25T15:27:46.754102+0000","last_unstale":"2024-04-25T15:27:46.757718+0000","last_undegraded":"2024-04-25T15:27:46.757718+0000","last_fullsized":"2024-04-25T15:27:46.757718+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:54.809375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742041+0000","last_change":"2024-04-25T15:27:46.742041+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742041+0000","last_undegraded":"2024-04-25T15:27:46.742041+0000","last_fullsized":"2024-04-25T15:27:46.742041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760460+0000","last_change":"2024-04-25T15:27:46.756182+0000","last_active":"2024-04-25T15:27:46.760460+0000","last_peered":"2024-04-25T15:27:46.760460+0000","last_clean":"2024-04-25T15:27:46.760460+0000","last_became_active":"2024-04-25T15:27:46.755789+0000","last_became_peered":"2024-04-25T15:27:46.755789+0000","last_unstale":"2024-04-25T15:27:46.760460+0000","last_undegraded":"2024-04-25T15:27:46.760460+0000","last_fullsized":"2024-04-25T15:27:46.760460+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.454298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743258+0000","last_change":"2024-04-25T15:27:46.743258+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743258+0000","last_undegraded":"2024-04-25T15:27:46.743258+0000","last_fullsized":"2024-04-25T15:27:46.743258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753687+0000","last_change":"2024-04-25T15:27:46.751621+0000","last_active":"2024-04-25T15:27:46.753687+0000","last_peered":"2024-04-25T15:27:46.753687+0000","last_clean":"2024-04-25T15:27:46.753687+0000","last_became_active":"2024-04-25T15:27:46.751328+0000","last_became_peered":"2024-04-25T15:27:46.751328+0000","last_unstale":"2024-04-25T15:27:46.753687+0000","last_undegraded":"2024-04-25T15:27:46.753687+0000","last_fullsized":"2024-04-25T15:27:46.753687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:07:28.385465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740778+0000","last_change":"2024-04-25T15:27:46.740778+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740778+0000","last_undegraded":"2024-04-25T15:27:46.740778+0000","last_fullsized":"2024-04-25T15:27:46.740778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755225+0000","last_change":"2024-04-25T15:27:46.748665+0000","last_active":"2024-04-25T15:27:46.755225+0000","last_peered":"2024-04-25T15:27:46.755225+0000","last_clean":"2024-04-25T15:27:46.755225+0000","last_became_active":"2024-04-25T15:27:46.748364+0000","last_became_peered":"2024-04-25T15:27:46.748364+0000","last_unstale":"2024-04-25T15:27:46.755225+0000","last_undegraded":"2024-04-25T15:27:46.755225+0000","last_fullsized":"2024-04-25T15:27:46.755225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:34.541226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734502+0000","last_change":"2024-04-25T15:27:46.734502+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734502+0000","last_undegraded":"2024-04-25T15:27:46.734502+0000","last_fullsized":"2024-04-25T15:27:46.734502+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760752+0000","last_change":"2024-04-25T15:27:46.759084+0000","last_active":"2024-04-25T15:27:46.760752+0000","last_peered":"2024-04-25T15:27:46.760752+0000","last_clean":"2024-04-25T15:27:46.760752+0000","last_became_active":"2024-04-25T15:27:46.758757+0000","last_became_peered":"2024-04-25T15:27:46.758757+0000","last_unstale":"2024-04-25T15:27:46.760752+0000","last_undegraded":"2024-04-25T15:27:46.760752+0000","last_fullsized":"2024-04-25T15:27:46.760752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:54.994195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739729+0000","last_change":"2024-04-25T15:27:46.739729+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739729+0000","last_undegraded":"2024-04-25T15:27:46.739729+0000","last_fullsized":"2024-04-25T15:27:46.739729+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755747+0000","last_change":"2024-04-25T15:27:46.753206+0000","last_active":"2024-04-25T15:27:46.755747+0000","last_peered":"2024-04-25T15:27:46.755747+0000","last_clean":"2024-04-25T15:27:46.755747+0000","last_became_active":"2024-04-25T15:27:46.752883+0000","last_became_peered":"2024-04-25T15:27:46.752883+0000","last_unstale":"2024-04-25T15:27:46.755747+0000","last_undegraded":"2024-04-25T15:27:46.755747+0000","last_fullsized":"2024-04-25T15:27:46.755747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:55.623358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.745243+0000","last_change":"2024-04-25T15:27:46.745243+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.745243+0000","last_undegraded":"2024-04-25T15:27:46.745243+0000","last_fullsized":"2024-04-25T15:27:46.745243+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755764+0000","last_change":"2024-04-25T15:27:46.753592+0000","last_active":"2024-04-25T15:27:46.755764+0000","last_peered":"2024-04-25T15:27:46.755764+0000","last_clean":"2024-04-25T15:27:46.755764+0000","last_became_active":"2024-04-25T15:27:46.753163+0000","last_became_peered":"2024-04-25T15:27:46.753163+0000","last_unstale":"2024-04-25T15:27:46.755764+0000","last_undegraded":"2024-04-25T15:27:46.755764+0000","last_fullsized":"2024-04-25T15:27:46.755764+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:19.025834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742091+0000","last_change":"2024-04-25T15:27:46.742091+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742091+0000","last_undegraded":"2024-04-25T15:27:46.742091+0000","last_fullsized":"2024-04-25T15:27:46.742091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752245+0000","last_change":"2024-04-25T15:27:46.744686+0000","last_active":"2024-04-25T15:27:46.752245+0000","last_peered":"2024-04-25T15:27:46.752245+0000","last_clean":"2024-04-25T15:27:46.752245+0000","last_became_active":"2024-04-25T15:27:46.744321+0000","last_became_peered":"2024-04-25T15:27:46.744321+0000","last_unstale":"2024-04-25T15:27:46.752245+0000","last_undegraded":"2024-04-25T15:27:46.752245+0000","last_fullsized":"2024-04-25T15:27:46.752245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:51.235314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739940+0000","last_change":"2024-04-25T15:27:46.739940+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739940+0000","last_undegraded":"2024-04-25T15:27:46.739940+0000","last_fullsized":"2024-04-25T15:27:46.739940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755161+0000","last_change":"2024-04-25T15:27:46.749552+0000","last_active":"2024-04-25T15:27:46.755161+0000","last_peered":"2024-04-25T15:27:46.755161+0000","last_clean":"2024-04-25T15:27:46.755161+0000","last_became_active":"2024-04-25T15:27:46.749242+0000","last_became_peered":"2024-04-25T15:27:46.749242+0000","last_unstale":"2024-04-25T15:27:46.755161+0000","last_undegraded":"2024-04-25T15:27:46.755161+0000","last_fullsized":"2024-04-25T15:27:46.755161+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:57:28.326266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743195+0000","last_change":"2024-04-25T15:27:46.743195+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743195+0000","last_undegraded":"2024-04-25T15:27:46.743195+0000","last_fullsized":"2024-04-25T15:27:46.743195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751802+0000","last_change":"2024-04-25T15:27:46.746473+0000","last_active":"2024-04-25T15:27:46.751802+0000","last_peered":"2024-04-25T15:27:46.751802+0000","last_clean":"2024-04-25T15:27:46.751802+0000","last_became_active":"2024-04-25T15:27:46.746083+0000","last_became_peered":"2024-04-25T15:27:46.746083+0000","last_unstale":"2024-04-25T15:27:46.751802+0000","last_undegraded":"2024-04-25T15:27:46.751802+0000","last_fullsized":"2024-04-25T15:27:46.751802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.454298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737498+0000","last_change":"2024-04-25T15:27:46.737498+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737498+0000","last_undegraded":"2024-04-25T15:27:46.737498+0000","last_fullsized":"2024-04-25T15:27:46.737498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752198+0000","last_change":"2024-04-25T15:27:46.745918+0000","last_active":"2024-04-25T15:27:46.752198+0000","last_peered":"2024-04-25T15:27:46.752198+0000","last_clean":"2024-04-25T15:27:46.752198+0000","last_became_active":"2024-04-25T15:27:46.745596+0000","last_became_peered":"2024-04-25T15:27:46.745596+0000","last_unstale":"2024-04-25T15:27:46.752198+0000","last_undegraded":"2024-04-25T15:27:46.752198+0000","last_fullsized":"2024-04-25T15:27:46.752198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:37.598034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739515+0000","last_change":"2024-04-25T15:27:46.739515+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739515+0000","last_undegraded":"2024-04-25T15:27:46.739515+0000","last_fullsized":"2024-04-25T15:27:46.739515+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.749424+0000","last_change":"2024-04-25T15:27:46.746186+0000","last_active":"2024-04-25T15:27:46.749424+0000","last_peered":"2024-04-25T15:27:46.749424+0000","last_clean":"2024-04-25T15:27:46.749424+0000","last_became_active":"2024-04-25T15:27:46.745867+0000","last_became_peered":"2024-04-25T15:27:46.745867+0000","last_unstale":"2024-04-25T15:27:46.749424+0000","last_undegraded":"2024-04-25T15:27:46.749424+0000","last_fullsized":"2024-04-25T15:27:46.749424+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.445845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741940+0000","last_change":"2024-04-25T15:27:46.741940+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741940+0000","last_undegraded":"2024-04-25T15:27:46.741940+0000","last_fullsized":"2024-04-25T15:27:46.741940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760410+0000","last_change":"2024-04-25T15:27:46.755483+0000","last_active":"2024-04-25T15:27:46.760410+0000","last_peered":"2024-04-25T15:27:46.760410+0000","last_clean":"2024-04-25T15:27:46.760410+0000","last_became_active":"2024-04-25T15:27:46.755160+0000","last_became_peered":"2024-04-25T15:27:46.755160+0000","last_unstale":"2024-04-25T15:27:46.760410+0000","last_undegraded":"2024-04-25T15:27:46.760410+0000","last_fullsized":"2024-04-25T15:27:46.760410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:40:46.657005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":191,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.743997+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:27:46.743997+0000","last_peered":"2024-04-25T15:27:46.743997+0000","last_clean":"2024-04-25T15:27:46.743997+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:27:46.743997+0000","last_undegraded":"2024-04-25T15:27:46.743997+0000","last_fullsized":"2024-04-25T15:27:46.743997+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343472,"statfs":{"total":96636764160,"available":96607715328,"internally_reserved":0,"allocated":1372160,"data_stored":912533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":328440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28376,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343464,"statfs":{"total":96636764160,"available":96607707136,"internally_reserved":0,"allocated":1372160,"data_stored":912533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:27:52.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:27:52.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.351+0000 7f1275625640 1 Processor -- start 2024-04-25T15:27:52.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.351+0000 7f1275625640 1 -- start start 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f1275625640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1270164b00 0x7f12700b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f1275625640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 0x7f12700b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f1275625640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12701615f0 con 0x7f127015df80 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f1275625640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f12701613b0 con 0x7f1270164b00 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f1275625640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1270164670 con 0x7f1270160fa0 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126e7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f127015df80 0x7f1270160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55620/0 (socket says 172.21.15.67:55620) 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126e7fc640 1 -- 172.21.15.67:0/3278816565 learned_addr learned my addr 172.21.15.67:0/3278816565 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3239405033 0 0) 0x7f12701615f0 con 0x7f127015df80 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126effd640 1 --2- 172.21.15.67:0/3278816565 >> v2:172.21.15.176:3300/0 conn(0x7f1270164b00 0x7f12700b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f125c003520 con 0x7f127015df80 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126f7fe640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 0x7f12700b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1681560600 0 0) 0x7f125c003520 con 0x7f127015df80 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 msgr2=0x7f12700b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 0x7f12700b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 >> v2:172.21.15.176:3300/0 conn(0x7f1270164b00 msgr2=0x7f12700b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 --2- 172.21.15.67:0/3278816565 >> v2:172.21.15.176:3300/0 conn(0x7f1270164b00 0x7f12700b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1270164020 con 0x7f127015df80 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f1258001670 con 0x7f127015df80 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 >> v1:172.21.15.67:6789/0 conn(0x7f127015df80 legacy=0x7f1270160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 0x7f125c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c003f30 0x7f125c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c004d40 0x7f125c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12701613b0 con 0x7f125c003f30 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1270164670 con 0x7f125c004d40 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.353+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f125c00ba10 con 0x7f1270160fa0 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126e7fc640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c003f30 0x7f125c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126effd640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 0x7f125c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126f7fe640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c004d40 0x7f125c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126e7fc640 1 -- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 msgr2=0x7f125c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126e7fc640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1270160fa0 0x7f125c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126e7fc640 1 -- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c004d40 msgr2=0x7f125c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126e7fc640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c004d40 0x7f125c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126e7fc640 1 -- 172.21.15.67:0/3278816565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c003520 con 0x7f125c003f30 2024-04-25T15:27:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126f7fe640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c004d40 0x7f125c005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.354+0000 7f126e7fc640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c003f30 0x7f125c004800 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1258001eb0 tx=0x7f125c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8997d7f07966d5b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258007d70 con 0x7f125c003f30 2024-04-25T15:27:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1258005af0 con 0x7f125c003f30 2024-04-25T15:27:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f126dffb640 1 -- 172.21.15.67:0/3278816565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258007070 con 0x7f125c003f30 2024-04-25T15:27:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f1275625640 1 -- 172.21.15.67:0/3278816565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c003f30 msgr2=0x7f125c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f1275625640 1 --2- 172.21.15.67:0/3278816565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f125c003f30 0x7f125c004800 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1258001eb0 tx=0x7f125c005830 comp rx=0 tx=0).stop 2024-04-25T15:27:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f126f7fe640 1 -- 172.21.15.67:0/3278816565 reap_dead start 2024-04-25T15:27:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f1275625640 1 -- 172.21.15.67:0/3278816565 shutdown_connections 2024-04-25T15:27:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f1275625640 1 -- 172.21.15.67:0/3278816565 >> 172.21.15.67:0/3278816565 conn(0x7f12700d2bc0 msgr2=0x7f12700d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f1275625640 1 -- 172.21.15.67:0/3278816565 shutdown_connections 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.355+0000 7f1275625640 1 -- 172.21.15.67:0/3278816565 wait complete. 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.357+0000 7f1275625640 1 Processor -- start 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.357+0000 7f1275625640 1 -- start start 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c003f30 0x7f12700cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12700ac700 0x7f12701f94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f127015e390 0x7f127015e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12700b8240 con 0x7f12700ac700 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f12700b80c0 con 0x7f125c003f30 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f12700b83c0 con 0x7f127015e390 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12700ac700 0x7f12701f94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12700ac700 0x7f12701f94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35210/0 (socket says 172.21.15.67:35210) 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 -- 172.21.15.67:0/1510923463 learned_addr learned my addr 172.21.15.67:0/1510923463 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 -- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f127015e390 msgr2=0x7f127015e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126f7fe640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f127015e390 0x7f127015e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126effd640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c003f30 0x7f12700cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f127015e390 0x7f127015e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 -- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c003f30 msgr2=0x7f12700cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c003f30 0x7f12700cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 -- 172.21.15.67:0/1510923463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127015ece0 con 0x7f12700ac700 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126f7fe640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f127015e390 0x7f127015e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126effd640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c003f30 0x7f12700cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f126e7fc640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12700ac700 0x7f12701f94e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f125802ae20 tx=0x7f1258000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f12677fe640 1 -- 172.21.15.67:0/1510923463 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258042070 con 0x7f12700ac700 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f12677fe640 1 -- 172.21.15.67:0/1510923463 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1258007480 con 0x7f12700ac700 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12700cd390 con 0x7f12700ac700 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f12677fe640 1 -- 172.21.15.67:0/1510923463 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258007d50 con 0x7f12700ac700 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.359+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12700cd950 con 0x7f12700ac700 2024-04-25T15:27:52.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.360+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12700c9530 con 0x7f12700ac700 2024-04-25T15:27:52.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.361+0000 7f12677fe640 1 -- 172.21.15.67:0/1510923463 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f125803b050 con 0x7f12700ac700 2024-04-25T15:27:52.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.362+0000 7f12677fe640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f12500744b0 0x7f12500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.362+0000 7f12677fe640 1 -- 172.21.15.67:0/1510923463 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f12580b5ab0 con 0x7f12700ac700 2024-04-25T15:27:52.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.362+0000 7f126effd640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f12500744b0 0x7f12500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.362+0000 7f126effd640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f12500744b0 0x7f12500769c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1268009b30 tx=0x7f1268008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:52.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.366+0000 7f12677fe640 1 -- 172.21.15.67:0/1510923463 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f127015ece0 con 0x7f12700ac700 2024-04-25T15:27:52.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.547+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1270050740 con 0x7f12500744b0 2024-04-25T15:27:52.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.554+0000 7f12677fe640 1 -- 172.21.15.67:0/1510923463 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246737 (secure 0 0 0) 0x7f1270050740 con 0x7f12500744b0 2024-04-25T15:27:52.557 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:27:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f12500744b0 msgr2=0x7f12500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f12500744b0 0x7f12500769c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1268009b30 tx=0x7f1268008040 comp rx=0 tx=0).stop 2024-04-25T15:27:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12700ac700 msgr2=0x7f12701f94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12700ac700 0x7f12701f94e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f125802ae20 tx=0x7f1258000fe0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 shutdown_connections 2024-04-25T15:27:52.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f12500744b0 0x7f12500769c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f127015e390 0x7f127015e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12700ac700 0x7f12701f94e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 --2- 172.21.15.67:0/1510923463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f125c003f30 0x7f12700cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 >> 172.21.15.67:0/1510923463 conn(0x7f12700d2bc0 msgr2=0x7f12700d5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:52.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 shutdown_connections 2024-04-25T15:27:52.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.557+0000 7f1275625640 1 -- 172.21.15.67:0/1510923463 wait complete. 2024-04-25T15:27:52.569 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-04-25T15:27:50.829478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223456,"kb_used_data":7252,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751264,"statfs":{"total":773094113280,"available":772865294336,"internally_reserved":0,"allocated":7426048,"data_stored":3764331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.099960"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742158+0000","last_change":"2024-04-25T15:27:46.742158+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742158+0000","last_undegraded":"2024-04-25T15:27:46.742158+0000","last_fullsized":"2024-04-25T15:27:46.742158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.746577+0000","last_change":"2024-04-25T15:27:46.746577+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.746577+0000","last_undegraded":"2024-04-25T15:27:46.746577+0000","last_fullsized":"2024-04-25T15:27:46.746577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741554+0000","last_change":"2024-04-25T15:27:46.741554+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741554+0000","last_undegraded":"2024-04-25T15:27:46.741554+0000","last_fullsized":"2024-04-25T15:27:46.741554+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739391+0000","last_change":"2024-04-25T15:27:46.739391+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739391+0000","last_undegraded":"2024-04-25T15:27:46.739391+0000","last_fullsized":"2024-04-25T15:27:46.739391+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741588+0000","last_change":"2024-04-25T15:27:46.741588+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741588+0000","last_undegraded":"2024-04-25T15:27:46.741588+0000","last_fullsized":"2024-04-25T15:27:46.741588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744353+0000","last_change":"2024-04-25T15:27:46.744353+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744353+0000","last_undegraded":"2024-04-25T15:27:46.744353+0000","last_fullsized":"2024-04-25T15:27:46.744353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741489+0000","last_change":"2024-04-25T15:27:46.741489+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741489+0000","last_undegraded":"2024-04-25T15:27:46.741489+0000","last_fullsized":"2024-04-25T15:27:46.741489+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741504+0000","last_change":"2024-04-25T15:27:46.741504+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741504+0000","last_undegraded":"2024-04-25T15:27:46.741504+0000","last_fullsized":"2024-04-25T15:27:46.741504+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743205+0000","last_change":"2024-04-25T15:27:46.743205+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743205+0000","last_undegraded":"2024-04-25T15:27:46.743205+0000","last_fullsized":"2024-04-25T15:27:46.743205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737954+0000","last_change":"2024-04-25T15:27:46.737954+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737954+0000","last_undegraded":"2024-04-25T15:27:46.737954+0000","last_fullsized":"2024-04-25T15:27:46.737954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743381+0000","last_change":"2024-04-25T15:27:46.743381+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743381+0000","last_undegraded":"2024-04-25T15:27:46.743381+0000","last_fullsized":"2024-04-25T15:27:46.743381+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738618+0000","last_change":"2024-04-25T15:27:46.738618+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738618+0000","last_undegraded":"2024-04-25T15:27:46.738618+0000","last_fullsized":"2024-04-25T15:27:46.738618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734472+0000","last_change":"2024-04-25T15:27:46.734472+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734472+0000","last_undegraded":"2024-04-25T15:27:46.734472+0000","last_fullsized":"2024-04-25T15:27:46.734472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740602+0000","last_change":"2024-04-25T15:27:46.740602+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740602+0000","last_undegraded":"2024-04-25T15:27:46.740602+0000","last_fullsized":"2024-04-25T15:27:46.740602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735281+0000","last_change":"2024-04-25T15:27:46.735281+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735281+0000","last_undegraded":"2024-04-25T15:27:46.735281+0000","last_fullsized":"2024-04-25T15:27:46.735281+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735174+0000","last_change":"2024-04-25T15:27:46.735174+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735174+0000","last_undegraded":"2024-04-25T15:27:46.735174+0000","last_fullsized":"2024-04-25T15:27:46.735174+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742694+0000","last_change":"2024-04-25T15:27:46.742694+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742694+0000","last_undegraded":"2024-04-25T15:27:46.742694+0000","last_fullsized":"2024-04-25T15:27:46.742694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738105+0000","last_change":"2024-04-25T15:27:46.738105+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738105+0000","last_undegraded":"2024-04-25T15:27:46.738105+0000","last_fullsized":"2024-04-25T15:27:46.738105+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742298+0000","last_change":"2024-04-25T15:27:46.742298+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742298+0000","last_undegraded":"2024-04-25T15:27:46.742298+0000","last_fullsized":"2024-04-25T15:27:46.742298+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734487+0000","last_change":"2024-04-25T15:27:46.734487+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734487+0000","last_undegraded":"2024-04-25T15:27:46.734487+0000","last_fullsized":"2024-04-25T15:27:46.734487+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.736254+0000","last_change":"2024-04-25T15:27:46.736254+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.736254+0000","last_undegraded":"2024-04-25T15:27:46.736254+0000","last_fullsized":"2024-04-25T15:27:46.736254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735290+0000","last_change":"2024-04-25T15:27:46.735290+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735290+0000","last_undegraded":"2024-04-25T15:27:46.735290+0000","last_fullsized":"2024-04-25T15:27:46.735290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740685+0000","last_change":"2024-04-25T15:27:46.740685+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740685+0000","last_undegraded":"2024-04-25T15:27:46.740685+0000","last_fullsized":"2024-04-25T15:27:46.740685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740976+0000","last_change":"2024-04-25T15:27:46.740976+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740976+0000","last_undegraded":"2024-04-25T15:27:46.740976+0000","last_fullsized":"2024-04-25T15:27:46.740976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737831+0000","last_change":"2024-04-25T15:27:46.737831+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737831+0000","last_undegraded":"2024-04-25T15:27:46.737831+0000","last_fullsized":"2024-04-25T15:27:46.737831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739762+0000","last_change":"2024-04-25T15:27:46.739762+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739762+0000","last_undegraded":"2024-04-25T15:27:46.739762+0000","last_fullsized":"2024-04-25T15:27:46.739762+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741068+0000","last_change":"2024-04-25T15:27:46.741068+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741068+0000","last_undegraded":"2024-04-25T15:27:46.741068+0000","last_fullsized":"2024-04-25T15:27:46.741068+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.736975+0000","last_change":"2024-04-25T15:27:46.736975+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.736975+0000","last_undegraded":"2024-04-25T15:27:46.736975+0000","last_fullsized":"2024-04-25T15:27:46.736975+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742531+0000","last_change":"2024-04-25T15:27:46.742531+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742531+0000","last_undegraded":"2024-04-25T15:27:46.742531+0000","last_fullsized":"2024-04-25T15:27:46.742531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744160+0000","last_change":"2024-04-25T15:27:46.744160+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744160+0000","last_undegraded":"2024-04-25T15:27:46.744160+0000","last_fullsized":"2024-04-25T15:27:46.744160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740651+0000","last_change":"2024-04-25T15:27:46.740651+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740651+0000","last_undegraded":"2024-04-25T15:27:46.740651+0000","last_fullsized":"2024-04-25T15:27:46.740651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740954+0000","last_change":"2024-04-25T15:27:46.740954+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740954+0000","last_undegraded":"2024-04-25T15:27:46.740954+0000","last_fullsized":"2024-04-25T15:27:46.740954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739785+0000","last_change":"2024-04-25T15:27:46.739785+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739785+0000","last_undegraded":"2024-04-25T15:27:46.739785+0000","last_fullsized":"2024-04-25T15:27:46.739785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760243+0000","last_change":"2024-04-25T15:27:46.756488+0000","last_active":"2024-04-25T15:27:46.760243+0000","last_peered":"2024-04-25T15:27:46.760243+0000","last_clean":"2024-04-25T15:27:46.760243+0000","last_became_active":"2024-04-25T15:27:46.756137+0000","last_became_peered":"2024-04-25T15:27:46.756137+0000","last_unstale":"2024-04-25T15:27:46.760243+0000","last_undegraded":"2024-04-25T15:27:46.760243+0000","last_fullsized":"2024-04-25T15:27:46.760243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:51.235314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742196+0000","last_change":"2024-04-25T15:27:46.742196+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742196+0000","last_undegraded":"2024-04-25T15:27:46.742196+0000","last_fullsized":"2024-04-25T15:27:46.742196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755354+0000","last_change":"2024-04-25T15:27:46.748183+0000","last_active":"2024-04-25T15:27:46.755354+0000","last_peered":"2024-04-25T15:27:46.755354+0000","last_clean":"2024-04-25T15:27:46.755354+0000","last_became_active":"2024-04-25T15:27:46.747854+0000","last_became_peered":"2024-04-25T15:27:46.747854+0000","last_unstale":"2024-04-25T15:27:46.755354+0000","last_undegraded":"2024-04-25T15:27:46.755354+0000","last_fullsized":"2024-04-25T15:27:46.755354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:52.385229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741214+0000","last_change":"2024-04-25T15:27:46.741214+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741214+0000","last_undegraded":"2024-04-25T15:27:46.741214+0000","last_fullsized":"2024-04-25T15:27:46.741214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753483+0000","last_change":"2024-04-25T15:27:46.750527+0000","last_active":"2024-04-25T15:27:46.753483+0000","last_peered":"2024-04-25T15:27:46.753483+0000","last_clean":"2024-04-25T15:27:46.753483+0000","last_became_active":"2024-04-25T15:27:46.749838+0000","last_became_peered":"2024-04-25T15:27:46.749838+0000","last_unstale":"2024-04-25T15:27:46.753483+0000","last_undegraded":"2024-04-25T15:27:46.753483+0000","last_fullsized":"2024-04-25T15:27:46.753483+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:44.420376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742915+0000","last_change":"2024-04-25T15:27:46.742915+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742915+0000","last_undegraded":"2024-04-25T15:27:46.742915+0000","last_fullsized":"2024-04-25T15:27:46.742915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752841+0000","last_change":"2024-04-25T15:27:46.747175+0000","last_active":"2024-04-25T15:27:46.752841+0000","last_peered":"2024-04-25T15:27:46.752841+0000","last_clean":"2024-04-25T15:27:46.752841+0000","last_became_active":"2024-04-25T15:27:46.746878+0000","last_became_peered":"2024-04-25T15:27:46.746878+0000","last_unstale":"2024-04-25T15:27:46.752841+0000","last_undegraded":"2024-04-25T15:27:46.752841+0000","last_fullsized":"2024-04-25T15:27:46.752841+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:42.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.738688+0000","last_change":"2024-04-25T15:27:46.738688+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.738688+0000","last_undegraded":"2024-04-25T15:27:46.738688+0000","last_fullsized":"2024-04-25T15:27:46.738688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751539+0000","last_change":"2024-04-25T15:27:46.748931+0000","last_active":"2024-04-25T15:27:46.751539+0000","last_peered":"2024-04-25T15:27:46.751539+0000","last_clean":"2024-04-25T15:27:46.751539+0000","last_became_active":"2024-04-25T15:27:46.748400+0000","last_became_peered":"2024-04-25T15:27:46.748400+0000","last_unstale":"2024-04-25T15:27:46.751539+0000","last_undegraded":"2024-04-25T15:27:46.751539+0000","last_fullsized":"2024-04-25T15:27:46.751539+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:32.140220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.735434+0000","last_change":"2024-04-25T15:27:46.735434+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.735434+0000","last_undegraded":"2024-04-25T15:27:46.735434+0000","last_fullsized":"2024-04-25T15:27:46.735434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.756042+0000","last_change":"2024-04-25T15:27:46.753325+0000","last_active":"2024-04-25T15:27:46.756042+0000","last_peered":"2024-04-25T15:27:46.756042+0000","last_clean":"2024-04-25T15:27:46.756042+0000","last_became_active":"2024-04-25T15:27:46.753006+0000","last_became_peered":"2024-04-25T15:27:46.753006+0000","last_unstale":"2024-04-25T15:27:46.756042+0000","last_undegraded":"2024-04-25T15:27:46.756042+0000","last_fullsized":"2024-04-25T15:27:46.756042+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:16.894533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.744277+0000","last_change":"2024-04-25T15:27:46.744277+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.744277+0000","last_undegraded":"2024-04-25T15:27:46.744277+0000","last_fullsized":"2024-04-25T15:27:46.744277+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.742751+0000","last_change":"2024-04-25T15:27:46.740274+0000","last_active":"2024-04-25T15:27:46.742751+0000","last_peered":"2024-04-25T15:27:46.742751+0000","last_clean":"2024-04-25T15:27:46.742751+0000","last_became_active":"2024-04-25T15:27:46.739275+0000","last_became_peered":"2024-04-25T15:27:46.739275+0000","last_unstale":"2024-04-25T15:27:46.742751+0000","last_undegraded":"2024-04-25T15:27:46.742751+0000","last_fullsized":"2024-04-25T15:27:46.742751+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:43.588775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734472+0000","last_change":"2024-04-25T15:27:46.734472+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734472+0000","last_undegraded":"2024-04-25T15:27:46.734472+0000","last_fullsized":"2024-04-25T15:27:46.734472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755355+0000","last_change":"2024-04-25T15:27:46.748972+0000","last_active":"2024-04-25T15:27:46.755355+0000","last_peered":"2024-04-25T15:27:46.755355+0000","last_clean":"2024-04-25T15:27:46.755355+0000","last_became_active":"2024-04-25T15:27:46.748630+0000","last_became_peered":"2024-04-25T15:27:46.748630+0000","last_unstale":"2024-04-25T15:27:46.755355+0000","last_undegraded":"2024-04-25T15:27:46.755355+0000","last_fullsized":"2024-04-25T15:27:46.755355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:01.865516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739213+0000","last_change":"2024-04-25T15:27:46.739213+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739213+0000","last_undegraded":"2024-04-25T15:27:46.739213+0000","last_fullsized":"2024-04-25T15:27:46.739213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757922+0000","last_change":"2024-04-25T15:27:46.753409+0000","last_active":"2024-04-25T15:27:46.757922+0000","last_peered":"2024-04-25T15:27:46.757922+0000","last_clean":"2024-04-25T15:27:46.757922+0000","last_became_active":"2024-04-25T15:27:46.753090+0000","last_became_peered":"2024-04-25T15:27:46.753090+0000","last_unstale":"2024-04-25T15:27:46.757922+0000","last_undegraded":"2024-04-25T15:27:46.757922+0000","last_fullsized":"2024-04-25T15:27:46.757922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:23.117643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741658+0000","last_change":"2024-04-25T15:27:46.741658+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741658+0000","last_undegraded":"2024-04-25T15:27:46.741658+0000","last_fullsized":"2024-04-25T15:27:46.741658+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755265+0000","last_change":"2024-04-25T15:27:46.751600+0000","last_active":"2024-04-25T15:27:46.755265+0000","last_peered":"2024-04-25T15:27:46.755265+0000","last_clean":"2024-04-25T15:27:46.755265+0000","last_became_active":"2024-04-25T15:27:46.751255+0000","last_became_peered":"2024-04-25T15:27:46.751255+0000","last_unstale":"2024-04-25T15:27:46.755265+0000","last_undegraded":"2024-04-25T15:27:46.755265+0000","last_fullsized":"2024-04-25T15:27:46.755265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.537162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740683+0000","last_change":"2024-04-25T15:27:46.740683+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740683+0000","last_undegraded":"2024-04-25T15:27:46.740683+0000","last_fullsized":"2024-04-25T15:27:46.740683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752242+0000","last_change":"2024-04-25T15:27:46.750192+0000","last_active":"2024-04-25T15:27:46.752242+0000","last_peered":"2024-04-25T15:27:46.752242+0000","last_clean":"2024-04-25T15:27:46.752242+0000","last_became_active":"2024-04-25T15:27:46.749859+0000","last_became_peered":"2024-04-25T15:27:46.749859+0000","last_unstale":"2024-04-25T15:27:46.752242+0000","last_undegraded":"2024-04-25T15:27:46.752242+0000","last_fullsized":"2024-04-25T15:27:46.752242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.634184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740931+0000","last_change":"2024-04-25T15:27:46.740931+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740931+0000","last_undegraded":"2024-04-25T15:27:46.740931+0000","last_fullsized":"2024-04-25T15:27:46.740931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.754580+0000","last_change":"2024-04-25T15:27:46.751593+0000","last_active":"2024-04-25T15:27:46.754580+0000","last_peered":"2024-04-25T15:27:46.754580+0000","last_clean":"2024-04-25T15:27:46.754580+0000","last_became_active":"2024-04-25T15:27:46.751123+0000","last_became_peered":"2024-04-25T15:27:46.751123+0000","last_unstale":"2024-04-25T15:27:46.754580+0000","last_undegraded":"2024-04-25T15:27:46.754580+0000","last_fullsized":"2024-04-25T15:27:46.754580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:51.896417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740179+0000","last_change":"2024-04-25T15:27:46.740179+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740179+0000","last_undegraded":"2024-04-25T15:27:46.740179+0000","last_fullsized":"2024-04-25T15:27:46.740179+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751660+0000","last_change":"2024-04-25T15:27:46.747059+0000","last_active":"2024-04-25T15:27:46.751660+0000","last_peered":"2024-04-25T15:27:46.751660+0000","last_clean":"2024-04-25T15:27:46.751660+0000","last_became_active":"2024-04-25T15:27:46.746774+0000","last_became_peered":"2024-04-25T15:27:46.746774+0000","last_unstale":"2024-04-25T15:27:46.751660+0000","last_undegraded":"2024-04-25T15:27:46.751660+0000","last_fullsized":"2024-04-25T15:27:46.751660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:54.994195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743383+0000","last_change":"2024-04-25T15:27:46.743383+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743383+0000","last_undegraded":"2024-04-25T15:27:46.743383+0000","last_fullsized":"2024-04-25T15:27:46.743383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760439+0000","last_change":"2024-04-25T15:27:46.755998+0000","last_active":"2024-04-25T15:27:46.760439+0000","last_peered":"2024-04-25T15:27:46.760439+0000","last_clean":"2024-04-25T15:27:46.760439+0000","last_became_active":"2024-04-25T15:27:46.755762+0000","last_became_peered":"2024-04-25T15:27:46.755762+0000","last_unstale":"2024-04-25T15:27:46.760439+0000","last_undegraded":"2024-04-25T15:27:46.760439+0000","last_fullsized":"2024-04-25T15:27:46.760439+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:43.588775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734455+0000","last_change":"2024-04-25T15:27:46.734455+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734455+0000","last_undegraded":"2024-04-25T15:27:46.734455+0000","last_fullsized":"2024-04-25T15:27:46.734455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755722+0000","last_change":"2024-04-25T15:27:46.753451+0000","last_active":"2024-04-25T15:27:46.755722+0000","last_peered":"2024-04-25T15:27:46.755722+0000","last_clean":"2024-04-25T15:27:46.755722+0000","last_became_active":"2024-04-25T15:27:46.753133+0000","last_became_peered":"2024-04-25T15:27:46.753133+0000","last_unstale":"2024-04-25T15:27:46.755722+0000","last_undegraded":"2024-04-25T15:27:46.755722+0000","last_fullsized":"2024-04-25T15:27:46.755722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:02.482063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743730+0000","last_change":"2024-04-25T15:27:46.743730+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743730+0000","last_undegraded":"2024-04-25T15:27:46.743730+0000","last_fullsized":"2024-04-25T15:27:46.743730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752208+0000","last_change":"2024-04-25T15:27:46.747236+0000","last_active":"2024-04-25T15:27:46.752208+0000","last_peered":"2024-04-25T15:27:46.752208+0000","last_clean":"2024-04-25T15:27:46.752208+0000","last_became_active":"2024-04-25T15:27:46.746401+0000","last_became_peered":"2024-04-25T15:27:46.746401+0000","last_unstale":"2024-04-25T15:27:46.752208+0000","last_undegraded":"2024-04-25T15:27:46.752208+0000","last_fullsized":"2024-04-25T15:27:46.752208+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:25.199824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737357+0000","last_change":"2024-04-25T15:27:46.737357+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737357+0000","last_undegraded":"2024-04-25T15:27:46.737357+0000","last_fullsized":"2024-04-25T15:27:46.737357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760768+0000","last_change":"2024-04-25T15:27:46.759114+0000","last_active":"2024-04-25T15:27:46.760768+0000","last_peered":"2024-04-25T15:27:46.760768+0000","last_clean":"2024-04-25T15:27:46.760768+0000","last_became_active":"2024-04-25T15:27:46.758785+0000","last_became_peered":"2024-04-25T15:27:46.758785+0000","last_unstale":"2024-04-25T15:27:46.760768+0000","last_undegraded":"2024-04-25T15:27:46.760768+0000","last_fullsized":"2024-04-25T15:27:46.760768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:56:51.896417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740337+0000","last_change":"2024-04-25T15:27:46.740337+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740337+0000","last_undegraded":"2024-04-25T15:27:46.740337+0000","last_fullsized":"2024-04-25T15:27:46.740337+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753377+0000","last_change":"2024-04-25T15:27:46.750313+0000","last_active":"2024-04-25T15:27:46.753377+0000","last_peered":"2024-04-25T15:27:46.753377+0000","last_clean":"2024-04-25T15:27:46.753377+0000","last_became_active":"2024-04-25T15:27:46.749988+0000","last_became_peered":"2024-04-25T15:27:46.749988+0000","last_unstale":"2024-04-25T15:27:46.753377+0000","last_undegraded":"2024-04-25T15:27:46.753377+0000","last_fullsized":"2024-04-25T15:27:46.753377+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:06.332293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737978+0000","last_change":"2024-04-25T15:27:46.737978+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737978+0000","last_undegraded":"2024-04-25T15:27:46.737978+0000","last_fullsized":"2024-04-25T15:27:46.737978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757737+0000","last_change":"2024-04-25T15:27:46.754470+0000","last_active":"2024-04-25T15:27:46.757737+0000","last_peered":"2024-04-25T15:27:46.757737+0000","last_clean":"2024-04-25T15:27:46.757737+0000","last_became_active":"2024-04-25T15:27:46.753975+0000","last_became_peered":"2024-04-25T15:27:46.753975+0000","last_unstale":"2024-04-25T15:27:46.757737+0000","last_undegraded":"2024-04-25T15:27:46.757737+0000","last_fullsized":"2024-04-25T15:27:46.757737+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:41.565463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740377+0000","last_change":"2024-04-25T15:27:46.740377+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740377+0000","last_undegraded":"2024-04-25T15:27:46.740377+0000","last_fullsized":"2024-04-25T15:27:46.740377+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.757718+0000","last_change":"2024-04-25T15:27:46.754424+0000","last_active":"2024-04-25T15:27:46.757718+0000","last_peered":"2024-04-25T15:27:46.757718+0000","last_clean":"2024-04-25T15:27:46.757718+0000","last_became_active":"2024-04-25T15:27:46.754102+0000","last_became_peered":"2024-04-25T15:27:46.754102+0000","last_unstale":"2024-04-25T15:27:46.757718+0000","last_undegraded":"2024-04-25T15:27:46.757718+0000","last_fullsized":"2024-04-25T15:27:46.757718+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:54.809375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742041+0000","last_change":"2024-04-25T15:27:46.742041+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742041+0000","last_undegraded":"2024-04-25T15:27:46.742041+0000","last_fullsized":"2024-04-25T15:27:46.742041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760460+0000","last_change":"2024-04-25T15:27:46.756182+0000","last_active":"2024-04-25T15:27:46.760460+0000","last_peered":"2024-04-25T15:27:46.760460+0000","last_clean":"2024-04-25T15:27:46.760460+0000","last_became_active":"2024-04-25T15:27:46.755789+0000","last_became_peered":"2024-04-25T15:27:46.755789+0000","last_unstale":"2024-04-25T15:27:46.760460+0000","last_undegraded":"2024-04-25T15:27:46.760460+0000","last_fullsized":"2024-04-25T15:27:46.760460+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.454298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743258+0000","last_change":"2024-04-25T15:27:46.743258+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743258+0000","last_undegraded":"2024-04-25T15:27:46.743258+0000","last_fullsized":"2024-04-25T15:27:46.743258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.753687+0000","last_change":"2024-04-25T15:27:46.751621+0000","last_active":"2024-04-25T15:27:46.753687+0000","last_peered":"2024-04-25T15:27:46.753687+0000","last_clean":"2024-04-25T15:27:46.753687+0000","last_became_active":"2024-04-25T15:27:46.751328+0000","last_became_peered":"2024-04-25T15:27:46.751328+0000","last_unstale":"2024-04-25T15:27:46.753687+0000","last_undegraded":"2024-04-25T15:27:46.753687+0000","last_fullsized":"2024-04-25T15:27:46.753687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:07:28.385465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.740778+0000","last_change":"2024-04-25T15:27:46.740778+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.740778+0000","last_undegraded":"2024-04-25T15:27:46.740778+0000","last_fullsized":"2024-04-25T15:27:46.740778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755225+0000","last_change":"2024-04-25T15:27:46.748665+0000","last_active":"2024-04-25T15:27:46.755225+0000","last_peered":"2024-04-25T15:27:46.755225+0000","last_clean":"2024-04-25T15:27:46.755225+0000","last_became_active":"2024-04-25T15:27:46.748364+0000","last_became_peered":"2024-04-25T15:27:46.748364+0000","last_unstale":"2024-04-25T15:27:46.755225+0000","last_undegraded":"2024-04-25T15:27:46.755225+0000","last_fullsized":"2024-04-25T15:27:46.755225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:34.541226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.734502+0000","last_change":"2024-04-25T15:27:46.734502+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.734502+0000","last_undegraded":"2024-04-25T15:27:46.734502+0000","last_fullsized":"2024-04-25T15:27:46.734502+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760752+0000","last_change":"2024-04-25T15:27:46.759084+0000","last_active":"2024-04-25T15:27:46.760752+0000","last_peered":"2024-04-25T15:27:46.760752+0000","last_clean":"2024-04-25T15:27:46.760752+0000","last_became_active":"2024-04-25T15:27:46.758757+0000","last_became_peered":"2024-04-25T15:27:46.758757+0000","last_unstale":"2024-04-25T15:27:46.760752+0000","last_undegraded":"2024-04-25T15:27:46.760752+0000","last_fullsized":"2024-04-25T15:27:46.760752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:54.994195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739729+0000","last_change":"2024-04-25T15:27:46.739729+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739729+0000","last_undegraded":"2024-04-25T15:27:46.739729+0000","last_fullsized":"2024-04-25T15:27:46.739729+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755747+0000","last_change":"2024-04-25T15:27:46.753206+0000","last_active":"2024-04-25T15:27:46.755747+0000","last_peered":"2024-04-25T15:27:46.755747+0000","last_clean":"2024-04-25T15:27:46.755747+0000","last_became_active":"2024-04-25T15:27:46.752883+0000","last_became_peered":"2024-04-25T15:27:46.752883+0000","last_unstale":"2024-04-25T15:27:46.755747+0000","last_undegraded":"2024-04-25T15:27:46.755747+0000","last_fullsized":"2024-04-25T15:27:46.755747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:55.623358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.745243+0000","last_change":"2024-04-25T15:27:46.745243+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.745243+0000","last_undegraded":"2024-04-25T15:27:46.745243+0000","last_fullsized":"2024-04-25T15:27:46.745243+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755764+0000","last_change":"2024-04-25T15:27:46.753592+0000","last_active":"2024-04-25T15:27:46.755764+0000","last_peered":"2024-04-25T15:27:46.755764+0000","last_clean":"2024-04-25T15:27:46.755764+0000","last_became_active":"2024-04-25T15:27:46.753163+0000","last_became_peered":"2024-04-25T15:27:46.753163+0000","last_unstale":"2024-04-25T15:27:46.755764+0000","last_undegraded":"2024-04-25T15:27:46.755764+0000","last_fullsized":"2024-04-25T15:27:46.755764+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:19.025834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.742091+0000","last_change":"2024-04-25T15:27:46.742091+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.742091+0000","last_undegraded":"2024-04-25T15:27:46.742091+0000","last_fullsized":"2024-04-25T15:27:46.742091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752245+0000","last_change":"2024-04-25T15:27:46.744686+0000","last_active":"2024-04-25T15:27:46.752245+0000","last_peered":"2024-04-25T15:27:46.752245+0000","last_clean":"2024-04-25T15:27:46.752245+0000","last_became_active":"2024-04-25T15:27:46.744321+0000","last_became_peered":"2024-04-25T15:27:46.744321+0000","last_unstale":"2024-04-25T15:27:46.752245+0000","last_undegraded":"2024-04-25T15:27:46.752245+0000","last_fullsized":"2024-04-25T15:27:46.752245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:51.235314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739940+0000","last_change":"2024-04-25T15:27:46.739940+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739940+0000","last_undegraded":"2024-04-25T15:27:46.739940+0000","last_fullsized":"2024-04-25T15:27:46.739940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.755161+0000","last_change":"2024-04-25T15:27:46.749552+0000","last_active":"2024-04-25T15:27:46.755161+0000","last_peered":"2024-04-25T15:27:46.755161+0000","last_clean":"2024-04-25T15:27:46.755161+0000","last_became_active":"2024-04-25T15:27:46.749242+0000","last_became_peered":"2024-04-25T15:27:46.749242+0000","last_unstale":"2024-04-25T15:27:46.755161+0000","last_undegraded":"2024-04-25T15:27:46.755161+0000","last_fullsized":"2024-04-25T15:27:46.755161+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:57:28.326266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.743195+0000","last_change":"2024-04-25T15:27:46.743195+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.743195+0000","last_undegraded":"2024-04-25T15:27:46.743195+0000","last_fullsized":"2024-04-25T15:27:46.743195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.751802+0000","last_change":"2024-04-25T15:27:46.746473+0000","last_active":"2024-04-25T15:27:46.751802+0000","last_peered":"2024-04-25T15:27:46.751802+0000","last_clean":"2024-04-25T15:27:46.751802+0000","last_became_active":"2024-04-25T15:27:46.746083+0000","last_became_peered":"2024-04-25T15:27:46.746083+0000","last_unstale":"2024-04-25T15:27:46.751802+0000","last_undegraded":"2024-04-25T15:27:46.751802+0000","last_fullsized":"2024-04-25T15:27:46.751802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.454298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.737498+0000","last_change":"2024-04-25T15:27:46.737498+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.737498+0000","last_undegraded":"2024-04-25T15:27:46.737498+0000","last_fullsized":"2024-04-25T15:27:46.737498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.752198+0000","last_change":"2024-04-25T15:27:46.745918+0000","last_active":"2024-04-25T15:27:46.752198+0000","last_peered":"2024-04-25T15:27:46.752198+0000","last_clean":"2024-04-25T15:27:46.752198+0000","last_became_active":"2024-04-25T15:27:46.745596+0000","last_became_peered":"2024-04-25T15:27:46.745596+0000","last_unstale":"2024-04-25T15:27:46.752198+0000","last_undegraded":"2024-04-25T15:27:46.752198+0000","last_fullsized":"2024-04-25T15:27:46.752198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:37.598034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.739515+0000","last_change":"2024-04-25T15:27:46.739515+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.739515+0000","last_undegraded":"2024-04-25T15:27:46.739515+0000","last_fullsized":"2024-04-25T15:27:46.739515+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.749424+0000","last_change":"2024-04-25T15:27:46.746186+0000","last_active":"2024-04-25T15:27:46.749424+0000","last_peered":"2024-04-25T15:27:46.749424+0000","last_clean":"2024-04-25T15:27:46.749424+0000","last_became_active":"2024-04-25T15:27:46.745867+0000","last_became_peered":"2024-04-25T15:27:46.745867+0000","last_unstale":"2024-04-25T15:27:46.749424+0000","last_undegraded":"2024-04-25T15:27:46.749424+0000","last_fullsized":"2024-04-25T15:27:46.749424+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.445845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-04-25T15:27:46.741940+0000","last_change":"2024-04-25T15:27:46.741940+0000","last_active":"2024-04-25T15:27:46.728373+0000","last_peered":"2024-04-25T15:27:46.728373+0000","last_clean":"2024-04-25T15:27:46.728373+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:27:46.741940+0000","last_undegraded":"2024-04-25T15:27:46.741940+0000","last_fullsized":"2024-04-25T15:27:46.741940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.760410+0000","last_change":"2024-04-25T15:27:46.755483+0000","last_active":"2024-04-25T15:27:46.760410+0000","last_peered":"2024-04-25T15:27:46.760410+0000","last_clean":"2024-04-25T15:27:46.760410+0000","last_became_active":"2024-04-25T15:27:46.755160+0000","last_became_peered":"2024-04-25T15:27:46.755160+0000","last_unstale":"2024-04-25T15:27:46.760410+0000","last_undegraded":"2024-04-25T15:27:46.760410+0000","last_fullsized":"2024-04-25T15:27:46.760410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:40:46.657005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":191,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T15:27:46.743997+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:27:46.743997+0000","last_peered":"2024-04-25T15:27:46.743997+0000","last_clean":"2024-04-25T15:27:46.743997+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:27:46.743997+0000","last_undegraded":"2024-04-25T15:27:46.743997+0000","last_fullsized":"2024-04-25T15:27:46.743997+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343472,"statfs":{"total":96636764160,"available":96607715328,"internally_reserved":0,"allocated":1372160,"data_stored":912533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":328440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":778240,"data_stored":322165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28376,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343464,"statfs":{"total":96636764160,"available":96607707136,"internally_reserved":0,"allocated":1372160,"data_stored":912533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:27:52.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3f is not active+clean 2024-04-25T15:27:52.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742158+0000', 'last_change': '2024-04-25T15:27:46.742158+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742158+0000', 'last_undegraded': '2024-04-25T15:27:46.742158+0000', 'last_fullsized': '2024-04-25T15:27:46.742158+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3e is not active+clean 2024-04-25T15:27:52.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.746577+0000', 'last_change': '2024-04-25T15:27:46.746577+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.746577+0000', 'last_undegraded': '2024-04-25T15:27:46.746577+0000', 'last_fullsized': '2024-04-25T15:27:46.746577+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3d is not active+clean 2024-04-25T15:27:52.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741554+0000', 'last_change': '2024-04-25T15:27:46.741554+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741554+0000', 'last_undegraded': '2024-04-25T15:27:46.741554+0000', 'last_fullsized': '2024-04-25T15:27:46.741554+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3c is not active+clean 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.739391+0000', 'last_change': '2024-04-25T15:27:46.739391+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.739391+0000', 'last_undegraded': '2024-04-25T15:27:46.739391+0000', 'last_fullsized': '2024-04-25T15:27:46.739391+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3b is not active+clean 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741588+0000', 'last_change': '2024-04-25T15:27:46.741588+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741588+0000', 'last_undegraded': '2024-04-25T15:27:46.741588+0000', 'last_fullsized': '2024-04-25T15:27:46.741588+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3a is not active+clean 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.744353+0000', 'last_change': '2024-04-25T15:27:46.744353+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.744353+0000', 'last_undegraded': '2024-04-25T15:27:46.744353+0000', 'last_fullsized': '2024-04-25T15:27:46.744353+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.39 is not active+clean 2024-04-25T15:27:52.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741489+0000', 'last_change': '2024-04-25T15:27:46.741489+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741489+0000', 'last_undegraded': '2024-04-25T15:27:46.741489+0000', 'last_fullsized': '2024-04-25T15:27:46.741489+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.38 is not active+clean 2024-04-25T15:27:52.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741504+0000', 'last_change': '2024-04-25T15:27:46.741504+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741504+0000', 'last_undegraded': '2024-04-25T15:27:46.741504+0000', 'last_fullsized': '2024-04-25T15:27:46.741504+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.37 is not active+clean 2024-04-25T15:27:52.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.743205+0000', 'last_change': '2024-04-25T15:27:46.743205+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.743205+0000', 'last_undegraded': '2024-04-25T15:27:46.743205+0000', 'last_fullsized': '2024-04-25T15:27:46.743205+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.36 is not active+clean 2024-04-25T15:27:52.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.737954+0000', 'last_change': '2024-04-25T15:27:46.737954+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.737954+0000', 'last_undegraded': '2024-04-25T15:27:46.737954+0000', 'last_fullsized': '2024-04-25T15:27:46.737954+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.35 is not active+clean 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.743381+0000', 'last_change': '2024-04-25T15:27:46.743381+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.743381+0000', 'last_undegraded': '2024-04-25T15:27:46.743381+0000', 'last_fullsized': '2024-04-25T15:27:46.743381+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.34 is not active+clean 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.738618+0000', 'last_change': '2024-04-25T15:27:46.738618+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.738618+0000', 'last_undegraded': '2024-04-25T15:27:46.738618+0000', 'last_fullsized': '2024-04-25T15:27:46.738618+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.33 is not active+clean 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.734472+0000', 'last_change': '2024-04-25T15:27:46.734472+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.734472+0000', 'last_undegraded': '2024-04-25T15:27:46.734472+0000', 'last_fullsized': '2024-04-25T15:27:46.734472+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.32 is not active+clean 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740602+0000', 'last_change': '2024-04-25T15:27:46.740602+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740602+0000', 'last_undegraded': '2024-04-25T15:27:46.740602+0000', 'last_fullsized': '2024-04-25T15:27:46.740602+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.31 is not active+clean 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.735281+0000', 'last_change': '2024-04-25T15:27:46.735281+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.735281+0000', 'last_undegraded': '2024-04-25T15:27:46.735281+0000', 'last_fullsized': '2024-04-25T15:27:46.735281+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.30 is not active+clean 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.735174+0000', 'last_change': '2024-04-25T15:27:46.735174+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.735174+0000', 'last_undegraded': '2024-04-25T15:27:46.735174+0000', 'last_fullsized': '2024-04-25T15:27:46.735174+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2f is not active+clean 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742694+0000', 'last_change': '2024-04-25T15:27:46.742694+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742694+0000', 'last_undegraded': '2024-04-25T15:27:46.742694+0000', 'last_fullsized': '2024-04-25T15:27:46.742694+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2e is not active+clean 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.738105+0000', 'last_change': '2024-04-25T15:27:46.738105+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.738105+0000', 'last_undegraded': '2024-04-25T15:27:46.738105+0000', 'last_fullsized': '2024-04-25T15:27:46.738105+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:27:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2d is not active+clean 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742298+0000', 'last_change': '2024-04-25T15:27:46.742298+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742298+0000', 'last_undegraded': '2024-04-25T15:27:46.742298+0000', 'last_fullsized': '2024-04-25T15:27:46.742298+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2c is not active+clean 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.734487+0000', 'last_change': '2024-04-25T15:27:46.734487+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.734487+0000', 'last_undegraded': '2024-04-25T15:27:46.734487+0000', 'last_fullsized': '2024-04-25T15:27:46.734487+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2b is not active+clean 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.736254+0000', 'last_change': '2024-04-25T15:27:46.736254+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.736254+0000', 'last_undegraded': '2024-04-25T15:27:46.736254+0000', 'last_fullsized': '2024-04-25T15:27:46.736254+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2a is not active+clean 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.735290+0000', 'last_change': '2024-04-25T15:27:46.735290+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.735290+0000', 'last_undegraded': '2024-04-25T15:27:46.735290+0000', 'last_fullsized': '2024-04-25T15:27:46.735290+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.29 is not active+clean 2024-04-25T15:27:52.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740685+0000', 'last_change': '2024-04-25T15:27:46.740685+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740685+0000', 'last_undegraded': '2024-04-25T15:27:46.740685+0000', 'last_fullsized': '2024-04-25T15:27:46.740685+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.28 is not active+clean 2024-04-25T15:27:52.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740976+0000', 'last_change': '2024-04-25T15:27:46.740976+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740976+0000', 'last_undegraded': '2024-04-25T15:27:46.740976+0000', 'last_fullsized': '2024-04-25T15:27:46.740976+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.27 is not active+clean 2024-04-25T15:27:52.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.737831+0000', 'last_change': '2024-04-25T15:27:46.737831+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.737831+0000', 'last_undegraded': '2024-04-25T15:27:46.737831+0000', 'last_fullsized': '2024-04-25T15:27:46.737831+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.26 is not active+clean 2024-04-25T15:27:52.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.739762+0000', 'last_change': '2024-04-25T15:27:46.739762+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.739762+0000', 'last_undegraded': '2024-04-25T15:27:46.739762+0000', 'last_fullsized': '2024-04-25T15:27:46.739762+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.25 is not active+clean 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741068+0000', 'last_change': '2024-04-25T15:27:46.741068+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741068+0000', 'last_undegraded': '2024-04-25T15:27:46.741068+0000', 'last_fullsized': '2024-04-25T15:27:46.741068+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.24 is not active+clean 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.736975+0000', 'last_change': '2024-04-25T15:27:46.736975+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.736975+0000', 'last_undegraded': '2024-04-25T15:27:46.736975+0000', 'last_fullsized': '2024-04-25T15:27:46.736975+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.23 is not active+clean 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742531+0000', 'last_change': '2024-04-25T15:27:46.742531+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742531+0000', 'last_undegraded': '2024-04-25T15:27:46.742531+0000', 'last_fullsized': '2024-04-25T15:27:46.742531+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.22 is not active+clean 2024-04-25T15:27:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.744160+0000', 'last_change': '2024-04-25T15:27:46.744160+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.744160+0000', 'last_undegraded': '2024-04-25T15:27:46.744160+0000', 'last_fullsized': '2024-04-25T15:27:46.744160+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.21 is not active+clean 2024-04-25T15:27:52.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740651+0000', 'last_change': '2024-04-25T15:27:46.740651+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740651+0000', 'last_undegraded': '2024-04-25T15:27:46.740651+0000', 'last_fullsized': '2024-04-25T15:27:46.740651+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.20 is not active+clean 2024-04-25T15:27:52.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740954+0000', 'last_change': '2024-04-25T15:27:46.740954+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740954+0000', 'last_undegraded': '2024-04-25T15:27:46.740954+0000', 'last_fullsized': '2024-04-25T15:27:46.740954+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-04-25T15:27:52.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.739785+0000', 'last_change': '2024-04-25T15:27:46.739785+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.739785+0000', 'last_undegraded': '2024-04-25T15:27:46.739785+0000', 'last_fullsized': '2024-04-25T15:27:46.739785+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-04-25T15:27:52.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742196+0000', 'last_change': '2024-04-25T15:27:46.742196+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742196+0000', 'last_undegraded': '2024-04-25T15:27:46.742196+0000', 'last_fullsized': '2024-04-25T15:27:46.742196+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-04-25T15:27:52.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741214+0000', 'last_change': '2024-04-25T15:27:46.741214+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741214+0000', 'last_undegraded': '2024-04-25T15:27:46.741214+0000', 'last_fullsized': '2024-04-25T15:27:46.741214+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-04-25T15:27:52.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742915+0000', 'last_change': '2024-04-25T15:27:46.742915+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742915+0000', 'last_undegraded': '2024-04-25T15:27:46.742915+0000', 'last_fullsized': '2024-04-25T15:27:46.742915+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-04-25T15:27:52.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.738688+0000', 'last_change': '2024-04-25T15:27:46.738688+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.738688+0000', 'last_undegraded': '2024-04-25T15:27:46.738688+0000', 'last_fullsized': '2024-04-25T15:27:46.738688+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.735434+0000', 'last_change': '2024-04-25T15:27:46.735434+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.735434+0000', 'last_undegraded': '2024-04-25T15:27:46.735434+0000', 'last_fullsized': '2024-04-25T15:27:46.735434+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.744277+0000', 'last_change': '2024-04-25T15:27:46.744277+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.744277+0000', 'last_undegraded': '2024-04-25T15:27:46.744277+0000', 'last_fullsized': '2024-04-25T15:27:46.744277+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.734472+0000', 'last_change': '2024-04-25T15:27:46.734472+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.734472+0000', 'last_undegraded': '2024-04-25T15:27:46.734472+0000', 'last_fullsized': '2024-04-25T15:27:46.734472+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-04-25T15:27:52.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.739213+0000', 'last_change': '2024-04-25T15:27:46.739213+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.739213+0000', 'last_undegraded': '2024-04-25T15:27:46.739213+0000', 'last_fullsized': '2024-04-25T15:27:46.739213+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-04-25T15:27:52.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741658+0000', 'last_change': '2024-04-25T15:27:46.741658+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741658+0000', 'last_undegraded': '2024-04-25T15:27:46.741658+0000', 'last_fullsized': '2024-04-25T15:27:46.741658+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-04-25T15:27:52.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740683+0000', 'last_change': '2024-04-25T15:27:46.740683+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740683+0000', 'last_undegraded': '2024-04-25T15:27:46.740683+0000', 'last_fullsized': '2024-04-25T15:27:46.740683+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-04-25T15:27:52.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740931+0000', 'last_change': '2024-04-25T15:27:46.740931+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740931+0000', 'last_undegraded': '2024-04-25T15:27:46.740931+0000', 'last_fullsized': '2024-04-25T15:27:46.740931+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740179+0000', 'last_change': '2024-04-25T15:27:46.740179+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740179+0000', 'last_undegraded': '2024-04-25T15:27:46.740179+0000', 'last_fullsized': '2024-04-25T15:27:46.740179+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.743383+0000', 'last_change': '2024-04-25T15:27:46.743383+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.743383+0000', 'last_undegraded': '2024-04-25T15:27:46.743383+0000', 'last_fullsized': '2024-04-25T15:27:46.743383+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.734455+0000', 'last_change': '2024-04-25T15:27:46.734455+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.734455+0000', 'last_undegraded': '2024-04-25T15:27:46.734455+0000', 'last_fullsized': '2024-04-25T15:27:46.734455+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-04-25T15:27:52.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.743730+0000', 'last_change': '2024-04-25T15:27:46.743730+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.743730+0000', 'last_undegraded': '2024-04-25T15:27:46.743730+0000', 'last_fullsized': '2024-04-25T15:27:46.743730+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-04-25T15:27:52.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.737357+0000', 'last_change': '2024-04-25T15:27:46.737357+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.737357+0000', 'last_undegraded': '2024-04-25T15:27:46.737357+0000', 'last_fullsized': '2024-04-25T15:27:46.737357+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:27:52.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-04-25T15:27:52.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740337+0000', 'last_change': '2024-04-25T15:27:46.740337+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740337+0000', 'last_undegraded': '2024-04-25T15:27:46.740337+0000', 'last_fullsized': '2024-04-25T15:27:46.740337+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-04-25T15:27:52.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.737978+0000', 'last_change': '2024-04-25T15:27:46.737978+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.737978+0000', 'last_undegraded': '2024-04-25T15:27:46.737978+0000', 'last_fullsized': '2024-04-25T15:27:46.737978+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740377+0000', 'last_change': '2024-04-25T15:27:46.740377+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740377+0000', 'last_undegraded': '2024-04-25T15:27:46.740377+0000', 'last_fullsized': '2024-04-25T15:27:46.740377+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742041+0000', 'last_change': '2024-04-25T15:27:46.742041+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742041+0000', 'last_undegraded': '2024-04-25T15:27:46.742041+0000', 'last_fullsized': '2024-04-25T15:27:46.742041+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.743258+0000', 'last_change': '2024-04-25T15:27:46.743258+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.743258+0000', 'last_undegraded': '2024-04-25T15:27:46.743258+0000', 'last_fullsized': '2024-04-25T15:27:46.743258+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-04-25T15:27:52.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.740778+0000', 'last_change': '2024-04-25T15:27:46.740778+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.740778+0000', 'last_undegraded': '2024-04-25T15:27:46.740778+0000', 'last_fullsized': '2024-04-25T15:27:46.740778+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:27:52.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-04-25T15:27:52.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.734502+0000', 'last_change': '2024-04-25T15:27:46.734502+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.734502+0000', 'last_undegraded': '2024-04-25T15:27:46.734502+0000', 'last_fullsized': '2024-04-25T15:27:46.734502+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:27:52.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-04-25T15:27:52.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.739729+0000', 'last_change': '2024-04-25T15:27:46.739729+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.739729+0000', 'last_undegraded': '2024-04-25T15:27:46.739729+0000', 'last_fullsized': '2024-04-25T15:27:46.739729+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:27:52.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-04-25T15:27:52.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.745243+0000', 'last_change': '2024-04-25T15:27:46.745243+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.745243+0000', 'last_undegraded': '2024-04-25T15:27:46.745243+0000', 'last_fullsized': '2024-04-25T15:27:46.745243+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-04-25T15:27:52.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.742091+0000', 'last_change': '2024-04-25T15:27:46.742091+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.742091+0000', 'last_undegraded': '2024-04-25T15:27:46.742091+0000', 'last_fullsized': '2024-04-25T15:27:46.742091+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:27:52.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-04-25T15:27:52.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.739940+0000', 'last_change': '2024-04-25T15:27:46.739940+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.739940+0000', 'last_undegraded': '2024-04-25T15:27:46.739940+0000', 'last_fullsized': '2024-04-25T15:27:46.739940+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:27:52.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-04-25T15:27:52.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.743195+0000', 'last_change': '2024-04-25T15:27:46.743195+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.743195+0000', 'last_undegraded': '2024-04-25T15:27:46.743195+0000', 'last_fullsized': '2024-04-25T15:27:46.743195+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-04-25T15:27:52.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.737498+0000', 'last_change': '2024-04-25T15:27:46.737498+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.737498+0000', 'last_undegraded': '2024-04-25T15:27:46.737498+0000', 'last_fullsized': '2024-04-25T15:27:46.737498+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:27:52.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-04-25T15:27:52.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.739515+0000', 'last_change': '2024-04-25T15:27:46.739515+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.739515+0000', 'last_undegraded': '2024-04-25T15:27:46.739515+0000', 'last_fullsized': '2024-04-25T15:27:46.739515+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:27:52.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-04-25T15:27:52.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:27:46.741940+0000', 'last_change': '2024-04-25T15:27:46.741940+0000', 'last_active': '2024-04-25T15:27:46.728373+0000', 'last_peered': '2024-04-25T15:27:46.728373+0000', 'last_clean': '2024-04-25T15:27:46.728373+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:27:46.741940+0000', 'last_undegraded': '2024-04-25T15:27:46.741940+0000', 'last_fullsized': '2024-04-25T15:27:46.741940+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'last_clean_scrub_stamp': '2024-04-25T15:27:46.728373+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:27:52.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-25T15:27:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.719+0000 7f760d585640 1 Processor -- start 2024-04-25T15:27:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.720+0000 7f760d585640 1 -- start start 2024-04-25T15:27:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f760d585640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76080b21a0 0x7f76080ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f760d585640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76080d1c10 con 0x7f76080b7840 2024-04-25T15:27:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f760d585640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76080b25b0 con 0x7f76080cfd40 2024-04-25T15:27:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f760d585640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f76080d27e0 con 0x7f76080b21a0 2024-04-25T15:27:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f76067fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76080b21a0 0x7f76080ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f76077fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76080b7840 0x7f76080b7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55626/0 (socket says 172.21.15.67:55626) 2024-04-25T15:27:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f76077fe640 1 -- 172.21.15.67:0/443483708 learned_addr learned my addr 172.21.15.67:0/443483708 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f76067fc640 1 -- 172.21.15.67:0/443483708 >> v1:172.21.15.176:6789/0 conn(0x7f76080cfd40 legacy=0x7f76080b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f76067fc640 1 -- 172.21.15.67:0/443483708 >> v1:172.21.15.67:6789/0 conn(0x7f76080b7840 legacy=0x7f76080b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f76067fc640 1 -- 172.21.15.67:0/443483708 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76080d1eb0 con 0x7f76080b21a0 2024-04-25T15:27:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.721+0000 7f7605ffb640 1 -- 172.21.15.67:0/443483708 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 583332418 0 0) 0x7f76080d1c10 con 0x7f76080b7840 2024-04-25T15:27:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.722+0000 7f76067fc640 1 --2- 172.21.15.67:0/443483708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080b21a0 0x7f76080ce110 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f75f00097a0 tx=0x7f75f002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b209784a4fcfad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.722+0000 7f7605ffb640 1 -- 172.21.15.67:0/443483708 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0037070 con 0x7f76080b21a0 2024-04-25T15:27:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.722+0000 7f7605ffb640 1 -- 172.21.15.67:0/443483708 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f0032440 con 0x7f76080b21a0 2024-04-25T15:27:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.722+0000 7f7605ffb640 1 -- 172.21.15.67:0/443483708 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0032810 con 0x7f76080b21a0 2024-04-25T15:27:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.723+0000 7f760d585640 1 -- 172.21.15.67:0/443483708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080b21a0 msgr2=0x7f76080ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.723+0000 7f760d585640 1 --2- 172.21.15.67:0/443483708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080b21a0 0x7f76080ce110 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f75f00097a0 tx=0x7f75f002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.723+0000 7f760d585640 1 -- 172.21.15.67:0/443483708 shutdown_connections 2024-04-25T15:27:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.723+0000 7f760d585640 1 --2- 172.21.15.67:0/443483708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080b21a0 0x7f76080ce110 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.723+0000 7f760d585640 1 -- 172.21.15.67:0/443483708 >> 172.21.15.67:0/443483708 conn(0x7f760809b7d0 msgr2=0x7f760809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.723+0000 7f760d585640 1 -- 172.21.15.67:0/443483708 shutdown_connections 2024-04-25T15:27:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.723+0000 7f760d585640 1 -- 172.21.15.67:0/443483708 wait complete. 2024-04-25T15:27:52.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.725+0000 7f760d585640 1 Processor -- start 2024-04-25T15:27:52.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.725+0000 7f760d585640 1 -- start start 2024-04-25T15:27:52.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f760d585640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76080b21a0 0x7f760815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f760d585640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76080b7840 0x7f76080b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f760d585640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080cfd40 0x7f76080b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f760d585640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76080ce650 con 0x7f76080cfd40 2024-04-25T15:27:52.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f760d585640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f76080d2470 con 0x7f76080b7840 2024-04-25T15:27:52.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f760d585640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f76080d2c70 con 0x7f76080b21a0 2024-04-25T15:27:52.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f76077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080cfd40 0x7f76080b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f76077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080cfd40 0x7f76080b0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35234/0 (socket says 172.21.15.67:35234) 2024-04-25T15:27:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.726+0000 7f76077fe640 1 -- 172.21.15.67:0/582269009 learned_addr learned my addr 172.21.15.67:0/582269009 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f7606ffd640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76080b21a0 0x7f760815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76067fc640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76080b7840 0x7f76080b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76077fe640 1 -- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76080b21a0 msgr2=0x7f760815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76077fe640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76080b21a0 0x7f760815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76077fe640 1 -- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76080b7840 msgr2=0x7f76080b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:52.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76077fe640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76080b7840 0x7f76080b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:52.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76077fe640 1 -- 172.21.15.67:0/582269009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7608161eb0 con 0x7f76080cfd40 2024-04-25T15:27:52.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f7606ffd640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76080b21a0 0x7f760815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:52.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76067fc640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76080b7840 0x7f76080b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:52.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f76077fe640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080cfd40 0x7f76080b0d70 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f760000c440 tx=0x7f760000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:52.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f75ff7fe640 1 -- 172.21.15.67:0/582269009 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7600017070 con 0x7f76080cfd40 2024-04-25T15:27:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f75ff7fe640 1 -- 172.21.15.67:0/582269009 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7600005ce0 con 0x7f76080cfd40 2024-04-25T15:27:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76080a0530 con 0x7f76080cfd40 2024-04-25T15:27:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f75ff7fe640 1 -- 172.21.15.67:0/582269009 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7600007310 con 0x7f76080cfd40 2024-04-25T15:27:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.727+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7608163410 con 0x7f76080cfd40 2024-04-25T15:27:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.728+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76080b37f0 con 0x7f76080cfd40 2024-04-25T15:27:52.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.729+0000 7f75ff7fe640 1 -- 172.21.15.67:0/582269009 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f760001a070 con 0x7f76080cfd40 2024-04-25T15:27:52.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.729+0000 7f75ff7fe640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f75e80744e0 0x7f75e80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:52.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.729+0000 7f75ff7fe640 1 -- 172.21.15.67:0/582269009 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f7600094440 con 0x7f76080cfd40 2024-04-25T15:27:52.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.730+0000 7f7606ffd640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f75e80744e0 0x7f75e80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:52.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.730+0000 7f7606ffd640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f75e80744e0 0x7f75e80769f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f75f80012e0 tx=0x7f75f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:52.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.734+0000 7f75ff7fe640 1 -- 172.21.15.67:0/582269009 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76080b37f0 con 0x7f76080cfd40 2024-04-25T15:27:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:52.999+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f7608161b90 con 0x7f76080cfd40 2024-04-25T15:27:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.000+0000 7f75ff7fe640 1 -- 172.21.15.67:0/582269009 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1453 (secure 0 0 0) 0x7f760005ecf0 con 0x7f76080cfd40 2024-04-25T15:27:53.001 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:53.001 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":168,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":67,"num_osds":8,"num_up_osds":8,"osd_up_since":1714058715,"num_in_osds":8,"osd_in_since":1714058706,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":228818944,"bytes_avail":772865294336,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":41,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6273}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-25T15:27:06.818678+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-25T15:27:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f75e80744e0 msgr2=0x7f75e80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f75e80744e0 0x7f75e80769f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f75f80012e0 tx=0x7f75f8009040 comp rx=0 tx=0).stop 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080cfd40 msgr2=0x7f76080b0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080cfd40 0x7f76080b0d70 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f760000c440 tx=0x7f760000c900 comp rx=0 tx=0).stop 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 shutdown_connections 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f75e80744e0 0x7f75e80769f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76080cfd40 0x7f76080b0d70 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76080b7840 0x7f76080b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 --2- 172.21.15.67:0/582269009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76080b21a0 0x7f760815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 >> 172.21.15.67:0/582269009 conn(0x7f760809b7d0 msgr2=0x7f76080716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 shutdown_connections 2024-04-25T15:27:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.002+0000 7f760d585640 1 -- 172.21.15.67:0/582269009 wait complete. 2024-04-25T15:27:53.014 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-25T15:27:53.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:27:53.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.124+0000 7fa0539f3640 1 Processor -- start 2024-04-25T15:27:53.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.125+0000 7fa0539f3640 1 -- start start 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa0539f3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa04c06d8e0 0x7fa04c0d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa0539f3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c0d12c0 con 0x7fa04c0d2d70 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa0539f3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04c0b3c10 con 0x7fa04c06dcf0 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa0539f3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa04c0b9520 con 0x7fa04c06d8e0 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa051f90640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa04c0d2d70 0x7fa04c0d0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55642/0 (socket says 172.21.15.67:55642) 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa051f90640 1 -- 172.21.15.67:0/1452686809 learned_addr learned my addr 172.21.15.67:0/1452686809 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa05178f640 1 --2- 172.21.15.67:0/1452686809 >> v2:172.21.15.176:3300/0 conn(0x7fa04c06d8e0 0x7fa04c0d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.126+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2199633963 0 0) 0x7fa04c0d12c0 con 0x7fa04c0d2d70 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa034003520 con 0x7fa04c0d2d70 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112629144 0 0) 0x7fa04c0b3c10 con 0x7fa04c06dcf0 2024-04-25T15:27:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa04c0d12c0 con 0x7fa04c06dcf0 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4109529557 0 0) 0x7fa034003520 con 0x7fa04c0d2d70 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 >> v2:172.21.15.176:3300/0 conn(0x7fa04c06d8e0 msgr2=0x7fa04c0d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 --2- 172.21.15.67:0/1452686809 >> v2:172.21.15.176:3300/0 conn(0x7fa04c06d8e0 0x7fa04c0d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 >> v1:172.21.15.176:6789/0 conn(0x7fa04c06dcf0 legacy=0x7fa04c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c0b3ef0 con 0x7fa04c0d2d70 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fa03c0019f0 con 0x7fa04c0d2d70 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 >> v1:172.21.15.67:6789/0 conn(0x7fa04c0d2d70 legacy=0x7fa04c0d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034003b40 0x7fa034003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa034004800 0x7fa034004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034005260 0x7fa03400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c0b9520 con 0x7fa034005260 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa03400c1c0 con 0x7fa034003b40 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa03400c3c0 con 0x7fa034004800 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa051f90640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034005260 0x7fa03400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa05178f640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034003b40 0x7fa034003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa050f8e640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa034004800 0x7fa034004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa051f90640 1 -- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa034004800 msgr2=0x7fa034004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa051f90640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa034004800 0x7fa034004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa051f90640 1 -- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034003b40 msgr2=0x7fa034003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa051f90640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034003b40 0x7fa034003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.127+0000 7fa051f90640 1 -- 172.21.15.67:0/1452686809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa034003520 con 0x7fa034005260 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.128+0000 7fa051f90640 1 -- 172.21.15.67:0/1452686809 reap_dead start 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.128+0000 7fa051f90640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034005260 0x7fa03400b9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0d4820 tx=0x7fa034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de96405d3a6be8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.128+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c00a410 con 0x7fa034005260 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.128+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa03c02f070 con 0x7fa034005260 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.128+0000 7fa04bfff640 1 -- 172.21.15.67:0/1452686809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c02bc00 con 0x7fa034005260 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.129+0000 7fa0539f3640 1 -- 172.21.15.67:0/1452686809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034005260 msgr2=0x7fa03400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.129+0000 7fa0539f3640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034005260 0x7fa03400b9c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0d4820 tx=0x7fa034005830 comp rx=0 tx=0).stop 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.129+0000 7fa0539f3640 1 -- 172.21.15.67:0/1452686809 shutdown_connections 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.129+0000 7fa0539f3640 1 --2- 172.21.15.67:0/1452686809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa034005260 0x7fa03400b9c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.129+0000 7fa0539f3640 1 -- 172.21.15.67:0/1452686809 >> 172.21.15.67:0/1452686809 conn(0x7fa04c08d110 msgr2=0x7fa04c08d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.129+0000 7fa0539f3640 1 -- 172.21.15.67:0/1452686809 shutdown_connections 2024-04-25T15:27:53.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.129+0000 7fa0539f3640 1 -- 172.21.15.67:0/1452686809 wait complete. 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.131+0000 7fa0539f3640 1 Processor -- start 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.131+0000 7fa0539f3640 1 -- start start 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa0539f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034005260 0x7fa04c09def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa0539f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa04c0cfb00 0x7fa04c15a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa0539f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa04c15efb0 0x7fa04c15f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa0539f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c0b44b0 con 0x7fa04c15efb0 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa0539f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa04c0b4820 con 0x7fa034005260 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa0539f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa04c0b38e0 con 0x7fa04c0cfb00 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa051f90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa04c15efb0 0x7fa04c15f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa051f90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa04c15efb0 0x7fa04c15f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35254/0 (socket says 172.21.15.67:35254) 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.132+0000 7fa051f90640 1 -- 172.21.15.67:0/2353864841 learned_addr learned my addr 172.21.15.67:0/2353864841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa050f8e640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa04c0cfb00 0x7fa04c15a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa05178f640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034005260 0x7fa04c09def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa051f90640 1 -- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa04c0cfb00 msgr2=0x7fa04c15a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa051f90640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa04c0cfb00 0x7fa04c15a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa051f90640 1 -- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034005260 msgr2=0x7fa04c09def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa051f90640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034005260 0x7fa04c09def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa051f90640 1 -- 172.21.15.67:0/2353864841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04c15f900 con 0x7fa04c15efb0 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa051f90640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa04c15efb0 0x7fa04c15f3c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa03c000f80 tx=0x7fa04c0b40c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa049ffb640 1 -- 172.21.15.67:0/2353864841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c001f70 con 0x7fa04c15efb0 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa049ffb640 1 -- 172.21.15.67:0/2353864841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa03c00a3f0 con 0x7fa04c15efb0 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa049ffb640 1 -- 172.21.15.67:0/2353864841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c02baf0 con 0x7fa04c15efb0 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04c0a0580 con 0x7fa04c15efb0 2024-04-25T15:27:53.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.133+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa04c0a0b40 con 0x7fa04c15efb0 2024-04-25T15:27:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.134+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04c050740 con 0x7fa04c15efb0 2024-04-25T15:27:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.135+0000 7fa049ffb640 1 -- 172.21.15.67:0/2353864841 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa03c03e030 con 0x7fa04c15efb0 2024-04-25T15:27:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.135+0000 7fa049ffb640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0380744e0 0x7fa0380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.136+0000 7fa049ffb640 1 -- 172.21.15.67:0/2353864841 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fa03c0b4ab0 con 0x7fa04c15efb0 2024-04-25T15:27:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.136+0000 7fa05178f640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0380744e0 0x7fa0380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.136+0000 7fa05178f640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0380744e0 0x7fa0380769f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0744c0 tx=0x7fa044008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:53.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.140+0000 7fa049ffb640 1 -- 172.21.15.67:0/2353864841 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa04c15f900 con 0x7fa04c15efb0 2024-04-25T15:27:53.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.321+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa04c0b8ac0 con 0x7fa0380744e0 2024-04-25T15:27:53.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.328+0000 7fa049ffb640 1 -- 172.21.15.67:0/2353864841 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252058 (secure 0 0 0) 0x7fa04c0b8ac0 con 0x7fa0380744e0 2024-04-25T15:27:53.331 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0380744e0 msgr2=0x7fa0380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0380744e0 0x7fa0380769f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0744c0 tx=0x7fa044008040 comp rx=0 tx=0).stop 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa04c15efb0 msgr2=0x7fa04c15f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa04c15efb0 0x7fa04c15f3c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa03c000f80 tx=0x7fa04c0b40c0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 shutdown_connections 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa0380744e0 0x7fa0380769f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa04c15efb0 0x7fa04c15f3c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa04c0cfb00 0x7fa04c15a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 --2- 172.21.15.67:0/2353864841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa034005260 0x7fa04c09def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:53.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 >> 172.21.15.67:0/2353864841 conn(0x7fa04c08d110 msgr2=0x7fa04c072160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:53.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 shutdown_connections 2024-04-25T15:27:53.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:53.331+0000 7fa0539f3640 1 -- 172.21.15.67:0/2353864841 wait complete. 2024-04-25T15:27:53.343 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-25T15:27:52.829965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223780,"kb_used_data":7592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754750940,"statfs":{"total":773094113280,"available":772864962560,"internally_reserved":0,"allocated":7774208,"data_stored":3917676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.097227"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745147+0000","last_change":"2024-04-25T15:27:47.784714+0000","last_active":"2024-04-25T15:27:49.745147+0000","last_peered":"2024-04-25T15:27:49.745147+0000","last_clean":"2024-04-25T15:27:49.745147+0000","last_became_active":"2024-04-25T15:27:47.784394+0000","last_became_peered":"2024-04-25T15:27:47.784394+0000","last_unstale":"2024-04-25T15:27:49.745147+0000","last_undegraded":"2024-04-25T15:27:49.745147+0000","last_fullsized":"2024-04-25T15:27:49.745147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:42.640130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213593+0000","last_change":"2024-04-25T15:27:47.784772+0000","last_active":"2024-04-25T15:27:49.213593+0000","last_peered":"2024-04-25T15:27:49.213593+0000","last_clean":"2024-04-25T15:27:49.213593+0000","last_became_active":"2024-04-25T15:27:47.784268+0000","last_became_peered":"2024-04-25T15:27:47.784268+0000","last_unstale":"2024-04-25T15:27:49.213593+0000","last_undegraded":"2024-04-25T15:27:49.213593+0000","last_fullsized":"2024-04-25T15:27:49.213593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:01.513008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418455+0000","last_change":"2024-04-25T15:27:47.771765+0000","last_active":"2024-04-25T15:27:50.418455+0000","last_peered":"2024-04-25T15:27:50.418455+0000","last_clean":"2024-04-25T15:27:50.418455+0000","last_became_active":"2024-04-25T15:27:47.771399+0000","last_became_peered":"2024-04-25T15:27:47.771399+0000","last_unstale":"2024-04-25T15:27:50.418455+0000","last_undegraded":"2024-04-25T15:27:50.418455+0000","last_fullsized":"2024-04-25T15:27:50.418455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:51.125252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774961+0000","last_change":"2024-04-25T15:27:47.775846+0000","last_active":"2024-04-25T15:27:48.774961+0000","last_peered":"2024-04-25T15:27:48.774961+0000","last_clean":"2024-04-25T15:27:48.774961+0000","last_became_active":"2024-04-25T15:27:47.775533+0000","last_became_peered":"2024-04-25T15:27:47.775533+0000","last_unstale":"2024-04-25T15:27:48.774961+0000","last_undegraded":"2024-04-25T15:27:48.774961+0000","last_fullsized":"2024-04-25T15:27:48.774961+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:38.496000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418370+0000","last_change":"2024-04-25T15:27:47.772773+0000","last_active":"2024-04-25T15:27:50.418370+0000","last_peered":"2024-04-25T15:27:50.418370+0000","last_clean":"2024-04-25T15:27:50.418370+0000","last_became_active":"2024-04-25T15:27:47.772468+0000","last_became_peered":"2024-04-25T15:27:47.772468+0000","last_unstale":"2024-04-25T15:27:50.418370+0000","last_undegraded":"2024-04-25T15:27:50.418370+0000","last_fullsized":"2024-04-25T15:27:50.418370+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:48.543108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213072+0000","last_change":"2024-04-25T15:27:47.781800+0000","last_active":"2024-04-25T15:27:49.213072+0000","last_peered":"2024-04-25T15:27:49.213072+0000","last_clean":"2024-04-25T15:27:49.213072+0000","last_became_active":"2024-04-25T15:27:47.781483+0000","last_became_peered":"2024-04-25T15:27:47.781483+0000","last_unstale":"2024-04-25T15:27:49.213072+0000","last_undegraded":"2024-04-25T15:27:49.213072+0000","last_fullsized":"2024-04-25T15:27:49.213072+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:04:59.972266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762097+0000","last_change":"2024-04-25T15:27:47.767384+0000","last_active":"2024-04-25T15:27:49.762097+0000","last_peered":"2024-04-25T15:27:49.762097+0000","last_clean":"2024-04-25T15:27:49.762097+0000","last_became_active":"2024-04-25T15:27:47.766973+0000","last_became_peered":"2024-04-25T15:27:47.766973+0000","last_unstale":"2024-04-25T15:27:49.762097+0000","last_undegraded":"2024-04-25T15:27:49.762097+0000","last_fullsized":"2024-04-25T15:27:49.762097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:53.388615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213254+0000","last_change":"2024-04-25T15:27:47.786432+0000","last_active":"2024-04-25T15:27:49.213254+0000","last_peered":"2024-04-25T15:27:49.213254+0000","last_clean":"2024-04-25T15:27:49.213254+0000","last_became_active":"2024-04-25T15:27:47.786105+0000","last_became_peered":"2024-04-25T15:27:47.786105+0000","last_unstale":"2024-04-25T15:27:49.213254+0000","last_undegraded":"2024-04-25T15:27:49.213254+0000","last_fullsized":"2024-04-25T15:27:49.213254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:52.241260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418447+0000","last_change":"2024-04-25T15:27:47.782985+0000","last_active":"2024-04-25T15:27:50.418447+0000","last_peered":"2024-04-25T15:27:50.418447+0000","last_clean":"2024-04-25T15:27:50.418447+0000","last_became_active":"2024-04-25T15:27:47.782663+0000","last_became_peered":"2024-04-25T15:27:47.782663+0000","last_unstale":"2024-04-25T15:27:50.418447+0000","last_undegraded":"2024-04-25T15:27:50.418447+0000","last_fullsized":"2024-04-25T15:27:50.418447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:17.900479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774518+0000","last_change":"2024-04-25T15:27:47.785807+0000","last_active":"2024-04-25T15:27:48.774518+0000","last_peered":"2024-04-25T15:27:48.774518+0000","last_clean":"2024-04-25T15:27:48.774518+0000","last_became_active":"2024-04-25T15:27:47.785440+0000","last_became_peered":"2024-04-25T15:27:47.785440+0000","last_unstale":"2024-04-25T15:27:48.774518+0000","last_undegraded":"2024-04-25T15:27:48.774518+0000","last_fullsized":"2024-04-25T15:27:48.774518+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:55.524808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745009+0000","last_change":"2024-04-25T15:27:47.771746+0000","last_active":"2024-04-25T15:27:49.745009+0000","last_peered":"2024-04-25T15:27:49.745009+0000","last_clean":"2024-04-25T15:27:49.745009+0000","last_became_active":"2024-04-25T15:27:47.771418+0000","last_became_peered":"2024-04-25T15:27:47.771418+0000","last_unstale":"2024-04-25T15:27:49.745009+0000","last_undegraded":"2024-04-25T15:27:49.745009+0000","last_fullsized":"2024-04-25T15:27:49.745009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:33.146166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775074+0000","last_change":"2024-04-25T15:27:47.784248+0000","last_active":"2024-04-25T15:27:48.775074+0000","last_peered":"2024-04-25T15:27:48.775074+0000","last_clean":"2024-04-25T15:27:48.775074+0000","last_became_active":"2024-04-25T15:27:47.783953+0000","last_became_peered":"2024-04-25T15:27:47.783953+0000","last_unstale":"2024-04-25T15:27:48.775074+0000","last_undegraded":"2024-04-25T15:27:48.775074+0000","last_fullsized":"2024-04-25T15:27:48.775074+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:34.828308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773727+0000","last_change":"2024-04-25T15:27:47.777009+0000","last_active":"2024-04-25T15:27:48.773727+0000","last_peered":"2024-04-25T15:27:48.773727+0000","last_clean":"2024-04-25T15:27:48.773727+0000","last_became_active":"2024-04-25T15:27:47.776670+0000","last_became_peered":"2024-04-25T15:27:47.776670+0000","last_unstale":"2024-04-25T15:27:48.773727+0000","last_undegraded":"2024-04-25T15:27:48.773727+0000","last_fullsized":"2024-04-25T15:27:48.773727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:02.213596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418208+0000","last_change":"2024-04-25T15:27:47.778816+0000","last_active":"2024-04-25T15:27:50.418208+0000","last_peered":"2024-04-25T15:27:50.418208+0000","last_clean":"2024-04-25T15:27:50.418208+0000","last_became_active":"2024-04-25T15:27:47.778428+0000","last_became_peered":"2024-04-25T15:27:47.778428+0000","last_unstale":"2024-04-25T15:27:50.418208+0000","last_undegraded":"2024-04-25T15:27:50.418208+0000","last_fullsized":"2024-04-25T15:27:50.418208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:14.451791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773448+0000","last_change":"2024-04-25T15:27:47.765971+0000","last_active":"2024-04-25T15:27:48.773448+0000","last_peered":"2024-04-25T15:27:48.773448+0000","last_clean":"2024-04-25T15:27:48.773448+0000","last_became_active":"2024-04-25T15:27:47.765617+0000","last_became_peered":"2024-04-25T15:27:47.765617+0000","last_unstale":"2024-04-25T15:27:48.773448+0000","last_undegraded":"2024-04-25T15:27:48.773448+0000","last_fullsized":"2024-04-25T15:27:48.773448+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:37.274297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773968+0000","last_change":"2024-04-25T15:27:47.777354+0000","last_active":"2024-04-25T15:27:48.773968+0000","last_peered":"2024-04-25T15:27:48.773968+0000","last_clean":"2024-04-25T15:27:48.773968+0000","last_became_active":"2024-04-25T15:27:47.777028+0000","last_became_peered":"2024-04-25T15:27:47.777028+0000","last_unstale":"2024-04-25T15:27:48.773968+0000","last_undegraded":"2024-04-25T15:27:48.773968+0000","last_fullsized":"2024-04-25T15:27:48.773968+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:49.491948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418608+0000","last_change":"2024-04-25T15:27:47.782281+0000","last_active":"2024-04-25T15:27:50.418608+0000","last_peered":"2024-04-25T15:27:50.418608+0000","last_clean":"2024-04-25T15:27:50.418608+0000","last_became_active":"2024-04-25T15:27:47.781895+0000","last_became_peered":"2024-04-25T15:27:47.781895+0000","last_unstale":"2024-04-25T15:27:50.418608+0000","last_undegraded":"2024-04-25T15:27:50.418608+0000","last_fullsized":"2024-04-25T15:27:50.418608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.571409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775202+0000","last_change":"2024-04-25T15:27:47.774789+0000","last_active":"2024-04-25T15:27:48.775202+0000","last_peered":"2024-04-25T15:27:48.775202+0000","last_clean":"2024-04-25T15:27:48.775202+0000","last_became_active":"2024-04-25T15:27:47.774000+0000","last_became_peered":"2024-04-25T15:27:47.774000+0000","last_unstale":"2024-04-25T15:27:48.775202+0000","last_undegraded":"2024-04-25T15:27:48.775202+0000","last_fullsized":"2024-04-25T15:27:48.775202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:41:25.750743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213244+0000","last_change":"2024-04-25T15:27:47.786710+0000","last_active":"2024-04-25T15:27:49.213244+0000","last_peered":"2024-04-25T15:27:49.213244+0000","last_clean":"2024-04-25T15:27:49.213244+0000","last_became_active":"2024-04-25T15:27:47.786335+0000","last_became_peered":"2024-04-25T15:27:47.786335+0000","last_unstale":"2024-04-25T15:27:49.213244+0000","last_undegraded":"2024-04-25T15:27:49.213244+0000","last_fullsized":"2024-04-25T15:27:49.213244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:55.815321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773849+0000","last_change":"2024-04-25T15:27:47.767587+0000","last_active":"2024-04-25T15:27:48.773849+0000","last_peered":"2024-04-25T15:27:48.773849+0000","last_clean":"2024-04-25T15:27:48.773849+0000","last_became_active":"2024-04-25T15:27:47.767230+0000","last_became_peered":"2024-04-25T15:27:47.767230+0000","last_unstale":"2024-04-25T15:27:48.773849+0000","last_undegraded":"2024-04-25T15:27:48.773849+0000","last_fullsized":"2024-04-25T15:27:48.773849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:25.374232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774541+0000","last_change":"2024-04-25T15:27:47.765967+0000","last_active":"2024-04-25T15:27:48.774541+0000","last_peered":"2024-04-25T15:27:48.774541+0000","last_clean":"2024-04-25T15:27:48.774541+0000","last_became_active":"2024-04-25T15:27:47.765603+0000","last_became_peered":"2024-04-25T15:27:47.765603+0000","last_unstale":"2024-04-25T15:27:48.774541+0000","last_undegraded":"2024-04-25T15:27:48.774541+0000","last_fullsized":"2024-04-25T15:27:48.774541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:55.815321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774196+0000","last_change":"2024-04-25T15:27:47.784518+0000","last_active":"2024-04-25T15:27:48.774196+0000","last_peered":"2024-04-25T15:27:48.774196+0000","last_clean":"2024-04-25T15:27:48.774196+0000","last_became_active":"2024-04-25T15:27:47.784199+0000","last_became_peered":"2024-04-25T15:27:47.784199+0000","last_unstale":"2024-04-25T15:27:48.774196+0000","last_undegraded":"2024-04-25T15:27:48.774196+0000","last_fullsized":"2024-04-25T15:27:48.774196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:24.352702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744707+0000","last_change":"2024-04-25T15:27:47.777682+0000","last_active":"2024-04-25T15:27:49.744707+0000","last_peered":"2024-04-25T15:27:49.744707+0000","last_clean":"2024-04-25T15:27:49.744707+0000","last_became_active":"2024-04-25T15:27:47.777377+0000","last_became_peered":"2024-04-25T15:27:47.777377+0000","last_unstale":"2024-04-25T15:27:49.744707+0000","last_undegraded":"2024-04-25T15:27:49.744707+0000","last_fullsized":"2024-04-25T15:27:49.744707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:25.374232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744636+0000","last_change":"2024-04-25T15:27:47.780015+0000","last_active":"2024-04-25T15:27:49.744636+0000","last_peered":"2024-04-25T15:27:49.744636+0000","last_clean":"2024-04-25T15:27:49.744636+0000","last_became_active":"2024-04-25T15:27:47.779610+0000","last_became_peered":"2024-04-25T15:27:47.779610+0000","last_unstale":"2024-04-25T15:27:49.744636+0000","last_undegraded":"2024-04-25T15:27:49.744636+0000","last_fullsized":"2024-04-25T15:27:49.744636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:02.213596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774260+0000","last_change":"2024-04-25T15:27:47.776091+0000","last_active":"2024-04-25T15:27:48.774260+0000","last_peered":"2024-04-25T15:27:48.774260+0000","last_clean":"2024-04-25T15:27:48.774260+0000","last_became_active":"2024-04-25T15:27:47.775811+0000","last_became_peered":"2024-04-25T15:27:47.775811+0000","last_unstale":"2024-04-25T15:27:48.774260+0000","last_undegraded":"2024-04-25T15:27:48.774260+0000","last_fullsized":"2024-04-25T15:27:48.774260+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:04:59.972266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762463+0000","last_change":"2024-04-25T15:27:47.783916+0000","last_active":"2024-04-25T15:27:49.762463+0000","last_peered":"2024-04-25T15:27:49.762463+0000","last_clean":"2024-04-25T15:27:49.762463+0000","last_became_active":"2024-04-25T15:27:47.783621+0000","last_became_peered":"2024-04-25T15:27:47.783621+0000","last_unstale":"2024-04-25T15:27:49.762463+0000","last_undegraded":"2024-04-25T15:27:49.762463+0000","last_fullsized":"2024-04-25T15:27:49.762463+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.749333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418530+0000","last_change":"2024-04-25T15:27:47.772770+0000","last_active":"2024-04-25T15:27:50.418530+0000","last_peered":"2024-04-25T15:27:50.418530+0000","last_clean":"2024-04-25T15:27:50.418530+0000","last_became_active":"2024-04-25T15:27:47.772438+0000","last_became_peered":"2024-04-25T15:27:47.772438+0000","last_unstale":"2024-04-25T15:27:50.418530+0000","last_undegraded":"2024-04-25T15:27:50.418530+0000","last_fullsized":"2024-04-25T15:27:50.418530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:12:50.108831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774387+0000","last_change":"2024-04-25T15:27:47.780616+0000","last_active":"2024-04-25T15:27:48.774387+0000","last_peered":"2024-04-25T15:27:48.774387+0000","last_clean":"2024-04-25T15:27:48.774387+0000","last_became_active":"2024-04-25T15:27:47.780282+0000","last_became_peered":"2024-04-25T15:27:47.780282+0000","last_unstale":"2024-04-25T15:27:48.774387+0000","last_undegraded":"2024-04-25T15:27:48.774387+0000","last_fullsized":"2024-04-25T15:27:48.774387+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:51.306682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417815+0000","last_change":"2024-04-25T15:27:47.775239+0000","last_active":"2024-04-25T15:27:50.417815+0000","last_peered":"2024-04-25T15:27:50.417815+0000","last_clean":"2024-04-25T15:27:50.417815+0000","last_became_active":"2024-04-25T15:27:47.774921+0000","last_became_peered":"2024-04-25T15:27:47.774921+0000","last_unstale":"2024-04-25T15:27:50.417815+0000","last_undegraded":"2024-04-25T15:27:50.417815+0000","last_fullsized":"2024-04-25T15:27:50.417815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:45.624627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762189+0000","last_change":"2024-04-25T15:27:47.773714+0000","last_active":"2024-04-25T15:27:49.762189+0000","last_peered":"2024-04-25T15:27:49.762189+0000","last_clean":"2024-04-25T15:27:49.762189+0000","last_became_active":"2024-04-25T15:27:47.773375+0000","last_became_peered":"2024-04-25T15:27:47.773375+0000","last_unstale":"2024-04-25T15:27:49.762189+0000","last_undegraded":"2024-04-25T15:27:49.762189+0000","last_fullsized":"2024-04-25T15:27:49.762189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:39.042562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762438+0000","last_change":"2024-04-25T15:27:47.783321+0000","last_active":"2024-04-25T15:27:49.762438+0000","last_peered":"2024-04-25T15:27:49.762438+0000","last_clean":"2024-04-25T15:27:49.762438+0000","last_became_active":"2024-04-25T15:27:47.782991+0000","last_became_peered":"2024-04-25T15:27:47.782991+0000","last_unstale":"2024-04-25T15:27:49.762438+0000","last_undegraded":"2024-04-25T15:27:49.762438+0000","last_fullsized":"2024-04-25T15:27:49.762438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:16.437778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213171+0000","last_change":"2024-04-25T15:27:47.780808+0000","last_active":"2024-04-25T15:27:49.213171+0000","last_peered":"2024-04-25T15:27:49.213171+0000","last_clean":"2024-04-25T15:27:49.213171+0000","last_became_active":"2024-04-25T15:27:47.780464+0000","last_became_peered":"2024-04-25T15:27:47.780464+0000","last_unstale":"2024-04-25T15:27:49.213171+0000","last_undegraded":"2024-04-25T15:27:49.213171+0000","last_fullsized":"2024-04-25T15:27:49.213171+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:51.306682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745152+0000","last_change":"2024-04-25T15:27:47.780047+0000","last_active":"2024-04-25T15:27:49.745152+0000","last_peered":"2024-04-25T15:27:49.745152+0000","last_clean":"2024-04-25T15:27:49.745152+0000","last_became_active":"2024-04-25T15:27:47.779713+0000","last_became_peered":"2024-04-25T15:27:47.779713+0000","last_unstale":"2024-04-25T15:27:49.745152+0000","last_undegraded":"2024-04-25T15:27:49.745152+0000","last_fullsized":"2024-04-25T15:27:49.745152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:38.522279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"66'4","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745343+0000","last_change":"2024-04-25T15:27:46.756488+0000","last_active":"2024-04-25T15:27:49.745343+0000","last_peered":"2024-04-25T15:27:49.745343+0000","last_clean":"2024-04-25T15:27:49.745343+0000","last_became_active":"2024-04-25T15:27:46.756137+0000","last_became_peered":"2024-04-25T15:27:46.756137+0000","last_unstale":"2024-04-25T15:27:49.745343+0000","last_undegraded":"2024-04-25T15:27:49.745343+0000","last_fullsized":"2024-04-25T15:27:49.745343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:34.541226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418778+0000","last_change":"2024-04-25T15:27:47.780333+0000","last_active":"2024-04-25T15:27:50.418778+0000","last_peered":"2024-04-25T15:27:50.418778+0000","last_clean":"2024-04-25T15:27:50.418778+0000","last_became_active":"2024-04-25T15:27:47.780000+0000","last_became_peered":"2024-04-25T15:27:47.780000+0000","last_unstale":"2024-04-25T15:27:50.418778+0000","last_undegraded":"2024-04-25T15:27:50.418778+0000","last_fullsized":"2024-04-25T15:27:50.418778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:40:47.662951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773969+0000","last_change":"2024-04-25T15:27:46.748183+0000","last_active":"2024-04-25T15:27:48.773969+0000","last_peered":"2024-04-25T15:27:48.773969+0000","last_clean":"2024-04-25T15:27:48.773969+0000","last_became_active":"2024-04-25T15:27:46.747854+0000","last_became_peered":"2024-04-25T15:27:46.747854+0000","last_unstale":"2024-04-25T15:27:48.773969+0000","last_undegraded":"2024-04-25T15:27:48.773969+0000","last_fullsized":"2024-04-25T15:27:48.773969+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:58.559219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213368+0000","last_change":"2024-04-25T15:27:47.775869+0000","last_active":"2024-04-25T15:27:49.213368+0000","last_peered":"2024-04-25T15:27:49.213368+0000","last_clean":"2024-04-25T15:27:49.213368+0000","last_became_active":"2024-04-25T15:27:47.775636+0000","last_became_peered":"2024-04-25T15:27:47.775636+0000","last_unstale":"2024-04-25T15:27:49.213368+0000","last_undegraded":"2024-04-25T15:27:49.213368+0000","last_fullsized":"2024-04-25T15:27:49.213368+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:53.391175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774297+0000","last_change":"2024-04-25T15:27:46.750527+0000","last_active":"2024-04-25T15:27:48.774297+0000","last_peered":"2024-04-25T15:27:48.774297+0000","last_clean":"2024-04-25T15:27:48.774297+0000","last_became_active":"2024-04-25T15:27:46.749838+0000","last_became_peered":"2024-04-25T15:27:46.749838+0000","last_unstale":"2024-04-25T15:27:48.774297+0000","last_undegraded":"2024-04-25T15:27:48.774297+0000","last_fullsized":"2024-04-25T15:27:48.774297+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:04.081872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745201+0000","last_change":"2024-04-25T15:27:47.770261+0000","last_active":"2024-04-25T15:27:49.745201+0000","last_peered":"2024-04-25T15:27:49.745201+0000","last_clean":"2024-04-25T15:27:49.745201+0000","last_became_active":"2024-04-25T15:27:47.769514+0000","last_became_peered":"2024-04-25T15:27:47.769514+0000","last_unstale":"2024-04-25T15:27:49.745201+0000","last_undegraded":"2024-04-25T15:27:49.745201+0000","last_fullsized":"2024-04-25T15:27:49.745201+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:15.809204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213725+0000","last_change":"2024-04-25T15:27:46.747175+0000","last_active":"2024-04-25T15:27:49.213725+0000","last_peered":"2024-04-25T15:27:49.213725+0000","last_clean":"2024-04-25T15:27:49.213725+0000","last_became_active":"2024-04-25T15:27:46.746878+0000","last_became_peered":"2024-04-25T15:27:46.746878+0000","last_unstale":"2024-04-25T15:27:49.213725+0000","last_undegraded":"2024-04-25T15:27:49.213725+0000","last_fullsized":"2024-04-25T15:27:49.213725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:02.088263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762349+0000","last_change":"2024-04-25T15:27:47.774555+0000","last_active":"2024-04-25T15:27:49.762349+0000","last_peered":"2024-04-25T15:27:49.762349+0000","last_clean":"2024-04-25T15:27:49.762349+0000","last_became_active":"2024-04-25T15:27:47.774254+0000","last_became_peered":"2024-04-25T15:27:47.774254+0000","last_unstale":"2024-04-25T15:27:49.762349+0000","last_undegraded":"2024-04-25T15:27:49.762349+0000","last_fullsized":"2024-04-25T15:27:49.762349+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:15:52.086564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418002+0000","last_change":"2024-04-25T15:27:46.748931+0000","last_active":"2024-04-25T15:27:50.418002+0000","last_peered":"2024-04-25T15:27:50.418002+0000","last_clean":"2024-04-25T15:27:50.418002+0000","last_became_active":"2024-04-25T15:27:46.748400+0000","last_became_peered":"2024-04-25T15:27:46.748400+0000","last_unstale":"2024-04-25T15:27:50.418002+0000","last_undegraded":"2024-04-25T15:27:50.418002+0000","last_fullsized":"2024-04-25T15:27:50.418002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:14.803258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774690+0000","last_change":"2024-04-25T15:27:47.776089+0000","last_active":"2024-04-25T15:27:48.774690+0000","last_peered":"2024-04-25T15:27:48.774690+0000","last_clean":"2024-04-25T15:27:48.774690+0000","last_became_active":"2024-04-25T15:27:47.775768+0000","last_became_peered":"2024-04-25T15:27:47.775768+0000","last_unstale":"2024-04-25T15:27:48.774690+0000","last_undegraded":"2024-04-25T15:27:48.774690+0000","last_fullsized":"2024-04-25T15:27:48.774690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:52.241260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.212951+0000","last_change":"2024-04-25T15:27:46.753325+0000","last_active":"2024-04-25T15:27:49.212951+0000","last_peered":"2024-04-25T15:27:49.212951+0000","last_clean":"2024-04-25T15:27:49.212951+0000","last_became_active":"2024-04-25T15:27:46.753006+0000","last_became_peered":"2024-04-25T15:27:46.753006+0000","last_unstale":"2024-04-25T15:27:49.212951+0000","last_undegraded":"2024-04-25T15:27:49.212951+0000","last_fullsized":"2024-04-25T15:27:49.212951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:37.598034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417912+0000","last_change":"2024-04-25T15:27:47.780022+0000","last_active":"2024-04-25T15:27:50.417912+0000","last_peered":"2024-04-25T15:27:50.417912+0000","last_clean":"2024-04-25T15:27:50.417912+0000","last_became_active":"2024-04-25T15:27:47.779594+0000","last_became_peered":"2024-04-25T15:27:47.779594+0000","last_unstale":"2024-04-25T15:27:50.417912+0000","last_undegraded":"2024-04-25T15:27:50.417912+0000","last_fullsized":"2024-04-25T15:27:50.417912+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:08:06.201716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774864+0000","last_change":"2024-04-25T15:27:46.740274+0000","last_active":"2024-04-25T15:27:48.774864+0000","last_peered":"2024-04-25T15:27:48.774864+0000","last_clean":"2024-04-25T15:27:48.774864+0000","last_became_active":"2024-04-25T15:27:46.739275+0000","last_became_peered":"2024-04-25T15:27:46.739275+0000","last_unstale":"2024-04-25T15:27:48.774864+0000","last_undegraded":"2024-04-25T15:27:48.774864+0000","last_fullsized":"2024-04-25T15:27:48.774864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:37.598034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773476+0000","last_change":"2024-04-25T15:27:47.769807+0000","last_active":"2024-04-25T15:27:48.773476+0000","last_peered":"2024-04-25T15:27:48.773476+0000","last_clean":"2024-04-25T15:27:48.773476+0000","last_became_active":"2024-04-25T15:27:47.769357+0000","last_became_peered":"2024-04-25T15:27:47.769357+0000","last_unstale":"2024-04-25T15:27:48.773476+0000","last_undegraded":"2024-04-25T15:27:48.773476+0000","last_fullsized":"2024-04-25T15:27:48.773476+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:53.115168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417499+0000","last_change":"2024-04-25T15:27:46.748972+0000","last_active":"2024-04-25T15:27:50.417499+0000","last_peered":"2024-04-25T15:27:50.417499+0000","last_clean":"2024-04-25T15:27:50.417499+0000","last_became_active":"2024-04-25T15:27:46.748630+0000","last_became_peered":"2024-04-25T15:27:46.748630+0000","last_unstale":"2024-04-25T15:27:50.417499+0000","last_undegraded":"2024-04-25T15:27:50.417499+0000","last_fullsized":"2024-04-25T15:27:50.417499+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:43.640246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762536+0000","last_change":"2024-04-25T15:27:47.770334+0000","last_active":"2024-04-25T15:27:49.762536+0000","last_peered":"2024-04-25T15:27:49.762536+0000","last_clean":"2024-04-25T15:27:49.762536+0000","last_became_active":"2024-04-25T15:27:47.769970+0000","last_became_peered":"2024-04-25T15:27:47.769970+0000","last_unstale":"2024-04-25T15:27:49.762536+0000","last_undegraded":"2024-04-25T15:27:49.762536+0000","last_fullsized":"2024-04-25T15:27:49.762536+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:11:10.445989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774396+0000","last_change":"2024-04-25T15:27:46.753409+0000","last_active":"2024-04-25T15:27:48.774396+0000","last_peered":"2024-04-25T15:27:48.774396+0000","last_clean":"2024-04-25T15:27:48.774396+0000","last_became_active":"2024-04-25T15:27:46.753090+0000","last_became_peered":"2024-04-25T15:27:46.753090+0000","last_unstale":"2024-04-25T15:27:48.774396+0000","last_undegraded":"2024-04-25T15:27:48.774396+0000","last_fullsized":"2024-04-25T15:27:48.774396+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:06.558947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745205+0000","last_change":"2024-04-25T15:27:47.781191+0000","last_active":"2024-04-25T15:27:49.745205+0000","last_peered":"2024-04-25T15:27:49.745205+0000","last_clean":"2024-04-25T15:27:49.745205+0000","last_became_active":"2024-04-25T15:27:47.780588+0000","last_became_peered":"2024-04-25T15:27:47.780588+0000","last_unstale":"2024-04-25T15:27:49.745205+0000","last_undegraded":"2024-04-25T15:27:49.745205+0000","last_fullsized":"2024-04-25T15:27:49.745205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:32:52.541061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213648+0000","last_change":"2024-04-25T15:27:46.751600+0000","last_active":"2024-04-25T15:27:49.213648+0000","last_peered":"2024-04-25T15:27:49.213648+0000","last_clean":"2024-04-25T15:27:49.213648+0000","last_became_active":"2024-04-25T15:27:46.751255+0000","last_became_peered":"2024-04-25T15:27:46.751255+0000","last_unstale":"2024-04-25T15:27:49.213648+0000","last_undegraded":"2024-04-25T15:27:49.213648+0000","last_fullsized":"2024-04-25T15:27:49.213648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:23.117643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213505+0000","last_change":"2024-04-25T15:27:47.781034+0000","last_active":"2024-04-25T15:27:49.213505+0000","last_peered":"2024-04-25T15:27:49.213505+0000","last_clean":"2024-04-25T15:27:49.213505+0000","last_became_active":"2024-04-25T15:27:47.780713+0000","last_became_peered":"2024-04-25T15:27:47.780713+0000","last_unstale":"2024-04-25T15:27:49.213505+0000","last_undegraded":"2024-04-25T15:27:49.213505+0000","last_fullsized":"2024-04-25T15:27:49.213505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:35.547172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"66'3","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418565+0000","last_change":"2024-04-25T15:27:46.750192+0000","last_active":"2024-04-25T15:27:50.418565+0000","last_peered":"2024-04-25T15:27:50.418565+0000","last_clean":"2024-04-25T15:27:50.418565+0000","last_became_active":"2024-04-25T15:27:46.749859+0000","last_became_peered":"2024-04-25T15:27:46.749859+0000","last_unstale":"2024-04-25T15:27:50.418565+0000","last_undegraded":"2024-04-25T15:27:50.418565+0000","last_fullsized":"2024-04-25T15:27:50.418565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:37.516333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417872+0000","last_change":"2024-04-25T15:27:47.781969+0000","last_active":"2024-04-25T15:27:50.417872+0000","last_peered":"2024-04-25T15:27:50.417872+0000","last_clean":"2024-04-25T15:27:50.417872+0000","last_became_active":"2024-04-25T15:27:47.781565+0000","last_became_peered":"2024-04-25T15:27:47.781565+0000","last_unstale":"2024-04-25T15:27:50.417872+0000","last_undegraded":"2024-04-25T15:27:50.417872+0000","last_fullsized":"2024-04-25T15:27:50.417872+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:52.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"66'6","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775201+0000","last_change":"2024-04-25T15:27:46.751593+0000","last_active":"2024-04-25T15:27:48.775201+0000","last_peered":"2024-04-25T15:27:48.775201+0000","last_clean":"2024-04-25T15:27:48.775201+0000","last_became_active":"2024-04-25T15:27:46.751123+0000","last_became_peered":"2024-04-25T15:27:46.751123+0000","last_unstale":"2024-04-25T15:27:48.775201+0000","last_undegraded":"2024-04-25T15:27:48.775201+0000","last_fullsized":"2024-04-25T15:27:48.775201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:06.332293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744788+0000","last_change":"2024-04-25T15:27:47.784323+0000","last_active":"2024-04-25T15:27:49.744788+0000","last_peered":"2024-04-25T15:27:49.744788+0000","last_clean":"2024-04-25T15:27:49.744788+0000","last_became_active":"2024-04-25T15:27:47.783947+0000","last_became_peered":"2024-04-25T15:27:47.783947+0000","last_unstale":"2024-04-25T15:27:49.744788+0000","last_undegraded":"2024-04-25T15:27:49.744788+0000","last_fullsized":"2024-04-25T15:27:49.744788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:03:38.884451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774979+0000","last_change":"2024-04-25T15:27:46.747059+0000","last_active":"2024-04-25T15:27:48.774979+0000","last_peered":"2024-04-25T15:27:48.774979+0000","last_clean":"2024-04-25T15:27:48.774979+0000","last_became_active":"2024-04-25T15:27:46.746774+0000","last_became_peered":"2024-04-25T15:27:46.746774+0000","last_unstale":"2024-04-25T15:27:48.774979+0000","last_undegraded":"2024-04-25T15:27:48.774979+0000","last_fullsized":"2024-04-25T15:27:48.774979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:25.199824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762364+0000","last_change":"2024-04-25T15:27:47.773677+0000","last_active":"2024-04-25T15:27:49.762364+0000","last_peered":"2024-04-25T15:27:49.762364+0000","last_clean":"2024-04-25T15:27:49.762364+0000","last_became_active":"2024-04-25T15:27:47.773338+0000","last_became_peered":"2024-04-25T15:27:47.773338+0000","last_unstale":"2024-04-25T15:27:49.762364+0000","last_undegraded":"2024-04-25T15:27:49.762364+0000","last_fullsized":"2024-04-25T15:27:49.762364+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:35:52.031977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213176+0000","last_change":"2024-04-25T15:27:46.755998+0000","last_active":"2024-04-25T15:27:49.213176+0000","last_peered":"2024-04-25T15:27:49.213176+0000","last_clean":"2024-04-25T15:27:49.213176+0000","last_became_active":"2024-04-25T15:27:46.755762+0000","last_became_peered":"2024-04-25T15:27:46.755762+0000","last_unstale":"2024-04-25T15:27:49.213176+0000","last_undegraded":"2024-04-25T15:27:49.213176+0000","last_fullsized":"2024-04-25T15:27:49.213176+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:25.199824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773402+0000","last_change":"2024-04-25T15:27:47.774124+0000","last_active":"2024-04-25T15:27:48.773402+0000","last_peered":"2024-04-25T15:27:48.773402+0000","last_clean":"2024-04-25T15:27:48.773402+0000","last_became_active":"2024-04-25T15:27:47.773798+0000","last_became_peered":"2024-04-25T15:27:47.773798+0000","last_unstale":"2024-04-25T15:27:48.773402+0000","last_undegraded":"2024-04-25T15:27:48.773402+0000","last_fullsized":"2024-04-25T15:27:48.773402+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:11:10.445989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762105+0000","last_change":"2024-04-25T15:27:46.753451+0000","last_active":"2024-04-25T15:27:49.762105+0000","last_peered":"2024-04-25T15:27:49.762105+0000","last_clean":"2024-04-25T15:27:49.762105+0000","last_became_active":"2024-04-25T15:27:46.753133+0000","last_became_peered":"2024-04-25T15:27:46.753133+0000","last_unstale":"2024-04-25T15:27:49.762105+0000","last_undegraded":"2024-04-25T15:27:49.762105+0000","last_fullsized":"2024-04-25T15:27:49.762105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:48.486002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417695+0000","last_change":"2024-04-25T15:27:47.779066+0000","last_active":"2024-04-25T15:27:50.417695+0000","last_peered":"2024-04-25T15:27:50.417695+0000","last_clean":"2024-04-25T15:27:50.417695+0000","last_became_active":"2024-04-25T15:27:47.778110+0000","last_became_peered":"2024-04-25T15:27:47.778110+0000","last_unstale":"2024-04-25T15:27:50.417695+0000","last_undegraded":"2024-04-25T15:27:50.417695+0000","last_fullsized":"2024-04-25T15:27:50.417695+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:21.619498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773975+0000","last_change":"2024-04-25T15:27:46.747236+0000","last_active":"2024-04-25T15:27:48.773975+0000","last_peered":"2024-04-25T15:27:48.773975+0000","last_clean":"2024-04-25T15:27:48.773975+0000","last_became_active":"2024-04-25T15:27:46.746401+0000","last_became_peered":"2024-04-25T15:27:46.746401+0000","last_unstale":"2024-04-25T15:27:48.773975+0000","last_undegraded":"2024-04-25T15:27:48.773975+0000","last_fullsized":"2024-04-25T15:27:48.773975+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:37.490054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774707+0000","last_change":"2024-04-25T15:27:47.761466+0000","last_active":"2024-04-25T15:27:48.774707+0000","last_peered":"2024-04-25T15:27:48.774707+0000","last_clean":"2024-04-25T15:27:48.774707+0000","last_became_active":"2024-04-25T15:27:47.761045+0000","last_became_peered":"2024-04-25T15:27:47.761045+0000","last_unstale":"2024-04-25T15:27:48.774707+0000","last_undegraded":"2024-04-25T15:27:48.774707+0000","last_fullsized":"2024-04-25T15:27:48.774707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:53.391175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213608+0000","last_change":"2024-04-25T15:27:46.759114+0000","last_active":"2024-04-25T15:27:49.213608+0000","last_peered":"2024-04-25T15:27:49.213608+0000","last_clean":"2024-04-25T15:27:49.213608+0000","last_became_active":"2024-04-25T15:27:46.758785+0000","last_became_peered":"2024-04-25T15:27:46.758785+0000","last_unstale":"2024-04-25T15:27:49.213608+0000","last_undegraded":"2024-04-25T15:27:49.213608+0000","last_fullsized":"2024-04-25T15:27:49.213608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:06.332293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745353+0000","last_change":"2024-04-25T15:27:47.778525+0000","last_active":"2024-04-25T15:27:49.745353+0000","last_peered":"2024-04-25T15:27:49.745353+0000","last_clean":"2024-04-25T15:27:49.745353+0000","last_became_active":"2024-04-25T15:27:47.778145+0000","last_became_peered":"2024-04-25T15:27:47.778145+0000","last_unstale":"2024-04-25T15:27:49.745353+0000","last_undegraded":"2024-04-25T15:27:49.745353+0000","last_fullsized":"2024-04-25T15:27:49.745353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:18.572778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774631+0000","last_change":"2024-04-25T15:27:46.750313+0000","last_active":"2024-04-25T15:27:48.774631+0000","last_peered":"2024-04-25T15:27:48.774631+0000","last_clean":"2024-04-25T15:27:48.774631+0000","last_became_active":"2024-04-25T15:27:46.749988+0000","last_became_peered":"2024-04-25T15:27:46.749988+0000","last_unstale":"2024-04-25T15:27:48.774631+0000","last_undegraded":"2024-04-25T15:27:48.774631+0000","last_fullsized":"2024-04-25T15:27:48.774631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:02.088263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775131+0000","last_change":"2024-04-25T15:27:47.779927+0000","last_active":"2024-04-25T15:27:48.775131+0000","last_peered":"2024-04-25T15:27:48.775131+0000","last_clean":"2024-04-25T15:27:48.775131+0000","last_became_active":"2024-04-25T15:27:47.779619+0000","last_became_peered":"2024-04-25T15:27:47.779619+0000","last_unstale":"2024-04-25T15:27:48.775131+0000","last_undegraded":"2024-04-25T15:27:48.775131+0000","last_fullsized":"2024-04-25T15:27:48.775131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:32.497676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213471+0000","last_change":"2024-04-25T15:27:46.754470+0000","last_active":"2024-04-25T15:27:49.213471+0000","last_peered":"2024-04-25T15:27:49.213471+0000","last_clean":"2024-04-25T15:27:49.213471+0000","last_became_active":"2024-04-25T15:27:46.753975+0000","last_became_peered":"2024-04-25T15:27:46.753975+0000","last_unstale":"2024-04-25T15:27:49.213471+0000","last_undegraded":"2024-04-25T15:27:49.213471+0000","last_fullsized":"2024-04-25T15:27:49.213471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:30.439782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418043+0000","last_change":"2024-04-25T15:27:47.779053+0000","last_active":"2024-04-25T15:27:50.418043+0000","last_peered":"2024-04-25T15:27:50.418043+0000","last_clean":"2024-04-25T15:27:50.418043+0000","last_became_active":"2024-04-25T15:27:47.777906+0000","last_became_peered":"2024-04-25T15:27:47.777906+0000","last_unstale":"2024-04-25T15:27:50.418043+0000","last_undegraded":"2024-04-25T15:27:50.418043+0000","last_fullsized":"2024-04-25T15:27:50.418043+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:06.828910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744979+0000","last_change":"2024-04-25T15:27:46.754424+0000","last_active":"2024-04-25T15:27:49.744979+0000","last_peered":"2024-04-25T15:27:49.744979+0000","last_clean":"2024-04-25T15:27:49.744979+0000","last_became_active":"2024-04-25T15:27:46.754102+0000","last_became_peered":"2024-04-25T15:27:46.754102+0000","last_unstale":"2024-04-25T15:27:49.744979+0000","last_undegraded":"2024-04-25T15:27:49.744979+0000","last_fullsized":"2024-04-25T15:27:49.744979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:52.385229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417996+0000","last_change":"2024-04-25T15:27:47.777884+0000","last_active":"2024-04-25T15:27:50.417996+0000","last_peered":"2024-04-25T15:27:50.417996+0000","last_clean":"2024-04-25T15:27:50.417996+0000","last_became_active":"2024-04-25T15:27:47.777532+0000","last_became_peered":"2024-04-25T15:27:47.777532+0000","last_unstale":"2024-04-25T15:27:50.417996+0000","last_undegraded":"2024-04-25T15:27:50.417996+0000","last_fullsized":"2024-04-25T15:27:50.417996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:43.448609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213421+0000","last_change":"2024-04-25T15:27:46.756182+0000","last_active":"2024-04-25T15:27:49.213421+0000","last_peered":"2024-04-25T15:27:49.213421+0000","last_clean":"2024-04-25T15:27:49.213421+0000","last_became_active":"2024-04-25T15:27:46.755789+0000","last_became_peered":"2024-04-25T15:27:46.755789+0000","last_unstale":"2024-04-25T15:27:49.213421+0000","last_undegraded":"2024-04-25T15:27:49.213421+0000","last_fullsized":"2024-04-25T15:27:49.213421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:04.081872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417663+0000","last_change":"2024-04-25T15:27:47.784326+0000","last_active":"2024-04-25T15:27:50.417663+0000","last_peered":"2024-04-25T15:27:50.417663+0000","last_clean":"2024-04-25T15:27:50.417663+0000","last_became_active":"2024-04-25T15:27:47.784004+0000","last_became_peered":"2024-04-25T15:27:47.784004+0000","last_unstale":"2024-04-25T15:27:50.417663+0000","last_undegraded":"2024-04-25T15:27:50.417663+0000","last_fullsized":"2024-04-25T15:27:50.417663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:28.663954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762408+0000","last_change":"2024-04-25T15:27:46.751621+0000","last_active":"2024-04-25T15:27:49.762408+0000","last_peered":"2024-04-25T15:27:49.762408+0000","last_clean":"2024-04-25T15:27:49.762408+0000","last_became_active":"2024-04-25T15:27:46.751328+0000","last_became_peered":"2024-04-25T15:27:46.751328+0000","last_unstale":"2024-04-25T15:27:49.762408+0000","last_undegraded":"2024-04-25T15:27:49.762408+0000","last_fullsized":"2024-04-25T15:27:49.762408+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:23.346756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418293+0000","last_change":"2024-04-25T15:27:47.785507+0000","last_active":"2024-04-25T15:27:50.418293+0000","last_peered":"2024-04-25T15:27:50.418293+0000","last_clean":"2024-04-25T15:27:50.418293+0000","last_became_active":"2024-04-25T15:27:47.785197+0000","last_became_peered":"2024-04-25T15:27:47.785197+0000","last_unstale":"2024-04-25T15:27:50.418293+0000","last_undegraded":"2024-04-25T15:27:50.418293+0000","last_fullsized":"2024-04-25T15:27:50.418293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:58.461399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773804+0000","last_change":"2024-04-25T15:27:46.748665+0000","last_active":"2024-04-25T15:27:48.773804+0000","last_peered":"2024-04-25T15:27:48.773804+0000","last_clean":"2024-04-25T15:27:48.773804+0000","last_became_active":"2024-04-25T15:27:46.748364+0000","last_became_peered":"2024-04-25T15:27:46.748364+0000","last_unstale":"2024-04-25T15:27:48.773804+0000","last_undegraded":"2024-04-25T15:27:48.773804+0000","last_fullsized":"2024-04-25T15:27:48.773804+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:30.416165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773844+0000","last_change":"2024-04-25T15:27:47.783356+0000","last_active":"2024-04-25T15:27:48.773844+0000","last_peered":"2024-04-25T15:27:48.773844+0000","last_clean":"2024-04-25T15:27:48.773844+0000","last_became_active":"2024-04-25T15:27:47.782999+0000","last_became_peered":"2024-04-25T15:27:47.782999+0000","last_unstale":"2024-04-25T15:27:48.773844+0000","last_undegraded":"2024-04-25T15:27:48.773844+0000","last_fullsized":"2024-04-25T15:27:48.773844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:03:38.884451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213676+0000","last_change":"2024-04-25T15:27:46.759084+0000","last_active":"2024-04-25T15:27:49.213676+0000","last_peered":"2024-04-25T15:27:49.213676+0000","last_clean":"2024-04-25T15:27:49.213676+0000","last_became_active":"2024-04-25T15:27:46.758757+0000","last_became_peered":"2024-04-25T15:27:46.758757+0000","last_unstale":"2024-04-25T15:27:49.213676+0000","last_undegraded":"2024-04-25T15:27:49.213676+0000","last_fullsized":"2024-04-25T15:27:49.213676+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:44.420376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762538+0000","last_change":"2024-04-25T15:27:47.782899+0000","last_active":"2024-04-25T15:27:49.762538+0000","last_peered":"2024-04-25T15:27:49.762538+0000","last_clean":"2024-04-25T15:27:49.762538+0000","last_became_active":"2024-04-25T15:27:47.782551+0000","last_became_peered":"2024-04-25T15:27:47.782551+0000","last_unstale":"2024-04-25T15:27:49.762538+0000","last_undegraded":"2024-04-25T15:27:49.762538+0000","last_fullsized":"2024-04-25T15:27:49.762538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:37.274297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762643+0000","last_change":"2024-04-25T15:27:46.753206+0000","last_active":"2024-04-25T15:27:49.762643+0000","last_peered":"2024-04-25T15:27:49.762643+0000","last_clean":"2024-04-25T15:27:49.762643+0000","last_became_active":"2024-04-25T15:27:46.752883+0000","last_became_peered":"2024-04-25T15:27:46.752883+0000","last_unstale":"2024-04-25T15:27:49.762643+0000","last_undegraded":"2024-04-25T15:27:49.762643+0000","last_fullsized":"2024-04-25T15:27:49.762643+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:04:58.966320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745011+0000","last_change":"2024-04-25T15:27:47.782136+0000","last_active":"2024-04-25T15:27:49.745011+0000","last_peered":"2024-04-25T15:27:49.745011+0000","last_clean":"2024-04-25T15:27:49.745011+0000","last_became_active":"2024-04-25T15:27:47.781773+0000","last_became_peered":"2024-04-25T15:27:47.781773+0000","last_unstale":"2024-04-25T15:27:49.745011+0000","last_undegraded":"2024-04-25T15:27:49.745011+0000","last_fullsized":"2024-04-25T15:27:49.745011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:41.235966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418144+0000","last_change":"2024-04-25T15:27:46.753592+0000","last_active":"2024-04-25T15:27:50.418144+0000","last_peered":"2024-04-25T15:27:50.418144+0000","last_clean":"2024-04-25T15:27:50.418144+0000","last_became_active":"2024-04-25T15:27:46.753163+0000","last_became_peered":"2024-04-25T15:27:46.753163+0000","last_unstale":"2024-04-25T15:27:50.418144+0000","last_undegraded":"2024-04-25T15:27:50.418144+0000","last_fullsized":"2024-04-25T15:27:50.418144+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:51:09.829896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745295+0000","last_change":"2024-04-25T15:27:47.770518+0000","last_active":"2024-04-25T15:27:49.745295+0000","last_peered":"2024-04-25T15:27:49.745295+0000","last_clean":"2024-04-25T15:27:49.745295+0000","last_became_active":"2024-04-25T15:27:47.769861+0000","last_became_peered":"2024-04-25T15:27:47.769861+0000","last_unstale":"2024-04-25T15:27:49.745295+0000","last_undegraded":"2024-04-25T15:27:49.745295+0000","last_fullsized":"2024-04-25T15:27:49.745295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:55.281915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773698+0000","last_change":"2024-04-25T15:27:46.744686+0000","last_active":"2024-04-25T15:27:48.773698+0000","last_peered":"2024-04-25T15:27:48.773698+0000","last_clean":"2024-04-25T15:27:48.773698+0000","last_became_active":"2024-04-25T15:27:46.744321+0000","last_became_peered":"2024-04-25T15:27:46.744321+0000","last_unstale":"2024-04-25T15:27:48.773698+0000","last_undegraded":"2024-04-25T15:27:48.773698+0000","last_fullsized":"2024-04-25T15:27:48.773698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:00.507062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418216+0000","last_change":"2024-04-25T15:27:47.775042+0000","last_active":"2024-04-25T15:27:50.418216+0000","last_peered":"2024-04-25T15:27:50.418216+0000","last_clean":"2024-04-25T15:27:50.418216+0000","last_became_active":"2024-04-25T15:27:47.774708+0000","last_became_peered":"2024-04-25T15:27:47.774708+0000","last_unstale":"2024-04-25T15:27:50.418216+0000","last_undegraded":"2024-04-25T15:27:50.418216+0000","last_fullsized":"2024-04-25T15:27:50.418216+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:43.212650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417721+0000","last_change":"2024-04-25T15:27:46.749552+0000","last_active":"2024-04-25T15:27:50.417721+0000","last_peered":"2024-04-25T15:27:50.417721+0000","last_clean":"2024-04-25T15:27:50.417721+0000","last_became_active":"2024-04-25T15:27:46.749242+0000","last_became_peered":"2024-04-25T15:27:46.749242+0000","last_unstale":"2024-04-25T15:27:50.417721+0000","last_undegraded":"2024-04-25T15:27:50.417721+0000","last_fullsized":"2024-04-25T15:27:50.417721+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:52.544769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213168+0000","last_change":"2024-04-25T15:27:47.777251+0000","last_active":"2024-04-25T15:27:49.213168+0000","last_peered":"2024-04-25T15:27:49.213168+0000","last_clean":"2024-04-25T15:27:49.213168+0000","last_became_active":"2024-04-25T15:27:47.776932+0000","last_became_peered":"2024-04-25T15:27:47.776932+0000","last_unstale":"2024-04-25T15:27:49.213168+0000","last_undegraded":"2024-04-25T15:27:49.213168+0000","last_fullsized":"2024-04-25T15:27:49.213168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:55.524808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774777+0000","last_change":"2024-04-25T15:27:46.746473+0000","last_active":"2024-04-25T15:27:48.774777+0000","last_peered":"2024-04-25T15:27:48.774777+0000","last_clean":"2024-04-25T15:27:48.774777+0000","last_became_active":"2024-04-25T15:27:46.746083+0000","last_became_peered":"2024-04-25T15:27:46.746083+0000","last_unstale":"2024-04-25T15:27:48.774777+0000","last_undegraded":"2024-04-25T15:27:48.774777+0000","last_fullsized":"2024-04-25T15:27:48.774777+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:41.597367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774944+0000","last_change":"2024-04-25T15:27:47.775285+0000","last_active":"2024-04-25T15:27:48.774944+0000","last_peered":"2024-04-25T15:27:48.774944+0000","last_clean":"2024-04-25T15:27:48.774944+0000","last_became_active":"2024-04-25T15:27:47.774923+0000","last_became_peered":"2024-04-25T15:27:47.774923+0000","last_unstale":"2024-04-25T15:27:48.774944+0000","last_undegraded":"2024-04-25T15:27:48.774944+0000","last_fullsized":"2024-04-25T15:27:48.774944+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:03.094209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773997+0000","last_change":"2024-04-25T15:27:46.745918+0000","last_active":"2024-04-25T15:27:48.773997+0000","last_peered":"2024-04-25T15:27:48.773997+0000","last_clean":"2024-04-25T15:27:48.773997+0000","last_became_active":"2024-04-25T15:27:46.745596+0000","last_became_peered":"2024-04-25T15:27:46.745596+0000","last_unstale":"2024-04-25T15:27:48.773997+0000","last_undegraded":"2024-04-25T15:27:48.773997+0000","last_fullsized":"2024-04-25T15:27:48.773997+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:30.439782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.212928+0000","last_change":"2024-04-25T15:27:47.785204+0000","last_active":"2024-04-25T15:27:49.212928+0000","last_peered":"2024-04-25T15:27:49.212928+0000","last_clean":"2024-04-25T15:27:49.212928+0000","last_became_active":"2024-04-25T15:27:47.784676+0000","last_became_peered":"2024-04-25T15:27:47.784676+0000","last_unstale":"2024-04-25T15:27:49.212928+0000","last_undegraded":"2024-04-25T15:27:49.212928+0000","last_fullsized":"2024-04-25T15:27:49.212928+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:24.352702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213338+0000","last_change":"2024-04-25T15:27:46.746186+0000","last_active":"2024-04-25T15:27:49.213338+0000","last_peered":"2024-04-25T15:27:49.213338+0000","last_clean":"2024-04-25T15:27:49.213338+0000","last_became_active":"2024-04-25T15:27:46.745867+0000","last_became_peered":"2024-04-25T15:27:46.745867+0000","last_unstale":"2024-04-25T15:27:49.213338+0000","last_undegraded":"2024-04-25T15:27:49.213338+0000","last_fullsized":"2024-04-25T15:27:49.213338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:37.490054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417865+0000","last_change":"2024-04-25T15:27:47.784218+0000","last_active":"2024-04-25T15:27:50.417865+0000","last_peered":"2024-04-25T15:27:50.417865+0000","last_clean":"2024-04-25T15:27:50.417865+0000","last_became_active":"2024-04-25T15:27:47.783851+0000","last_became_peered":"2024-04-25T15:27:47.783851+0000","last_unstale":"2024-04-25T15:27:50.417865+0000","last_undegraded":"2024-04-25T15:27:50.417865+0000","last_fullsized":"2024-04-25T15:27:50.417865+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:06.431534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213046+0000","last_change":"2024-04-25T15:27:46.755483+0000","last_active":"2024-04-25T15:27:49.213046+0000","last_peered":"2024-04-25T15:27:49.213046+0000","last_clean":"2024-04-25T15:27:49.213046+0000","last_became_active":"2024-04-25T15:27:46.755160+0000","last_became_peered":"2024-04-25T15:27:46.755160+0000","last_unstale":"2024-04-25T15:27:49.213046+0000","last_undegraded":"2024-04-25T15:27:49.213046+0000","last_fullsized":"2024-04-25T15:27:49.213046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:06.558947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744656+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:27:49.744656+0000","last_peered":"2024-04-25T15:27:49.744656+0000","last_clean":"2024-04-25T15:27:49.744656+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:27:49.744656+0000","last_undegraded":"2024-04-25T15:27:49.744656+0000","last_fullsized":"2024-04-25T15:27:49.744656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":1417216,"data_stored":932499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":341607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":341605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":341585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":341627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":827392,"data_stored":343127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":835584,"data_stored":343743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":1409024,"data_stored":931883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:27:56.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:27:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.473+0000 7f962c1b4640 1 Processor -- start 2024-04-25T15:27:56.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.473+0000 7f962c1b4640 1 -- start start 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f962c1b4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 0x7f962415e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f962c1b4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96240ce930 con 0x7f96240ce520 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f962c1b4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f962406c9d0 con 0x7f96240b7f50 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f962c1b4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f962415e860 con 0x7f962406c5c0 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f9629f50640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96240ce520 0x7f96240b7840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55648/0 (socket says 172.21.15.67:55648) 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f9629f50640 1 -- 172.21.15.67:0/651525929 learned_addr learned my addr 172.21.15.67:0/651525929 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f962a751640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 0x7f962415e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591649371 0 0) 0x7f96240ce930 con 0x7f96240ce520 2024-04-25T15:27:56.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.475+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9610003520 con 0x7f96240ce520 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 85426615 0 0) 0x7f9614001110 con 0x7f96240b7f50 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96240ce930 con 0x7f96240b7f50 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1106561271 0 0) 0x7f9610003520 con 0x7f96240ce520 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 msgr2=0x7f962415e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 0x7f962415e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 >> v1:172.21.15.176:6789/0 conn(0x7f96240b7f50 legacy=0x7f962406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96240d0450 con 0x7f96240ce520 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f96200019f0 con 0x7f96240ce520 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 >> v1:172.21.15.67:6789/0 conn(0x7f96240ce520 legacy=0x7f96240b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003b70 0x7f9610003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 0x7f9610004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610004d40 0x7f9610005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f962415e860 con 0x7f9610003b70 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f961000bbb0 con 0x7f9610004d40 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f961000bdb0 con 0x7f962406c5c0 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9629f50640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003b70 0x7f9610003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f962974f640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 0x7f9610004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f962a751640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610004d40 0x7f9610005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9629f50640 1 -- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 msgr2=0x7f9610004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9629f50640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962406c5c0 0x7f9610004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9629f50640 1 -- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610004d40 msgr2=0x7f9610005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9629f50640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610004d40 0x7f9610005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f9629f50640 1 -- 172.21.15.67:0/651525929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9610003520 con 0x7f9610003b70 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.476+0000 7f962a751640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610004d40 0x7f9610005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.477+0000 7f9629f50640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003b70 0x7f9610003fc0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f962000a7f0 tx=0x7f9610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f46e275b081c17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.477+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962000a450 con 0x7f9610003b70 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.477+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f962002f070 con 0x7f9610003b70 2024-04-25T15:27:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.477+0000 7f9628f4e640 1 -- 172.21.15.67:0/651525929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962002bc70 con 0x7f9610003b70 2024-04-25T15:27:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.478+0000 7f962c1b4640 1 -- 172.21.15.67:0/651525929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003b70 msgr2=0x7f9610003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.478+0000 7f962c1b4640 1 --2- 172.21.15.67:0/651525929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003b70 0x7f9610003fc0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f962000a7f0 tx=0x7f9610005830 comp rx=0 tx=0).stop 2024-04-25T15:27:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.478+0000 7f962a751640 1 -- 172.21.15.67:0/651525929 reap_dead start 2024-04-25T15:27:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.478+0000 7f962c1b4640 1 -- 172.21.15.67:0/651525929 shutdown_connections 2024-04-25T15:27:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.478+0000 7f962c1b4640 1 -- 172.21.15.67:0/651525929 >> 172.21.15.67:0/651525929 conn(0x7f962409b7d0 msgr2=0x7f962409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.478+0000 7f962c1b4640 1 -- 172.21.15.67:0/651525929 shutdown_connections 2024-04-25T15:27:56.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.478+0000 7f962c1b4640 1 -- 172.21.15.67:0/651525929 wait complete. 2024-04-25T15:27:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.480+0000 7f962c1b4640 1 Processor -- start 2024-04-25T15:27:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.480+0000 7f962c1b4640 1 -- start start 2024-04-25T15:27:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962c1b4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610003b70 0x7f96240b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962c1b4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962409c9d0 0x7f96240cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962c1b4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96240b2790 0x7f962409e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962c1b4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f962406d210 con 0x7f962409c9d0 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962c1b4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f962406cd10 con 0x7f9610003b70 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962c1b4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f962406d4f0 con 0x7f96240b2790 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962974f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962409c9d0 0x7f96240cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962974f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962409c9d0 0x7f96240cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35262/0 (socket says 172.21.15.67:35262) 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962974f640 1 -- 172.21.15.67:0/1979955940 learned_addr learned my addr 172.21.15.67:0/1979955940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f962a751640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96240b2790 0x7f962409e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.481+0000 7f9629f50640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610003b70 0x7f96240b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962974f640 1 -- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96240b2790 msgr2=0x7f962409e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962974f640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96240b2790 0x7f962409e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962974f640 1 -- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610003b70 msgr2=0x7f96240b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962974f640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610003b70 0x7f96240b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962974f640 1 -- 172.21.15.67:0/1979955940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96240b0bd0 con 0x7f962409c9d0 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962974f640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962409c9d0 0x7f96240cb760 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f961400f7a0 tx=0x7f961400fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f961a7fc640 1 -- 172.21.15.67:0/1979955940 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961400fe50 con 0x7f962409c9d0 2024-04-25T15:27:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f961a7fc640 1 -- 172.21.15.67:0/1979955940 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9614018070 con 0x7f962409c9d0 2024-04-25T15:27:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f961a7fc640 1 -- 172.21.15.67:0/1979955940 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961400d730 con 0x7f962409c9d0 2024-04-25T15:27:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96240b0eb0 con 0x7f962409c9d0 2024-04-25T15:27:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.482+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96240b2270 con 0x7f962409c9d0 2024-04-25T15:27:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.483+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9624050740 con 0x7f962409c9d0 2024-04-25T15:27:56.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.484+0000 7f961a7fc640 1 -- 172.21.15.67:0/1979955940 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f961400d8d0 con 0x7f962409c9d0 2024-04-25T15:27:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.484+0000 7f961a7fc640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96080744b0 0x7f96080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.484+0000 7f961a7fc640 1 -- 172.21.15.67:0/1979955940 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f96140950f0 con 0x7f962409c9d0 2024-04-25T15:27:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.484+0000 7f9629f50640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96080744b0 0x7f96080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.485+0000 7f9629f50640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96080744b0 0x7f96080769c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9620033000 tx=0x7f962002d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.489+0000 7f961a7fc640 1 -- 172.21.15.67:0/1979955940 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f962406c9d0 con 0x7f962409c9d0 2024-04-25T15:27:56.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.672+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f96240b0980 con 0x7f96080744b0 2024-04-25T15:27:56.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.679+0000 7f961a7fc640 1 -- 172.21.15.67:0/1979955940 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252058 (secure 0 0 0) 0x7f96240b0980 con 0x7f96080744b0 2024-04-25T15:27:56.683 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:56.684 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96080744b0 msgr2=0x7f96080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96080744b0 0x7f96080769c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9620033000 tx=0x7f962002d000 comp rx=0 tx=0).stop 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962409c9d0 msgr2=0x7f96240cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962409c9d0 0x7f96240cb760 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f961400f7a0 tx=0x7f961400fc60 comp rx=0 tx=0).stop 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 shutdown_connections 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96080744b0 0x7f96080769c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96240b2790 0x7f962409e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962409c9d0 0x7f96240cb760 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 --2- 172.21.15.67:0/1979955940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9610003b70 0x7f96240b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 >> 172.21.15.67:0/1979955940 conn(0x7f962409b7d0 msgr2=0x7f96240b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 shutdown_connections 2024-04-25T15:27:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.682+0000 7f962c1b4640 1 -- 172.21.15.67:0/1979955940 wait complete. 2024-04-25T15:27:56.694 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-04-25T15:27:54.830704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223780,"kb_used_data":7592,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754750940,"statfs":{"total":773094113280,"available":772864962560,"internally_reserved":0,"allocated":7774208,"data_stored":3917676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002156"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745147+0000","last_change":"2024-04-25T15:27:47.784714+0000","last_active":"2024-04-25T15:27:49.745147+0000","last_peered":"2024-04-25T15:27:49.745147+0000","last_clean":"2024-04-25T15:27:49.745147+0000","last_became_active":"2024-04-25T15:27:47.784394+0000","last_became_peered":"2024-04-25T15:27:47.784394+0000","last_unstale":"2024-04-25T15:27:49.745147+0000","last_undegraded":"2024-04-25T15:27:49.745147+0000","last_fullsized":"2024-04-25T15:27:49.745147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:42.640130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213593+0000","last_change":"2024-04-25T15:27:47.784772+0000","last_active":"2024-04-25T15:27:49.213593+0000","last_peered":"2024-04-25T15:27:49.213593+0000","last_clean":"2024-04-25T15:27:49.213593+0000","last_became_active":"2024-04-25T15:27:47.784268+0000","last_became_peered":"2024-04-25T15:27:47.784268+0000","last_unstale":"2024-04-25T15:27:49.213593+0000","last_undegraded":"2024-04-25T15:27:49.213593+0000","last_fullsized":"2024-04-25T15:27:49.213593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:01.513008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418455+0000","last_change":"2024-04-25T15:27:47.771765+0000","last_active":"2024-04-25T15:27:50.418455+0000","last_peered":"2024-04-25T15:27:50.418455+0000","last_clean":"2024-04-25T15:27:50.418455+0000","last_became_active":"2024-04-25T15:27:47.771399+0000","last_became_peered":"2024-04-25T15:27:47.771399+0000","last_unstale":"2024-04-25T15:27:50.418455+0000","last_undegraded":"2024-04-25T15:27:50.418455+0000","last_fullsized":"2024-04-25T15:27:50.418455+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:51.125252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774961+0000","last_change":"2024-04-25T15:27:47.775846+0000","last_active":"2024-04-25T15:27:48.774961+0000","last_peered":"2024-04-25T15:27:48.774961+0000","last_clean":"2024-04-25T15:27:48.774961+0000","last_became_active":"2024-04-25T15:27:47.775533+0000","last_became_peered":"2024-04-25T15:27:47.775533+0000","last_unstale":"2024-04-25T15:27:48.774961+0000","last_undegraded":"2024-04-25T15:27:48.774961+0000","last_fullsized":"2024-04-25T15:27:48.774961+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:38.496000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418370+0000","last_change":"2024-04-25T15:27:47.772773+0000","last_active":"2024-04-25T15:27:50.418370+0000","last_peered":"2024-04-25T15:27:50.418370+0000","last_clean":"2024-04-25T15:27:50.418370+0000","last_became_active":"2024-04-25T15:27:47.772468+0000","last_became_peered":"2024-04-25T15:27:47.772468+0000","last_unstale":"2024-04-25T15:27:50.418370+0000","last_undegraded":"2024-04-25T15:27:50.418370+0000","last_fullsized":"2024-04-25T15:27:50.418370+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:48.543108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213072+0000","last_change":"2024-04-25T15:27:47.781800+0000","last_active":"2024-04-25T15:27:49.213072+0000","last_peered":"2024-04-25T15:27:49.213072+0000","last_clean":"2024-04-25T15:27:49.213072+0000","last_became_active":"2024-04-25T15:27:47.781483+0000","last_became_peered":"2024-04-25T15:27:47.781483+0000","last_unstale":"2024-04-25T15:27:49.213072+0000","last_undegraded":"2024-04-25T15:27:49.213072+0000","last_fullsized":"2024-04-25T15:27:49.213072+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:04:59.972266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762097+0000","last_change":"2024-04-25T15:27:47.767384+0000","last_active":"2024-04-25T15:27:49.762097+0000","last_peered":"2024-04-25T15:27:49.762097+0000","last_clean":"2024-04-25T15:27:49.762097+0000","last_became_active":"2024-04-25T15:27:47.766973+0000","last_became_peered":"2024-04-25T15:27:47.766973+0000","last_unstale":"2024-04-25T15:27:49.762097+0000","last_undegraded":"2024-04-25T15:27:49.762097+0000","last_fullsized":"2024-04-25T15:27:49.762097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:53.388615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213254+0000","last_change":"2024-04-25T15:27:47.786432+0000","last_active":"2024-04-25T15:27:49.213254+0000","last_peered":"2024-04-25T15:27:49.213254+0000","last_clean":"2024-04-25T15:27:49.213254+0000","last_became_active":"2024-04-25T15:27:47.786105+0000","last_became_peered":"2024-04-25T15:27:47.786105+0000","last_unstale":"2024-04-25T15:27:49.213254+0000","last_undegraded":"2024-04-25T15:27:49.213254+0000","last_fullsized":"2024-04-25T15:27:49.213254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:52.241260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418447+0000","last_change":"2024-04-25T15:27:47.782985+0000","last_active":"2024-04-25T15:27:50.418447+0000","last_peered":"2024-04-25T15:27:50.418447+0000","last_clean":"2024-04-25T15:27:50.418447+0000","last_became_active":"2024-04-25T15:27:47.782663+0000","last_became_peered":"2024-04-25T15:27:47.782663+0000","last_unstale":"2024-04-25T15:27:50.418447+0000","last_undegraded":"2024-04-25T15:27:50.418447+0000","last_fullsized":"2024-04-25T15:27:50.418447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:17.900479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774518+0000","last_change":"2024-04-25T15:27:47.785807+0000","last_active":"2024-04-25T15:27:48.774518+0000","last_peered":"2024-04-25T15:27:48.774518+0000","last_clean":"2024-04-25T15:27:48.774518+0000","last_became_active":"2024-04-25T15:27:47.785440+0000","last_became_peered":"2024-04-25T15:27:47.785440+0000","last_unstale":"2024-04-25T15:27:48.774518+0000","last_undegraded":"2024-04-25T15:27:48.774518+0000","last_fullsized":"2024-04-25T15:27:48.774518+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:55.524808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745009+0000","last_change":"2024-04-25T15:27:47.771746+0000","last_active":"2024-04-25T15:27:49.745009+0000","last_peered":"2024-04-25T15:27:49.745009+0000","last_clean":"2024-04-25T15:27:49.745009+0000","last_became_active":"2024-04-25T15:27:47.771418+0000","last_became_peered":"2024-04-25T15:27:47.771418+0000","last_unstale":"2024-04-25T15:27:49.745009+0000","last_undegraded":"2024-04-25T15:27:49.745009+0000","last_fullsized":"2024-04-25T15:27:49.745009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:33.146166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775074+0000","last_change":"2024-04-25T15:27:47.784248+0000","last_active":"2024-04-25T15:27:48.775074+0000","last_peered":"2024-04-25T15:27:48.775074+0000","last_clean":"2024-04-25T15:27:48.775074+0000","last_became_active":"2024-04-25T15:27:47.783953+0000","last_became_peered":"2024-04-25T15:27:47.783953+0000","last_unstale":"2024-04-25T15:27:48.775074+0000","last_undegraded":"2024-04-25T15:27:48.775074+0000","last_fullsized":"2024-04-25T15:27:48.775074+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:34.828308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773727+0000","last_change":"2024-04-25T15:27:47.777009+0000","last_active":"2024-04-25T15:27:48.773727+0000","last_peered":"2024-04-25T15:27:48.773727+0000","last_clean":"2024-04-25T15:27:48.773727+0000","last_became_active":"2024-04-25T15:27:47.776670+0000","last_became_peered":"2024-04-25T15:27:47.776670+0000","last_unstale":"2024-04-25T15:27:48.773727+0000","last_undegraded":"2024-04-25T15:27:48.773727+0000","last_fullsized":"2024-04-25T15:27:48.773727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:02.213596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418208+0000","last_change":"2024-04-25T15:27:47.778816+0000","last_active":"2024-04-25T15:27:50.418208+0000","last_peered":"2024-04-25T15:27:50.418208+0000","last_clean":"2024-04-25T15:27:50.418208+0000","last_became_active":"2024-04-25T15:27:47.778428+0000","last_became_peered":"2024-04-25T15:27:47.778428+0000","last_unstale":"2024-04-25T15:27:50.418208+0000","last_undegraded":"2024-04-25T15:27:50.418208+0000","last_fullsized":"2024-04-25T15:27:50.418208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:14.451791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773448+0000","last_change":"2024-04-25T15:27:47.765971+0000","last_active":"2024-04-25T15:27:48.773448+0000","last_peered":"2024-04-25T15:27:48.773448+0000","last_clean":"2024-04-25T15:27:48.773448+0000","last_became_active":"2024-04-25T15:27:47.765617+0000","last_became_peered":"2024-04-25T15:27:47.765617+0000","last_unstale":"2024-04-25T15:27:48.773448+0000","last_undegraded":"2024-04-25T15:27:48.773448+0000","last_fullsized":"2024-04-25T15:27:48.773448+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:37.274297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773968+0000","last_change":"2024-04-25T15:27:47.777354+0000","last_active":"2024-04-25T15:27:48.773968+0000","last_peered":"2024-04-25T15:27:48.773968+0000","last_clean":"2024-04-25T15:27:48.773968+0000","last_became_active":"2024-04-25T15:27:47.777028+0000","last_became_peered":"2024-04-25T15:27:47.777028+0000","last_unstale":"2024-04-25T15:27:48.773968+0000","last_undegraded":"2024-04-25T15:27:48.773968+0000","last_fullsized":"2024-04-25T15:27:48.773968+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:49.491948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418608+0000","last_change":"2024-04-25T15:27:47.782281+0000","last_active":"2024-04-25T15:27:50.418608+0000","last_peered":"2024-04-25T15:27:50.418608+0000","last_clean":"2024-04-25T15:27:50.418608+0000","last_became_active":"2024-04-25T15:27:47.781895+0000","last_became_peered":"2024-04-25T15:27:47.781895+0000","last_unstale":"2024-04-25T15:27:50.418608+0000","last_undegraded":"2024-04-25T15:27:50.418608+0000","last_fullsized":"2024-04-25T15:27:50.418608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.571409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775202+0000","last_change":"2024-04-25T15:27:47.774789+0000","last_active":"2024-04-25T15:27:48.775202+0000","last_peered":"2024-04-25T15:27:48.775202+0000","last_clean":"2024-04-25T15:27:48.775202+0000","last_became_active":"2024-04-25T15:27:47.774000+0000","last_became_peered":"2024-04-25T15:27:47.774000+0000","last_unstale":"2024-04-25T15:27:48.775202+0000","last_undegraded":"2024-04-25T15:27:48.775202+0000","last_fullsized":"2024-04-25T15:27:48.775202+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:41:25.750743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213244+0000","last_change":"2024-04-25T15:27:47.786710+0000","last_active":"2024-04-25T15:27:49.213244+0000","last_peered":"2024-04-25T15:27:49.213244+0000","last_clean":"2024-04-25T15:27:49.213244+0000","last_became_active":"2024-04-25T15:27:47.786335+0000","last_became_peered":"2024-04-25T15:27:47.786335+0000","last_unstale":"2024-04-25T15:27:49.213244+0000","last_undegraded":"2024-04-25T15:27:49.213244+0000","last_fullsized":"2024-04-25T15:27:49.213244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:55.815321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773849+0000","last_change":"2024-04-25T15:27:47.767587+0000","last_active":"2024-04-25T15:27:48.773849+0000","last_peered":"2024-04-25T15:27:48.773849+0000","last_clean":"2024-04-25T15:27:48.773849+0000","last_became_active":"2024-04-25T15:27:47.767230+0000","last_became_peered":"2024-04-25T15:27:47.767230+0000","last_unstale":"2024-04-25T15:27:48.773849+0000","last_undegraded":"2024-04-25T15:27:48.773849+0000","last_fullsized":"2024-04-25T15:27:48.773849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:25.374232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774541+0000","last_change":"2024-04-25T15:27:47.765967+0000","last_active":"2024-04-25T15:27:48.774541+0000","last_peered":"2024-04-25T15:27:48.774541+0000","last_clean":"2024-04-25T15:27:48.774541+0000","last_became_active":"2024-04-25T15:27:47.765603+0000","last_became_peered":"2024-04-25T15:27:47.765603+0000","last_unstale":"2024-04-25T15:27:48.774541+0000","last_undegraded":"2024-04-25T15:27:48.774541+0000","last_fullsized":"2024-04-25T15:27:48.774541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:55.815321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774196+0000","last_change":"2024-04-25T15:27:47.784518+0000","last_active":"2024-04-25T15:27:48.774196+0000","last_peered":"2024-04-25T15:27:48.774196+0000","last_clean":"2024-04-25T15:27:48.774196+0000","last_became_active":"2024-04-25T15:27:47.784199+0000","last_became_peered":"2024-04-25T15:27:47.784199+0000","last_unstale":"2024-04-25T15:27:48.774196+0000","last_undegraded":"2024-04-25T15:27:48.774196+0000","last_fullsized":"2024-04-25T15:27:48.774196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:24.352702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744707+0000","last_change":"2024-04-25T15:27:47.777682+0000","last_active":"2024-04-25T15:27:49.744707+0000","last_peered":"2024-04-25T15:27:49.744707+0000","last_clean":"2024-04-25T15:27:49.744707+0000","last_became_active":"2024-04-25T15:27:47.777377+0000","last_became_peered":"2024-04-25T15:27:47.777377+0000","last_unstale":"2024-04-25T15:27:49.744707+0000","last_undegraded":"2024-04-25T15:27:49.744707+0000","last_fullsized":"2024-04-25T15:27:49.744707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:25.374232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744636+0000","last_change":"2024-04-25T15:27:47.780015+0000","last_active":"2024-04-25T15:27:49.744636+0000","last_peered":"2024-04-25T15:27:49.744636+0000","last_clean":"2024-04-25T15:27:49.744636+0000","last_became_active":"2024-04-25T15:27:47.779610+0000","last_became_peered":"2024-04-25T15:27:47.779610+0000","last_unstale":"2024-04-25T15:27:49.744636+0000","last_undegraded":"2024-04-25T15:27:49.744636+0000","last_fullsized":"2024-04-25T15:27:49.744636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:02.213596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774260+0000","last_change":"2024-04-25T15:27:47.776091+0000","last_active":"2024-04-25T15:27:48.774260+0000","last_peered":"2024-04-25T15:27:48.774260+0000","last_clean":"2024-04-25T15:27:48.774260+0000","last_became_active":"2024-04-25T15:27:47.775811+0000","last_became_peered":"2024-04-25T15:27:47.775811+0000","last_unstale":"2024-04-25T15:27:48.774260+0000","last_undegraded":"2024-04-25T15:27:48.774260+0000","last_fullsized":"2024-04-25T15:27:48.774260+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:04:59.972266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762463+0000","last_change":"2024-04-25T15:27:47.783916+0000","last_active":"2024-04-25T15:27:49.762463+0000","last_peered":"2024-04-25T15:27:49.762463+0000","last_clean":"2024-04-25T15:27:49.762463+0000","last_became_active":"2024-04-25T15:27:47.783621+0000","last_became_peered":"2024-04-25T15:27:47.783621+0000","last_unstale":"2024-04-25T15:27:49.762463+0000","last_undegraded":"2024-04-25T15:27:49.762463+0000","last_fullsized":"2024-04-25T15:27:49.762463+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:33:36.749333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418530+0000","last_change":"2024-04-25T15:27:47.772770+0000","last_active":"2024-04-25T15:27:50.418530+0000","last_peered":"2024-04-25T15:27:50.418530+0000","last_clean":"2024-04-25T15:27:50.418530+0000","last_became_active":"2024-04-25T15:27:47.772438+0000","last_became_peered":"2024-04-25T15:27:47.772438+0000","last_unstale":"2024-04-25T15:27:50.418530+0000","last_undegraded":"2024-04-25T15:27:50.418530+0000","last_fullsized":"2024-04-25T15:27:50.418530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:12:50.108831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774387+0000","last_change":"2024-04-25T15:27:47.780616+0000","last_active":"2024-04-25T15:27:48.774387+0000","last_peered":"2024-04-25T15:27:48.774387+0000","last_clean":"2024-04-25T15:27:48.774387+0000","last_became_active":"2024-04-25T15:27:47.780282+0000","last_became_peered":"2024-04-25T15:27:47.780282+0000","last_unstale":"2024-04-25T15:27:48.774387+0000","last_undegraded":"2024-04-25T15:27:48.774387+0000","last_fullsized":"2024-04-25T15:27:48.774387+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:51.306682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417815+0000","last_change":"2024-04-25T15:27:47.775239+0000","last_active":"2024-04-25T15:27:50.417815+0000","last_peered":"2024-04-25T15:27:50.417815+0000","last_clean":"2024-04-25T15:27:50.417815+0000","last_became_active":"2024-04-25T15:27:47.774921+0000","last_became_peered":"2024-04-25T15:27:47.774921+0000","last_unstale":"2024-04-25T15:27:50.417815+0000","last_undegraded":"2024-04-25T15:27:50.417815+0000","last_fullsized":"2024-04-25T15:27:50.417815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:45.624627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762189+0000","last_change":"2024-04-25T15:27:47.773714+0000","last_active":"2024-04-25T15:27:49.762189+0000","last_peered":"2024-04-25T15:27:49.762189+0000","last_clean":"2024-04-25T15:27:49.762189+0000","last_became_active":"2024-04-25T15:27:47.773375+0000","last_became_peered":"2024-04-25T15:27:47.773375+0000","last_unstale":"2024-04-25T15:27:49.762189+0000","last_undegraded":"2024-04-25T15:27:49.762189+0000","last_fullsized":"2024-04-25T15:27:49.762189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:39.042562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762438+0000","last_change":"2024-04-25T15:27:47.783321+0000","last_active":"2024-04-25T15:27:49.762438+0000","last_peered":"2024-04-25T15:27:49.762438+0000","last_clean":"2024-04-25T15:27:49.762438+0000","last_became_active":"2024-04-25T15:27:47.782991+0000","last_became_peered":"2024-04-25T15:27:47.782991+0000","last_unstale":"2024-04-25T15:27:49.762438+0000","last_undegraded":"2024-04-25T15:27:49.762438+0000","last_fullsized":"2024-04-25T15:27:49.762438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:16.437778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213171+0000","last_change":"2024-04-25T15:27:47.780808+0000","last_active":"2024-04-25T15:27:49.213171+0000","last_peered":"2024-04-25T15:27:49.213171+0000","last_clean":"2024-04-25T15:27:49.213171+0000","last_became_active":"2024-04-25T15:27:47.780464+0000","last_became_peered":"2024-04-25T15:27:47.780464+0000","last_unstale":"2024-04-25T15:27:49.213171+0000","last_undegraded":"2024-04-25T15:27:49.213171+0000","last_fullsized":"2024-04-25T15:27:49.213171+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:51.306682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745152+0000","last_change":"2024-04-25T15:27:47.780047+0000","last_active":"2024-04-25T15:27:49.745152+0000","last_peered":"2024-04-25T15:27:49.745152+0000","last_clean":"2024-04-25T15:27:49.745152+0000","last_became_active":"2024-04-25T15:27:47.779713+0000","last_became_peered":"2024-04-25T15:27:47.779713+0000","last_unstale":"2024-04-25T15:27:49.745152+0000","last_undegraded":"2024-04-25T15:27:49.745152+0000","last_fullsized":"2024-04-25T15:27:49.745152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:38.522279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"66'4","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745343+0000","last_change":"2024-04-25T15:27:46.756488+0000","last_active":"2024-04-25T15:27:49.745343+0000","last_peered":"2024-04-25T15:27:49.745343+0000","last_clean":"2024-04-25T15:27:49.745343+0000","last_became_active":"2024-04-25T15:27:46.756137+0000","last_became_peered":"2024-04-25T15:27:46.756137+0000","last_unstale":"2024-04-25T15:27:49.745343+0000","last_undegraded":"2024-04-25T15:27:49.745343+0000","last_fullsized":"2024-04-25T15:27:49.745343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:34.541226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418778+0000","last_change":"2024-04-25T15:27:47.780333+0000","last_active":"2024-04-25T15:27:50.418778+0000","last_peered":"2024-04-25T15:27:50.418778+0000","last_clean":"2024-04-25T15:27:50.418778+0000","last_became_active":"2024-04-25T15:27:47.780000+0000","last_became_peered":"2024-04-25T15:27:47.780000+0000","last_unstale":"2024-04-25T15:27:50.418778+0000","last_undegraded":"2024-04-25T15:27:50.418778+0000","last_fullsized":"2024-04-25T15:27:50.418778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:40:47.662951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773969+0000","last_change":"2024-04-25T15:27:46.748183+0000","last_active":"2024-04-25T15:27:48.773969+0000","last_peered":"2024-04-25T15:27:48.773969+0000","last_clean":"2024-04-25T15:27:48.773969+0000","last_became_active":"2024-04-25T15:27:46.747854+0000","last_became_peered":"2024-04-25T15:27:46.747854+0000","last_unstale":"2024-04-25T15:27:48.773969+0000","last_undegraded":"2024-04-25T15:27:48.773969+0000","last_fullsized":"2024-04-25T15:27:48.773969+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:58.559219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213368+0000","last_change":"2024-04-25T15:27:47.775869+0000","last_active":"2024-04-25T15:27:49.213368+0000","last_peered":"2024-04-25T15:27:49.213368+0000","last_clean":"2024-04-25T15:27:49.213368+0000","last_became_active":"2024-04-25T15:27:47.775636+0000","last_became_peered":"2024-04-25T15:27:47.775636+0000","last_unstale":"2024-04-25T15:27:49.213368+0000","last_undegraded":"2024-04-25T15:27:49.213368+0000","last_fullsized":"2024-04-25T15:27:49.213368+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:53.391175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774297+0000","last_change":"2024-04-25T15:27:46.750527+0000","last_active":"2024-04-25T15:27:48.774297+0000","last_peered":"2024-04-25T15:27:48.774297+0000","last_clean":"2024-04-25T15:27:48.774297+0000","last_became_active":"2024-04-25T15:27:46.749838+0000","last_became_peered":"2024-04-25T15:27:46.749838+0000","last_unstale":"2024-04-25T15:27:48.774297+0000","last_undegraded":"2024-04-25T15:27:48.774297+0000","last_fullsized":"2024-04-25T15:27:48.774297+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:04.081872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745201+0000","last_change":"2024-04-25T15:27:47.770261+0000","last_active":"2024-04-25T15:27:49.745201+0000","last_peered":"2024-04-25T15:27:49.745201+0000","last_clean":"2024-04-25T15:27:49.745201+0000","last_became_active":"2024-04-25T15:27:47.769514+0000","last_became_peered":"2024-04-25T15:27:47.769514+0000","last_unstale":"2024-04-25T15:27:49.745201+0000","last_undegraded":"2024-04-25T15:27:49.745201+0000","last_fullsized":"2024-04-25T15:27:49.745201+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:15.809204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213725+0000","last_change":"2024-04-25T15:27:46.747175+0000","last_active":"2024-04-25T15:27:49.213725+0000","last_peered":"2024-04-25T15:27:49.213725+0000","last_clean":"2024-04-25T15:27:49.213725+0000","last_became_active":"2024-04-25T15:27:46.746878+0000","last_became_peered":"2024-04-25T15:27:46.746878+0000","last_unstale":"2024-04-25T15:27:49.213725+0000","last_undegraded":"2024-04-25T15:27:49.213725+0000","last_fullsized":"2024-04-25T15:27:49.213725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:02.088263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762349+0000","last_change":"2024-04-25T15:27:47.774555+0000","last_active":"2024-04-25T15:27:49.762349+0000","last_peered":"2024-04-25T15:27:49.762349+0000","last_clean":"2024-04-25T15:27:49.762349+0000","last_became_active":"2024-04-25T15:27:47.774254+0000","last_became_peered":"2024-04-25T15:27:47.774254+0000","last_unstale":"2024-04-25T15:27:49.762349+0000","last_undegraded":"2024-04-25T15:27:49.762349+0000","last_fullsized":"2024-04-25T15:27:49.762349+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:15:52.086564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418002+0000","last_change":"2024-04-25T15:27:46.748931+0000","last_active":"2024-04-25T15:27:50.418002+0000","last_peered":"2024-04-25T15:27:50.418002+0000","last_clean":"2024-04-25T15:27:50.418002+0000","last_became_active":"2024-04-25T15:27:46.748400+0000","last_became_peered":"2024-04-25T15:27:46.748400+0000","last_unstale":"2024-04-25T15:27:50.418002+0000","last_undegraded":"2024-04-25T15:27:50.418002+0000","last_fullsized":"2024-04-25T15:27:50.418002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:14.803258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774690+0000","last_change":"2024-04-25T15:27:47.776089+0000","last_active":"2024-04-25T15:27:48.774690+0000","last_peered":"2024-04-25T15:27:48.774690+0000","last_clean":"2024-04-25T15:27:48.774690+0000","last_became_active":"2024-04-25T15:27:47.775768+0000","last_became_peered":"2024-04-25T15:27:47.775768+0000","last_unstale":"2024-04-25T15:27:48.774690+0000","last_undegraded":"2024-04-25T15:27:48.774690+0000","last_fullsized":"2024-04-25T15:27:48.774690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:52.241260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.212951+0000","last_change":"2024-04-25T15:27:46.753325+0000","last_active":"2024-04-25T15:27:49.212951+0000","last_peered":"2024-04-25T15:27:49.212951+0000","last_clean":"2024-04-25T15:27:49.212951+0000","last_became_active":"2024-04-25T15:27:46.753006+0000","last_became_peered":"2024-04-25T15:27:46.753006+0000","last_unstale":"2024-04-25T15:27:49.212951+0000","last_undegraded":"2024-04-25T15:27:49.212951+0000","last_fullsized":"2024-04-25T15:27:49.212951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:37.598034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417912+0000","last_change":"2024-04-25T15:27:47.780022+0000","last_active":"2024-04-25T15:27:50.417912+0000","last_peered":"2024-04-25T15:27:50.417912+0000","last_clean":"2024-04-25T15:27:50.417912+0000","last_became_active":"2024-04-25T15:27:47.779594+0000","last_became_peered":"2024-04-25T15:27:47.779594+0000","last_unstale":"2024-04-25T15:27:50.417912+0000","last_undegraded":"2024-04-25T15:27:50.417912+0000","last_fullsized":"2024-04-25T15:27:50.417912+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:08:06.201716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774864+0000","last_change":"2024-04-25T15:27:46.740274+0000","last_active":"2024-04-25T15:27:48.774864+0000","last_peered":"2024-04-25T15:27:48.774864+0000","last_clean":"2024-04-25T15:27:48.774864+0000","last_became_active":"2024-04-25T15:27:46.739275+0000","last_became_peered":"2024-04-25T15:27:46.739275+0000","last_unstale":"2024-04-25T15:27:48.774864+0000","last_undegraded":"2024-04-25T15:27:48.774864+0000","last_fullsized":"2024-04-25T15:27:48.774864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:37.598034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773476+0000","last_change":"2024-04-25T15:27:47.769807+0000","last_active":"2024-04-25T15:27:48.773476+0000","last_peered":"2024-04-25T15:27:48.773476+0000","last_clean":"2024-04-25T15:27:48.773476+0000","last_became_active":"2024-04-25T15:27:47.769357+0000","last_became_peered":"2024-04-25T15:27:47.769357+0000","last_unstale":"2024-04-25T15:27:48.773476+0000","last_undegraded":"2024-04-25T15:27:48.773476+0000","last_fullsized":"2024-04-25T15:27:48.773476+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:53.115168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417499+0000","last_change":"2024-04-25T15:27:46.748972+0000","last_active":"2024-04-25T15:27:50.417499+0000","last_peered":"2024-04-25T15:27:50.417499+0000","last_clean":"2024-04-25T15:27:50.417499+0000","last_became_active":"2024-04-25T15:27:46.748630+0000","last_became_peered":"2024-04-25T15:27:46.748630+0000","last_unstale":"2024-04-25T15:27:50.417499+0000","last_undegraded":"2024-04-25T15:27:50.417499+0000","last_fullsized":"2024-04-25T15:27:50.417499+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:43.640246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762536+0000","last_change":"2024-04-25T15:27:47.770334+0000","last_active":"2024-04-25T15:27:49.762536+0000","last_peered":"2024-04-25T15:27:49.762536+0000","last_clean":"2024-04-25T15:27:49.762536+0000","last_became_active":"2024-04-25T15:27:47.769970+0000","last_became_peered":"2024-04-25T15:27:47.769970+0000","last_unstale":"2024-04-25T15:27:49.762536+0000","last_undegraded":"2024-04-25T15:27:49.762536+0000","last_fullsized":"2024-04-25T15:27:49.762536+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:11:10.445989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774396+0000","last_change":"2024-04-25T15:27:46.753409+0000","last_active":"2024-04-25T15:27:48.774396+0000","last_peered":"2024-04-25T15:27:48.774396+0000","last_clean":"2024-04-25T15:27:48.774396+0000","last_became_active":"2024-04-25T15:27:46.753090+0000","last_became_peered":"2024-04-25T15:27:46.753090+0000","last_unstale":"2024-04-25T15:27:48.774396+0000","last_undegraded":"2024-04-25T15:27:48.774396+0000","last_fullsized":"2024-04-25T15:27:48.774396+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:06.558947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745205+0000","last_change":"2024-04-25T15:27:47.781191+0000","last_active":"2024-04-25T15:27:49.745205+0000","last_peered":"2024-04-25T15:27:49.745205+0000","last_clean":"2024-04-25T15:27:49.745205+0000","last_became_active":"2024-04-25T15:27:47.780588+0000","last_became_peered":"2024-04-25T15:27:47.780588+0000","last_unstale":"2024-04-25T15:27:49.745205+0000","last_undegraded":"2024-04-25T15:27:49.745205+0000","last_fullsized":"2024-04-25T15:27:49.745205+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:32:52.541061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213648+0000","last_change":"2024-04-25T15:27:46.751600+0000","last_active":"2024-04-25T15:27:49.213648+0000","last_peered":"2024-04-25T15:27:49.213648+0000","last_clean":"2024-04-25T15:27:49.213648+0000","last_became_active":"2024-04-25T15:27:46.751255+0000","last_became_peered":"2024-04-25T15:27:46.751255+0000","last_unstale":"2024-04-25T15:27:49.213648+0000","last_undegraded":"2024-04-25T15:27:49.213648+0000","last_fullsized":"2024-04-25T15:27:49.213648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:23.117643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213505+0000","last_change":"2024-04-25T15:27:47.781034+0000","last_active":"2024-04-25T15:27:49.213505+0000","last_peered":"2024-04-25T15:27:49.213505+0000","last_clean":"2024-04-25T15:27:49.213505+0000","last_became_active":"2024-04-25T15:27:47.780713+0000","last_became_peered":"2024-04-25T15:27:47.780713+0000","last_unstale":"2024-04-25T15:27:49.213505+0000","last_undegraded":"2024-04-25T15:27:49.213505+0000","last_fullsized":"2024-04-25T15:27:49.213505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:35.547172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"66'3","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418565+0000","last_change":"2024-04-25T15:27:46.750192+0000","last_active":"2024-04-25T15:27:50.418565+0000","last_peered":"2024-04-25T15:27:50.418565+0000","last_clean":"2024-04-25T15:27:50.418565+0000","last_became_active":"2024-04-25T15:27:46.749859+0000","last_became_peered":"2024-04-25T15:27:46.749859+0000","last_unstale":"2024-04-25T15:27:50.418565+0000","last_undegraded":"2024-04-25T15:27:50.418565+0000","last_fullsized":"2024-04-25T15:27:50.418565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:41:37.516333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417872+0000","last_change":"2024-04-25T15:27:47.781969+0000","last_active":"2024-04-25T15:27:50.417872+0000","last_peered":"2024-04-25T15:27:50.417872+0000","last_clean":"2024-04-25T15:27:50.417872+0000","last_became_active":"2024-04-25T15:27:47.781565+0000","last_became_peered":"2024-04-25T15:27:47.781565+0000","last_unstale":"2024-04-25T15:27:50.417872+0000","last_undegraded":"2024-04-25T15:27:50.417872+0000","last_fullsized":"2024-04-25T15:27:50.417872+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:52.873645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"66'6","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775201+0000","last_change":"2024-04-25T15:27:46.751593+0000","last_active":"2024-04-25T15:27:48.775201+0000","last_peered":"2024-04-25T15:27:48.775201+0000","last_clean":"2024-04-25T15:27:48.775201+0000","last_became_active":"2024-04-25T15:27:46.751123+0000","last_became_peered":"2024-04-25T15:27:46.751123+0000","last_unstale":"2024-04-25T15:27:48.775201+0000","last_undegraded":"2024-04-25T15:27:48.775201+0000","last_fullsized":"2024-04-25T15:27:48.775201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:06.332293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744788+0000","last_change":"2024-04-25T15:27:47.784323+0000","last_active":"2024-04-25T15:27:49.744788+0000","last_peered":"2024-04-25T15:27:49.744788+0000","last_clean":"2024-04-25T15:27:49.744788+0000","last_became_active":"2024-04-25T15:27:47.783947+0000","last_became_peered":"2024-04-25T15:27:47.783947+0000","last_unstale":"2024-04-25T15:27:49.744788+0000","last_undegraded":"2024-04-25T15:27:49.744788+0000","last_fullsized":"2024-04-25T15:27:49.744788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:03:38.884451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774979+0000","last_change":"2024-04-25T15:27:46.747059+0000","last_active":"2024-04-25T15:27:48.774979+0000","last_peered":"2024-04-25T15:27:48.774979+0000","last_clean":"2024-04-25T15:27:48.774979+0000","last_became_active":"2024-04-25T15:27:46.746774+0000","last_became_peered":"2024-04-25T15:27:46.746774+0000","last_unstale":"2024-04-25T15:27:48.774979+0000","last_undegraded":"2024-04-25T15:27:48.774979+0000","last_fullsized":"2024-04-25T15:27:48.774979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:25.199824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762364+0000","last_change":"2024-04-25T15:27:47.773677+0000","last_active":"2024-04-25T15:27:49.762364+0000","last_peered":"2024-04-25T15:27:49.762364+0000","last_clean":"2024-04-25T15:27:49.762364+0000","last_became_active":"2024-04-25T15:27:47.773338+0000","last_became_peered":"2024-04-25T15:27:47.773338+0000","last_unstale":"2024-04-25T15:27:49.762364+0000","last_undegraded":"2024-04-25T15:27:49.762364+0000","last_fullsized":"2024-04-25T15:27:49.762364+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:35:52.031977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213176+0000","last_change":"2024-04-25T15:27:46.755998+0000","last_active":"2024-04-25T15:27:49.213176+0000","last_peered":"2024-04-25T15:27:49.213176+0000","last_clean":"2024-04-25T15:27:49.213176+0000","last_became_active":"2024-04-25T15:27:46.755762+0000","last_became_peered":"2024-04-25T15:27:46.755762+0000","last_unstale":"2024-04-25T15:27:49.213176+0000","last_undegraded":"2024-04-25T15:27:49.213176+0000","last_fullsized":"2024-04-25T15:27:49.213176+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:25.199824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773402+0000","last_change":"2024-04-25T15:27:47.774124+0000","last_active":"2024-04-25T15:27:48.773402+0000","last_peered":"2024-04-25T15:27:48.773402+0000","last_clean":"2024-04-25T15:27:48.773402+0000","last_became_active":"2024-04-25T15:27:47.773798+0000","last_became_peered":"2024-04-25T15:27:47.773798+0000","last_unstale":"2024-04-25T15:27:48.773402+0000","last_undegraded":"2024-04-25T15:27:48.773402+0000","last_fullsized":"2024-04-25T15:27:48.773402+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:11:10.445989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762105+0000","last_change":"2024-04-25T15:27:46.753451+0000","last_active":"2024-04-25T15:27:49.762105+0000","last_peered":"2024-04-25T15:27:49.762105+0000","last_clean":"2024-04-25T15:27:49.762105+0000","last_became_active":"2024-04-25T15:27:46.753133+0000","last_became_peered":"2024-04-25T15:27:46.753133+0000","last_unstale":"2024-04-25T15:27:49.762105+0000","last_undegraded":"2024-04-25T15:27:49.762105+0000","last_fullsized":"2024-04-25T15:27:49.762105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:48.486002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417695+0000","last_change":"2024-04-25T15:27:47.779066+0000","last_active":"2024-04-25T15:27:50.417695+0000","last_peered":"2024-04-25T15:27:50.417695+0000","last_clean":"2024-04-25T15:27:50.417695+0000","last_became_active":"2024-04-25T15:27:47.778110+0000","last_became_peered":"2024-04-25T15:27:47.778110+0000","last_unstale":"2024-04-25T15:27:50.417695+0000","last_undegraded":"2024-04-25T15:27:50.417695+0000","last_fullsized":"2024-04-25T15:27:50.417695+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:21.619498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773975+0000","last_change":"2024-04-25T15:27:46.747236+0000","last_active":"2024-04-25T15:27:48.773975+0000","last_peered":"2024-04-25T15:27:48.773975+0000","last_clean":"2024-04-25T15:27:48.773975+0000","last_became_active":"2024-04-25T15:27:46.746401+0000","last_became_peered":"2024-04-25T15:27:46.746401+0000","last_unstale":"2024-04-25T15:27:48.773975+0000","last_undegraded":"2024-04-25T15:27:48.773975+0000","last_fullsized":"2024-04-25T15:27:48.773975+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:37.490054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774707+0000","last_change":"2024-04-25T15:27:47.761466+0000","last_active":"2024-04-25T15:27:48.774707+0000","last_peered":"2024-04-25T15:27:48.774707+0000","last_clean":"2024-04-25T15:27:48.774707+0000","last_became_active":"2024-04-25T15:27:47.761045+0000","last_became_peered":"2024-04-25T15:27:47.761045+0000","last_unstale":"2024-04-25T15:27:48.774707+0000","last_undegraded":"2024-04-25T15:27:48.774707+0000","last_fullsized":"2024-04-25T15:27:48.774707+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:53.391175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213608+0000","last_change":"2024-04-25T15:27:46.759114+0000","last_active":"2024-04-25T15:27:49.213608+0000","last_peered":"2024-04-25T15:27:49.213608+0000","last_clean":"2024-04-25T15:27:49.213608+0000","last_became_active":"2024-04-25T15:27:46.758785+0000","last_became_peered":"2024-04-25T15:27:46.758785+0000","last_unstale":"2024-04-25T15:27:49.213608+0000","last_undegraded":"2024-04-25T15:27:49.213608+0000","last_fullsized":"2024-04-25T15:27:49.213608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:06.332293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745353+0000","last_change":"2024-04-25T15:27:47.778525+0000","last_active":"2024-04-25T15:27:49.745353+0000","last_peered":"2024-04-25T15:27:49.745353+0000","last_clean":"2024-04-25T15:27:49.745353+0000","last_became_active":"2024-04-25T15:27:47.778145+0000","last_became_peered":"2024-04-25T15:27:47.778145+0000","last_unstale":"2024-04-25T15:27:49.745353+0000","last_undegraded":"2024-04-25T15:27:49.745353+0000","last_fullsized":"2024-04-25T15:27:49.745353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:18.572778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774631+0000","last_change":"2024-04-25T15:27:46.750313+0000","last_active":"2024-04-25T15:27:48.774631+0000","last_peered":"2024-04-25T15:27:48.774631+0000","last_clean":"2024-04-25T15:27:48.774631+0000","last_became_active":"2024-04-25T15:27:46.749988+0000","last_became_peered":"2024-04-25T15:27:46.749988+0000","last_unstale":"2024-04-25T15:27:48.774631+0000","last_undegraded":"2024-04-25T15:27:48.774631+0000","last_fullsized":"2024-04-25T15:27:48.774631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:02.088263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.775131+0000","last_change":"2024-04-25T15:27:47.779927+0000","last_active":"2024-04-25T15:27:48.775131+0000","last_peered":"2024-04-25T15:27:48.775131+0000","last_clean":"2024-04-25T15:27:48.775131+0000","last_became_active":"2024-04-25T15:27:47.779619+0000","last_became_peered":"2024-04-25T15:27:47.779619+0000","last_unstale":"2024-04-25T15:27:48.775131+0000","last_undegraded":"2024-04-25T15:27:48.775131+0000","last_fullsized":"2024-04-25T15:27:48.775131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:32.497676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213471+0000","last_change":"2024-04-25T15:27:46.754470+0000","last_active":"2024-04-25T15:27:49.213471+0000","last_peered":"2024-04-25T15:27:49.213471+0000","last_clean":"2024-04-25T15:27:49.213471+0000","last_became_active":"2024-04-25T15:27:46.753975+0000","last_became_peered":"2024-04-25T15:27:46.753975+0000","last_unstale":"2024-04-25T15:27:49.213471+0000","last_undegraded":"2024-04-25T15:27:49.213471+0000","last_fullsized":"2024-04-25T15:27:49.213471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:30.439782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418043+0000","last_change":"2024-04-25T15:27:47.779053+0000","last_active":"2024-04-25T15:27:50.418043+0000","last_peered":"2024-04-25T15:27:50.418043+0000","last_clean":"2024-04-25T15:27:50.418043+0000","last_became_active":"2024-04-25T15:27:47.777906+0000","last_became_peered":"2024-04-25T15:27:47.777906+0000","last_unstale":"2024-04-25T15:27:50.418043+0000","last_undegraded":"2024-04-25T15:27:50.418043+0000","last_fullsized":"2024-04-25T15:27:50.418043+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:06.828910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744979+0000","last_change":"2024-04-25T15:27:46.754424+0000","last_active":"2024-04-25T15:27:49.744979+0000","last_peered":"2024-04-25T15:27:49.744979+0000","last_clean":"2024-04-25T15:27:49.744979+0000","last_became_active":"2024-04-25T15:27:46.754102+0000","last_became_peered":"2024-04-25T15:27:46.754102+0000","last_unstale":"2024-04-25T15:27:49.744979+0000","last_undegraded":"2024-04-25T15:27:49.744979+0000","last_fullsized":"2024-04-25T15:27:49.744979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:52.385229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417996+0000","last_change":"2024-04-25T15:27:47.777884+0000","last_active":"2024-04-25T15:27:50.417996+0000","last_peered":"2024-04-25T15:27:50.417996+0000","last_clean":"2024-04-25T15:27:50.417996+0000","last_became_active":"2024-04-25T15:27:47.777532+0000","last_became_peered":"2024-04-25T15:27:47.777532+0000","last_unstale":"2024-04-25T15:27:50.417996+0000","last_undegraded":"2024-04-25T15:27:50.417996+0000","last_fullsized":"2024-04-25T15:27:50.417996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:43.448609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213421+0000","last_change":"2024-04-25T15:27:46.756182+0000","last_active":"2024-04-25T15:27:49.213421+0000","last_peered":"2024-04-25T15:27:49.213421+0000","last_clean":"2024-04-25T15:27:49.213421+0000","last_became_active":"2024-04-25T15:27:46.755789+0000","last_became_peered":"2024-04-25T15:27:46.755789+0000","last_unstale":"2024-04-25T15:27:49.213421+0000","last_undegraded":"2024-04-25T15:27:49.213421+0000","last_fullsized":"2024-04-25T15:27:49.213421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:04.081872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417663+0000","last_change":"2024-04-25T15:27:47.784326+0000","last_active":"2024-04-25T15:27:50.417663+0000","last_peered":"2024-04-25T15:27:50.417663+0000","last_clean":"2024-04-25T15:27:50.417663+0000","last_became_active":"2024-04-25T15:27:47.784004+0000","last_became_peered":"2024-04-25T15:27:47.784004+0000","last_unstale":"2024-04-25T15:27:50.417663+0000","last_undegraded":"2024-04-25T15:27:50.417663+0000","last_fullsized":"2024-04-25T15:27:50.417663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:28.663954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762408+0000","last_change":"2024-04-25T15:27:46.751621+0000","last_active":"2024-04-25T15:27:49.762408+0000","last_peered":"2024-04-25T15:27:49.762408+0000","last_clean":"2024-04-25T15:27:49.762408+0000","last_became_active":"2024-04-25T15:27:46.751328+0000","last_became_peered":"2024-04-25T15:27:46.751328+0000","last_unstale":"2024-04-25T15:27:49.762408+0000","last_undegraded":"2024-04-25T15:27:49.762408+0000","last_fullsized":"2024-04-25T15:27:49.762408+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:23.346756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418293+0000","last_change":"2024-04-25T15:27:47.785507+0000","last_active":"2024-04-25T15:27:50.418293+0000","last_peered":"2024-04-25T15:27:50.418293+0000","last_clean":"2024-04-25T15:27:50.418293+0000","last_became_active":"2024-04-25T15:27:47.785197+0000","last_became_peered":"2024-04-25T15:27:47.785197+0000","last_unstale":"2024-04-25T15:27:50.418293+0000","last_undegraded":"2024-04-25T15:27:50.418293+0000","last_fullsized":"2024-04-25T15:27:50.418293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:58.461399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773804+0000","last_change":"2024-04-25T15:27:46.748665+0000","last_active":"2024-04-25T15:27:48.773804+0000","last_peered":"2024-04-25T15:27:48.773804+0000","last_clean":"2024-04-25T15:27:48.773804+0000","last_became_active":"2024-04-25T15:27:46.748364+0000","last_became_peered":"2024-04-25T15:27:46.748364+0000","last_unstale":"2024-04-25T15:27:48.773804+0000","last_undegraded":"2024-04-25T15:27:48.773804+0000","last_fullsized":"2024-04-25T15:27:48.773804+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:30.416165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773844+0000","last_change":"2024-04-25T15:27:47.783356+0000","last_active":"2024-04-25T15:27:48.773844+0000","last_peered":"2024-04-25T15:27:48.773844+0000","last_clean":"2024-04-25T15:27:48.773844+0000","last_became_active":"2024-04-25T15:27:47.782999+0000","last_became_peered":"2024-04-25T15:27:47.782999+0000","last_unstale":"2024-04-25T15:27:48.773844+0000","last_undegraded":"2024-04-25T15:27:48.773844+0000","last_fullsized":"2024-04-25T15:27:48.773844+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:03:38.884451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213676+0000","last_change":"2024-04-25T15:27:46.759084+0000","last_active":"2024-04-25T15:27:49.213676+0000","last_peered":"2024-04-25T15:27:49.213676+0000","last_clean":"2024-04-25T15:27:49.213676+0000","last_became_active":"2024-04-25T15:27:46.758757+0000","last_became_peered":"2024-04-25T15:27:46.758757+0000","last_unstale":"2024-04-25T15:27:49.213676+0000","last_undegraded":"2024-04-25T15:27:49.213676+0000","last_fullsized":"2024-04-25T15:27:49.213676+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:44.420376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762538+0000","last_change":"2024-04-25T15:27:47.782899+0000","last_active":"2024-04-25T15:27:49.762538+0000","last_peered":"2024-04-25T15:27:49.762538+0000","last_clean":"2024-04-25T15:27:49.762538+0000","last_became_active":"2024-04-25T15:27:47.782551+0000","last_became_peered":"2024-04-25T15:27:47.782551+0000","last_unstale":"2024-04-25T15:27:49.762538+0000","last_undegraded":"2024-04-25T15:27:49.762538+0000","last_fullsized":"2024-04-25T15:27:49.762538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:37.274297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.762643+0000","last_change":"2024-04-25T15:27:46.753206+0000","last_active":"2024-04-25T15:27:49.762643+0000","last_peered":"2024-04-25T15:27:49.762643+0000","last_clean":"2024-04-25T15:27:49.762643+0000","last_became_active":"2024-04-25T15:27:46.752883+0000","last_became_peered":"2024-04-25T15:27:46.752883+0000","last_unstale":"2024-04-25T15:27:49.762643+0000","last_undegraded":"2024-04-25T15:27:49.762643+0000","last_fullsized":"2024-04-25T15:27:49.762643+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:04:58.966320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745011+0000","last_change":"2024-04-25T15:27:47.782136+0000","last_active":"2024-04-25T15:27:49.745011+0000","last_peered":"2024-04-25T15:27:49.745011+0000","last_clean":"2024-04-25T15:27:49.745011+0000","last_became_active":"2024-04-25T15:27:47.781773+0000","last_became_peered":"2024-04-25T15:27:47.781773+0000","last_unstale":"2024-04-25T15:27:49.745011+0000","last_undegraded":"2024-04-25T15:27:49.745011+0000","last_fullsized":"2024-04-25T15:27:49.745011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:41.235966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418144+0000","last_change":"2024-04-25T15:27:46.753592+0000","last_active":"2024-04-25T15:27:50.418144+0000","last_peered":"2024-04-25T15:27:50.418144+0000","last_clean":"2024-04-25T15:27:50.418144+0000","last_became_active":"2024-04-25T15:27:46.753163+0000","last_became_peered":"2024-04-25T15:27:46.753163+0000","last_unstale":"2024-04-25T15:27:50.418144+0000","last_undegraded":"2024-04-25T15:27:50.418144+0000","last_fullsized":"2024-04-25T15:27:50.418144+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:51:09.829896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.745295+0000","last_change":"2024-04-25T15:27:47.770518+0000","last_active":"2024-04-25T15:27:49.745295+0000","last_peered":"2024-04-25T15:27:49.745295+0000","last_clean":"2024-04-25T15:27:49.745295+0000","last_became_active":"2024-04-25T15:27:47.769861+0000","last_became_peered":"2024-04-25T15:27:47.769861+0000","last_unstale":"2024-04-25T15:27:49.745295+0000","last_undegraded":"2024-04-25T15:27:49.745295+0000","last_fullsized":"2024-04-25T15:27:49.745295+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:55.281915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773698+0000","last_change":"2024-04-25T15:27:46.744686+0000","last_active":"2024-04-25T15:27:48.773698+0000","last_peered":"2024-04-25T15:27:48.773698+0000","last_clean":"2024-04-25T15:27:48.773698+0000","last_became_active":"2024-04-25T15:27:46.744321+0000","last_became_peered":"2024-04-25T15:27:46.744321+0000","last_unstale":"2024-04-25T15:27:48.773698+0000","last_undegraded":"2024-04-25T15:27:48.773698+0000","last_fullsized":"2024-04-25T15:27:48.773698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:00.507062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.418216+0000","last_change":"2024-04-25T15:27:47.775042+0000","last_active":"2024-04-25T15:27:50.418216+0000","last_peered":"2024-04-25T15:27:50.418216+0000","last_clean":"2024-04-25T15:27:50.418216+0000","last_became_active":"2024-04-25T15:27:47.774708+0000","last_became_peered":"2024-04-25T15:27:47.774708+0000","last_unstale":"2024-04-25T15:27:50.418216+0000","last_undegraded":"2024-04-25T15:27:50.418216+0000","last_fullsized":"2024-04-25T15:27:50.418216+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:14:43.212650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417721+0000","last_change":"2024-04-25T15:27:46.749552+0000","last_active":"2024-04-25T15:27:50.417721+0000","last_peered":"2024-04-25T15:27:50.417721+0000","last_clean":"2024-04-25T15:27:50.417721+0000","last_became_active":"2024-04-25T15:27:46.749242+0000","last_became_peered":"2024-04-25T15:27:46.749242+0000","last_unstale":"2024-04-25T15:27:50.417721+0000","last_undegraded":"2024-04-25T15:27:50.417721+0000","last_fullsized":"2024-04-25T15:27:50.417721+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:52.544769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213168+0000","last_change":"2024-04-25T15:27:47.777251+0000","last_active":"2024-04-25T15:27:49.213168+0000","last_peered":"2024-04-25T15:27:49.213168+0000","last_clean":"2024-04-25T15:27:49.213168+0000","last_became_active":"2024-04-25T15:27:47.776932+0000","last_became_peered":"2024-04-25T15:27:47.776932+0000","last_unstale":"2024-04-25T15:27:49.213168+0000","last_undegraded":"2024-04-25T15:27:49.213168+0000","last_fullsized":"2024-04-25T15:27:49.213168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:39:55.524808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774777+0000","last_change":"2024-04-25T15:27:46.746473+0000","last_active":"2024-04-25T15:27:48.774777+0000","last_peered":"2024-04-25T15:27:48.774777+0000","last_clean":"2024-04-25T15:27:48.774777+0000","last_became_active":"2024-04-25T15:27:46.746083+0000","last_became_peered":"2024-04-25T15:27:46.746083+0000","last_unstale":"2024-04-25T15:27:48.774777+0000","last_undegraded":"2024-04-25T15:27:48.774777+0000","last_fullsized":"2024-04-25T15:27:48.774777+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:41.597367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.774944+0000","last_change":"2024-04-25T15:27:47.775285+0000","last_active":"2024-04-25T15:27:48.774944+0000","last_peered":"2024-04-25T15:27:48.774944+0000","last_clean":"2024-04-25T15:27:48.774944+0000","last_became_active":"2024-04-25T15:27:47.774923+0000","last_became_peered":"2024-04-25T15:27:47.774923+0000","last_unstale":"2024-04-25T15:27:48.774944+0000","last_undegraded":"2024-04-25T15:27:48.774944+0000","last_fullsized":"2024-04-25T15:27:48.774944+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:03.094209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:48.773997+0000","last_change":"2024-04-25T15:27:46.745918+0000","last_active":"2024-04-25T15:27:48.773997+0000","last_peered":"2024-04-25T15:27:48.773997+0000","last_clean":"2024-04-25T15:27:48.773997+0000","last_became_active":"2024-04-25T15:27:46.745596+0000","last_became_peered":"2024-04-25T15:27:46.745596+0000","last_unstale":"2024-04-25T15:27:48.773997+0000","last_undegraded":"2024-04-25T15:27:48.773997+0000","last_fullsized":"2024-04-25T15:27:48.773997+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:25:30.439782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.212928+0000","last_change":"2024-04-25T15:27:47.785204+0000","last_active":"2024-04-25T15:27:49.212928+0000","last_peered":"2024-04-25T15:27:49.212928+0000","last_clean":"2024-04-25T15:27:49.212928+0000","last_became_active":"2024-04-25T15:27:47.784676+0000","last_became_peered":"2024-04-25T15:27:47.784676+0000","last_unstale":"2024-04-25T15:27:49.212928+0000","last_undegraded":"2024-04-25T15:27:49.212928+0000","last_fullsized":"2024-04-25T15:27:49.212928+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:24.352702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213338+0000","last_change":"2024-04-25T15:27:46.746186+0000","last_active":"2024-04-25T15:27:49.213338+0000","last_peered":"2024-04-25T15:27:49.213338+0000","last_clean":"2024-04-25T15:27:49.213338+0000","last_became_active":"2024-04-25T15:27:46.745867+0000","last_became_peered":"2024-04-25T15:27:46.745867+0000","last_unstale":"2024-04-25T15:27:49.213338+0000","last_undegraded":"2024-04-25T15:27:49.213338+0000","last_fullsized":"2024-04-25T15:27:49.213338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:37.490054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:50.417865+0000","last_change":"2024-04-25T15:27:47.784218+0000","last_active":"2024-04-25T15:27:50.417865+0000","last_peered":"2024-04-25T15:27:50.417865+0000","last_clean":"2024-04-25T15:27:50.417865+0000","last_became_active":"2024-04-25T15:27:47.783851+0000","last_became_peered":"2024-04-25T15:27:47.783851+0000","last_unstale":"2024-04-25T15:27:50.417865+0000","last_undegraded":"2024-04-25T15:27:50.417865+0000","last_fullsized":"2024-04-25T15:27:50.417865+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:46.728373+0000","last_clean_scrub_stamp":"2024-04-25T15:27:46.728373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:06.431534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.213046+0000","last_change":"2024-04-25T15:27:46.755483+0000","last_active":"2024-04-25T15:27:49.213046+0000","last_peered":"2024-04-25T15:27:49.213046+0000","last_clean":"2024-04-25T15:27:49.213046+0000","last_became_active":"2024-04-25T15:27:46.755160+0000","last_became_peered":"2024-04-25T15:27:46.755160+0000","last_unstale":"2024-04-25T15:27:49.213046+0000","last_undegraded":"2024-04-25T15:27:49.213046+0000","last_fullsized":"2024-04-25T15:27:49.213046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:27:45.722427+0000","last_clean_scrub_stamp":"2024-04-25T15:27:45.722427+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:06.558947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":195,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-25T15:27:49.744656+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:27:49.744656+0000","last_peered":"2024-04-25T15:27:49.744656+0000","last_clean":"2024-04-25T15:27:49.744656+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:27:49.744656+0000","last_undegraded":"2024-04-25T15:27:49.744656+0000","last_fullsized":"2024-04-25T15:27:49.744656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":1417216,"data_stored":932499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":341607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":341605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":341585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":341627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":827392,"data_stored":343127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":835584,"data_stored":343743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":1409024,"data_stored":931883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:27:56.699 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:27:56.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:27:56.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.854+0000 7f5584ae7640 1 Processor -- start 2024-04-25T15:27:56.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.854+0000 7f5584ae7640 1 -- start start 2024-04-25T15:27:56.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.855+0000 7f5584ae7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 0x7f558015e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.855+0000 7f5584ae7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55801609b0 con 0x7f5580163fa0 2024-04-25T15:27:56.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.855+0000 7f5584ae7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5580164cd0 con 0x7f558006c5c0 2024-04-25T15:27:56.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.855+0000 7f5584ae7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5580160c30 con 0x7f5580161710 2024-04-25T15:27:56.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.855+0000 7f557ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5580163fa0 0x7f55801643b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55660/0 (socket says 172.21.15.67:55660) 2024-04-25T15:27:56.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.855+0000 7f557ed9d640 1 -- 172.21.15.67:0/3340038872 learned_addr learned my addr 172.21.15.67:0/3340038872 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557e59c640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 0x7f558015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1949998979 0 0) 0x7f5578001490 con 0x7f5580163fa0 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f556c003520 con 0x7f5580163fa0 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 160293260 0 0) 0x7f556c003520 con 0x7f5580163fa0 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 msgr2=0x7f558015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 0x7f558015e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 >> v1:172.21.15.176:6789/0 conn(0x7f558006c5c0 legacy=0x7f558006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5580160f10 con 0x7f5580163fa0 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f5578001bc0 con 0x7f5580163fa0 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 >> v1:172.21.15.67:6789/0 conn(0x7f5580163fa0 legacy=0x7f55801643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 0x7f556c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f556c003f60 0x7f556c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f556c004d40 0x7f556c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5580160c30 con 0x7f556c004d40 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f556c00bc00 con 0x7f556c003f60 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f556c00be00 con 0x7f5580161710 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557ed9d640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f556c004d40 0x7f556c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557e59c640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 0x7f556c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f5577fff640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f556c003f60 0x7f556c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557ed9d640 1 -- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 msgr2=0x7f556c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557ed9d640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5580161710 0x7f556c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557ed9d640 1 -- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f556c003f60 msgr2=0x7f556c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557ed9d640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f556c003f60 0x7f556c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.856+0000 7f557ed9d640 1 -- 172.21.15.67:0/3340038872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556c003520 con 0x7f556c004d40 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.857+0000 7f557ed9d640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f556c004d40 0x7f556c005150 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f55800d0300 tx=0x7f556c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac84255efd55289c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.857+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557800a650 con 0x7f556c004d40 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.857+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557802f070 con 0x7f556c004d40 2024-04-25T15:27:56.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.857+0000 7f557dd9b640 1 -- 172.21.15.67:0/3340038872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578034470 con 0x7f556c004d40 2024-04-25T15:27:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.858+0000 7f5584ae7640 1 -- 172.21.15.67:0/3340038872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f556c004d40 msgr2=0x7f556c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.858+0000 7f5584ae7640 1 --2- 172.21.15.67:0/3340038872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f556c004d40 0x7f556c005150 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f55800d0300 tx=0x7f556c005830 comp rx=0 tx=0).stop 2024-04-25T15:27:56.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.858+0000 7f557ed9d640 1 -- 172.21.15.67:0/3340038872 reap_dead start 2024-04-25T15:27:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.858+0000 7f5584ae7640 1 -- 172.21.15.67:0/3340038872 shutdown_connections 2024-04-25T15:27:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.858+0000 7f5584ae7640 1 -- 172.21.15.67:0/3340038872 >> 172.21.15.67:0/3340038872 conn(0x7f55800d5d90 msgr2=0x7f55800d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.858+0000 7f5584ae7640 1 -- 172.21.15.67:0/3340038872 shutdown_connections 2024-04-25T15:27:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.858+0000 7f5584ae7640 1 -- 172.21.15.67:0/3340038872 wait complete. 2024-04-25T15:27:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.860+0000 7f5584ae7640 1 Processor -- start 2024-04-25T15:27:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.860+0000 7f5584ae7640 1 -- start start 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5584ae7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55800ca0d0 0x7f558014b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5584ae7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55800cf8d0 0x7f55800cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5584ae7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f558014b9a0 0x7f558014bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5584ae7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f558015e200 con 0x7f55800cf8d0 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5584ae7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f558015e080 con 0x7f55800ca0d0 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5584ae7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f558015e380 con 0x7f558014b9a0 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5577fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55800cf8d0 0x7f55800cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5577fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55800cf8d0 0x7f55800cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35284/0 (socket says 172.21.15.67:35284) 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.861+0000 7f5577fff640 1 -- 172.21.15.67:0/2491987455 learned_addr learned my addr 172.21.15.67:0/2491987455 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f557e59c640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55800ca0d0 0x7f558014b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f557ed9d640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f558014b9a0 0x7f558014bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5577fff640 1 -- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f558014b9a0 msgr2=0x7f558014bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5577fff640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f558014b9a0 0x7f558014bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5577fff640 1 -- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55800ca0d0 msgr2=0x7f558014b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5577fff640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55800ca0d0 0x7f558014b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5577fff640 1 -- 172.21.15.67:0/2491987455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55800ceb30 con 0x7f55800cf8d0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5577fff640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55800cf8d0 0x7f55800cfce0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f556800f610 tx=0x7f556800fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5576ffd640 1 -- 172.21.15.67:0/2491987455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55680052e0 con 0x7f55800cf8d0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5576ffd640 1 -- 172.21.15.67:0/2491987455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5568018070 con 0x7f55800cf8d0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55800cee10 con 0x7f55800cf8d0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5576ffd640 1 -- 172.21.15.67:0/2491987455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568003830 con 0x7f55800cf8d0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.862+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55800cf3d0 con 0x7f55800cf8d0 2024-04-25T15:27:56.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.863+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5580050740 con 0x7f55800cf8d0 2024-04-25T15:27:56.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.864+0000 7f5576ffd640 1 -- 172.21.15.67:0/2491987455 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f55680039d0 con 0x7f55800cf8d0 2024-04-25T15:27:56.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.865+0000 7f5576ffd640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55600744b0 0x7f55600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:56.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.865+0000 7f5576ffd640 1 -- 172.21.15.67:0/2491987455 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f5568094e70 con 0x7f55800cf8d0 2024-04-25T15:27:56.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.865+0000 7f557e59c640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55600744b0 0x7f55600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:56.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.865+0000 7f557e59c640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55600744b0 0x7f55600769c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f55800ca800 tx=0x7f5570009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:56.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:56.869+0000 7f5576ffd640 1 -- 172.21.15.67:0/2491987455 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f556805f720 con 0x7f55800cf8d0 2024-04-25T15:27:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.126+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f558006cb00 con 0x7f55800cf8d0 2024-04-25T15:27:57.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.131+0000 7f5576ffd640 1 -- 172.21.15.67:0/2491987455 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7f556805f020 con 0x7f55800cf8d0 2024-04-25T15:27:57.132 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:27:57.132 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:27:57.132 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-25T15:27:57.132 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-25T15:27:57.132 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:27:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-25T15:27:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55600744b0 msgr2=0x7f55600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55600744b0 0x7f55600769c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f55800ca800 tx=0x7f5570009190 comp rx=0 tx=0).stop 2024-04-25T15:27:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55800cf8d0 msgr2=0x7f55800cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55800cf8d0 0x7f55800cfce0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f556800f610 tx=0x7f556800fa10 comp rx=0 tx=0).stop 2024-04-25T15:27:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 shutdown_connections 2024-04-25T15:27:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55600744b0 0x7f55600769c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f558014b9a0 0x7f558014bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55800cf8d0 0x7f55800cfce0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 --2- 172.21.15.67:0/2491987455 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55800ca0d0 0x7f558014b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 >> 172.21.15.67:0/2491987455 conn(0x7f55800d5d90 msgr2=0x7f558015fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 shutdown_connections 2024-04-25T15:27:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.133+0000 7f5584ae7640 1 -- 172.21.15.67:0/2491987455 wait complete. 2024-04-25T15:27:57.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:27:57.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.256+0000 7f25361a8640 1 Processor -- start 2024-04-25T15:27:57.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.256+0000 7f25361a8640 1 -- start start 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f25361a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25300b4830 0x7f253006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f25361a8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f25300b2dc0 0x7f25300b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f25361a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253006d870 con 0x7f25300d4570 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f25361a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f25300b4c40 con 0x7f25300b4830 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f25361a8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f253006e460 con 0x7f25300b2dc0 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25300b4830 0x7f253006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25300b4830 0x7f253006e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35292/0 (socket says 172.21.15.67:35292) 2024-04-25T15:27:57.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f25300d4570 0x7f25300d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55666/0 (socket says 172.21.15.67:55666) 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252effd640 1 -- 172.21.15.67:0/4141337284 learned_addr learned my addr 172.21.15.67:0/4141337284 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252ffff640 1 --2- 172.21.15.67:0/4141337284 >> v2:172.21.15.176:3300/0 conn(0x7f25300b2dc0 0x7f25300b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252effd640 1 -- 172.21.15.67:0/4141337284 >> v2:172.21.15.176:3300/0 conn(0x7f25300b2dc0 msgr2=0x7f25300b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252effd640 1 --2- 172.21.15.67:0/4141337284 >> v2:172.21.15.176:3300/0 conn(0x7f25300b2dc0 0x7f25300b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252effd640 1 -- 172.21.15.67:0/4141337284 >> v1:172.21.15.67:6789/0 conn(0x7f25300d4570 legacy=0x7f25300d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252effd640 1 -- 172.21.15.67:0/4141337284 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253006db10 con 0x7f25300b4830 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252e7fc640 1 -- 172.21.15.67:0/4141337284 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732680767 0 0) 0x7f253006d870 con 0x7f25300d4570 2024-04-25T15:27:57.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.258+0000 7f252ffff640 1 --2- 172.21.15.67:0/4141337284 >> v2:172.21.15.176:3300/0 conn(0x7f25300b2dc0 0x7f25300b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.259+0000 7f252effd640 1 --2- 172.21.15.67:0/4141337284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300b4830 0x7f253006e8f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f253006e5e0 tx=0x7f252402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67673273b2cf636 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.259+0000 7f252e7fc640 1 -- 172.21.15.67:0/4141337284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524037070 con 0x7f25300b4830 2024-04-25T15:27:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.259+0000 7f252e7fc640 1 -- 172.21.15.67:0/4141337284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2524032440 con 0x7f25300b4830 2024-04-25T15:27:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.259+0000 7f252e7fc640 1 -- 172.21.15.67:0/4141337284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524032810 con 0x7f25300b4830 2024-04-25T15:27:57.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 -- 172.21.15.67:0/4141337284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300b4830 msgr2=0x7f253006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 --2- 172.21.15.67:0/4141337284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300b4830 0x7f253006e8f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f253006e5e0 tx=0x7f252402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 -- 172.21.15.67:0/4141337284 shutdown_connections 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 --2- 172.21.15.67:0/4141337284 >> v2:172.21.15.176:3300/0 conn(0x7f25300b2dc0 0x7f25300b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 --2- 172.21.15.67:0/4141337284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300b4830 0x7f253006e8f0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 -- 172.21.15.67:0/4141337284 >> 172.21.15.67:0/4141337284 conn(0x7f25300d3a50 msgr2=0x7f25300d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 -- 172.21.15.67:0/4141337284 shutdown_connections 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.260+0000 7f25361a8640 1 -- 172.21.15.67:0/4141337284 wait complete. 2024-04-25T15:27:57.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.262+0000 7f25361a8640 1 Processor -- start 2024-04-25T15:27:57.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.262+0000 7f25361a8640 1 -- start start 2024-04-25T15:27:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f25361a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25300b2dc0 0x7f253009d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f25361a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25300b4830 0x7f25301569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f25361a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300d4570 0x7f25300b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f25361a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25300d0fb0 con 0x7f25300d4570 2024-04-25T15:27:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f25361a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f25300d0bc0 con 0x7f25300b4830 2024-04-25T15:27:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f25361a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f25300b3710 con 0x7f25300b2dc0 2024-04-25T15:27:57.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f252ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300d4570 0x7f25300b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f252ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300d4570 0x7f25300b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35294/0 (socket says 172.21.15.67:35294) 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f252ffff640 1 -- 172.21.15.67:0/2422912302 learned_addr learned my addr 172.21.15.67:0/2422912302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f252f7fe640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25300b2dc0 0x7f253009d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f252effd640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25300b4830 0x7f25301569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f252ffff640 1 -- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25300b2dc0 msgr2=0x7f253009d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.263+0000 7f252ffff640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25300b2dc0 0x7f253009d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f252ffff640 1 -- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25300b4830 msgr2=0x7f25301569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f252ffff640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25300b4830 0x7f25301569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f252ffff640 1 -- 172.21.15.67:0/2422912302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253009df80 con 0x7f25300d4570 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f252effd640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25300b4830 0x7f25301569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f252ffff640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300d4570 0x7f25300b17c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f252800cdb0 tx=0x7f252800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f2517fff640 1 -- 172.21.15.67:0/2422912302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252800f680 con 0x7f25300d4570 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f2517fff640 1 -- 172.21.15.67:0/2422912302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252800f840 con 0x7f25300d4570 2024-04-25T15:27:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253009e260 con 0x7f25300d4570 2024-04-25T15:27:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f2517fff640 1 -- 172.21.15.67:0/2422912302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252800fc30 con 0x7f25300d4570 2024-04-25T15:27:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.264+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253009e820 con 0x7f25300d4570 2024-04-25T15:27:57.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.265+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2530050740 con 0x7f25300d4570 2024-04-25T15:27:57.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.266+0000 7f2517fff640 1 -- 172.21.15.67:0/2422912302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f25280213f0 con 0x7f25300d4570 2024-04-25T15:27:57.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.266+0000 7f2517fff640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f251c0744e0 0x7f251c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:57.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.266+0000 7f2517fff640 1 -- 172.21.15.67:0/2422912302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f2528095fd0 con 0x7f25300d4570 2024-04-25T15:27:57.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.266+0000 7f252f7fe640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f251c0744e0 0x7f251c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:57.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.267+0000 7f252f7fe640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f251c0744e0 0x7f251c0769f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2520001020 tx=0x7f2520009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:57.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.271+0000 7f2517fff640 1 -- 172.21.15.67:0/2422912302 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2528014030 con 0x7f25300d4570 2024-04-25T15:27:57.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.493+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f253009dd10 con 0x7f25300d4570 2024-04-25T15:27:57.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.494+0000 7f2517fff640 1 -- 172.21.15.67:0/2422912302 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5189 (secure 0 0 0) 0x7f2528060760 con 0x7f25300d4570 2024-04-25T15:27:57.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:27:57.495 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2076054232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2076054232},{"type":"v1","addr":"172.21.15.176:6836","nonce":2076054232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":5537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3930910548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3930910548},{"type":"v1","addr":"172.21.15.176:6837","nonce":3930910548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2322157634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322157634},{"type":"v1","addr":"172.21.15.67:6834","nonce":2322157634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:47.123622+0000","modified":"2024-04-25T15:27:50.154657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6273":{"gid":6273,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3903876028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3903876028},{"type":"v1","addr":"172.21.15.67:6835","nonce":3903876028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6273,"qdb_cluster":[6273]},"id":5}]} 2024-04-25T15:27:57.495 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.495+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f251c0744e0 msgr2=0x7f251c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.495+0000 7f25361a8640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f251c0744e0 0x7f251c0769f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2520001020 tx=0x7f2520009040 comp rx=0 tx=0).stop 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.495+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300d4570 msgr2=0x7f25300b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.495+0000 7f25361a8640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300d4570 0x7f25300b17c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f252800cdb0 tx=0x7f252800a650 comp rx=0 tx=0).stop 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 shutdown_connections 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f251c0744e0 0x7f251c0769f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25300d4570 0x7f25300b17c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25300b4830 0x7f25301569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 --2- 172.21.15.67:0/2422912302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25300b2dc0 0x7f253009d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 >> 172.21.15.67:0/2422912302 conn(0x7f25300d3a50 msgr2=0x7f25300cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 shutdown_connections 2024-04-25T15:27:57.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:57.496+0000 7f25361a8640 1 -- 172.21.15.67:0/2422912302 wait complete. 2024-04-25T15:27:57.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:27:47.123622+0000', 'modified': '2024-04-25T15:27:50.154657+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6273': {'gid': 6273, 'name': 'a', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/3903876028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 3903876028}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3903876028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6273, 'qdb_cluster': [6273]} 2024-04-25T15:27:57.911 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:27:57.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T15:27:58.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.030+0000 7f2d10892640 1 Processor -- start 2024-04-25T15:27:58.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.031+0000 7f2d10892640 1 -- start start 2024-04-25T15:27:58.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d10892640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2d0c164b00 0x7f2d0c0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d10892640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d0c15df80 0x7f2d0c0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d10892640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d0c1615f0 con 0x7f2d0c160890 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d10892640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0c1613b0 con 0x7f2d0c15df80 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d10892640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0c164670 con 0x7f2d0c164b00 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d09d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d0c15df80 0x7f2d0c0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d09d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d0c15df80 0x7f2d0c0b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39284/0 (socket says 172.21.15.67:39284) 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d09d9b640 1 -- 172.21.15.67:0/2573845707 learned_addr learned my addr 172.21.15.67:0/2573845707 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.032+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/2573845707 >> v2:172.21.15.176:3300/0 conn(0x7f2d0c164b00 0x7f2d0c0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d09d9b640 1 -- 172.21.15.67:0/2573845707 >> v2:172.21.15.176:3300/0 conn(0x7f2d0c164b00 msgr2=0x7f2d0c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/2573845707 >> v2:172.21.15.176:3300/0 conn(0x7f2d0c164b00 0x7f2d0c0b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d09d9b640 1 -- 172.21.15.67:0/2573845707 >> v1:172.21.15.176:6789/0 conn(0x7f2d0c160890 legacy=0x7f2d0c160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d09d9b640 1 -- 172.21.15.67:0/2573845707 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0c164020 con 0x7f2d0c15df80 2024-04-25T15:27:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/2573845707 >> v2:172.21.15.176:3300/0 conn(0x7f2d0c164b00 0x7f2d0c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/2573845707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c15df80 0x7f2d0c0b80c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c08e410 tx=0x7f2cf402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1534a844169056 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d0959a640 1 -- 172.21.15.67:0/2573845707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4037070 con 0x7f2d0c15df80 2024-04-25T15:27:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.033+0000 7f2d0959a640 1 -- 172.21.15.67:0/2573845707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf4032440 con 0x7f2d0c15df80 2024-04-25T15:27:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d0959a640 1 -- 172.21.15.67:0/2573845707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4032810 con 0x7f2d0c15df80 2024-04-25T15:27:58.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 -- 172.21.15.67:0/2573845707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c15df80 msgr2=0x7f2d0c0b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 --2- 172.21.15.67:0/2573845707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c15df80 0x7f2d0c0b80c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c08e410 tx=0x7f2cf402a230 comp rx=0 tx=0).stop 2024-04-25T15:27:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 -- 172.21.15.67:0/2573845707 shutdown_connections 2024-04-25T15:27:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 --2- 172.21.15.67:0/2573845707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c15df80 0x7f2d0c0b80c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 --2- 172.21.15.67:0/2573845707 >> v2:172.21.15.176:3300/0 conn(0x7f2d0c164b00 0x7f2d0c0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 -- 172.21.15.67:0/2573845707 >> 172.21.15.67:0/2573845707 conn(0x7f2d0c0d2bc0 msgr2=0x7f2d0c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 -- 172.21.15.67:0/2573845707 shutdown_connections 2024-04-25T15:27:58.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.034+0000 7f2d10892640 1 -- 172.21.15.67:0/2573845707 wait complete. 2024-04-25T15:27:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.036+0000 7f2d10892640 1 Processor -- start 2024-04-25T15:27:58.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.036+0000 7f2d10892640 1 -- start start 2024-04-25T15:27:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.037+0000 7f2d10892640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c15df80 0x7f2d0c148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.037+0000 7f2d10892640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c160890 0x7f2d0c1f9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.037+0000 7f2d10892640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c164b00 0x7f2d0c14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d10892640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c0b4a00 con 0x7f2d0c164b00 2024-04-25T15:27:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d10892640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c0b4880 con 0x7f2d0c160890 2024-04-25T15:27:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d10892640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0c0b4b80 con 0x7f2d0c15df80 2024-04-25T15:27:58.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c164b00 0x7f2d0c14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c164b00 0x7f2d0c14ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39294/0 (socket says 172.21.15.67:39294) 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/299710506 learned_addr learned my addr 172.21.15.67:0/299710506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c15df80 0x7f2d0c148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c160890 0x7f2d0c1f9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c15df80 msgr2=0x7f2d0c148bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c15df80 0x7f2d0c148bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c160890 msgr2=0x7f2d0c1f9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c160890 0x7f2d0c1f9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/299710506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0c14b910 con 0x7f2d0c164b00 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c164b00 0x7f2d0c14ac50 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00ba10 tx=0x7f2cfc00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d02ffd640 1 -- 172.21.15.67:0/299710506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc019070 con 0x7f2d0c164b00 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d02ffd640 1 -- 172.21.15.67:0/299710506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cfc00cb40 con 0x7f2d0c164b00 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d02ffd640 1 -- 172.21.15.67:0/299710506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc014440 con 0x7f2d0c164b00 2024-04-25T15:27:58.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.038+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0c0c99d0 con 0x7f2d0c164b00 2024-04-25T15:27:58.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.039+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d0c1f9ac0 con 0x7f2d0c164b00 2024-04-25T15:27:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.040+0000 7f2d02ffd640 1 -- 172.21.15.67:0/299710506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2cfc01c070 con 0x7f2d0c164b00 2024-04-25T15:27:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.041+0000 7f2d02ffd640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2cec0744b0 0x7f2cec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.041+0000 7f2d02ffd640 1 -- 172.21.15.67:0/299710506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f2cfc094d40 con 0x7f2d0c164b00 2024-04-25T15:27:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.041+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2cec0744b0 0x7f2cec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.041+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2cec0744b0 0x7f2cec0769c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400aac0 tx=0x7f2d04008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.042+0000 7f2d17816740 1 Processor -- start 2024-04-25T15:27:58.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.042+0000 7f2d17816740 1 -- start start 2024-04-25T15:27:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d17816740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be281600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be27dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be257260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d17816740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b6be1bb740 con 0x55b6bdeb5270 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55b6be1573a0 con 0x55b6bde94590 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55b6be1b7700 con 0x55b6bde8c3b0 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be281600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be281600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39300/0 (socket says 172.21.15.67:39300) 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0a59c640 1 -- 172.21.15.67:0/1584936737 learned_addr learned my addr 172.21.15.67:0/1584936737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0a59c640 1 -- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 msgr2=0x55b6be27dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be27dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be257260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be27dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0a59c640 1 -- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 msgr2=0x55b6be257260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.043+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be257260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.044+0000 7f2d0a59c640 1 -- 172.21.15.67:0/1584936737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6be173750 con 0x55b6bdeb5270 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.044+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be281600 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400fd50 tx=0x7f2d04012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3ef99a6fefeb63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.044+0000 7f2cebfff640 1 -- 172.21.15.67:0/1584936737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04023070 con 0x55b6bdeb5270 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.044+0000 7f2cebfff640 1 -- 172.21.15.67:0/1584936737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d04019070 con 0x55b6bdeb5270 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.044+0000 7f2cebfff640 1 -- 172.21.15.67:0/1584936737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04013820 con 0x55b6bdeb5270 2024-04-25T15:27:58.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 -- 172.21.15.67:0/1584936737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 msgr2=0x55b6be281600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be281600 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400fd50 tx=0x7f2d04012000 comp rx=0 tx=0).stop 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 -- 172.21.15.67:0/1584936737 shutdown_connections 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be257260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be27dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 --2- 172.21.15.67:0/1584936737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be281600 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 -- 172.21.15.67:0/1584936737 >> 172.21.15.67:0/1584936737 conn(0x55b6bdea93c0 msgr2=0x55b6be2569f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 -- 172.21.15.67:0/1584936737 shutdown_connections 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.045+0000 7f2d17816740 1 -- 172.21.15.67:0/1584936737 wait complete. 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.046+0000 7f2d17816740 1 -- start start 2024-04-25T15:27:58.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.046+0000 7f2d17816740 1 Processor -- start 2024-04-25T15:27:58.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d17816740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be27c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be27ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d17816740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b6be142fe0 con 0x55b6bde8c3b0 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55b6be284b30 con 0x55b6bde94590 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55b6be29cdb0 con 0x55b6bdeb5270 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d17816740 10 client.0 _ll_register_callbacks cb 0x55b6be159150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39310/0 (socket says 172.21.15.67:39310) 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.047+0000 7f2d0a59c640 1 -- 172.21.15.67:0/603833438 learned_addr learned my addr 172.21.15.67:0/603833438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d0a59c640 1 -- 172.21.15.67:0/603833438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 msgr2=0x55b6be27ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be27ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be27ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d0a59c640 1 -- 172.21.15.67:0/603833438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 msgr2=0x55b6be27c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be27c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d0a59c640 1 -- 172.21.15.67:0/603833438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6bdecd2d0 con 0x55b6bde8c3b0 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27c250 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2d04008010 tx=0x7f2d04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb42bc7d27fd736 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2ce97fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2ce97fa640 1 -- 172.21.15.67:0/603833438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04023040 con 0x55b6bde8c3b0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2ce97fa640 1 -- 172.21.15.67:0/603833438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f2d04005b90 con 0x55b6bde8c3b0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d04019070 con 0x55b6bde8c3b0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2ce97fa640 1 -- 172.21.15.67:0/603833438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2d04005d90 con 0x55b6bde8c3b0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d17816740 10 client.6435 fetch_fsmap learned FSMap version 41 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) -- 0x55b6bdf976b0 con 0x55b6bde8c3b0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.048+0000 7f2ce97fa640 1 -- 172.21.15.67:0/603833438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d04019070 con 0x55b6bde8c3b0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2ce97fa640 1 -- 172.21.15.67:0/603833438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 41) ==== 3494+0+0 (secure 0 0 0) 0x7f2d0403a050 con 0x55b6bde8c3b0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d17816740 10 client.6435 fetch_fsmap finished waiting for FSMap version 41 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d17816740 10 client.6435 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d17816740 20 client.6435 populate_metadata read hostname 'smithi067' 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d17816740 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6bde73e40 0x55b6be2b9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d17816740 4 client.6435 mds_command: new command op to 6273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 --> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6be1c9350 con 0x55b6bde73e40 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6bde73e40 0x55b6be2b9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2ce97fa640 10 client.6435 ms_handle_connect on v2:172.21.15.67:6833/3903876028 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.049+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6bde73e40 0x55b6be2b9130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85d531bdc816ae19 server_cookie=15415768949e1529 in_seq=0 out_seq=0 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2ce97fa640 1 -- 172.21.15.67:0/603833438 <== mds.0 v2:172.21.15.67:6833/3903876028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b6be1c9350 con 0x55b6bde73e40 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2ce97fa640 10 client.6435 handle_command_reply: tid=0 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 client.6435 shutdown 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 msgr2=0x55b6be27c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27c250 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f2d04008010 tx=0x7f2d04004730 comp rx=0 tx=0).stop 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 shutdown_connections 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6bde73e40 0x55b6be2b9130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be27ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be27c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 --2- 172.21.15.67:0/603833438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27c250 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2ce97fa640 0 client.6435 ms_handle_reset on v2:172.21.15.67:6833/3903876028 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 >> 172.21.15.67:0/603833438 conn(0x55b6bdea93c0 msgr2=0x55b6be2569f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 shutdown_connections 2024-04-25T15:27:58.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 1 -- 172.21.15.67:0/603833438 wait complete. 2024-04-25T15:27:58.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.050+0000 7f2d17816740 20 client.6435 trim_cache size 0 max 16384 2024-04-25T15:27:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.069+0000 7f2d17816740 1 Processor -- start 2024-04-25T15:27:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.070+0000 7f2d17816740 1 -- start start 2024-04-25T15:27:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d17816740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be291a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be291fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d17816740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c1eaa30 con 0x55b6bde8c3b0 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c1e9d90 con 0x55b6bde94590 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0c1ea3e0 con 0x55b6bdeb5270 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39322/0 (socket says 172.21.15.67:39322) 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 -- 172.21.15.67:0/628525505 learned_addr learned my addr 172.21.15.67:0/628525505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:58.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be291a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be291fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 -- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 msgr2=0x55b6be291fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be291fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 -- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 msgr2=0x55b6be291a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be291a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 -- 172.21.15.67:0/628525505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6be2566e0 con 0x55b6bde8c3b0 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be291fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.071+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27d430 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400f6b0 tx=0x7f2d0402a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a544e7160ad3cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2ceb7fe640 1 -- 172.21.15.67:0/628525505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04023050 con 0x55b6bde8c3b0 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2ceb7fe640 1 -- 172.21.15.67:0/628525505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d04041070 con 0x55b6bde8c3b0 2024-04-25T15:27:58.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2ceb7fe640 1 -- 172.21.15.67:0/628525505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04005bb0 con 0x55b6bde8c3b0 2024-04-25T15:27:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 -- 172.21.15.67:0/628525505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 msgr2=0x55b6be27d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27d430 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400f6b0 tx=0x7f2d0402a040 comp rx=0 tx=0).stop 2024-04-25T15:27:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 -- 172.21.15.67:0/628525505 shutdown_connections 2024-04-25T15:27:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bdeb5270 0x55b6be291fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be291a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 --2- 172.21.15.67:0/628525505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bde8c3b0 0x55b6be27d430 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 -- 172.21.15.67:0/628525505 >> 172.21.15.67:0/628525505 conn(0x55b6bdea93c0 msgr2=0x55b6be29aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 -- 172.21.15.67:0/628525505 shutdown_connections 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.072+0000 7f2d17816740 1 -- 172.21.15.67:0/628525505 wait complete. 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.073+0000 7f2d17816740 1 -- start start 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.073+0000 7f2d17816740 1 Processor -- start 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be319050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be2964f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be296a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b6be2ab380 con 0x55b6bdeb5270 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55b6be29cb40 con 0x55b6bde94590 2024-04-25T15:27:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55b6be283ef0 con 0x55b6bde8c3b0 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 10 client.0 _ll_register_callbacks cb 0x7f2d0400fc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be296a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be296a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39338/0 (socket says 172.21.15.67:39338) 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/2437064223 learned_addr learned my addr 172.21.15.67:0/2437064223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be319050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d09d9b640 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be2964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 msgr2=0x55b6be319050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be319050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 msgr2=0x55b6be2964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be2964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 -- 172.21.15.67:0/2437064223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6be256050 con 0x55b6bdeb5270 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d0ad9d640 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be296a30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc05af00 tx=0x7f2cfc05f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a2f9ecc700edc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2ce97fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2ce97fa640 1 -- 172.21.15.67:0/2437064223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc062040 con 0x55b6bdeb5270 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2ce97fa640 1 -- 172.21.15.67:0/2437064223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f2cfc05a7f0 con 0x55b6bdeb5270 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.075+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d04041070 con 0x55b6bdeb5270 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2ce97fa640 1 -- 172.21.15.67:0/2437064223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2cfc063180 con 0x55b6bdeb5270 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 10 client.6441 fetch_fsmap learned FSMap version 41 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) -- 0x55b6be31a970 con 0x55b6bdeb5270 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2ce97fa640 1 -- 172.21.15.67:0/2437064223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d04041070 con 0x55b6bdeb5270 2024-04-25T15:27:58.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2ce97fa640 1 -- 172.21.15.67:0/2437064223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 41) ==== 3494+0+0 (secure 0 0 0) 0x7f2cfc05a9f0 con 0x55b6bdeb5270 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 10 client.6441 fetch_fsmap finished waiting for FSMap version 41 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 10 client.6441 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 20 client.6441 populate_metadata read hostname 'smithi067' 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6be2956f0 0x55b6be3198f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 4 client.6441 mds_command: new command op to 6273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 --> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] -- command(tid 0: {"prefix": "status"}) -- 0x55b6be251f60 con 0x55b6be2956f0 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6be2956f0 0x55b6be3198f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.076+0000 7f2ce97fa640 10 client.6441 ms_handle_connect on v2:172.21.15.67:6833/3903876028 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d0a59c640 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6be2956f0 0x55b6be3198f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2ecd26408215557 server_cookie=eb166a36ecd89c2d in_seq=0 out_seq=0 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2ce97fa640 1 -- 172.21.15.67:0/2437064223 <== mds.0 v2:172.21.15.67:6833/3903876028 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55b6be251f60 con 0x55b6be2956f0 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2ce97fa640 10 client.6441 handle_command_reply: tid=0 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 client.6441 shutdown 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 msgr2=0x55b6be296a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be296a30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc05af00 tx=0x7f2cfc05f4a0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 shutdown_connections 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:6833/3903876028,v1:172.21.15.67:6835/3903876028] conn(0x55b6be2956f0 0x55b6be3198f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b6bdeb5270 0x55b6be296a30 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55b6bde94590 0x55b6be2964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2ce97fa640 0 client.6441 ms_handle_reset on v2:172.21.15.67:6833/3903876028 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 --2- 172.21.15.67:0/2437064223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55b6bde8c3b0 0x55b6be319050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 >> 172.21.15.67:0/2437064223 conn(0x55b6bdea93c0 msgr2=0x55b6be27d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 shutdown_connections 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 1 -- 172.21.15.67:0/2437064223 wait complete. 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.077+0000 7f2d17816740 20 client.6441 trim_cache size 0 max 16384 2024-04-25T15:27:58.085 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b43d0acb-252a-4f7a-bf14-34d789b94b95", 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6273, 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.325886544999999, 2024-04-25T15:27:58.086 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 41, 2024-04-25T15:27:58.087 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 67, 2024-04-25T15:27:58.087 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 67, 2024-04-25T15:27:58.087 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 15.291076264000001 2024-04-25T15:27:58.087 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2cec0744b0 msgr2=0x7f2cec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2cec0744b0 0x7f2cec0769c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400aac0 tx=0x7f2d04008040 comp rx=0 tx=0).stop 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c164b00 msgr2=0x7f2d0c14ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c164b00 0x7f2d0c14ac50 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00ba10 tx=0x7f2cfc00bed0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 shutdown_connections 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2cec0744b0 0x7f2cec0769c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d0c164b00 0x7f2d0c14ac50 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c160890 0x7f2d0c1f9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 --2- 172.21.15.67:0/299710506 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c15df80 0x7f2d0c148bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 >> 172.21.15.67:0/299710506 conn(0x7f2d0c0d2bc0 msgr2=0x7f2d0c0d5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 shutdown_connections 2024-04-25T15:27:58.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.078+0000 7f2d10892640 1 -- 172.21.15.67:0/299710506 wait complete. 2024-04-25T15:27:58.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:27:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.200+0000 7f7947af9640 1 Processor -- start 2024-04-25T15:27:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.200+0000 7f7947af9640 1 -- start start 2024-04-25T15:27:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7947af9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f79400d4130 0x7f79400d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7947af9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79400b3c10 con 0x7f794006d8e0 2024-04-25T15:27:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7947af9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79400b3ef0 con 0x7f794006dcf0 2024-04-25T15:27:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7947af9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f79400b9520 con 0x7f79400d4130 2024-04-25T15:27:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7945895640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f794006d8e0 0x7f79400d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40782/0 (socket says 172.21.15.67:40782) 2024-04-25T15:27:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7945895640 1 -- 172.21.15.67:0/2770778548 learned_addr learned my addr 172.21.15.67:0/2770778548 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7945094640 1 --2- 172.21.15.67:0/2770778548 >> v2:172.21.15.176:3300/0 conn(0x7f79400d4130 0x7f79400d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7944893640 1 -- 172.21.15.67:0/2770778548 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877783449 0 0) 0x7f79400b3c10 con 0x7f794006d8e0 2024-04-25T15:27:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7944893640 1 -- 172.21.15.67:0/2770778548 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f792c003520 con 0x7f794006d8e0 2024-04-25T15:27:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7945094640 1 -- 172.21.15.67:0/2770778548 >> v1:172.21.15.176:6789/0 conn(0x7f794006dcf0 legacy=0x7f79400d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7945094640 1 -- 172.21.15.67:0/2770778548 >> v1:172.21.15.67:6789/0 conn(0x7f794006d8e0 legacy=0x7f79400d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.202+0000 7f7945094640 1 -- 172.21.15.67:0/2770778548 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79400b41d0 con 0x7f79400d4130 2024-04-25T15:27:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.203+0000 7f7945094640 1 --2- 172.21.15.67:0/2770778548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79400d4130 0x7f79400d1d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f79300097a0 tx=0x7f793002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdc359e3517b3cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.203+0000 7f7944893640 1 -- 172.21.15.67:0/2770778548 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7930037070 con 0x7f79400d4130 2024-04-25T15:27:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.203+0000 7f7944893640 1 -- 172.21.15.67:0/2770778548 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7930032440 con 0x7f79400d4130 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.203+0000 7f7944893640 1 -- 172.21.15.67:0/2770778548 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79300327f0 con 0x7f79400d4130 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.204+0000 7f7947af9640 1 -- 172.21.15.67:0/2770778548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79400d4130 msgr2=0x7f79400d1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.204+0000 7f7947af9640 1 --2- 172.21.15.67:0/2770778548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79400d4130 0x7f79400d1d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f79300097a0 tx=0x7f793002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.204+0000 7f7947af9640 1 -- 172.21.15.67:0/2770778548 shutdown_connections 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.204+0000 7f7947af9640 1 --2- 172.21.15.67:0/2770778548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79400d4130 0x7f79400d1d60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.204+0000 7f7947af9640 1 -- 172.21.15.67:0/2770778548 >> 172.21.15.67:0/2770778548 conn(0x7f794008d110 msgr2=0x7f794008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.204+0000 7f7947af9640 1 -- 172.21.15.67:0/2770778548 shutdown_connections 2024-04-25T15:27:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.204+0000 7f7947af9640 1 -- 172.21.15.67:0/2770778548 wait complete. 2024-04-25T15:27:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.206+0000 7f7947af9640 1 Processor -- start 2024-04-25T15:27:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.206+0000 7f7947af9640 1 -- start start 2024-04-25T15:27:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7947af9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794006d8e0 0x7f79400b7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7947af9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794006dcf0 0x7f79400b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7947af9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794015d4d0 0x7f79401e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7947af9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79400b38e0 con 0x7f794006d8e0 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7947af9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f79400b4820 con 0x7f794006dcf0 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7947af9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79400d29b0 con 0x7f794015d4d0 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7945895640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794006d8e0 0x7f79400b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7945895640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794006d8e0 0x7f79400b7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39354/0 (socket says 172.21.15.67:39354) 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.207+0000 7f7945895640 1 -- 172.21.15.67:0/4110609762 learned_addr learned my addr 172.21.15.67:0/4110609762 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7945094640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794006dcf0 0x7f79400b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7946096640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794015d4d0 0x7f79401e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7945895640 1 -- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794015d4d0 msgr2=0x7f79401e7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7945895640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794015d4d0 0x7f79401e7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7945895640 1 -- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794006dcf0 msgr2=0x7f79400b7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7945895640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794006dcf0 0x7f79400b7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7945895640 1 -- 172.21.15.67:0/4110609762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794009f0f0 con 0x7f794006d8e0 2024-04-25T15:27:58.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7946096640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794015d4d0 0x7f79401e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:27:58.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7945895640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794006d8e0 0x7f79400b7700 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f793c00cbe0 tx=0x7f793c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f79367fc640 1 -- 172.21.15.67:0/4110609762 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c017070 con 0x7f794006d8e0 2024-04-25T15:27:58.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f79367fc640 1 -- 172.21.15.67:0/4110609762 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f793c00aca0 con 0x7f794006d8e0 2024-04-25T15:27:58.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794009f3d0 con 0x7f794006d8e0 2024-04-25T15:27:58.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f79367fc640 1 -- 172.21.15.67:0/4110609762 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c012600 con 0x7f794006d8e0 2024-04-25T15:27:58.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.208+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794009f990 con 0x7f794006d8e0 2024-04-25T15:27:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.209+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7940050740 con 0x7f794006d8e0 2024-04-25T15:27:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.210+0000 7f79367fc640 1 -- 172.21.15.67:0/4110609762 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f793c01a070 con 0x7f794006d8e0 2024-04-25T15:27:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.210+0000 7f79367fc640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79240744b0 0x7f79240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.210+0000 7f79367fc640 1 -- 172.21.15.67:0/4110609762 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f793c094e90 con 0x7f794006d8e0 2024-04-25T15:27:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.210+0000 7f7945094640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79240744b0 0x7f79240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.211+0000 7f7945094640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79240744b0 0x7f79240769c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7930034000 tx=0x7f793002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:58.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.215+0000 7f79367fc640 1 -- 172.21.15.67:0/4110609762 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f793c05f060 con 0x7f794006d8e0 2024-04-25T15:27:58.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.437+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7940072ad0 con 0x7f794006d8e0 2024-04-25T15:27:58.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.438+0000 7f79367fc640 1 -- 172.21.15.67:0/4110609762 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) ==== 53+0+73 (secure 0 0 0) 0x7f793c063360 con 0x7f794006d8e0 2024-04-25T15:27:58.439 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79240744b0 msgr2=0x7f79240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79240744b0 0x7f79240769c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7930034000 tx=0x7f793002e000 comp rx=0 tx=0).stop 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794006d8e0 msgr2=0x7f79400b7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794006d8e0 0x7f79400b7700 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f793c00cbe0 tx=0x7f793c00a060 comp rx=0 tx=0).stop 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 shutdown_connections 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79240744b0 0x7f79240769c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794015d4d0 0x7f79401e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794006dcf0 0x7f79400b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 --2- 172.21.15.67:0/4110609762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794006d8e0 0x7f79400b7700 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 >> 172.21.15.67:0/4110609762 conn(0x7f794008d110 msgr2=0x7f794006e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 shutdown_connections 2024-04-25T15:27:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58.440+0000 7f7947af9640 1 -- 172.21.15.67:0/4110609762 wait complete. 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:27:58.452 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:27:58.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:58.453 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:27:58.468 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:58.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42620sec preferred_lft 42620sec 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7482:42ff:fe30:5ffc/64 scope link 2024-04-25T15:27:58.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::885:bff:fe04:f6cb/64 scope link 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ca:af:51:1f:d5:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8af:51ff:fe1f:d5b0/64 scope link 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:58.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:58.471 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:27:58.531 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:27:58.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:58.533 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:27:58.588 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:27:58.588 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:27:58.588 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:27:58.588 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:27:58.588 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:27:58.588 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:27:58.588 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:27:58.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:27:58.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:27:58.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:27:58.589 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:27:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:27:58.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:27:58.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:58.714 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:27:58.729 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:27:58.730 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:27:58.730 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:27:58.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:58.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:27:58.794 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:27:58.794 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:58.849 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:27:58.850 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:27:58.850 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:58.904 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T15:27:58.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:58.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:58.998 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:58.998 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:27:59.007 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:27:59.007 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:27:59.007 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:27:59.007 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:27:59.007 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:27:59.008 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T15:27:59.008 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:27:59.008 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:27:59.008 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:27:59.008 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:27:59.010 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:27:59.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:59.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:27:59.039 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:27:59.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:27:59.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:27:59.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:27:59.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:27:59.145 INFO:teuthology.orchestra.run.smithi067.stdout:client5624 2024-04-25T15:27:59.145 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:27:59.145 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:27:59.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:27:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.377+0000 7f4f8dda8640 1 Processor -- start 2024-04-25T15:27:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.377+0000 7f4f8dda8640 1 -- start start 2024-04-25T15:27:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f8dda8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4f881443c0 0x7f4f8806fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f8dda8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f880cfcf0 0x7f4f8806c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f8dda8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f8815f5f0 con 0x7f4f88070370 2024-04-25T15:27:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f8dda8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4f880708d0 con 0x7f4f881443c0 2024-04-25T15:27:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f8dda8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8815f160 con 0x7f4f880cfcf0 2024-04-25T15:27:59.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f87fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f880cfcf0 0x7f4f8806c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f877fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4f881443c0 0x7f4f8806fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f86ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f4f88070370 0x7f4f880cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:50198/0 (socket says 172.21.15.67:50198) 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f86ffd640 1 -- 172.21.15.67:0/1386076033 learned_addr learned my addr 172.21.15.67:0/1386076033 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f87fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f880cfcf0 0x7f4f8806c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:57220/0 (socket says 172.21.15.67:57220) 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f87fff640 1 -- 172.21.15.67:0/1386076033 >> v2:172.21.15.176:3300/0 conn(0x7f4f881443c0 msgr2=0x7f4f8806fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.379+0000 7f4f87fff640 1 --2- 172.21.15.67:0/1386076033 >> v2:172.21.15.176:3300/0 conn(0x7f4f881443c0 0x7f4f8806fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.380+0000 7f4f87fff640 1 -- 172.21.15.67:0/1386076033 >> v1:172.21.15.176:6789/0 conn(0x7f4f88070370 legacy=0x7f4f880cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.380+0000 7f4f867fc640 1 -- 172.21.15.67:0/1386076033 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2276416146 0 0) 0x7f4f8815f5f0 con 0x7f4f88070370 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.380+0000 7f4f87fff640 1 -- 172.21.15.67:0/1386076033 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8815e830 con 0x7f4f880cfcf0 2024-04-25T15:27:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.380+0000 7f4f877fe640 1 --2- 172.21.15.67:0/1386076033 >> v2:172.21.15.176:3300/0 conn(0x7f4f881443c0 0x7f4f8806fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:27:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.380+0000 7f4f87fff640 1 --2- 172.21.15.67:0/1386076033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f880cfcf0 0x7f4f8806c5c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009d90 tx=0x7f4f8002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0c1f785d9bde13c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f867fc640 1 -- 172.21.15.67:0/1386076033 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80007740 con 0x7f4f880cfcf0 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f867fc640 1 -- 172.21.15.67:0/1386076033 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f800078e0 con 0x7f4f880cfcf0 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f867fc640 1 -- 172.21.15.67:0/1386076033 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80007cb0 con 0x7f4f880cfcf0 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 -- 172.21.15.67:0/1386076033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f880cfcf0 msgr2=0x7f4f8806c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/1386076033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f880cfcf0 0x7f4f8806c5c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f80009d90 tx=0x7f4f8002a660 comp rx=0 tx=0).stop 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 -- 172.21.15.67:0/1386076033 shutdown_connections 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/1386076033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f880cfcf0 0x7f4f8806c5c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/1386076033 >> v2:172.21.15.176:3300/0 conn(0x7f4f881443c0 0x7f4f8806fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 -- 172.21.15.67:0/1386076033 >> 172.21.15.67:0/1386076033 conn(0x7f4f880d1ae0 msgr2=0x7f4f880d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 -- 172.21.15.67:0/1386076033 shutdown_connections 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.381+0000 7f4f8dda8640 1 -- 172.21.15.67:0/1386076033 wait complete. 2024-04-25T15:27:59.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.383+0000 7f4f8dda8640 1 Processor -- start 2024-04-25T15:27:59.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.383+0000 7f4f8dda8640 1 -- start start 2024-04-25T15:27:59.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.384+0000 7f4f8dda8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f88070370 0x7f4f880c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.384+0000 7f4f8dda8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f881443c0 0x7f4f881f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.384+0000 7f4f8dda8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f881f84f0 0x7f4f880b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f8dda8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f881447d0 con 0x7f4f881443c0 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f8dda8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8815edf0 con 0x7f4f88070370 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f8dda8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f88162b00 con 0x7f4f881f84f0 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f881443c0 0x7f4f881f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f881443c0 0x7f4f881f70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39366/0 (socket says 172.21.15.67:39366) 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 -- 172.21.15.67:0/4130263889 learned_addr learned my addr 172.21.15.67:0/4130263889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f877fe640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f88070370 0x7f4f880c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f87fff640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f881f84f0 0x7f4f880b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 -- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f881f84f0 msgr2=0x7f4f880b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:59.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f881f84f0 0x7f4f880b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 -- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f88070370 msgr2=0x7f4f880c80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f88070370 0x7f4f880c80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 -- 172.21.15.67:0/4130263889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f880ad5d0 con 0x7f4f881443c0 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f86ffd640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f881443c0 0x7f4f881f70a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c3e0 tx=0x7f4f7c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f6ffff640 1 -- 172.21.15.67:0/4130263889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0074e0 con 0x7f4f881443c0 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f6ffff640 1 -- 172.21.15.67:0/4130263889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0047e0 con 0x7f4f881443c0 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f880ad870 con 0x7f4f881443c0 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.385+0000 7f4f6ffff640 1 -- 172.21.15.67:0/4130263889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c004bd0 con 0x7f4f881443c0 2024-04-25T15:27:59.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.386+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f880ade30 con 0x7f4f881443c0 2024-04-25T15:27:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.387+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f88050740 con 0x7f4f881443c0 2024-04-25T15:27:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.387+0000 7f4f6ffff640 1 -- 172.21.15.67:0/4130263889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4f7c011070 con 0x7f4f881443c0 2024-04-25T15:27:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.388+0000 7f4f6ffff640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f740744b0 0x7f4f740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:27:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.388+0000 7f4f6ffff640 1 -- 172.21.15.67:0/4130263889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f4f7c094cb0 con 0x7f4f881443c0 2024-04-25T15:27:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.388+0000 7f4f877fe640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f740744b0 0x7f4f740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:27:59.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.388+0000 7f4f877fe640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f740744b0 0x7f4f740769c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009560 tx=0x7f4f78009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:27:59.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.392+0000 7f4f6ffff640 1 -- 172.21.15.67:0/4130263889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f7c05edf0 con 0x7f4f881443c0 2024-04-25T15:27:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.615+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4f880c8660 con 0x7f4f881443c0 2024-04-25T15:27:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.616+0000 7f4f6ffff640 1 -- 172.21.15.67:0/4130263889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) ==== 53+0+73 (secure 0 0 0) 0x7f4f7c0630f0 con 0x7f4f881443c0 2024-04-25T15:27:59.617 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:27:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f740744b0 msgr2=0x7f4f740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f740744b0 0x7f4f740769c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009560 tx=0x7f4f78009190 comp rx=0 tx=0).stop 2024-04-25T15:27:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f881443c0 msgr2=0x7f4f881f70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f881443c0 0x7f4f881f70a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c3e0 tx=0x7f4f7c00c8a0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 shutdown_connections 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4f740744b0 0x7f4f740769c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f881f84f0 0x7f4f880b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f881443c0 0x7f4f881f70a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 --2- 172.21.15.67:0/4130263889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f88070370 0x7f4f880c80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.617+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 >> 172.21.15.67:0/4130263889 conn(0x7f4f880d1ae0 msgr2=0x7f4f880d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.618+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 shutdown_connections 2024-04-25T15:27:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:27:59.618+0000 7f4f8dda8640 1 -- 172.21.15.67:0/4130263889 wait complete. 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:27:59.630 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:27:59.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:59.630 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42609sec preferred_lft 42609sec 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:27:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::248e:2bff:fe69:7f63/64 scope link 2024-04-25T15:27:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:27:59.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:59.655 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:27:59.716 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:27:59.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:59.717 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:27:59.775 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:27:59.775 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:27:59.775 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:27:59.775 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:27:59.775 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:27:59.775 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:27:59.775 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:27:59.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:27:59.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:27:59.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:27:59.776 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:27:59.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:27:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:27:59.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:27:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:27:59.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:59.904 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:27:59.919 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:27:59.920 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:27:59.920 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:27:59.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:27:59.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:27:59.983 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:27:59.984 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:00.039 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:28:00.039 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:28:00.039 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:00.094 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T15:28:00.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:00.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:00.186 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:00.186 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:00.205 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:00.205 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:00.205 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:28:00.205 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:28:00.205 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:28:00.206 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:28:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:28:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:28:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:28:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:28:00.207 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:28:00.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:00.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:00.230 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:28:00.319 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:00 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:28:00.321 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:28:00.326 INFO:teuthology.orchestra.run.smithi176.stdout:client6127 2024-04-25T15:28:00.327 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:00.327 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:00.341 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:00.342 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:00.397 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:00.397 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:00.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.577+0000 7fabdd51f640 1 Processor -- start 2024-04-25T15:28:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.577+0000 7fabdd51f640 1 -- start start 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.578+0000 7fabdd51f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabd815abd0 0x7fabd806c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.578+0000 7fabdd51f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd8164750 0x7fabd80d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.578+0000 7fabdd51f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabd806cf10 con 0x7fabd806cb00 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.578+0000 7fabdd51f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fabd8161380 con 0x7fabd815abd0 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.578+0000 7fabdd51f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fabd8164b60 con 0x7fabd8164750 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabd815abd0 0x7fabd806c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabd815abd0 0x7fabd806c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39372/0 (socket says 172.21.15.67:39372) 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd6ffd640 1 -- 172.21.15.67:0/1907488031 learned_addr learned my addr 172.21.15.67:0/1907488031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabcffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fabd806cb00 0x7fabd8164040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40786/0 (socket says 172.21.15.67:40786) 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd77fe640 1 --2- 172.21.15.67:0/1907488031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd8164750 0x7fabd80d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd6ffd640 1 -- 172.21.15.67:0/1907488031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd8164750 msgr2=0x7fabd80d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd6ffd640 1 --2- 172.21.15.67:0/1907488031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd8164750 0x7fabd80d4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd6ffd640 1 -- 172.21.15.67:0/1907488031 >> v1:172.21.15.67:6789/0 conn(0x7fabd806cb00 legacy=0x7fabd8164040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd67fc640 1 -- 172.21.15.67:0/1907488031 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 734053432 0 0) 0x7fabc0001110 con 0x7fabd806cb00 2024-04-25T15:28:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd6ffd640 1 -- 172.21.15.67:0/1907488031 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd8164db0 con 0x7fabd815abd0 2024-04-25T15:28:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.579+0000 7fabd77fe640 1 --2- 172.21.15.67:0/1907488031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd8164750 0x7fabd80d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.580+0000 7fabd6ffd640 1 --2- 172.21.15.67:0/1907488031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd815abd0 0x7fabd806c5c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fabc8009870 tx=0x7fabc802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa048387f49f58e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.580+0000 7fabd67fc640 1 -- 172.21.15.67:0/1907488031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8037070 con 0x7fabd815abd0 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.580+0000 7fabd67fc640 1 -- 172.21.15.67:0/1907488031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc8032440 con 0x7fabd815abd0 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.580+0000 7fabd67fc640 1 -- 172.21.15.67:0/1907488031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8032810 con 0x7fabd815abd0 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 -- 172.21.15.67:0/1907488031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd815abd0 msgr2=0x7fabd806c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 --2- 172.21.15.67:0/1907488031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd815abd0 0x7fabd806c5c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fabc8009870 tx=0x7fabc802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 -- 172.21.15.67:0/1907488031 shutdown_connections 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 --2- 172.21.15.67:0/1907488031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd8164750 0x7fabd80d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 --2- 172.21.15.67:0/1907488031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd815abd0 0x7fabd806c5c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 -- 172.21.15.67:0/1907488031 >> 172.21.15.67:0/1907488031 conn(0x7fabd80d5df0 msgr2=0x7fabd80d6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 -- 172.21.15.67:0/1907488031 shutdown_connections 2024-04-25T15:28:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.581+0000 7fabdd51f640 1 -- 172.21.15.67:0/1907488031 wait complete. 2024-04-25T15:28:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.583+0000 7fabdd51f640 1 Processor -- start 2024-04-25T15:28:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.583+0000 7fabdd51f640 1 -- start start 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabdd51f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabd806cb00 0x7fabd814ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabdd51f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd815abd0 0x7fabd80b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabdd51f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd8164750 0x7fabd80b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabdd51f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fabd81609e0 con 0x7fabd8164750 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabdd51f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fabd8161010 con 0x7fabd806cb00 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabdd51f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fabd8161810 con 0x7fabd815abd0 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd8164750 0x7fabd80b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd8164750 0x7fabd80b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39388/0 (socket says 172.21.15.67:39388) 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 -- 172.21.15.67:0/2307141521 learned_addr learned my addr 172.21.15.67:0/2307141521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd6ffd640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabd806cb00 0x7fabd814ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabcffff640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd815abd0 0x7fabd80b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 -- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd815abd0 msgr2=0x7fabd80b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd815abd0 0x7fabd80b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 -- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabd806cb00 msgr2=0x7fabd814ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabd806cb00 0x7fabd814ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.584+0000 7fabd77fe640 1 -- 172.21.15.67:0/2307141521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd80b8140 con 0x7fabd8164750 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.585+0000 7fabcffff640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd815abd0 0x7fabd80b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.585+0000 7fabd77fe640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd8164750 0x7fabd80b5850 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fabd0007e20 tx=0x7fabd000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.585+0000 7fabcf7fe640 1 -- 172.21.15.67:0/2307141521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd001a070 con 0x7fabd8164750 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.585+0000 7fabcf7fe640 1 -- 172.21.15.67:0/2307141521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd0003b90 con 0x7fabd8164750 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.585+0000 7fabcf7fe640 1 -- 172.21.15.67:0/2307141521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd000de80 con 0x7fabd8164750 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.585+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabd80b7530 con 0x7fabd8164750 2024-04-25T15:28:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.585+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabd80b7af0 con 0x7fabd8164750 2024-04-25T15:28:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.586+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabd8050740 con 0x7fabd8164750 2024-04-25T15:28:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.587+0000 7fabcf7fe640 1 -- 172.21.15.67:0/2307141521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fabd001d070 con 0x7fabd8164750 2024-04-25T15:28:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.587+0000 7fabcf7fe640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fabb80744e0 0x7fabb80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.587+0000 7fabcf7fe640 1 -- 172.21.15.67:0/2307141521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fabd0095330 con 0x7fabd8164750 2024-04-25T15:28:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.587+0000 7fabd6ffd640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fabb80744e0 0x7fabb80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.588+0000 7fabd6ffd640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fabb80744e0 0x7fabb80769f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fabc8034000 tx=0x7fabc802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:00.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.592+0000 7fabcf7fe640 1 -- 172.21.15.67:0/2307141521 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fabd0010070 con 0x7fabd8164750 2024-04-25T15:28:00.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.777+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabd8150d10 con 0x7fabd8164750 2024-04-25T15:28:00.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.778+0000 7fabcf7fe640 1 -- 172.21.15.67:0/2307141521 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16632 (secure 0 0 0) 0x7fabd005fbe0 con 0x7fabd8164750 2024-04-25T15:28:00.779 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:00.779 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:48.764450+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fabb80744e0 msgr2=0x7fabb80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fabb80744e0 0x7fabb80769f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fabc8034000 tx=0x7fabc802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd8164750 msgr2=0x7fabd80b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd8164750 0x7fabd80b5850 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fabd0007e20 tx=0x7fabd000d4a0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 shutdown_connections 2024-04-25T15:28:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fabb80744e0 0x7fabb80769f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd8164750 0x7fabd80b5850 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabd815abd0 0x7fabd80b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 --2- 172.21.15.67:0/2307141521 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabd806cb00 0x7fabd814ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 >> 172.21.15.67:0/2307141521 conn(0x7fabd80d5df0 msgr2=0x7fabd81635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 shutdown_connections 2024-04-25T15:28:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.780+0000 7fabdd51f640 1 -- 172.21.15.67:0/2307141521 wait complete. 2024-04-25T15:28:00.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:27:45.453204+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:27:46.073423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:00.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:00.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.907+0000 7f0dc737d640 1 Processor -- start 2024-04-25T15:28:00.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.907+0000 7f0dc737d640 1 -- start start 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc737d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0dc0161710 0x7f0dc015e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc737d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 0x7f0dc006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc737d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc01609b0 con 0x7f0dc0163fa0 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc737d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0dc006cf10 con 0x7f0dc0161710 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc737d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc0164cd0 con 0x7f0dc006c5c0 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc591a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0dc0163fa0 0x7f0dc01643b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40802/0 (socket says 172.21.15.67:40802) 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc591a640 1 -- 172.21.15.67:0/3050066342 learned_addr learned my addr 172.21.15.67:0/3050066342 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2263708216 0 0) 0x7f0db8001110 con 0x7f0dc0163fa0 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc5119640 1 --2- 172.21.15.67:0/3050066342 >> v2:172.21.15.176:3300/0 conn(0x7f0dc0161710 0x7f0dc015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0dc4918640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 0x7f0dc006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dac003520 con 0x7f0dc0163fa0 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3100149081 0 0) 0x7f0dac003520 con 0x7f0dc0163fa0 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 msgr2=0x7f0dc006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 0x7f0dc006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 >> v2:172.21.15.176:3300/0 conn(0x7f0dc0161710 msgr2=0x7f0dc015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 --2- 172.21.15.67:0/3050066342 >> v2:172.21.15.176:3300/0 conn(0x7f0dc0161710 0x7f0dc015e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc0160c30 con 0x7f0dc0163fa0 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.909+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f0db8001840 con 0x7f0dc0163fa0 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 >> v1:172.21.15.67:6789/0 conn(0x7f0dc0163fa0 legacy=0x7f0dc01643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0db7fff640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac0039b0 0x7f0dac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0db7fff640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 0x7f0dac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0db7fff640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac004d40 0x7f0dac005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc006cf10 con 0x7f0dac0039b0 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc0164cd0 con 0x7f0dac004d40 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0dac00ba40 con 0x7f0dc006c5c0 2024-04-25T15:28:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc5119640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac0039b0 0x7f0dac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc4918640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 0x7f0dac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc591a640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac004d40 0x7f0dac005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc5119640 1 -- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 msgr2=0x7f0dac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc5119640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc006c5c0 0x7f0dac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc5119640 1 -- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac004d40 msgr2=0x7f0dac005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc5119640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac004d40 0x7f0dac005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.910+0000 7f0dc5119640 1 -- 172.21.15.67:0/3050066342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dac003520 con 0x7f0dac0039b0 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.911+0000 7f0dc5119640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac0039b0 0x7f0dac003e20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009f90 tx=0x7f0dac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de9f9fc608ed8d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.911+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc009c10 con 0x7f0dac0039b0 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.911+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dbc036070 con 0x7f0dac0039b0 2024-04-25T15:28:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.911+0000 7f0db7fff640 1 -- 172.21.15.67:0/3050066342 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc031840 con 0x7f0dac0039b0 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.912+0000 7f0dc737d640 1 -- 172.21.15.67:0/3050066342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac0039b0 msgr2=0x7f0dac003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.912+0000 7f0dc737d640 1 --2- 172.21.15.67:0/3050066342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac0039b0 0x7f0dac003e20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009f90 tx=0x7f0dac005830 comp rx=0 tx=0).stop 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.912+0000 7f0dc591a640 1 -- 172.21.15.67:0/3050066342 reap_dead start 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.912+0000 7f0dc737d640 1 -- 172.21.15.67:0/3050066342 shutdown_connections 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.912+0000 7f0dc737d640 1 -- 172.21.15.67:0/3050066342 >> 172.21.15.67:0/3050066342 conn(0x7f0dc00d5d90 msgr2=0x7f0dc00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.912+0000 7f0dc737d640 1 -- 172.21.15.67:0/3050066342 shutdown_connections 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.912+0000 7f0dc737d640 1 -- 172.21.15.67:0/3050066342 wait complete. 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.914+0000 7f0dc737d640 1 Processor -- start 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.914+0000 7f0dc737d640 1 -- start start 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc737d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac0039b0 0x7f0dc0152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc737d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc0153370 0x7f0dc01fa160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc737d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc01fa6a0 0x7f0dc00cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc737d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc006c9d0 con 0x7f0dc0153370 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc737d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc01611f0 con 0x7f0dac0039b0 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc737d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc006cb50 con 0x7f0dc01fa6a0 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc0153370 0x7f0dc01fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc0153370 0x7f0dc01fa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39412/0 (socket says 172.21.15.67:39412) 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 -- 172.21.15.67:0/4237954297 learned_addr learned my addr 172.21.15.67:0/4237954297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc5119640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac0039b0 0x7f0dc0152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc591a640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc01fa6a0 0x7f0dc00cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 -- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc01fa6a0 msgr2=0x7f0dc00cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc01fa6a0 0x7f0dc00cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 -- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac0039b0 msgr2=0x7f0dc0152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac0039b0 0x7f0dc0152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.915+0000 7f0dc4918640 1 -- 172.21.15.67:0/4237954297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc0155ec0 con 0x7f0dc0153370 2024-04-25T15:28:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.916+0000 7f0dc4918640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc0153370 0x7f0dc01fa160 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0db00119a0 tx=0x7f0db0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.916+0000 7f0db5ffb640 1 -- 172.21.15.67:0/4237954297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db000faf0 con 0x7f0dc0153370 2024-04-25T15:28:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.916+0000 7f0db5ffb640 1 -- 172.21.15.67:0/4237954297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db000fc90 con 0x7f0dc0153370 2024-04-25T15:28:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.916+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc00cf8e0 con 0x7f0dc0153370 2024-04-25T15:28:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.916+0000 7f0db5ffb640 1 -- 172.21.15.67:0/4237954297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0014440 con 0x7f0dc0153370 2024-04-25T15:28:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.916+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc00ce060 con 0x7f0dc0153370 2024-04-25T15:28:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.917+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc0153850 con 0x7f0dc0153370 2024-04-25T15:28:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.918+0000 7f0db5ffb640 1 -- 172.21.15.67:0/4237954297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0db0012040 con 0x7f0dc0153370 2024-04-25T15:28:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.918+0000 7f0db5ffb640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0da40744e0 0x7f0da40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.918+0000 7f0db5ffb640 1 -- 172.21.15.67:0/4237954297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f0db0095730 con 0x7f0dc0153370 2024-04-25T15:28:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.919+0000 7f0dc5119640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0da40744e0 0x7f0da40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.919+0000 7f0dc5119640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0da40744e0 0x7f0da40769f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc035000 tx=0x7f0dbc02f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:00.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:00.923+0000 7f0db5ffb640 1 -- 172.21.15.67:0/4237954297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0db0019030 con 0x7f0dc0153370 2024-04-25T15:28:01.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.110+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0dc0050740 con 0x7f0dc0153370 2024-04-25T15:28:01.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.110+0000 7f0db5ffb640 1 -- 172.21.15.67:0/4237954297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7f0db005f870 con 0x7f0dc0153370 2024-04-25T15:28:01.111 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0da40744e0 msgr2=0x7f0da40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0da40744e0 0x7f0da40769f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc035000 tx=0x7f0dbc02f000 comp rx=0 tx=0).stop 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc0153370 msgr2=0x7f0dc01fa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc0153370 0x7f0dc01fa160 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0db00119a0 tx=0x7f0db0011e60 comp rx=0 tx=0).stop 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 shutdown_connections 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0da40744e0 0x7f0da40769f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dc01fa6a0 0x7f0dc00cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc0153370 0x7f0dc01fa160 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 --2- 172.21.15.67:0/4237954297 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dac0039b0 0x7f0dc0152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 >> 172.21.15.67:0/4237954297 conn(0x7f0dc00d5d90 msgr2=0x7f0dc015fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 shutdown_connections 2024-04-25T15:28:01.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.113+0000 7f0dc737d640 1 -- 172.21.15.67:0/4237954297 wait complete. 2024-04-25T15:28:01.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:28:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.238+0000 7f78a358f640 1 Processor -- start 2024-04-25T15:28:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.239+0000 7f78a358f640 1 -- start start 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a358f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f789c0b9270 0x7f789c0d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a358f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789c0cf810 con 0x7f789c162a10 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a358f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789c0b0690 con 0x7f789c161ef0 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a358f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f789c162e20 con 0x7f789c0b9270 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a132b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f789c0b9270 0x7f789c0d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a1b2c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f789c162a10 0x7f789c0b0e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40808/0 (socket says 172.21.15.67:40808) 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a1b2c640 1 -- 172.21.15.67:0/2631855843 learned_addr learned my addr 172.21.15.67:0/2631855843 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a132b640 1 -- 172.21.15.67:0/2631855843 >> v1:172.21.15.176:6789/0 conn(0x7f789c161ef0 legacy=0x7f789c162300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a132b640 1 -- 172.21.15.67:0/2631855843 >> v1:172.21.15.67:6789/0 conn(0x7f789c162a10 legacy=0x7f789c0b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f78a132b640 1 -- 172.21.15.67:0/2631855843 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789c0cfab0 con 0x7f789c0b9270 2024-04-25T15:28:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.240+0000 7f789bfff640 1 -- 172.21.15.67:0/2631855843 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221993241 0 0) 0x7f789c0cf810 con 0x7f789c162a10 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.241+0000 7f78a132b640 1 --2- 172.21.15.67:0/2631855843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c0b9270 0x7f789c0d15e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f78940097a0 tx=0x7f789402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ba137febc78cab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.241+0000 7f789bfff640 1 -- 172.21.15.67:0/2631855843 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894037070 con 0x7f789c0b9270 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.241+0000 7f789bfff640 1 -- 172.21.15.67:0/2631855843 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f789402adc0 con 0x7f789c0b9270 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.241+0000 7f789bfff640 1 -- 172.21.15.67:0/2631855843 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7894032660 con 0x7f789c0b9270 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.242+0000 7f78a358f640 1 -- 172.21.15.67:0/2631855843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c0b9270 msgr2=0x7f789c0d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.242+0000 7f78a358f640 1 --2- 172.21.15.67:0/2631855843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c0b9270 0x7f789c0d15e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f78940097a0 tx=0x7f789402a0b0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.242+0000 7f78a358f640 1 -- 172.21.15.67:0/2631855843 shutdown_connections 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.242+0000 7f78a358f640 1 --2- 172.21.15.67:0/2631855843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c0b9270 0x7f789c0d15e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.242+0000 7f78a358f640 1 -- 172.21.15.67:0/2631855843 >> 172.21.15.67:0/2631855843 conn(0x7f789c0d3aa0 msgr2=0x7f789c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:01.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.242+0000 7f78a358f640 1 -- 172.21.15.67:0/2631855843 shutdown_connections 2024-04-25T15:28:01.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.242+0000 7f78a358f640 1 -- 172.21.15.67:0/2631855843 wait complete. 2024-04-25T15:28:01.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.244+0000 7f78a358f640 1 Processor -- start 2024-04-25T15:28:01.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.244+0000 7f78a358f640 1 -- start start 2024-04-25T15:28:01.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a358f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f789c0b9270 0x7f789c0cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a358f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c161ef0 0x7f789c15b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a358f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f789c162a10 0x7f789c1e9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a358f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f789c0b1700 con 0x7f789c161ef0 2024-04-25T15:28:01.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a358f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f789c0b1580 con 0x7f789c0b9270 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a358f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f789c0b1880 con 0x7f789c162a10 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a0b2a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c161ef0 0x7f789c15b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a0b2a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c161ef0 0x7f789c15b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39434/0 (socket says 172.21.15.67:39434) 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a0b2a640 1 -- 172.21.15.67:0/2226989944 learned_addr learned my addr 172.21.15.67:0/2226989944 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a132b640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f789c0b9270 0x7f789c0cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.245+0000 7f78a1b2c640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f789c162a10 0x7f789c1e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a0b2a640 1 -- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f789c162a10 msgr2=0x7f789c1e9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a0b2a640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f789c162a10 0x7f789c1e9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a0b2a640 1 -- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f789c0b9270 msgr2=0x7f789c0cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a0b2a640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f789c0b9270 0x7f789c0cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a0b2a640 1 -- 172.21.15.67:0/2226989944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789c0caf70 con 0x7f789c161ef0 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a0b2a640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c161ef0 0x7f789c15b510 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f789000b3c0 tx=0x7f789000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f7899ffb640 1 -- 172.21.15.67:0/2226989944 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890015020 con 0x7f789c161ef0 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f7899ffb640 1 -- 172.21.15.67:0/2226989944 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7890011830 con 0x7f789c161ef0 2024-04-25T15:28:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f7899ffb640 1 -- 172.21.15.67:0/2226989944 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890011c60 con 0x7f789c161ef0 2024-04-25T15:28:01.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f789c0cb250 con 0x7f789c161ef0 2024-04-25T15:28:01.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.246+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f789c0cb810 con 0x7f789c161ef0 2024-04-25T15:28:01.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.247+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f789c050740 con 0x7f789c161ef0 2024-04-25T15:28:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.248+0000 7f7899ffb640 1 -- 172.21.15.67:0/2226989944 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7890018020 con 0x7f789c161ef0 2024-04-25T15:28:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.248+0000 7f7899ffb640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f78880744b0 0x7f78880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.248+0000 7f7899ffb640 1 -- 172.21.15.67:0/2226989944 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f7890094db0 con 0x7f789c161ef0 2024-04-25T15:28:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.248+0000 7f78a132b640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f78880744b0 0x7f78880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.249+0000 7f78a132b640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f78880744b0 0x7f78880769c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7894034000 tx=0x7f789402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.253+0000 7f7899ffb640 1 -- 172.21.15.67:0/2226989944 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f789001f030 con 0x7f789c161ef0 2024-04-25T15:28:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.441+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f789c0d4770 con 0x7f789c161ef0 2024-04-25T15:28:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.441+0000 7f7899ffb640 1 -- 172.21.15.67:0/2226989944 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f789005eef0 con 0x7f789c161ef0 2024-04-25T15:28:01.442 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:28:01.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f78880744b0 msgr2=0x7f78880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f78880744b0 0x7f78880769c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7894034000 tx=0x7f789402e000 comp rx=0 tx=0).stop 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c161ef0 msgr2=0x7f789c15b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c161ef0 0x7f789c15b510 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f789000b3c0 tx=0x7f789000b880 comp rx=0 tx=0).stop 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 shutdown_connections 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f78880744b0 0x7f78880769c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f789c162a10 0x7f789c1e9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f789c161ef0 0x7f789c15b510 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 --2- 172.21.15.67:0/2226989944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f789c0b9270 0x7f789c0cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.443+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 >> 172.21.15.67:0/2226989944 conn(0x7f789c0d3aa0 msgr2=0x7f789c072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.444+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 shutdown_connections 2024-04-25T15:28:01.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.444+0000 7f78a358f640 1 -- 172.21.15.67:0/2226989944 wait complete. 2024-04-25T15:28:01.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:28:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.569+0000 7f851230e640 1 Processor -- start 2024-04-25T15:28:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.570+0000 7f851230e640 1 -- start start 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f851230e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f850c0b7ef0 0x7f850c0b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f851230e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f850c0b3f30 con 0x7f850c06ecd0 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f851230e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f850c0b4210 con 0x7f850c0d0190 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f851230e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f850c0b23e0 con 0x7f850c0b7ef0 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f85108ab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f850c0b7ef0 0x7f850c0b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f850b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f850c06ecd0 0x7f850c0b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40814/0 (socket says 172.21.15.67:40814) 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f850b7fe640 1 -- 172.21.15.67:0/1117266568 learned_addr learned my addr 172.21.15.67:0/1117266568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:01.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f85108ab640 1 -- 172.21.15.67:0/1117266568 >> v1:172.21.15.176:6789/0 conn(0x7f850c0d0190 legacy=0x7f850c0d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f85108ab640 1 -- 172.21.15.67:0/1117266568 >> v1:172.21.15.67:6789/0 conn(0x7f850c06ecd0 legacy=0x7f850c0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.571+0000 7f85108ab640 1 -- 172.21.15.67:0/1117266568 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c0b2630 con 0x7f850c0b7ef0 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.572+0000 7f85108ab640 1 --2- 172.21.15.67:0/1117266568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b7ef0 0x7f850c0b2e30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8504009d90 tx=0x7f850402a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedd189a2f83b1aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.572+0000 7f850affd640 1 -- 172.21.15.67:0/1117266568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504007740 con 0x7f850c0b7ef0 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.572+0000 7f850affd640 1 -- 172.21.15.67:0/1117266568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85040078e0 con 0x7f850c0b7ef0 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.572+0000 7f850affd640 1 -- 172.21.15.67:0/1117266568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504007cb0 con 0x7f850c0b7ef0 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.573+0000 7f851230e640 1 -- 172.21.15.67:0/1117266568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b7ef0 msgr2=0x7f850c0b2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.573+0000 7f851230e640 1 --2- 172.21.15.67:0/1117266568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b7ef0 0x7f850c0b2e30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8504009d90 tx=0x7f850402a580 comp rx=0 tx=0).stop 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.573+0000 7f851230e640 1 -- 172.21.15.67:0/1117266568 shutdown_connections 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.573+0000 7f851230e640 1 --2- 172.21.15.67:0/1117266568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c0b7ef0 0x7f850c0b2e30 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.573+0000 7f851230e640 1 -- 172.21.15.67:0/1117266568 >> 172.21.15.67:0/1117266568 conn(0x7f850c09b7d0 msgr2=0x7f850c09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:01.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.573+0000 7f851230e640 1 -- 172.21.15.67:0/1117266568 shutdown_connections 2024-04-25T15:28:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.573+0000 7f851230e640 1 -- 172.21.15.67:0/1117266568 wait complete. 2024-04-25T15:28:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.575+0000 7f851230e640 1 Processor -- start 2024-04-25T15:28:01.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.575+0000 7f851230e640 1 -- start start 2024-04-25T15:28:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f851230e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f850c06ecd0 0x7f850c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f851230e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f850c0d0190 0x7f850c163200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f851230e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c06d610 0x7f850c06da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f851230e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f850c0b3cd0 con 0x7f850c06d610 2024-04-25T15:28:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f851230e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f850c0b3b50 con 0x7f850c0d0190 2024-04-25T15:28:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f851230e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f850c160640 con 0x7f850c06ecd0 2024-04-25T15:28:01.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f85108ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c06d610 0x7f850c06da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f85108ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c06d610 0x7f850c06da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39440/0 (socket says 172.21.15.67:39440) 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f85108ab640 1 -- 172.21.15.67:0/1722586287 learned_addr learned my addr 172.21.15.67:0/1722586287 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f850b7fe640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f850c0d0190 0x7f850c163200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.576+0000 7f850bfff640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f850c06ecd0 0x7f850c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f85108ab640 1 -- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f850c06ecd0 msgr2=0x7f850c157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f85108ab640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f850c06ecd0 0x7f850c157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f85108ab640 1 -- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f850c0d0190 msgr2=0x7f850c163200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f85108ab640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f850c0d0190 0x7f850c163200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f85108ab640 1 -- 172.21.15.67:0/1722586287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c06df60 con 0x7f850c06d610 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f850b7fe640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f850c0d0190 0x7f850c163200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f850bfff640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f850c06ecd0 0x7f850c157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f85108ab640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c06d610 0x7f850c06da20 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8504009eb0 tx=0x7f8504004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f8508ff9640 1 -- 172.21.15.67:0/1722586287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504040070 con 0x7f850c06d610 2024-04-25T15:28:01.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f8508ff9640 1 -- 172.21.15.67:0/1722586287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8504031c90 con 0x7f850c06d610 2024-04-25T15:28:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850c0cac90 con 0x7f850c06d610 2024-04-25T15:28:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f8508ff9640 1 -- 172.21.15.67:0/1722586287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850403c640 con 0x7f850c06d610 2024-04-25T15:28:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.577+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850c0cb250 con 0x7f850c06d610 2024-04-25T15:28:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.578+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850c050740 con 0x7f850c06d610 2024-04-25T15:28:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.579+0000 7f8508ff9640 1 -- 172.21.15.67:0/1722586287 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8504039040 con 0x7f850c06d610 2024-04-25T15:28:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.580+0000 7f8508ff9640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84f80744e0 0x7f84f80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.580+0000 7f8508ff9640 1 -- 172.21.15.67:0/1722586287 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f85040b6470 con 0x7f850c06d610 2024-04-25T15:28:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.580+0000 7f850bfff640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84f80744e0 0x7f84f80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.581+0000 7f850bfff640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84f80744e0 0x7f84f80769f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f84fc006fd0 tx=0x7f84fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.584+0000 7f8508ff9640 1 -- 172.21.15.67:0/1722586287 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f850c06df60 con 0x7f850c06d610 2024-04-25T15:28:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.769+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f850c0cf560 con 0x7f850c06d610 2024-04-25T15:28:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.770+0000 7f8508ff9640 1 -- 172.21.15.67:0/1722586287 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f8504080c40 con 0x7f850c06d610 2024-04-25T15:28:01.771 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84f80744e0 msgr2=0x7f84f80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84f80744e0 0x7f84f80769f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f84fc006fd0 tx=0x7f84fc008040 comp rx=0 tx=0).stop 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c06d610 msgr2=0x7f850c06da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c06d610 0x7f850c06da20 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8504009eb0 tx=0x7f8504004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 shutdown_connections 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f84f80744e0 0x7f84f80769f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f850c06d610 0x7f850c06da20 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f850c0d0190 0x7f850c163200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 --2- 172.21.15.67:0/1722586287 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f850c06ecd0 0x7f850c157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 >> 172.21.15.67:0/1722586287 conn(0x7f850c09b7d0 msgr2=0x7f850c070310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:01.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 shutdown_connections 2024-04-25T15:28:01.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.772+0000 7f851230e640 1 -- 172.21.15.67:0/1722586287 wait complete. 2024-04-25T15:28:01.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:28:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.895+0000 7f050916c640 1 Processor -- start 2024-04-25T15:28:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.895+0000 7f050916c640 1 -- start start 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050916c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504164b00 0x7f05040b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050916c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f050415df80 0x7f05040b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050916c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05041615f0 con 0x7f0504160890 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050916c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f05041613b0 con 0x7f050415df80 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050916c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0504164670 con 0x7f0504164b00 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f050415df80 0x7f05040b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f050415df80 0x7f05040b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39442/0 (socket says 172.21.15.67:39442) 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.896+0000 7f050259c640 1 -- 172.21.15.67:0/1445340920 learned_addr learned my addr 172.21.15.67:0/1445340920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.897+0000 7f0502d9d640 1 --2- 172.21.15.67:0/1445340920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504164b00 0x7f05040b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.897+0000 7f050259c640 1 -- 172.21.15.67:0/1445340920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504164b00 msgr2=0x7f05040b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.897+0000 7f050259c640 1 --2- 172.21.15.67:0/1445340920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504164b00 0x7f05040b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.897+0000 7f050259c640 1 -- 172.21.15.67:0/1445340920 >> v1:172.21.15.176:6789/0 conn(0x7f0504160890 legacy=0x7f0504160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.897+0000 7f050259c640 1 -- 172.21.15.67:0/1445340920 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0504164020 con 0x7f050415df80 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050259c640 1 --2- 172.21.15.67:0/1445340920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f05040b80c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009870 tx=0x7f04ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc79626a98760273 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f0501d9b640 1 -- 172.21.15.67:0/1445340920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec037070 con 0x7f050415df80 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f0501d9b640 1 -- 172.21.15.67:0/1445340920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04ec032440 con 0x7f050415df80 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f0501d9b640 1 -- 172.21.15.67:0/1445340920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec032810 con 0x7f050415df80 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 -- 172.21.15.67:0/1445340920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 msgr2=0x7f05040b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 --2- 172.21.15.67:0/1445340920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f05040b80c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009870 tx=0x7f04ec02a270 comp rx=0 tx=0).stop 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 -- 172.21.15.67:0/1445340920 shutdown_connections 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 --2- 172.21.15.67:0/1445340920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f05040b80c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 --2- 172.21.15.67:0/1445340920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504164b00 0x7f05040b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 -- 172.21.15.67:0/1445340920 >> 172.21.15.67:0/1445340920 conn(0x7f05040d2bc0 msgr2=0x7f05040d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 -- 172.21.15.67:0/1445340920 shutdown_connections 2024-04-25T15:28:01.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.898+0000 7f050916c640 1 -- 172.21.15.67:0/1445340920 wait complete. 2024-04-25T15:28:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.901+0000 7f050916c640 1 Processor -- start 2024-04-25T15:28:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.901+0000 7f050916c640 1 -- start start 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050916c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f0504157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050916c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504160890 0x7f0504158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050916c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0504164b00 0x7f05041586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050916c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f05040b4a00 con 0x7f050415df80 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050916c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f05040b4880 con 0x7f0504164b00 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050916c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f05040b4b80 con 0x7f0504160890 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f0504157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f0504157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39452/0 (socket says 172.21.15.67:39452) 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 -- 172.21.15.67:0/4256768980 learned_addr learned my addr 172.21.15.67:0/4256768980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:01.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050259c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504160890 0x7f0504158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f050359e640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0504164b00 0x7f05041586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 -- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504160890 msgr2=0x7f0504158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504160890 0x7f0504158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 -- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0504164b00 msgr2=0x7f05041586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0504164b00 0x7f05041586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.902+0000 7f0502d9d640 1 -- 172.21.15.67:0/4256768980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050415a370 con 0x7f050415df80 2024-04-25T15:28:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.903+0000 7f0502d9d640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f0504157c20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00e870 tx=0x7f04fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.905+0000 7f04fb7fe640 1 -- 172.21.15.67:0/4256768980 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc017070 con 0x7f050415df80 2024-04-25T15:28:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.905+0000 7f04fb7fe640 1 -- 172.21.15.67:0/4256768980 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04fc00c940 con 0x7f050415df80 2024-04-25T15:28:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.905+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050415a650 con 0x7f050415df80 2024-04-25T15:28:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.905+0000 7f04fb7fe640 1 -- 172.21.15.67:0/4256768980 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc00cd30 con 0x7f050415df80 2024-04-25T15:28:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.905+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f050415ac10 con 0x7f050415df80 2024-04-25T15:28:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.906+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0504050740 con 0x7f050415df80 2024-04-25T15:28:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.907+0000 7f04fb7fe640 1 -- 172.21.15.67:0/4256768980 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f04fc01a070 con 0x7f050415df80 2024-04-25T15:28:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.907+0000 7f04fb7fe640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04e40744b0 0x7f04e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.907+0000 7f04fb7fe640 1 -- 172.21.15.67:0/4256768980 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f04fc0954e0 con 0x7f050415df80 2024-04-25T15:28:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.908+0000 7f050259c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04e40744b0 0x7f04e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.908+0000 7f050259c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04e40744b0 0x7f04e40769c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f04ec034000 tx=0x7f04ec02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:01.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:01.912+0000 7f04fb7fe640 1 -- 172.21.15.67:0/4256768980 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04fc05f580 con 0x7f050415df80 2024-04-25T15:28:02.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.107+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f05040d4170 con 0x7f050415df80 2024-04-25T15:28:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.108+0000 7f04fb7fe640 1 -- 172.21.15.67:0/4256768980 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) ==== 80+0+0 (secure 0 0 0) 0x7f04fc063880 con 0x7f050415df80 2024-04-25T15:28:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.109+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04e40744b0 msgr2=0x7f04e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.109+0000 7f050916c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04e40744b0 0x7f04e40769c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f04ec034000 tx=0x7f04ec02e000 comp rx=0 tx=0).stop 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.109+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 msgr2=0x7f0504157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.109+0000 7f050916c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f0504157c20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00e870 tx=0x7f04fc00ed30 comp rx=0 tx=0).stop 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 shutdown_connections 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f04e40744b0 0x7f04e40769c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0504164b00 0x7f05041586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0504160890 0x7f0504158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 --2- 172.21.15.67:0/4256768980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050415df80 0x7f0504157c20 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 >> 172.21.15.67:0/4256768980 conn(0x7f05040d2bc0 msgr2=0x7f05040d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 shutdown_connections 2024-04-25T15:28:02.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.110+0000 7f050916c640 1 -- 172.21.15.67:0/4256768980 wait complete. 2024-04-25T15:28:02.123 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:28:02.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:02.155 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:28:02.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.156 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:28:02.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.214 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-25T15:28:02.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.233 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 134217728 ./subdir) 2024-04-25T15:28:02.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:28:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.441+0000 7f802fb72640 1 Processor -- start 2024-04-25T15:28:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.442+0000 7f802fb72640 1 -- start start 2024-04-25T15:28:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802fb72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028164b00 0x7f80280b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802fb72640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f802815df80 0x7f80280b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802fb72640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80281615f0 con 0x7f8028160890 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802fb72640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f80281613b0 con 0x7f802815df80 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802fb72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8028164670 con 0x7f8028164b00 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802d10d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f802815df80 0x7f80280b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802d10d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f802815df80 0x7f80280b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39458/0 (socket says 172.21.15.67:39458) 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802e10f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8028160890 0x7f8028160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40826/0 (socket says 172.21.15.67:40826) 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.443+0000 7f802d10d640 1 -- 172.21.15.67:0/2225867547 learned_addr learned my addr 172.21.15.67:0/2225867547 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802d10d640 1 -- 172.21.15.67:0/2225867547 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028164b00 msgr2=0x7f80280b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802d10d640 1 --2- 172.21.15.67:0/2225867547 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028164b00 0x7f80280b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802d10d640 1 -- 172.21.15.67:0/2225867547 >> v1:172.21.15.67:6789/0 conn(0x7f8028160890 legacy=0x7f8028160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802d10d640 1 -- 172.21.15.67:0/2225867547 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8028164020 con 0x7f802815df80 2024-04-25T15:28:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802d10d640 1 --2- 172.21.15.67:0/2225867547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802815df80 0x7f80280b80c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f8018009870 tx=0x7f801802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c0c5d7f0f7c1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802c90c640 1 -- 172.21.15.67:0/2225867547 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018037070 con 0x7f802815df80 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802c90c640 1 -- 172.21.15.67:0/2225867547 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8018032440 con 0x7f802815df80 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.444+0000 7f802c90c640 1 -- 172.21.15.67:0/2225867547 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018032810 con 0x7f802815df80 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 -- 172.21.15.67:0/2225867547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802815df80 msgr2=0x7f80280b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 --2- 172.21.15.67:0/2225867547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802815df80 0x7f80280b80c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f8018009870 tx=0x7f801802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 -- 172.21.15.67:0/2225867547 shutdown_connections 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 --2- 172.21.15.67:0/2225867547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f802815df80 0x7f80280b80c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 --2- 172.21.15.67:0/2225867547 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028164b00 0x7f80280b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 -- 172.21.15.67:0/2225867547 >> 172.21.15.67:0/2225867547 conn(0x7f80280d2bc0 msgr2=0x7f80280d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 -- 172.21.15.67:0/2225867547 shutdown_connections 2024-04-25T15:28:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.445+0000 7f802fb72640 1 -- 172.21.15.67:0/2225867547 wait complete. 2024-04-25T15:28:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.447+0000 7f802fb72640 1 Processor -- start 2024-04-25T15:28:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.447+0000 7f802fb72640 1 -- start start 2024-04-25T15:28:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802fb72640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f802815df80 0x7f80280c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802fb72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028160890 0x7f80280cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802fb72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8028164b00 0x7f80280cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802fb72640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80280b4a00 con 0x7f8028164b00 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802fb72640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f80280b4880 con 0x7f802815df80 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802fb72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f80280b4b80 con 0x7f8028160890 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802e10f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8028164b00 0x7f80280cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802e10f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8028164b00 0x7f80280cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39462/0 (socket says 172.21.15.67:39462) 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.448+0000 7f802e10f640 1 -- 172.21.15.67:0/4013825558 learned_addr learned my addr 172.21.15.67:0/4013825558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802d90e640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f802815df80 0x7f80280c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802e10f640 1 -- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028160890 msgr2=0x7f80280cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802d10d640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028160890 0x7f80280cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802e10f640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028160890 0x7f80280cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802e10f640 1 -- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f802815df80 msgr2=0x7f80280c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802e10f640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f802815df80 0x7f80280c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802e10f640 1 -- 172.21.15.67:0/4013825558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80280ce800 con 0x7f8028164b00 2024-04-25T15:28:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802d90e640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f802815df80 0x7f80280c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802e10f640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8028164b00 0x7f80280cf8a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f802000c760 tx=0x7f802000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f801e7fc640 1 -- 172.21.15.67:0/4013825558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020017070 con 0x7f8028164b00 2024-04-25T15:28:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f801e7fc640 1 -- 172.21.15.67:0/4013825558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f802000ce10 con 0x7f8028164b00 2024-04-25T15:28:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80280ceae0 con 0x7f8028164b00 2024-04-25T15:28:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f801e7fc640 1 -- 172.21.15.67:0/4013825558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802000a7e0 con 0x7f8028164b00 2024-04-25T15:28:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.449+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80280cf0a0 con 0x7f8028164b00 2024-04-25T15:28:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.450+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8028050740 con 0x7f8028164b00 2024-04-25T15:28:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.451+0000 7f801e7fc640 1 -- 172.21.15.67:0/4013825558 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f802001a070 con 0x7f8028164b00 2024-04-25T15:28:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.451+0000 7f801e7fc640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f800c0744b0 0x7f800c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.451+0000 7f801e7fc640 1 -- 172.21.15.67:0/4013825558 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f8020094690 con 0x7f8028164b00 2024-04-25T15:28:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.452+0000 7f802d90e640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f800c0744b0 0x7f800c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.452+0000 7f802d90e640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f800c0744b0 0x7f800c0769c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f802400a870 tx=0x7f8024008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:02.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.456+0000 7f801e7fc640 1 -- 172.21.15.67:0/4013825558 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f802005e7d0 con 0x7f8028164b00 2024-04-25T15:28:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.681+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f802815cfd0 con 0x7f8028164b00 2024-04-25T15:28:02.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.681+0000 7f801e7fc640 1 -- 172.21.15.67:0/4013825558 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) ==== 53+0+73 (secure 0 0 0) 0x7f8020062ad0 con 0x7f8028164b00 2024-04-25T15:28:02.683 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f800c0744b0 msgr2=0x7f800c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f800c0744b0 0x7f800c0769c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f802400a870 tx=0x7f8024008040 comp rx=0 tx=0).stop 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8028164b00 msgr2=0x7f80280cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8028164b00 0x7f80280cf8a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f802000c760 tx=0x7f802000cc20 comp rx=0 tx=0).stop 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 shutdown_connections 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f800c0744b0 0x7f800c0769c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8028164b00 0x7f80280cf8a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8028160890 0x7f80280cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 --2- 172.21.15.67:0/4013825558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f802815df80 0x7f80280c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 >> 172.21.15.67:0/4013825558 conn(0x7f80280d2bc0 msgr2=0x7f802815f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 shutdown_connections 2024-04-25T15:28:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:02.683+0000 7f802fb72640 1 -- 172.21.15.67:0/4013825558 wait complete. 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /subdir 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:28:02.696 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:28:02.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.697 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:28:02.715 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:28:02.715 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:28:02.715 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:28:02.715 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:28:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42606sec preferred_lft 42606sec 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:28:02.717 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:28:02.718 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::248e:2bff:fe69:7f63/64 scope link 2024-04-25T15:28:02.718 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:02.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.718 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:28:02.782 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:28:02.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.784 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:28:02.841 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:28:02.841 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:28:02.841 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:28:02.842 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:28:02.842 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:28:02.842 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:28:02.842 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:28:02.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:28:02.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:28:02.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:28:02.842 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:28:02.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:28:02.967 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:28:02.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.973 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:28:02.988 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:28:02.988 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:28:02.988 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:28:02.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:02.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:28:03.052 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:28:03.053 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:03.107 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:28:03.108 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:28:03.108 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:03.162 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/subdir 2024-04-25T15:28:03.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:03.163 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/subdir /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:03.255 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:03.255 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:03.264 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:03.264 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:03.264 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:28:03.264 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/subdir 2024-04-25T15:28:03.264 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:28:03.265 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:28:03.265 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:28:03.265 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:28:03.265 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:28:03.265 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:28:03.266 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:28:03.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:03.266 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:03.295 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:28:03.385 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:28:03.392 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:28:03.397 INFO:teuthology.orchestra.run.smithi176.stdout:client6151 2024-04-25T15:28:03.398 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:03.398 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:03.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:03.413 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec df -B1 .) 2024-04-25T15:28:03.471 INFO:teuthology.orchestra.run.smithi176.stdout:Filesystem 1B-blocks Used Available Use% Mounted on 2024-04-25T15:28:03.472 INFO:teuthology.orchestra.run.smithi176.stdout:1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/subdir 134217728 0 134217728 0% /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:03.472 WARNING:tasks.cephfs.mount:['Filesystem 1B-blocks Used Available Use% Mounted on', '1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/subdir 134217728 0 134217728 0% /home/ubuntu/cephtest/mnt.1'] 2024-04-25T15:28:03.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:03.472 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 268435456 ./subdir) 2024-04-25T15:28:03.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:03.495 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec df -B1 .) 2024-04-25T15:28:03.527 INFO:teuthology.orchestra.run.smithi176.stdout:Filesystem 1B-blocks Used Available Use% Mounted on 2024-04-25T15:28:03.527 INFO:teuthology.orchestra.run.smithi176.stdout:1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/subdir 268435456 0 268435456 0% /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:03.527 WARNING:tasks.cephfs.mount:['Filesystem 1B-blocks Used Available Use% Mounted on', '1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/subdir 268435456 0 268435456 0% /home/ubuntu/cephtest/mnt.1'] 2024-04-25T15:28:03.527 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:28:03.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:03.579 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:03.612 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:28:03.612 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:03.612 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:03.613 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:03.636 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:03.636 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:03.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:03.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.846+0000 7fde93e6e640 1 Processor -- start 2024-04-25T15:28:03.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.846+0000 7fde93e6e640 1 -- start start 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde93e6e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde8c0d0e00 0x7fde8c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde93e6e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fde8c0d4a50 0x7fde8c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde93e6e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8c0d1850 con 0x7fde8c06c5c0 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde93e6e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fde8c160de0 con 0x7fde8c0d0e00 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde93e6e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fde8c1610c0 con 0x7fde8c0d4a50 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde91c0a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde8c0d0e00 0x7fde8c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde91c0a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde8c0d0e00 0x7fde8c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39470/0 (socket says 172.21.15.67:39470) 2024-04-25T15:28:03.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.847+0000 7fde91c0a640 1 -- 172.21.15.67:0/1962125000 learned_addr learned my addr 172.21.15.67:0/1962125000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.848+0000 7fde9240b640 1 --2- 172.21.15.67:0/1962125000 >> v2:172.21.15.176:3300/0 conn(0x7fde8c0d4a50 0x7fde8c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.848+0000 7fde91c0a640 1 -- 172.21.15.67:0/1962125000 >> v2:172.21.15.176:3300/0 conn(0x7fde8c0d4a50 msgr2=0x7fde8c0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.848+0000 7fde91c0a640 1 --2- 172.21.15.67:0/1962125000 >> v2:172.21.15.176:3300/0 conn(0x7fde8c0d4a50 0x7fde8c0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.848+0000 7fde91c0a640 1 -- 172.21.15.67:0/1962125000 >> v1:172.21.15.176:6789/0 conn(0x7fde8c06c5c0 legacy=0x7fde8c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.848+0000 7fde91c0a640 1 -- 172.21.15.67:0/1962125000 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c1613a0 con 0x7fde8c0d0e00 2024-04-25T15:28:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde91c0a640 1 --2- 172.21.15.67:0/1962125000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d0e00 0x7fde8c0d1210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0d1750 tx=0x7fde8802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b7565b5632e5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde90c08640 1 -- 172.21.15.67:0/1962125000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88037070 con 0x7fde8c0d0e00 2024-04-25T15:28:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde90c08640 1 -- 172.21.15.67:0/1962125000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde88032440 con 0x7fde8c0d0e00 2024-04-25T15:28:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde90c08640 1 -- 172.21.15.67:0/1962125000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88032810 con 0x7fde8c0d0e00 2024-04-25T15:28:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde93e6e640 1 -- 172.21.15.67:0/1962125000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d0e00 msgr2=0x7fde8c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1962125000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d0e00 0x7fde8c0d1210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0d1750 tx=0x7fde8802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde93e6e640 1 -- 172.21.15.67:0/1962125000 shutdown_connections 2024-04-25T15:28:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1962125000 >> v2:172.21.15.176:3300/0 conn(0x7fde8c0d4a50 0x7fde8c0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1962125000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d0e00 0x7fde8c0d1210 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.849+0000 7fde93e6e640 1 -- 172.21.15.67:0/1962125000 >> 172.21.15.67:0/1962125000 conn(0x7fde8c0d5d90 msgr2=0x7fde8c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.850+0000 7fde93e6e640 1 -- 172.21.15.67:0/1962125000 shutdown_connections 2024-04-25T15:28:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.850+0000 7fde93e6e640 1 -- 172.21.15.67:0/1962125000 wait complete. 2024-04-25T15:28:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.851+0000 7fde93e6e640 1 Processor -- start 2024-04-25T15:28:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.851+0000 7fde93e6e640 1 -- start start 2024-04-25T15:28:03.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde93e6e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde8c06c5c0 0x7fde8c15dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde93e6e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde8c0d0e00 0x7fde8c15d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde93e6e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d4a50 0x7fde8c15d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde93e6e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c164e00 con 0x7fde8c0d4a50 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde93e6e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c0b5320 con 0x7fde8c0d0e00 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde93e6e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c164f80 con 0x7fde8c06c5c0 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d4a50 0x7fde8c15d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde91409640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde8c0d0e00 0x7fde8c15d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde91c0a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde8c06c5c0 0x7fde8c15dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde91409640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde8c0d0e00 0x7fde8c15d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:33818/0 (socket says 172.21.15.67:33818) 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d4a50 0x7fde8c15d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39478/0 (socket says 172.21.15.67:39478) 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde91409640 1 -- 172.21.15.67:0/1074323851 learned_addr learned my addr 172.21.15.67:0/1074323851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde91c0a640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde8c06c5c0 0x7fde8c15dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:57318/0 (socket says 172.21.15.67:57318) 2024-04-25T15:28:03.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 -- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde8c06c5c0 msgr2=0x7fde8c15dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde8c06c5c0 0x7fde8c15dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 -- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde8c0d0e00 msgr2=0x7fde8c15d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde8c0d0e00 0x7fde8c15d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 -- 172.21.15.67:0/1074323851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c15e2a0 con 0x7fde8c0d4a50 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde91409640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde8c0d0e00 0x7fde8c15d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde9240b640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d4a50 0x7fde8c15d700 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fde8400ebd0 tx=0x7fde8400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.853+0000 7fde827fc640 1 -- 172.21.15.67:0/1074323851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84018020 con 0x7fde8c0d4a50 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.854+0000 7fde827fc640 1 -- 172.21.15.67:0/1074323851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8400cd20 con 0x7fde8c0d4a50 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.854+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8c1f6fa0 con 0x7fde8c0d4a50 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.854+0000 7fde827fc640 1 -- 172.21.15.67:0/1074323851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde840146c0 con 0x7fde8c0d4a50 2024-04-25T15:28:03.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.854+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8c1f7560 con 0x7fde8c0d4a50 2024-04-25T15:28:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.854+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8c050740 con 0x7fde8c0d4a50 2024-04-25T15:28:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.855+0000 7fde827fc640 1 -- 172.21.15.67:0/1074323851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fde84005d10 con 0x7fde8c0d4a50 2024-04-25T15:28:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.856+0000 7fde827fc640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fde700744b0 0x7fde700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:03.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.856+0000 7fde827fc640 1 -- 172.21.15.67:0/1074323851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fde84095c90 con 0x7fde8c0d4a50 2024-04-25T15:28:03.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.856+0000 7fde91c0a640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fde700744b0 0x7fde700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:03.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.856+0000 7fde91c0a640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fde700744b0 0x7fde700769c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde88034000 tx=0x7fde8802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:03.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:03.860+0000 7fde827fc640 1 -- 172.21.15.67:0/1074323851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde84022030 con 0x7fde8c0d4a50 2024-04-25T15:28:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.045+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde8c0c8ea0 con 0x7fde8c0d4a50 2024-04-25T15:28:04.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.046+0000 7fde827fc640 1 -- 172.21.15.67:0/1074323851 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16632 (secure 0 0 0) 0x7fde84010040 con 0x7fde8c0d4a50 2024-04-25T15:28:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:04.048 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:48.764450+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fde700744b0 msgr2=0x7fde700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fde700744b0 0x7fde700769c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fde88034000 tx=0x7fde8802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d4a50 msgr2=0x7fde8c15d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d4a50 0x7fde8c15d700 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fde8400ebd0 tx=0x7fde8400c490 comp rx=0 tx=0).stop 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 shutdown_connections 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fde700744b0 0x7fde700769c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde8c0d4a50 0x7fde8c15d700 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde8c0d0e00 0x7fde8c15d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 --2- 172.21.15.67:0/1074323851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde8c06c5c0 0x7fde8c15dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 >> 172.21.15.67:0/1074323851 conn(0x7fde8c0d5d90 msgr2=0x7fde8c0d4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 shutdown_connections 2024-04-25T15:28:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.048+0000 7fde93e6e640 1 -- 172.21.15.67:0/1074323851 wait complete. 2024-04-25T15:28:04.063 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:27:45.453204+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:27:46.073423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:04.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:04.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.176+0000 7ff5da74d640 1 Processor -- start 2024-04-25T15:28:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.176+0000 7ff5da74d640 1 -- start start 2024-04-25T15:28:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5da74d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5d4161710 0x7ff5d415e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5da74d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5da74d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5d41609b0 con 0x7ff5d4163fa0 2024-04-25T15:28:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5da74d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff5d406cf10 con 0x7ff5d4161710 2024-04-25T15:28:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5da74d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d4164cd0 con 0x7ff5d406c5c0 2024-04-25T15:28:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5d4161710 0x7ff5d415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5d4161710 0x7ff5d415e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39486/0 (socket says 172.21.15.67:39486) 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d8cea640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff5d4163fa0 0x7ff5d41643b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40828/0 (socket says 172.21.15.67:40828) 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d3fff640 1 -- 172.21.15.67:0/24379654 learned_addr learned my addr 172.21.15.67:0/24379654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d37fe640 1 --2- 172.21.15.67:0/24379654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d3fff640 1 -- 172.21.15.67:0/24379654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 msgr2=0x7ff5d406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d3fff640 1 --2- 172.21.15.67:0/24379654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.177+0000 7ff5d3fff640 1 -- 172.21.15.67:0/24379654 >> v1:172.21.15.67:6789/0 conn(0x7ff5d4163fa0 legacy=0x7ff5d41643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.178+0000 7ff5d3fff640 1 -- 172.21.15.67:0/24379654 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d4160c30 con 0x7ff5d4161710 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.178+0000 7ff5d2ffd640 1 -- 172.21.15.67:0/24379654 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131706558 0 0) 0x7ff5cc001490 con 0x7ff5d4163fa0 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.178+0000 7ff5d37fe640 1 --2- 172.21.15.67:0/24379654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.178+0000 7ff5d3fff640 1 --2- 172.21.15.67:0/24379654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4161710 0x7ff5d415e080 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009870 tx=0x7ff5bc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7c491b29590508 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.178+0000 7ff5d2ffd640 1 -- 172.21.15.67:0/24379654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc037070 con 0x7ff5d4161710 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.178+0000 7ff5d2ffd640 1 -- 172.21.15.67:0/24379654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5bc032440 con 0x7ff5d4161710 2024-04-25T15:28:04.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.178+0000 7ff5d2ffd640 1 -- 172.21.15.67:0/24379654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc032810 con 0x7ff5d4161710 2024-04-25T15:28:04.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 -- 172.21.15.67:0/24379654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4161710 msgr2=0x7ff5d415e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 --2- 172.21.15.67:0/24379654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4161710 0x7ff5d415e080 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009870 tx=0x7ff5bc02a1c0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 -- 172.21.15.67:0/24379654 shutdown_connections 2024-04-25T15:28:04.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 --2- 172.21.15.67:0/24379654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 --2- 172.21.15.67:0/24379654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4161710 0x7ff5d415e080 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 -- 172.21.15.67:0/24379654 >> 172.21.15.67:0/24379654 conn(0x7ff5d40d5d90 msgr2=0x7ff5d40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 -- 172.21.15.67:0/24379654 shutdown_connections 2024-04-25T15:28:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.179+0000 7ff5da74d640 1 -- 172.21.15.67:0/24379654 wait complete. 2024-04-25T15:28:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.181+0000 7ff5da74d640 1 Processor -- start 2024-04-25T15:28:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.181+0000 7ff5da74d640 1 -- start start 2024-04-25T15:28:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.182+0000 7ff5da74d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d40cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.182+0000 7ff5da74d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5d4161710 0x7ff5d4151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.182+0000 7ff5da74d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4163fa0 0x7ff5d40ba3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.182+0000 7ff5da74d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d40d49f0 con 0x7ff5d4163fa0 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.182+0000 7ff5da74d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d41611f0 con 0x7ff5d4161710 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.182+0000 7ff5da74d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d40d4b70 con 0x7ff5d406c5c0 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4163fa0 0x7ff5d40ba3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4163fa0 0x7ff5d40ba3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39496/0 (socket says 172.21.15.67:39496) 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 -- 172.21.15.67:0/3302071408 learned_addr learned my addr 172.21.15.67:0/3302071408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d3fff640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d40cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d37fe640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5d4161710 0x7ff5d4151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 -- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 msgr2=0x7ff5d40cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d40cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 -- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5d4161710 msgr2=0x7ff5d4151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5d4161710 0x7ff5d4151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 -- 172.21.15.67:0/3302071408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d40cf560 con 0x7ff5d4163fa0 2024-04-25T15:28:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d8cea640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4163fa0 0x7ff5d40ba3f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00c6d0 tx=0x7ff5cc00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d0ff9640 1 -- 172.21.15.67:0/3302071408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc018070 con 0x7ff5d4163fa0 2024-04-25T15:28:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d0ff9640 1 -- 172.21.15.67:0/3302071408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5cc004d60 con 0x7ff5d4163fa0 2024-04-25T15:28:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5d0ff9640 1 -- 172.21.15.67:0/3302071408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc00a880 con 0x7ff5d4163fa0 2024-04-25T15:28:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d40cf840 con 0x7ff5d4163fa0 2024-04-25T15:28:04.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.183+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d40cfe00 con 0x7ff5d4163fa0 2024-04-25T15:28:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.184+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d4050740 con 0x7ff5d4163fa0 2024-04-25T15:28:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.185+0000 7ff5d0ff9640 1 -- 172.21.15.67:0/3302071408 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff5cc01b070 con 0x7ff5d4163fa0 2024-04-25T15:28:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.186+0000 7ff5d0ff9640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5b40744b0 0x7ff5b40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.186+0000 7ff5d0ff9640 1 -- 172.21.15.67:0/3302071408 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7ff5cc094910 con 0x7ff5d4163fa0 2024-04-25T15:28:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.186+0000 7ff5d3fff640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5b40744b0 0x7ff5b40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.186+0000 7ff5d3fff640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5b40744b0 0x7ff5b40769c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc034000 tx=0x7ff5bc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.191+0000 7ff5d0ff9640 1 -- 172.21.15.67:0/3302071408 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff5d41609b0 con 0x7ff5d4163fa0 2024-04-25T15:28:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.373+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff5d40d0c00 con 0x7ff5d4163fa0 2024-04-25T15:28:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.373+0000 7ff5d0ff9640 1 -- 172.21.15.67:0/3302071408 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7ff5cc067120 con 0x7ff5d4163fa0 2024-04-25T15:28:04.375 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5b40744b0 msgr2=0x7ff5b40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5b40744b0 0x7ff5b40769c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc034000 tx=0x7ff5bc02e000 comp rx=0 tx=0).stop 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4163fa0 msgr2=0x7ff5d40ba3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4163fa0 0x7ff5d40ba3f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00c6d0 tx=0x7ff5cc00cb90 comp rx=0 tx=0).stop 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 shutdown_connections 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5b40744b0 0x7ff5b40769c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5d4163fa0 0x7ff5d40ba3f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5d4161710 0x7ff5d4151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 --2- 172.21.15.67:0/3302071408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5d406c5c0 0x7ff5d40cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.375+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 >> 172.21.15.67:0/3302071408 conn(0x7ff5d40d5d90 msgr2=0x7ff5d415e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.376+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 shutdown_connections 2024-04-25T15:28:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.376+0000 7ff5da74d640 1 -- 172.21.15.67:0/3302071408 wait complete. 2024-04-25T15:28:04.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:28:04.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.502+0000 7f8ab2927640 1 Processor -- start 2024-04-25T15:28:04.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.503+0000 7f8ab2927640 1 -- start start 2024-04-25T15:28:04.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab2927640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac0d0e00 0x7f8aac0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab2927640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aac0d4a50 0x7f8aac0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab2927640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aac0d1850 con 0x7f8aac06c5c0 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab2927640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8aac160de0 con 0x7f8aac0d4a50 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab2927640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac1610c0 con 0x7f8aac0d0e00 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab0ec4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aac0d4a50 0x7f8aac0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab0ec4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8aac0d4a50 0x7f8aac0d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39512/0 (socket says 172.21.15.67:39512) 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/1178465012 learned_addr learned my addr 172.21.15.67:0/1178465012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8aabfff640 1 --2- 172.21.15.67:0/1178465012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac0d0e00 0x7f8aac0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/1178465012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac0d0e00 msgr2=0x7f8aac0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab0ec4640 1 --2- 172.21.15.67:0/1178465012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac0d0e00 0x7f8aac0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/1178465012 >> v1:172.21.15.176:6789/0 conn(0x7f8aac06c5c0 legacy=0x7f8aac06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/1178465012 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac1613a0 con 0x7f8aac0d4a50 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.504+0000 7f8aabfff640 1 --2- 172.21.15.67:0/1178465012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac0d0e00 0x7f8aac0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:04.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.505+0000 7f8ab0ec4640 1 --2- 172.21.15.67:0/1178465012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac0d4e60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009e50 tx=0x7f8aa402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6786f9ea7ee8efb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.505+0000 7f8aaaffd640 1 -- 172.21.15.67:0/1178465012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4036070 con 0x7f8aac0d4a50 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.505+0000 7f8aaaffd640 1 -- 172.21.15.67:0/1178465012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa4031600 con 0x7f8aac0d4a50 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.505+0000 7f8aaaffd640 1 -- 172.21.15.67:0/1178465012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40319d0 con 0x7f8aac0d4a50 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 -- 172.21.15.67:0/1178465012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 msgr2=0x7f8aac0d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 --2- 172.21.15.67:0/1178465012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac0d4e60 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009e50 tx=0x7f8aa402a710 comp rx=0 tx=0).stop 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 -- 172.21.15.67:0/1178465012 shutdown_connections 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 --2- 172.21.15.67:0/1178465012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac0d4e60 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 --2- 172.21.15.67:0/1178465012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac0d0e00 0x7f8aac0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 -- 172.21.15.67:0/1178465012 >> 172.21.15.67:0/1178465012 conn(0x7f8aac0d5d90 msgr2=0x7f8aac0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 -- 172.21.15.67:0/1178465012 shutdown_connections 2024-04-25T15:28:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.506+0000 7f8ab2927640 1 -- 172.21.15.67:0/1178465012 wait complete. 2024-04-25T15:28:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.508+0000 7f8ab2927640 1 Processor -- start 2024-04-25T15:28:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.508+0000 7f8ab2927640 1 -- start start 2024-04-25T15:28:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab2927640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac06c5c0 0x7f8aac1f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab2927640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aac0d0e00 0x7f8aac148c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab2927640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac1fa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab2927640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac164e00 con 0x7f8aac0d4a50 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab2927640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0b5320 con 0x7f8aac0d0e00 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab2927640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac164f80 con 0x7f8aac06c5c0 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab0ec4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac1fa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab0ec4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac1fa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39522/0 (socket says 172.21.15.67:39522) 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.509+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/3385772230 learned_addr learned my addr 172.21.15.67:0/3385772230 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8aabfff640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac06c5c0 0x7f8aac1f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8aab7fe640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aac0d0e00 0x7f8aac148c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac06c5c0 msgr2=0x7f8aac1f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab0ec4640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac06c5c0 0x7f8aac1f7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aac0d0e00 msgr2=0x7f8aac148c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab0ec4640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aac0d0e00 0x7f8aac148c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab0ec4640 1 -- 172.21.15.67:0/3385772230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac1f7b60 con 0x7f8aac0d4a50 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab0ec4640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac1fa1e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009f70 tx=0x7f8aa4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8aa8ff9640 1 -- 172.21.15.67:0/3385772230 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4041020 con 0x7f8aac0d4a50 2024-04-25T15:28:04.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8aa8ff9640 1 -- 172.21.15.67:0/3385772230 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa4007780 con 0x7f8aac0d4a50 2024-04-25T15:28:04.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aac1ea420 con 0x7f8aac0d4a50 2024-04-25T15:28:04.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8aa8ff9640 1 -- 172.21.15.67:0/3385772230 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4036040 con 0x7f8aac0d4a50 2024-04-25T15:28:04.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.510+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aac0b6c30 con 0x7f8aac0d4a50 2024-04-25T15:28:04.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.511+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac0b6740 con 0x7f8aac0d4a50 2024-04-25T15:28:04.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.512+0000 7f8aa8ff9640 1 -- 172.21.15.67:0/3385772230 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8aa4044020 con 0x7f8aac0d4a50 2024-04-25T15:28:04.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.513+0000 7f8aa8ff9640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a8c0744b0 0x7f8a8c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.513+0000 7f8aa8ff9640 1 -- 172.21.15.67:0/3385772230 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f8aa40b5e40 con 0x7f8aac0d4a50 2024-04-25T15:28:04.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.513+0000 7f8aabfff640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a8c0744b0 0x7f8a8c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.513+0000 7f8aabfff640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a8c0744b0 0x7f8a8c0769c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a94006fd0 tx=0x7f8a94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.518+0000 7f8aa8ff9640 1 -- 172.21.15.67:0/3385772230 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8aa404b030 con 0x7f8aac0d4a50 2024-04-25T15:28:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.703+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8aac050740 con 0x7f8aac0d4a50 2024-04-25T15:28:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.703+0000 7f8aa8ff9640 1 -- 172.21.15.67:0/3385772230 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f8aa407ff80 con 0x7f8aac0d4a50 2024-04-25T15:28:04.705 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:28:04.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a8c0744b0 msgr2=0x7f8a8c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a8c0744b0 0x7f8a8c0769c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a94006fd0 tx=0x7f8a94008040 comp rx=0 tx=0).stop 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 msgr2=0x7f8aac1fa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac1fa1e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009f70 tx=0x7f8aa4004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 shutdown_connections 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8a8c0744b0 0x7f8a8c0769c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aac0d4a50 0x7f8aac1fa1e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aac0d0e00 0x7f8aac148c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 --2- 172.21.15.67:0/3385772230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aac06c5c0 0x7f8aac1f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.705+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 >> 172.21.15.67:0/3385772230 conn(0x7f8aac0d5d90 msgr2=0x7f8aac0ba510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.706+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 shutdown_connections 2024-04-25T15:28:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.706+0000 7f8ab2927640 1 -- 172.21.15.67:0/3385772230 wait complete. 2024-04-25T15:28:04.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:28:04.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.832+0000 7fa435e97640 1 Processor -- start 2024-04-25T15:28:04.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.832+0000 7fa435e97640 1 -- start start 2024-04-25T15:28:04.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.833+0000 7fa435e97640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa4300b8290 0x7fa430160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.833+0000 7fa435e97640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4301615f0 con 0x7fa430164b00 2024-04-25T15:28:04.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.833+0000 7fa435e97640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa430164020 con 0x7fa43015df80 2024-04-25T15:28:04.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.833+0000 7fa435e97640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa430164670 con 0x7fa4300b8290 2024-04-25T15:28:04.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.833+0000 7fa42f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa430164b00 0x7fa4300b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40838/0 (socket says 172.21.15.67:40838) 2024-04-25T15:28:04.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.833+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2173577728 learned_addr learned my addr 172.21.15.67:0/2173577728 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:04.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 964143771 0 0) 0x7fa4301615f0 con 0x7fa430164b00 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42effd640 1 --2- 172.21.15.67:0/2173577728 >> v2:172.21.15.176:3300/0 conn(0x7fa4300b8290 0x7fa430160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa418003520 con 0x7fa430164b00 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3061407023 0 0) 0x7fa418003520 con 0x7fa430164b00 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 >> v2:172.21.15.176:3300/0 conn(0x7fa4300b8290 msgr2=0x7fa430160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 --2- 172.21.15.67:0/2173577728 >> v2:172.21.15.176:3300/0 conn(0x7fa4300b8290 0x7fa430160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 >> v1:172.21.15.176:6789/0 conn(0x7fa43015df80 legacy=0x7fa430160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa430164300 con 0x7fa430164b00 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fa420001670 con 0x7fa430164b00 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 >> v1:172.21.15.67:6789/0 conn(0x7fa430164b00 legacy=0x7fa4300b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa418003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa418004800 0x7fa418004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa418005270 0x7fa41800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa430164670 con 0x7fa4180039b0 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa41800c190 con 0x7fa418004800 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa41800c390 con 0x7fa418005270 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42f7fe640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa418003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42effd640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa418004800 0x7fa418004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42ffff640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa418005270 0x7fa41800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa418005270 msgr2=0x7fa41800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.834+0000 7fa42f7fe640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa418005270 0x7fa41800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.835+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa418004800 msgr2=0x7fa418004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.835+0000 7fa42f7fe640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa418004800 0x7fa418004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.835+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2173577728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa418003520 con 0x7fa4180039b0 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.835+0000 7fa42ffff640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa418005270 0x7fa41800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.835+0000 7fa42ffff640 1 -- 172.21.15.67:0/2173577728 reap_dead start 2024-04-25T15:28:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.835+0000 7fa42f7fe640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa418003e40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c240 tx=0x7fa418005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4642da1c8518aff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.836+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420007d10 con 0x7fa4180039b0 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.836+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa420005ad0 con 0x7fa4180039b0 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.836+0000 7fa42e7fc640 1 -- 172.21.15.67:0/2173577728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420007070 con 0x7fa4180039b0 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.836+0000 7fa435e97640 1 -- 172.21.15.67:0/2173577728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 msgr2=0x7fa418003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.836+0000 7fa435e97640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa418003e40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c240 tx=0x7fa418005830 comp rx=0 tx=0).stop 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.837+0000 7fa435e97640 1 -- 172.21.15.67:0/2173577728 shutdown_connections 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.837+0000 7fa435e97640 1 --2- 172.21.15.67:0/2173577728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa418003e40 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.837+0000 7fa435e97640 1 -- 172.21.15.67:0/2173577728 >> 172.21.15.67:0/2173577728 conn(0x7fa4300d2bc0 msgr2=0x7fa4300d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.837+0000 7fa435e97640 1 -- 172.21.15.67:0/2173577728 shutdown_connections 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.837+0000 7fa435e97640 1 -- 172.21.15.67:0/2173577728 wait complete. 2024-04-25T15:28:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.838+0000 7fa435e97640 1 Processor -- start 2024-04-25T15:28:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.839+0000 7fa435e97640 1 -- start start 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa435e97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa430158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa435e97640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa430157be0 0x7fa430155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa435e97640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4300d4650 0x7fa4300d4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa435e97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4300b7f10 con 0x7fa4180039b0 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa435e97640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4300b7d90 con 0x7fa4300d4650 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa435e97640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4300b8090 con 0x7fa430157be0 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa430158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa430158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39552/0 (socket says 172.21.15.67:39552) 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2122654556 learned_addr learned my addr 172.21.15.67:0/2122654556 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42effd640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa430157be0 0x7fa430155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42ffff640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4300d4650 0x7fa4300d4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa430157be0 msgr2=0x7fa430155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa430157be0 0x7fa430155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4300d4650 msgr2=0x7fa4300d4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4300d4650 0x7fa4300d4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42f7fe640 1 -- 172.21.15.67:0/2122654556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4300d4fa0 con 0x7fa4180039b0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42effd640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa430157be0 0x7fa430155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.840+0000 7fa42ffff640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4300d4650 0x7fa4300d4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.841+0000 7fa42f7fe640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa430158200 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c240 tx=0x7fa4301645e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.841+0000 7fa417fff640 1 -- 172.21.15.67:0/2122654556 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420041070 con 0x7fa4180039b0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.841+0000 7fa417fff640 1 -- 172.21.15.67:0/2122654556 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa420004e20 con 0x7fa4180039b0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.841+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa430158b70 con 0x7fa4180039b0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.841+0000 7fa417fff640 1 -- 172.21.15.67:0/2122654556 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420007d10 con 0x7fa4180039b0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.841+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4300c9d60 con 0x7fa4180039b0 2024-04-25T15:28:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.842+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa430050740 con 0x7fa4180039b0 2024-04-25T15:28:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.843+0000 7fa417fff640 1 -- 172.21.15.67:0/2122654556 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa420044070 con 0x7fa4180039b0 2024-04-25T15:28:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.843+0000 7fa417fff640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa41c0744b0 0x7fa41c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.843+0000 7fa417fff640 1 -- 172.21.15.67:0/2122654556 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fa4200b5a70 con 0x7fa4180039b0 2024-04-25T15:28:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.843+0000 7fa42effd640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa41c0744b0 0x7fa41c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.844+0000 7fa42effd640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa41c0744b0 0x7fa41c0769c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa4300cf070 tx=0x7fa424008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:04.848+0000 7fa417fff640 1 -- 172.21.15.67:0/2122654556 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4300d4fa0 con 0x7fa4180039b0 2024-04-25T15:28:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.033+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa4301612f0 con 0x7fa4180039b0 2024-04-25T15:28:05.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.033+0000 7fa417fff640 1 -- 172.21.15.67:0/2122654556 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7fa4200802e0 con 0x7fa4180039b0 2024-04-25T15:28:05.035 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:28:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa41c0744b0 msgr2=0x7fa41c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa41c0744b0 0x7fa41c0769c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa4300cf070 tx=0x7fa424008040 comp rx=0 tx=0).stop 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 msgr2=0x7fa430158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa430158200 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa42000c240 tx=0x7fa4301645e0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 shutdown_connections 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa41c0744b0 0x7fa41c0769c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4300d4650 0x7fa4300d4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa430157be0 0x7fa430155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 --2- 172.21.15.67:0/2122654556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4180039b0 0x7fa430158200 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.035+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 >> 172.21.15.67:0/2122654556 conn(0x7fa4300d2bc0 msgr2=0x7fa43015f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.036+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 shutdown_connections 2024-04-25T15:28:05.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.036+0000 7fa435e97640 1 -- 172.21.15.67:0/2122654556 wait complete. 2024-04-25T15:28:05.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:28:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.159+0000 7f93e8620640 1 Processor -- start 2024-04-25T15:28:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.159+0000 7f93e8620640 1 -- start start 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e8620640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f93e006d8e0 0x7f93e00d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e8620640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93e00d2d70 0x7f93e00d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e8620640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e00b3c10 con 0x7f93e006dcf0 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e8620640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f93e00b9520 con 0x7f93e00d2d70 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e8620640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f93e00b4820 con 0x7f93e006d8e0 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e6bbd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93e00d2d70 0x7f93e00d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e6bbd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f93e00d2d70 0x7f93e00d3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39562/0 (socket says 172.21.15.67:39562) 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e6bbd640 1 -- 172.21.15.67:0/715987191 learned_addr learned my addr 172.21.15.67:0/715987191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e63bc640 1 --2- 172.21.15.67:0/715987191 >> v2:172.21.15.176:3300/0 conn(0x7f93e006d8e0 0x7f93e00d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e6bbd640 1 -- 172.21.15.67:0/715987191 >> v2:172.21.15.176:3300/0 conn(0x7f93e006d8e0 msgr2=0x7f93e00d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e6bbd640 1 --2- 172.21.15.67:0/715987191 >> v2:172.21.15.176:3300/0 conn(0x7f93e006d8e0 0x7f93e00d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e6bbd640 1 -- 172.21.15.67:0/715987191 >> v1:172.21.15.176:6789/0 conn(0x7f93e006dcf0 legacy=0x7f93e00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.161+0000 7f93e6bbd640 1 -- 172.21.15.67:0/715987191 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e00b3ef0 con 0x7f93e00d2d70 2024-04-25T15:28:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.162+0000 7f93e6bbd640 1 --2- 172.21.15.67:0/715987191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e00d2d70 0x7f93e00d3f60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f93e00b38b0 tx=0x7f93d802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8ff36f7f2ecdbb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.162+0000 7f93e53ba640 1 -- 172.21.15.67:0/715987191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d8036070 con 0x7f93e00d2d70 2024-04-25T15:28:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.162+0000 7f93e53ba640 1 -- 172.21.15.67:0/715987191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d8031600 con 0x7f93e00d2d70 2024-04-25T15:28:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.162+0000 7f93e53ba640 1 -- 172.21.15.67:0/715987191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d80319d0 con 0x7f93e00d2d70 2024-04-25T15:28:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 -- 172.21.15.67:0/715987191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e00d2d70 msgr2=0x7f93e00d3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 --2- 172.21.15.67:0/715987191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e00d2d70 0x7f93e00d3f60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f93e00b38b0 tx=0x7f93d802a660 comp rx=0 tx=0).stop 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 -- 172.21.15.67:0/715987191 shutdown_connections 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 --2- 172.21.15.67:0/715987191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e00d2d70 0x7f93e00d3f60 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 --2- 172.21.15.67:0/715987191 >> v2:172.21.15.176:3300/0 conn(0x7f93e006d8e0 0x7f93e00d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 -- 172.21.15.67:0/715987191 >> 172.21.15.67:0/715987191 conn(0x7f93e008d110 msgr2=0x7f93e008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 -- 172.21.15.67:0/715987191 shutdown_connections 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.163+0000 7f93e8620640 1 -- 172.21.15.67:0/715987191 wait complete. 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.165+0000 7f93e8620640 1 Processor -- start 2024-04-25T15:28:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.165+0000 7f93e8620640 1 -- start start 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e8620640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e006d8e0 0x7f93e01f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e8620640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e006dcf0 0x7f93e01e7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e8620640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00d2d70 0x7f93e00a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e8620640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f93e00b38e0 con 0x7f93e006d8e0 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e8620640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f93e00b44b0 con 0x7f93e00d2d70 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e8620640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f93e00d0bb0 con 0x7f93e006dcf0 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e006d8e0 0x7f93e01f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e006d8e0 0x7f93e01f5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39566/0 (socket says 172.21.15.67:39566) 2024-04-25T15:28:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 -- 172.21.15.67:0/3382213146 learned_addr learned my addr 172.21.15.67:0/3382213146 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e6bbd640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00d2d70 0x7f93e00a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e5bbb640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e006dcf0 0x7f93e01e7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 -- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e006dcf0 msgr2=0x7f93e01e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e006dcf0 0x7f93e01e7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 -- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00d2d70 msgr2=0x7f93e00a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00d2d70 0x7f93e00a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.166+0000 7f93e63bc640 1 -- 172.21.15.67:0/3382213146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e01e8a90 con 0x7f93e006d8e0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.167+0000 7f93e63bc640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e006d8e0 0x7f93e01f5e60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00c820 tx=0x7f93dc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.167+0000 7f93d6ffd640 1 -- 172.21.15.67:0/3382213146 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc004030 con 0x7f93e006d8e0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.167+0000 7f93d6ffd640 1 -- 172.21.15.67:0/3382213146 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93dc0041d0 con 0x7f93e006d8e0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.167+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e01e8020 con 0x7f93e006d8e0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.167+0000 7f93d6ffd640 1 -- 172.21.15.67:0/3382213146 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc007830 con 0x7f93e006d8e0 2024-04-25T15:28:05.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.167+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e009e5c0 con 0x7f93e006d8e0 2024-04-25T15:28:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.168+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e0050740 con 0x7f93e006d8e0 2024-04-25T15:28:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.169+0000 7f93d6ffd640 1 -- 172.21.15.67:0/3382213146 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f93dc0079d0 con 0x7f93e006d8e0 2024-04-25T15:28:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.169+0000 7f93d6ffd640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f93c40744e0 0x7f93c40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.169+0000 7f93d6ffd640 1 -- 172.21.15.67:0/3382213146 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f93dc095120 con 0x7f93e006d8e0 2024-04-25T15:28:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.169+0000 7f93e5bbb640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f93c40744e0 0x7f93c40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.170+0000 7f93e5bbb640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f93c40744e0 0x7f93c40769f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f93d0006fd0 tx=0x7f93d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:05.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.174+0000 7f93d6ffd640 1 -- 172.21.15.67:0/3382213146 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f93dc015020 con 0x7f93e006d8e0 2024-04-25T15:28:05.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.374+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f93e0073120 con 0x7f93e006d8e0 2024-04-25T15:28:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.375+0000 7f93d6ffd640 1 -- 172.21.15.67:0/3382213146 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) ==== 80+0+0 (secure 0 0 0) 0x7f93dc05f8b0 con 0x7f93e006d8e0 2024-04-25T15:28:05.377 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f93c40744e0 msgr2=0x7f93c40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f93c40744e0 0x7f93c40769f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f93d0006fd0 tx=0x7f93d0008040 comp rx=0 tx=0).stop 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e006d8e0 msgr2=0x7f93e01f5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e006d8e0 0x7f93e01f5e60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00c820 tx=0x7f93dc00cce0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 shutdown_connections 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f93c40744e0 0x7f93c40769f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00d2d70 0x7f93e00a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e006dcf0 0x7f93e01e7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 --2- 172.21.15.67:0/3382213146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93e006d8e0 0x7f93e01f5e60 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 >> 172.21.15.67:0/3382213146 conn(0x7f93e008d110 msgr2=0x7f93e00b8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 shutdown_connections 2024-04-25T15:28:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.377+0000 7f93e8620640 1 -- 172.21.15.67:0/3382213146 wait complete. 2024-04-25T15:28:05.390 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T15:28:05.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:05.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T15:28:05.417 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:28:05.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:05.417 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:28:05.473 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:05.473 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:05.473 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:05.492 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:05.492 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:05.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.660+0000 7fa15b615640 1 Processor -- start 2024-04-25T15:28:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.661+0000 7fa15b615640 1 -- start start 2024-04-25T15:28:05.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa15b615640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1540ce800 0x7fa1540d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa15b615640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa154163130 0x7fa1540b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa15b615640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1540b0770 con 0x7fa1540d1620 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa15b615640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa1540b10a0 con 0x7fa1540ce800 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa15b615640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa1540d2100 con 0x7fa154163130 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa1593b1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1540ce800 0x7fa1540d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa1593b1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1540ce800 0x7fa1540d1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39582/0 (socket says 172.21.15.67:39582) 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa1593b1640 1 -- 172.21.15.67:0/3534406293 learned_addr learned my addr 172.21.15.67:0/3534406293 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa159bb2640 1 --2- 172.21.15.67:0/3534406293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa154163130 0x7fa1540b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa1593b1640 1 -- 172.21.15.67:0/3534406293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa154163130 msgr2=0x7fa1540b2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa1593b1640 1 --2- 172.21.15.67:0/3534406293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa154163130 0x7fa1540b2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.662+0000 7fa1593b1640 1 -- 172.21.15.67:0/3534406293 >> v1:172.21.15.176:6789/0 conn(0x7fa1540d1620 legacy=0x7fa154162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.663+0000 7fa1593b1640 1 -- 172.21.15.67:0/3534406293 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1540b0a50 con 0x7fa1540ce800 2024-04-25T15:28:05.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.663+0000 7fa159bb2640 1 --2- 172.21.15.67:0/3534406293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa154163130 0x7fa1540b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:05.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.663+0000 7fa1593b1640 1 --2- 172.21.15.67:0/3534406293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1540ce800 0x7fa1540d1bc0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa144009ea0 tx=0x7fa14402a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215b753af16d77ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.663+0000 7fa153fff640 1 -- 172.21.15.67:0/3534406293 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa144036070 con 0x7fa1540ce800 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.663+0000 7fa153fff640 1 -- 172.21.15.67:0/3534406293 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa144031600 con 0x7fa1540ce800 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.663+0000 7fa153fff640 1 -- 172.21.15.67:0/3534406293 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1440319d0 con 0x7fa1540ce800 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 -- 172.21.15.67:0/3534406293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1540ce800 msgr2=0x7fa1540d1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 --2- 172.21.15.67:0/3534406293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1540ce800 0x7fa1540d1bc0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa144009ea0 tx=0x7fa14402a640 comp rx=0 tx=0).stop 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 -- 172.21.15.67:0/3534406293 shutdown_connections 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 --2- 172.21.15.67:0/3534406293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa154163130 0x7fa1540b2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 --2- 172.21.15.67:0/3534406293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1540ce800 0x7fa1540d1bc0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 -- 172.21.15.67:0/3534406293 >> 172.21.15.67:0/3534406293 conn(0x7fa1540d3a50 msgr2=0x7fa1540d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 -- 172.21.15.67:0/3534406293 shutdown_connections 2024-04-25T15:28:05.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.664+0000 7fa15b615640 1 -- 172.21.15.67:0/3534406293 wait complete. 2024-04-25T15:28:05.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.666+0000 7fa15b615640 1 Processor -- start 2024-04-25T15:28:05.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.666+0000 7fa15b615640 1 -- start start 2024-04-25T15:28:05.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa15b615640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa1540ce800 0x7fa1540cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa15b615640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1540d1620 0x7fa1540b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa15b615640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa154163130 0x7fa15415b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa15b615640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa154163540 con 0x7fa154163130 2024-04-25T15:28:05.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa15b615640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa1540b51c0 con 0x7fa1540d1620 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa15b615640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa15415be40 con 0x7fa1540ce800 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa159bb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa154163130 0x7fa15415b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa159bb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa154163130 0x7fa15415b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39594/0 (socket says 172.21.15.67:39594) 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.667+0000 7fa159bb2640 1 -- 172.21.15.67:0/3175611781 learned_addr learned my addr 172.21.15.67:0/3175611781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa1593b1640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa1540ce800 0x7fa1540cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa158bb0640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1540d1620 0x7fa1540b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa159bb2640 1 -- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa1540ce800 msgr2=0x7fa1540cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa159bb2640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa1540ce800 0x7fa1540cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa159bb2640 1 -- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1540d1620 msgr2=0x7fa1540b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa159bb2640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1540d1620 0x7fa1540b2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa159bb2640 1 -- 172.21.15.67:0/3175611781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1540b40d0 con 0x7fa154163130 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa158bb0640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1540d1620 0x7fa1540b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa159bb2640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa154163130 0x7fa15415b230 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00c5f0 tx=0x7fa14c00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa151ffb640 1 -- 172.21.15.67:0/3175611781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c015070 con 0x7fa154163130 2024-04-25T15:28:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa151ffb640 1 -- 172.21.15.67:0/3175611781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa14c011820 con 0x7fa154163130 2024-04-25T15:28:05.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1540b43b0 con 0x7fa154163130 2024-04-25T15:28:05.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa151ffb640 1 -- 172.21.15.67:0/3175611781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14c011c10 con 0x7fa154163130 2024-04-25T15:28:05.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.668+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1540b2350 con 0x7fa154163130 2024-04-25T15:28:05.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.669+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa154050740 con 0x7fa154163130 2024-04-25T15:28:05.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.670+0000 7fa151ffb640 1 -- 172.21.15.67:0/3175611781 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa14c018070 con 0x7fa154163130 2024-04-25T15:28:05.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.671+0000 7fa151ffb640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1400744b0 0x7fa1400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.671+0000 7fa151ffb640 1 -- 172.21.15.67:0/3175611781 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fa14c094fb0 con 0x7fa154163130 2024-04-25T15:28:05.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.671+0000 7fa1593b1640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1400744b0 0x7fa1400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.672+0000 7fa1593b1640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1400744b0 0x7fa1400769c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa144033000 tx=0x7fa144009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:05.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.676+0000 7fa151ffb640 1 -- 172.21.15.67:0/3175611781 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa14c05f190 con 0x7fa154163130 2024-04-25T15:28:05.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.860+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1540d1220 con 0x7fa154163130 2024-04-25T15:28:05.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.862+0000 7fa151ffb640 1 -- 172.21.15.67:0/3175611781 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16632 (secure 0 0 0) 0x7fa14c063490 con 0x7fa154163130 2024-04-25T15:28:05.864 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:05.864 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:48.764450+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.863+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1400744b0 msgr2=0x7fa1400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.863+0000 7fa15b615640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1400744b0 0x7fa1400769c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa144033000 tx=0x7fa144009000 comp rx=0 tx=0).stop 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.863+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa154163130 msgr2=0x7fa15415b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.863+0000 7fa15b615640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa154163130 0x7fa15415b230 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00c5f0 tx=0x7fa14c00cab0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 shutdown_connections 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1400744b0 0x7fa1400769c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa154163130 0x7fa15415b230 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1540d1620 0x7fa1540b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 --2- 172.21.15.67:0/3175611781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa1540ce800 0x7fa1540cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 >> 172.21.15.67:0/3175611781 conn(0x7fa1540d3a50 msgr2=0x7fa1540d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 shutdown_connections 2024-04-25T15:28:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.864+0000 7fa15b615640 1 -- 172.21.15.67:0/3175611781 wait complete. 2024-04-25T15:28:05.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:27:45.453204+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:27:46.073423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:05.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:05.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.992+0000 7f27b97b2640 1 Processor -- start 2024-04-25T15:28:05.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.992+0000 7f27b97b2640 1 -- start start 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.993+0000 7f27b97b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f27b40d0e00 0x7f27b40d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.993+0000 7f27b97b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27b406c5c0 0x7f27b406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.993+0000 7f27b97b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b406cf10 con 0x7f27b40b7810 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.993+0000 7f27b97b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f27b40d5a60 con 0x7f27b406c5c0 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.993+0000 7f27b97b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f27b40d1850 con 0x7f27b40d0e00 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27abfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27b406c5c0 0x7f27b406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27abfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27b406c5c0 0x7f27b406c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39602/0 (socket says 172.21.15.67:39602) 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27abfff640 1 -- 172.21.15.67:0/3260490408 learned_addr learned my addr 172.21.15.67:0/3260490408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27b2ffd640 1 --2- 172.21.15.67:0/3260490408 >> v2:172.21.15.176:3300/0 conn(0x7f27b40d0e00 0x7f27b40d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27abfff640 1 -- 172.21.15.67:0/3260490408 >> v2:172.21.15.176:3300/0 conn(0x7f27b40d0e00 msgr2=0x7f27b40d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:05.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27abfff640 1 --2- 172.21.15.67:0/3260490408 >> v2:172.21.15.176:3300/0 conn(0x7f27b40d0e00 0x7f27b40d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27abfff640 1 -- 172.21.15.67:0/3260490408 >> v1:172.21.15.176:6789/0 conn(0x7f27b40b7810 legacy=0x7f27b40b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27abfff640 1 -- 172.21.15.67:0/3260490408 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b4160de0 con 0x7f27b406c5c0 2024-04-25T15:28:05.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.994+0000 7f27b2ffd640 1 --2- 172.21.15.67:0/3260490408 >> v2:172.21.15.176:3300/0 conn(0x7f27b40d0e00 0x7f27b40d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.995+0000 7f27abfff640 1 --2- 172.21.15.67:0/3260490408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b406c5c0 0x7f27b406c9d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f27b40d5be0 tx=0x7f279c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73739c7a087f67ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.995+0000 7f27b27fc640 1 -- 172.21.15.67:0/3260490408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c037070 con 0x7f27b406c5c0 2024-04-25T15:28:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.995+0000 7f27b27fc640 1 -- 172.21.15.67:0/3260490408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f279c032440 con 0x7f27b406c5c0 2024-04-25T15:28:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.995+0000 7f27b27fc640 1 -- 172.21.15.67:0/3260490408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c032810 con 0x7f27b406c5c0 2024-04-25T15:28:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 -- 172.21.15.67:0/3260490408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b406c5c0 msgr2=0x7f27b406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 --2- 172.21.15.67:0/3260490408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b406c5c0 0x7f27b406c9d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f27b40d5be0 tx=0x7f279c02a180 comp rx=0 tx=0).stop 2024-04-25T15:28:05.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 -- 172.21.15.67:0/3260490408 shutdown_connections 2024-04-25T15:28:05.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 --2- 172.21.15.67:0/3260490408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b406c5c0 0x7f27b406c9d0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 --2- 172.21.15.67:0/3260490408 >> v2:172.21.15.176:3300/0 conn(0x7f27b40d0e00 0x7f27b40d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:05.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 -- 172.21.15.67:0/3260490408 >> 172.21.15.67:0/3260490408 conn(0x7f27b40d5d90 msgr2=0x7f27b40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:05.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 -- 172.21.15.67:0/3260490408 shutdown_connections 2024-04-25T15:28:05.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.996+0000 7f27b97b2640 1 -- 172.21.15.67:0/3260490408 wait complete. 2024-04-25T15:28:05.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.997+0000 7f27b97b2640 1 Processor -- start 2024-04-25T15:28:05.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.997+0000 7f27b97b2640 1 -- start start 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b97b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27b406c5c0 0x7f27b40ce180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b97b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b40b7810 0x7f27b415a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b97b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27b40d0e00 0x7f27b40cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b97b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27b4160ad0 con 0x7f27b40b7810 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b97b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f27b41613a0 con 0x7f27b406c5c0 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b97b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27b40b5320 con 0x7f27b40d0e00 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b40b7810 0x7f27b415a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b40b7810 0x7f27b415a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39610/0 (socket says 172.21.15.67:39610) 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 -- 172.21.15.67:0/1587945412 learned_addr learned my addr 172.21.15.67:0/1587945412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b37fe640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27b40d0e00 0x7f27b40cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27b2ffd640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27b406c5c0 0x7f27b40ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 -- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27b40d0e00 msgr2=0x7f27b40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27b40d0e00 0x7f27b40cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 -- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27b406c5c0 msgr2=0x7f27b40ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27b406c5c0 0x7f27b40ce180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 -- 172.21.15.67:0/1587945412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b41ea300 con 0x7f27b40b7810 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:05.999+0000 7f27abfff640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b40b7810 0x7f27b415a600 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f279c02a1c0 tx=0x7f279c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.000+0000 7f27ab7fe640 1 -- 172.21.15.67:0/1587945412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c037040 con 0x7f27b40b7810 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.000+0000 7f27ab7fe640 1 -- 172.21.15.67:0/1587945412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f279c042070 con 0x7f27b40b7810 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.000+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b41ea5e0 con 0x7f27b40b7810 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.000+0000 7f27ab7fe640 1 -- 172.21.15.67:0/1587945412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279c030bb0 con 0x7f27b40b7810 2024-04-25T15:28:06.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.000+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b41eaba0 con 0x7f27b40b7810 2024-04-25T15:28:06.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.001+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b41ea070 con 0x7f27b40b7810 2024-04-25T15:28:06.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.001+0000 7f27ab7fe640 1 -- 172.21.15.67:0/1587945412 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f279c047460 con 0x7f27b40b7810 2024-04-25T15:28:06.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.002+0000 7f27ab7fe640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27940744b0 0x7f27940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.002+0000 7f27ab7fe640 1 -- 172.21.15.67:0/1587945412 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f279c0b5e50 con 0x7f27b40b7810 2024-04-25T15:28:06.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.002+0000 7f27b2ffd640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27940744b0 0x7f27940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.003+0000 7f27b2ffd640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27940744b0 0x7f27940769c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f27a400aac0 tx=0x7f27a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.006+0000 7f27ab7fe640 1 -- 172.21.15.67:0/1587945412 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27b406cf10 con 0x7f27b40b7810 2024-04-25T15:28:06.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.193+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f27b4050740 con 0x7f27b40b7810 2024-04-25T15:28:06.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.193+0000 7f27ab7fe640 1 -- 172.21.15.67:0/1587945412 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7f279c07ff10 con 0x7f27b40b7810 2024-04-25T15:28:06.195 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27940744b0 msgr2=0x7f27940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27940744b0 0x7f27940769c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f27a400aac0 tx=0x7f27a4008040 comp rx=0 tx=0).stop 2024-04-25T15:28:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b40b7810 msgr2=0x7f27b415a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b40b7810 0x7f27b415a600 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f279c02a1c0 tx=0x7f279c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 shutdown_connections 2024-04-25T15:28:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f27940744b0 0x7f27940769c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27b40d0e00 0x7f27b40cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27b40b7810 0x7f27b415a600 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 --2- 172.21.15.67:0/1587945412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27b406c5c0 0x7f27b40ce180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 >> 172.21.15.67:0/1587945412 conn(0x7f27b40d5d90 msgr2=0x7f27b40b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 shutdown_connections 2024-04-25T15:28:06.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.195+0000 7f27b97b2640 1 -- 172.21.15.67:0/1587945412 wait complete. 2024-04-25T15:28:06.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:28:06.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.321+0000 7f57395ee640 1 Processor -- start 2024-04-25T15:28:06.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.321+0000 7f57395ee640 1 -- start start 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f57395ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f57340cec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f57395ee640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57340d2580 0x7f57340d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f57395ee640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f57340d2ed0 0x7f57340ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f57395ee640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f57340d32e0 con 0x7f57340d2580 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f57395ee640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f57340ce5d0 con 0x7f57340d2ed0 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f57395ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f57340cf3c0 con 0x7f57340ce1c0 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f572bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57340d2580 0x7f57340d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f572bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57340d2580 0x7f57340d2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39622/0 (socket says 172.21.15.67:39622) 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.322+0000 7f572bfff640 1 -- 172.21.15.67:0/1153487403 learned_addr learned my addr 172.21.15.67:0/1153487403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f5732ffd640 1 --2- 172.21.15.67:0/1153487403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f57340cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f57337fe640 1 --2- 172.21.15.67:0/1153487403 >> v2:172.21.15.176:3300/0 conn(0x7f57340d2ed0 0x7f57340ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f572bfff640 1 -- 172.21.15.67:0/1153487403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 msgr2=0x7f57340cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:06.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f572bfff640 1 --2- 172.21.15.67:0/1153487403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f57340cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f572bfff640 1 -- 172.21.15.67:0/1153487403 >> v2:172.21.15.176:3300/0 conn(0x7f57340d2ed0 msgr2=0x7f57340ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:06.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f572bfff640 1 --2- 172.21.15.67:0/1153487403 >> v2:172.21.15.176:3300/0 conn(0x7f57340d2ed0 0x7f57340ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f572bfff640 1 -- 172.21.15.67:0/1153487403 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57340cf6a0 con 0x7f57340d2580 2024-04-25T15:28:06.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f5732ffd640 1 --2- 172.21.15.67:0/1153487403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f57340cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:06.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.323+0000 7f572bfff640 1 --2- 172.21.15.67:0/1153487403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2580 0x7f57340d2990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5734074050 tx=0x7f571c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88c180ae8d054da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57327fc640 1 -- 172.21.15.67:0/1153487403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c037070 con 0x7f57340d2580 2024-04-25T15:28:06.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57327fc640 1 -- 172.21.15.67:0/1153487403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571c032440 con 0x7f57340d2580 2024-04-25T15:28:06.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57327fc640 1 -- 172.21.15.67:0/1153487403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c032810 con 0x7f57340d2580 2024-04-25T15:28:06.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 -- 172.21.15.67:0/1153487403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2580 msgr2=0x7f57340d2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 --2- 172.21.15.67:0/1153487403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2580 0x7f57340d2990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5734074050 tx=0x7f571c02a140 comp rx=0 tx=0).stop 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 -- 172.21.15.67:0/1153487403 shutdown_connections 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 --2- 172.21.15.67:0/1153487403 >> v2:172.21.15.176:3300/0 conn(0x7f57340d2ed0 0x7f57340ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 --2- 172.21.15.67:0/1153487403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2580 0x7f57340d2990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 --2- 172.21.15.67:0/1153487403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f57340cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 -- 172.21.15.67:0/1153487403 >> 172.21.15.67:0/1153487403 conn(0x7f573409b7d0 msgr2=0x7f573409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 -- 172.21.15.67:0/1153487403 shutdown_connections 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.324+0000 7f57395ee640 1 -- 172.21.15.67:0/1153487403 wait complete. 2024-04-25T15:28:06.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.326+0000 7f57395ee640 1 Processor -- start 2024-04-25T15:28:06.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.326+0000 7f57395ee640 1 -- start start 2024-04-25T15:28:06.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.327+0000 7f57395ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f573415f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.327+0000 7f57395ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57340d2580 0x7f5734155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.327+0000 7f57395ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2ed0 0x7f5734162730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.327+0000 7f57395ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f57340cd530 con 0x7f57340d2ed0 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.327+0000 7f57395ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f57340cfc60 con 0x7f57340d2580 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.327+0000 7f57395ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f57340cd6b0 con 0x7f57340ce1c0 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2ed0 0x7f5734162730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2ed0 0x7f5734162730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39628/0 (socket says 172.21.15.67:39628) 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 -- 172.21.15.67:0/4276239602 learned_addr learned my addr 172.21.15.67:0/4276239602 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f572bfff640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57340d2580 0x7f5734155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f5732ffd640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f573415f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 -- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 msgr2=0x7f573415f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f573415f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 -- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57340d2580 msgr2=0x7f5734155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57340d2580 0x7f5734155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 -- 172.21.15.67:0/4276239602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57340c6a80 con 0x7f57340d2ed0 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f572bfff640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57340d2580 0x7f5734155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57337fe640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2ed0 0x7f5734162730 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f572c00efc0 tx=0x7f572c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f572b7fe640 1 -- 172.21.15.67:0/4276239602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c018020 con 0x7f57340d2ed0 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f572b7fe640 1 -- 172.21.15.67:0/4276239602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f572c014480 con 0x7f57340d2ed0 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57340c6d60 con 0x7f57340d2ed0 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f572b7fe640 1 -- 172.21.15.67:0/4276239602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c009e60 con 0x7f57340d2ed0 2024-04-25T15:28:06.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.328+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57340c7320 con 0x7f57340d2ed0 2024-04-25T15:28:06.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.330+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5734050740 con 0x7f57340d2ed0 2024-04-25T15:28:06.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.330+0000 7f572b7fe640 1 -- 172.21.15.67:0/4276239602 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f572c005d10 con 0x7f57340d2ed0 2024-04-25T15:28:06.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.331+0000 7f572b7fe640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57140744b0 0x7f57140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.331+0000 7f572b7fe640 1 -- 172.21.15.67:0/4276239602 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f572c095f40 con 0x7f57340d2ed0 2024-04-25T15:28:06.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.331+0000 7f5732ffd640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57140744b0 0x7f57140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.331+0000 7f5732ffd640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57140744b0 0x7f57140769c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5734073a20 tx=0x7f5724008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.335+0000 7f572b7fe640 1 -- 172.21.15.67:0/4276239602 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f572c022030 con 0x7f57340d2ed0 2024-04-25T15:28:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.520+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f573406f210 con 0x7f57340d2ed0 2024-04-25T15:28:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.521+0000 7f572b7fe640 1 -- 172.21.15.67:0/4276239602 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f572c010040 con 0x7f57340d2ed0 2024-04-25T15:28:06.523 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:28:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57140744b0 msgr2=0x7f57140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57140744b0 0x7f57140769c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5734073a20 tx=0x7f5724008040 comp rx=0 tx=0).stop 2024-04-25T15:28:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2ed0 msgr2=0x7f5734162730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2ed0 0x7f5734162730 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f572c00efc0 tx=0x7f572c00c490 comp rx=0 tx=0).stop 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 shutdown_connections 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57140744b0 0x7f57140769c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57340d2ed0 0x7f5734162730 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57340d2580 0x7f5734155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 --2- 172.21.15.67:0/4276239602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57340ce1c0 0x7f573415f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 >> 172.21.15.67:0/4276239602 conn(0x7f573409b7d0 msgr2=0x7f57340b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 shutdown_connections 2024-04-25T15:28:06.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.523+0000 7f57395ee640 1 -- 172.21.15.67:0/4276239602 wait complete. 2024-04-25T15:28:06.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:28:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.649+0000 7f5119fab640 1 Processor -- start 2024-04-25T15:28:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.649+0000 7f5119fab640 1 -- start start 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.650+0000 7f5119fab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51140b8290 0x7f5114160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.650+0000 7f5119fab640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51141615f0 con 0x7f5114164b00 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.650+0000 7f5119fab640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5114164020 con 0x7f511415df80 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.650+0000 7f5119fab640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5114164670 con 0x7f51140b8290 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.651+0000 7f51137fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5114164b00 0x7f51140b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40846/0 (socket says 172.21.15.67:40846) 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.651+0000 7f51137fe640 1 -- 172.21.15.67:0/4202289898 learned_addr learned my addr 172.21.15.67:0/4202289898 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.651+0000 7f5112ffd640 1 --2- 172.21.15.67:0/4202289898 >> v2:172.21.15.67:3300/0 conn(0x7f51140b8290 0x7f5114160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.651+0000 7f5112ffd640 1 -- 172.21.15.67:0/4202289898 >> v1:172.21.15.176:6789/0 conn(0x7f511415df80 legacy=0x7f5114160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.651+0000 7f5112ffd640 1 -- 172.21.15.67:0/4202289898 >> v1:172.21.15.67:6789/0 conn(0x7f5114164b00 legacy=0x7f51140b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.651+0000 7f5112ffd640 1 -- 172.21.15.67:0/4202289898 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5114164300 con 0x7f51140b8290 2024-04-25T15:28:06.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.651+0000 7f51127fc640 1 -- 172.21.15.67:0/4202289898 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262786806 0 0) 0x7f51141615f0 con 0x7f5114164b00 2024-04-25T15:28:06.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.652+0000 7f5112ffd640 1 --2- 172.21.15.67:0/4202289898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51140b8290 0x7f5114160890 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f51141647f0 tx=0x7f510802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42761e4c27a6f0bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.652+0000 7f51127fc640 1 -- 172.21.15.67:0/4202289898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5108037070 con 0x7f51140b8290 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.652+0000 7f51127fc640 1 -- 172.21.15.67:0/4202289898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5108032440 con 0x7f51140b8290 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.652+0000 7f51127fc640 1 -- 172.21.15.67:0/4202289898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5108032810 con 0x7f51140b8290 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.652+0000 7f5119fab640 1 -- 172.21.15.67:0/4202289898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51140b8290 msgr2=0x7f5114160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.653+0000 7f5119fab640 1 --2- 172.21.15.67:0/4202289898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51140b8290 0x7f5114160890 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f51141647f0 tx=0x7f510802a210 comp rx=0 tx=0).stop 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.653+0000 7f5119fab640 1 -- 172.21.15.67:0/4202289898 shutdown_connections 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.653+0000 7f5119fab640 1 --2- 172.21.15.67:0/4202289898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51140b8290 0x7f5114160890 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.653+0000 7f5119fab640 1 -- 172.21.15.67:0/4202289898 >> 172.21.15.67:0/4202289898 conn(0x7f51140d2bc0 msgr2=0x7f51140d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.653+0000 7f5119fab640 1 -- 172.21.15.67:0/4202289898 shutdown_connections 2024-04-25T15:28:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.653+0000 7f5119fab640 1 -- 172.21.15.67:0/4202289898 wait complete. 2024-04-25T15:28:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.654+0000 7f5119fab640 1 Processor -- start 2024-04-25T15:28:06.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.655+0000 7f5119fab640 1 -- start start 2024-04-25T15:28:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5119fab640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f511415df80 0x7f511415a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5119fab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5114164b00 0x7f5114163290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5119fab640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5114157ce0 0x7f51141580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5119fab640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f51140b4c90 con 0x7f5114164b00 2024-04-25T15:28:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5119fab640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f51140b4b10 con 0x7f511415df80 2024-04-25T15:28:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5119fab640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f51140b4e10 con 0x7f5114157ce0 2024-04-25T15:28:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5114164b00 0x7f5114163290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5114164b00 0x7f5114163290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39656/0 (socket says 172.21.15.67:39656) 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 -- 172.21.15.67:0/2198088616 learned_addr learned my addr 172.21.15.67:0/2198088616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f51137fe640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f511415df80 0x7f511415a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5113fff640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5114157ce0 0x7f51141580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 -- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5114157ce0 msgr2=0x7f51141580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5114157ce0 0x7f51141580f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 -- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f511415df80 msgr2=0x7f511415a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f511415df80 0x7f511415a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.656+0000 7f5112ffd640 1 -- 172.21.15.67:0/2198088616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5114158630 con 0x7f5114164b00 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.657+0000 7f5112ffd640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5114164b00 0x7f5114163290 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5108000c00 tx=0x7f51080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.657+0000 7f50fbfff640 1 -- 172.21.15.67:0/2198088616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5108037050 con 0x7f5114164b00 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.657+0000 7f50fbfff640 1 -- 172.21.15.67:0/2198088616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5108038040 con 0x7f5114164b00 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.657+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51141f9e10 con 0x7f5114164b00 2024-04-25T15:28:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.657+0000 7f50fbfff640 1 -- 172.21.15.67:0/2198088616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5108004450 con 0x7f5114164b00 2024-04-25T15:28:06.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.657+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51141fa3d0 con 0x7f5114164b00 2024-04-25T15:28:06.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.658+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5114050740 con 0x7f5114164b00 2024-04-25T15:28:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.658+0000 7f50fbfff640 1 -- 172.21.15.67:0/2198088616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5108030900 con 0x7f5114164b00 2024-04-25T15:28:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.659+0000 7f50fbfff640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f51000744b0 0x7f51000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.659+0000 7f50fbfff640 1 -- 172.21.15.67:0/2198088616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f51080b5c30 con 0x7f5114164b00 2024-04-25T15:28:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.659+0000 7f51137fe640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f51000744b0 0x7f51000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.660+0000 7f51137fe640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f51000744b0 0x7f51000769c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5104000f60 tx=0x7f5104007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.663+0000 7f50fbfff640 1 -- 172.21.15.67:0/2198088616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f510804c020 con 0x7f5114164b00 2024-04-25T15:28:06.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.849+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f51140d4170 con 0x7f5114164b00 2024-04-25T15:28:06.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.849+0000 7f50fbfff640 1 -- 172.21.15.67:0/2198088616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f5114158630 con 0x7f5114164b00 2024-04-25T15:28:06.852 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f51000744b0 msgr2=0x7f51000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f51000744b0 0x7f51000769c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5104000f60 tx=0x7f5104007e40 comp rx=0 tx=0).stop 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5114164b00 msgr2=0x7f5114163290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5114164b00 0x7f5114163290 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5108000c00 tx=0x7f51080046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 shutdown_connections 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f51000744b0 0x7f51000769c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5114157ce0 0x7f51141580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5114164b00 0x7f5114163290 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 --2- 172.21.15.67:0/2198088616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f511415df80 0x7f511415a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.851+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 >> 172.21.15.67:0/2198088616 conn(0x7f51140d2bc0 msgr2=0x7f511415b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.852+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 shutdown_connections 2024-04-25T15:28:06.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.852+0000 7f5119fab640 1 -- 172.21.15.67:0/2198088616 wait complete. 2024-04-25T15:28:06.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:28:06.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.976+0000 7f8832106640 1 Processor -- start 2024-04-25T15:28:06.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.976+0000 7f8832106640 1 -- start start 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.977+0000 7f8832106640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f882c0d0e00 0x7f882c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.977+0000 7f8832106640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.977+0000 7f8832106640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f882c0b7810 0x7f882c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f8832106640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f882c06cf10 con 0x7f882c0b7810 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f8832106640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f882c0d1850 con 0x7f882c0d0e00 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f8832106640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f882c160de0 con 0x7f882c06c5c0 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f882c0b7810 0x7f882c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f882c0b7810 0x7f882c0b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39668/0 (socket says 172.21.15.67:39668) 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 -- 172.21.15.67:0/1380046926 learned_addr learned my addr 172.21.15.67:0/1380046926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882affd640 1 --2- 172.21.15.67:0/1380046926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882b7fe640 1 --2- 172.21.15.67:0/1380046926 >> v2:172.21.15.176:3300/0 conn(0x7f882c0d0e00 0x7f882c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 -- 172.21.15.67:0/1380046926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 msgr2=0x7f882c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 --2- 172.21.15.67:0/1380046926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 -- 172.21.15.67:0/1380046926 >> v2:172.21.15.176:3300/0 conn(0x7f882c0d0e00 msgr2=0x7f882c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 --2- 172.21.15.67:0/1380046926 >> v2:172.21.15.176:3300/0 conn(0x7f882c0d0e00 0x7f882c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.978+0000 7f882bfff640 1 -- 172.21.15.67:0/1380046926 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c1610c0 con 0x7f882c0b7810 2024-04-25T15:28:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.979+0000 7f882bfff640 1 --2- 172.21.15.67:0/1380046926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0b7810 0x7f882c0b7c20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f882c0d3e60 tx=0x7f882402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b5316afeaa61d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.979+0000 7f882a7fc640 1 -- 172.21.15.67:0/1380046926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8824036070 con 0x7f882c0b7810 2024-04-25T15:28:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.979+0000 7f882a7fc640 1 -- 172.21.15.67:0/1380046926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8824031600 con 0x7f882c0b7810 2024-04-25T15:28:06.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.979+0000 7f882a7fc640 1 -- 172.21.15.67:0/1380046926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88240319d0 con 0x7f882c0b7810 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 -- 172.21.15.67:0/1380046926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0b7810 msgr2=0x7f882c0b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 --2- 172.21.15.67:0/1380046926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0b7810 0x7f882c0b7c20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f882c0d3e60 tx=0x7f882402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 -- 172.21.15.67:0/1380046926 shutdown_connections 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 --2- 172.21.15.67:0/1380046926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0b7810 0x7f882c0b7c20 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 --2- 172.21.15.67:0/1380046926 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 --2- 172.21.15.67:0/1380046926 >> v2:172.21.15.176:3300/0 conn(0x7f882c0d0e00 0x7f882c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 -- 172.21.15.67:0/1380046926 >> 172.21.15.67:0/1380046926 conn(0x7f882c0d5d90 msgr2=0x7f882c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 -- 172.21.15.67:0/1380046926 shutdown_connections 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.980+0000 7f8832106640 1 -- 172.21.15.67:0/1380046926 wait complete. 2024-04-25T15:28:06.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.981+0000 7f8832106640 1 Processor -- start 2024-04-25T15:28:06.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.982+0000 7f8832106640 1 -- start start 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f8832106640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c0cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f8832106640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f882c0b7810 0x7f882c157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f8832106640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0d0e00 0x7f882c1628f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f8832106640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f882c0b5320 con 0x7f882c0d0e00 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f8832106640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f882c160ad0 con 0x7f882c0b7810 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f8832106640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f882c164e00 con 0x7f882c06c5c0 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0d0e00 0x7f882c1628f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0d0e00 0x7f882c1628f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39684/0 (socket says 172.21.15.67:39684) 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 -- 172.21.15.67:0/3405953454 learned_addr learned my addr 172.21.15.67:0/3405953454 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:06.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882b7fe640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c0cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882affd640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f882c0b7810 0x7f882c157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 -- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 msgr2=0x7f882c0cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c0cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 -- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f882c0b7810 msgr2=0x7f882c157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f882c0b7810 0x7f882c157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.983+0000 7f882bfff640 1 -- 172.21.15.67:0/3405953454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c0c9f30 con 0x7f882c0d0e00 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.984+0000 7f882bfff640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0d0e00 0x7f882c1628f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f882402ac90 tx=0x7f8824004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.984+0000 7f8813fff640 1 -- 172.21.15.67:0/3405953454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882403f020 con 0x7f882c0d0e00 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.984+0000 7f8813fff640 1 -- 172.21.15.67:0/3405953454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8824037040 con 0x7f882c0d0e00 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.984+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f882c0ca210 con 0x7f882c0d0e00 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.984+0000 7f8813fff640 1 -- 172.21.15.67:0/3405953454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8824036070 con 0x7f882c0d0e00 2024-04-25T15:28:06.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.984+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f882c15c8f0 con 0x7f882c0d0e00 2024-04-25T15:28:06.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.985+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f882c050740 con 0x7f882c0d0e00 2024-04-25T15:28:06.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.986+0000 7f8813fff640 1 -- 172.21.15.67:0/3405953454 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8824042020 con 0x7f882c0d0e00 2024-04-25T15:28:06.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.986+0000 7f8813fff640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88180744b0 0x7f88180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:06.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.986+0000 7f8813fff640 1 -- 172.21.15.67:0/3405953454 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f88240b58b0 con 0x7f882c0d0e00 2024-04-25T15:28:06.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.987+0000 7f882b7fe640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88180744b0 0x7f88180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:06.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.987+0000 7f882b7fe640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88180744b0 0x7f88180769c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f88200079a0 tx=0x7f8820008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:06.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:06.991+0000 7f8813fff640 1 -- 172.21.15.67:0/3405953454 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8824049020 con 0x7f882c0d0e00 2024-04-25T15:28:07.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.190+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f882c0c9cc0 con 0x7f882c0d0e00 2024-04-25T15:28:07.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.190+0000 7f8813fff640 1 -- 172.21.15.67:0/3405953454 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) ==== 80+0+0 (secure 0 0 0) 0x7f882407fa80 con 0x7f882c0d0e00 2024-04-25T15:28:07.192 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:28:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88180744b0 msgr2=0x7f88180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88180744b0 0x7f88180769c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f88200079a0 tx=0x7f8820008040 comp rx=0 tx=0).stop 2024-04-25T15:28:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0d0e00 msgr2=0x7f882c1628f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0d0e00 0x7f882c1628f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f882402ac90 tx=0x7f8824004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 shutdown_connections 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f88180744b0 0x7f88180769c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f882c0d0e00 0x7f882c1628f0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f882c0b7810 0x7f882c157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 --2- 172.21.15.67:0/3405953454 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f882c06c5c0 0x7f882c0cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 >> 172.21.15.67:0/3405953454 conn(0x7f882c0d5d90 msgr2=0x7f882c0d4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 shutdown_connections 2024-04-25T15:28:07.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.192+0000 7f8832106640 1 -- 172.21.15.67:0/3405953454 wait complete. 2024-04-25T15:28:07.205 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:28:07.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:07.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:07.236 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:28:07.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:07.236 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:28:07.292 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:07.293 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:07.293 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:07.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:28:07.309 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:07.309 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:07.309 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:07.349 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:28:07.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.460+0000 7f5c9fa67640 1 Processor -- start 2024-04-25T15:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.460+0000 7f5c9fa67640 1 -- start start 2024-04-25T15:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9fa67640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c980b8be0 0x7f5c980b9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9fa67640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5c980b4ff0 0x7f5c980b6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9fa67640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c980b6e50 con 0x7f5c980d04b0 2024-04-25T15:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9fa67640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5c980cee00 con 0x7f5c980b8be0 2024-04-25T15:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9fa67640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5c980b5400 con 0x7f5c980b4ff0 2024-04-25T15:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9d803640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c980b8be0 0x7f5c980b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9e004640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c980d04b0 0x7f5c980d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40854/0 (socket says 172.21.15.67:40854) 2024-04-25T15:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9e004640 1 -- 172.21.15.67:0/702035972 learned_addr learned my addr 172.21.15.67:0/702035972 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9d002640 1 --2- 172.21.15.67:0/702035972 >> v2:172.21.15.176:3300/0 conn(0x7f5c980b4ff0 0x7f5c980b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9d803640 1 -- 172.21.15.67:0/702035972 >> v2:172.21.15.176:3300/0 conn(0x7f5c980b4ff0 msgr2=0x7f5c980b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9d803640 1 --2- 172.21.15.67:0/702035972 >> v2:172.21.15.176:3300/0 conn(0x7f5c980b4ff0 0x7f5c980b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9d803640 1 -- 172.21.15.67:0/702035972 >> v1:172.21.15.67:6789/0 conn(0x7f5c980d04b0 legacy=0x7f5c980d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9d803640 1 -- 172.21.15.67:0/702035972 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c980d2720 con 0x7f5c980b8be0 2024-04-25T15:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.462+0000 7f5c9c801640 1 -- 172.21.15.67:0/702035972 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774371432 0 0) 0x7f5c980b6e50 con 0x7f5c980d04b0 2024-04-25T15:28:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.463+0000 7f5c9d803640 1 --2- 172.21.15.67:0/702035972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980b8be0 0x7f5c980b9110 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5c9808d0e0 tx=0x7f5c9002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc56aee2cc908b97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.463+0000 7f5c9c801640 1 -- 172.21.15.67:0/702035972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90037070 con 0x7f5c980b8be0 2024-04-25T15:28:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.463+0000 7f5c9c801640 1 -- 172.21.15.67:0/702035972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c90032440 con 0x7f5c980b8be0 2024-04-25T15:28:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.463+0000 7f5c9c801640 1 -- 172.21.15.67:0/702035972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90032810 con 0x7f5c980b8be0 2024-04-25T15:28:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 -- 172.21.15.67:0/702035972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980b8be0 msgr2=0x7f5c980b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/702035972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980b8be0 0x7f5c980b9110 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5c9808d0e0 tx=0x7f5c9002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 -- 172.21.15.67:0/702035972 shutdown_connections 2024-04-25T15:28:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/702035972 >> v2:172.21.15.176:3300/0 conn(0x7f5c980b4ff0 0x7f5c980b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/702035972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980b8be0 0x7f5c980b9110 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 -- 172.21.15.67:0/702035972 >> 172.21.15.67:0/702035972 conn(0x7f5c980d3ac0 msgr2=0x7f5c980d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 -- 172.21.15.67:0/702035972 shutdown_connections 2024-04-25T15:28:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.464+0000 7f5c9fa67640 1 -- 172.21.15.67:0/702035972 wait complete. 2024-04-25T15:28:07.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.466+0000 7f5c9fa67640 1 Processor -- start 2024-04-25T15:28:07.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.466+0000 7f5c9fa67640 1 -- start start 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.467+0000 7f5c9fa67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c980b4ff0 0x7f5c980cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.467+0000 7f5c9fa67640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c980b8be0 0x7f5c980cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.467+0000 7f5c9fa67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980d04b0 0x7f5c980ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9fa67640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c980d23d0 con 0x7f5c980d04b0 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9fa67640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c980cea90 con 0x7f5c980b8be0 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9fa67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c980ce740 con 0x7f5c980b4ff0 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980d04b0 0x7f5c980ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980d04b0 0x7f5c980ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39692/0 (socket says 172.21.15.67:39692) 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 -- 172.21.15.67:0/2704777356 learned_addr learned my addr 172.21.15.67:0/2704777356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9d002640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c980b8be0 0x7f5c980cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9d803640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c980b4ff0 0x7f5c980cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 -- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c980b4ff0 msgr2=0x7f5c980cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c980b4ff0 0x7f5c980cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 -- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c980b8be0 msgr2=0x7f5c980cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c980b8be0 0x7f5c980cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 -- 172.21.15.67:0/2704777356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9806e770 con 0x7f5c980d04b0 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c9e004640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980d04b0 0x7f5c980ccd60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800c700 tx=0x7f5c8800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.468+0000 7f5c967fc640 1 -- 172.21.15.67:0/2704777356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88018070 con 0x7f5c980d04b0 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.469+0000 7f5c967fc640 1 -- 172.21.15.67:0/2704777356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c8800ce10 con 0x7f5c980d04b0 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.469+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9806ea50 con 0x7f5c980d04b0 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.469+0000 7f5c967fc640 1 -- 172.21.15.67:0/2704777356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8800a880 con 0x7f5c980d04b0 2024-04-25T15:28:07.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.469+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9806f010 con 0x7f5c980d04b0 2024-04-25T15:28:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.470+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c98050740 con 0x7f5c980d04b0 2024-04-25T15:28:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.470+0000 7f5c967fc640 1 -- 172.21.15.67:0/2704777356 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5c8800aa20 con 0x7f5c980d04b0 2024-04-25T15:28:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.471+0000 7f5c967fc640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5c840744b0 0x7f5c840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.471+0000 7f5c967fc640 1 -- 172.21.15.67:0/2704777356 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f5c88094790 con 0x7f5c980d04b0 2024-04-25T15:28:07.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.471+0000 7f5c9d803640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5c840744b0 0x7f5c840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.472+0000 7f5c9d803640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5c840744b0 0x7f5c840769c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5c90034000 tx=0x7f5c9002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:07.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.476+0000 7f5c967fc640 1 -- 172.21.15.67:0/2704777356 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c8805e8c0 con 0x7f5c980d04b0 2024-04-25T15:28:07.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.699+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c9815f000 con 0x7f5c980d04b0 2024-04-25T15:28:07.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.699+0000 7f5c967fc640 1 -- 172.21.15.67:0/2704777356 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5189 (secure 0 0 0) 0x7f5c88062bc0 con 0x7f5c980d04b0 2024-04-25T15:28:07.701 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:07.701 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2076054232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2076054232},{"type":"v1","addr":"172.21.15.176:6836","nonce":2076054232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":5537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3930910548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3930910548},{"type":"v1","addr":"172.21.15.176:6837","nonce":3930910548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2322157634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322157634},{"type":"v1","addr":"172.21.15.67:6834","nonce":2322157634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:47.123622+0000","modified":"2024-04-25T15:27:50.154657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6273":{"gid":6273,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3903876028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3903876028},{"type":"v1","addr":"172.21.15.67:6835","nonce":3903876028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6273,"qdb_cluster":[6273]},"id":5}]} 2024-04-25T15:28:07.702 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-25T15:28:07.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5c840744b0 msgr2=0x7f5c840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5c840744b0 0x7f5c840769c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5c90034000 tx=0x7f5c9002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:07.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980d04b0 msgr2=0x7f5c980ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980d04b0 0x7f5c980ccd60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800c700 tx=0x7f5c8800cbc0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 shutdown_connections 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5c840744b0 0x7f5c840769c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c980d04b0 0x7f5c980ccd60 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c980b8be0 0x7f5c980cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 --2- 172.21.15.67:0/2704777356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c980b4ff0 0x7f5c980cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 >> 172.21.15.67:0/2704777356 conn(0x7f5c980d3ac0 msgr2=0x7f5c98071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 shutdown_connections 2024-04-25T15:28:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.701+0000 7f5c9fa67640 1 -- 172.21.15.67:0/2704777356 wait complete. 2024-04-25T15:28:07.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:07.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.825+0000 7fa3ce984640 1 Processor -- start 2024-04-25T15:28:07.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.825+0000 7fa3ce984640 1 -- start start 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.826+0000 7fa3ce984640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81612f0 0x7fa3c80d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.826+0000 7fa3ce984640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c806f3e0 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.826+0000 7fa3ce984640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c806f6c0 con 0x7fa3c81434d0 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.826+0000 7fa3ce984640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c81438e0 con 0x7fa3c81612f0 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3c77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3c80d0ed0 0x7fa3c8160be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52074/0 (socket says 172.21.15.67:52074) 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3c77fe640 1 -- 172.21.15.67:0/722985345 learned_addr learned my addr 172.21.15.67:0/722985345 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3ccf21640 1 --2- 172.21.15.67:0/722985345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81612f0 0x7fa3c80d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435532512 0 0) 0x7fa3c806f3e0 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3b4003520 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1854200861 0 0) 0x7fa3c806f6c0 con 0x7fa3c81434d0 2024-04-25T15:28:07.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3c806f3e0 con 0x7fa3c81434d0 2024-04-25T15:28:07.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3449312786 0 0) 0x7fa3b4003520 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3ccf21640 1 -- 172.21.15.67:0/722985345 >> v1:172.21.15.176:6789/0 conn(0x7fa3c81434d0 legacy=0x7fa3c8070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3ccf21640 1 -- 172.21.15.67:0/722985345 >> v1:172.21.15.67:6789/0 conn(0x7fa3c80d0ed0 legacy=0x7fa3c8160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.827+0000 7fa3ccf21640 1 -- 172.21.15.67:0/722985345 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c806f9a0 con 0x7fa3c81612f0 2024-04-25T15:28:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.828+0000 7fa3ccf21640 1 --2- 172.21.15.67:0/722985345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81612f0 0x7fa3c80d12e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009dc0 tx=0x7fa3c002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=443a564e93086ac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.828+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0001670 con 0x7fa3c81612f0 2024-04-25T15:28:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.828+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c0007540 con 0x7fa3c81612f0 2024-04-25T15:28:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.828+0000 7fa3c6ffd640 1 -- 172.21.15.67:0/722985345 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0007910 con 0x7fa3c81612f0 2024-04-25T15:28:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.829+0000 7fa3ce984640 1 -- 172.21.15.67:0/722985345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81612f0 msgr2=0x7fa3c80d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.829+0000 7fa3ce984640 1 --2- 172.21.15.67:0/722985345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81612f0 0x7fa3c80d12e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009dc0 tx=0x7fa3c002a660 comp rx=0 tx=0).stop 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.829+0000 7fa3ce984640 1 -- 172.21.15.67:0/722985345 shutdown_connections 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.829+0000 7fa3ce984640 1 --2- 172.21.15.67:0/722985345 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81612f0 0x7fa3c80d12e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.829+0000 7fa3ce984640 1 -- 172.21.15.67:0/722985345 >> 172.21.15.67:0/722985345 conn(0x7fa3c80d1ae0 msgr2=0x7fa3c80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.829+0000 7fa3ce984640 1 -- 172.21.15.67:0/722985345 shutdown_connections 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.829+0000 7fa3ce984640 1 -- 172.21.15.67:0/722985345 wait complete. 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.831+0000 7fa3ce984640 1 Processor -- start 2024-04-25T15:28:07.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.831+0000 7fa3ce984640 1 -- start start 2024-04-25T15:28:07.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3ce984640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c80d0ed0 0x7fa3c80aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3ce984640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81434d0 0x7fa3c81f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3ce984640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c81f5330 0x7fa3c806ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3ce984640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c80b51a0 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3ce984640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c806f090 con 0x7fa3c81f5330 2024-04-25T15:28:07.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3ce984640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c80b5320 con 0x7fa3c81434d0 2024-04-25T15:28:07.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3c7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c80d0ed0 0x7fa3c80aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3c7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c80d0ed0 0x7fa3c80aa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41300/0 (socket says 172.21.15.67:41300) 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3c7fff640 1 -- 172.21.15.67:0/2136658264 learned_addr learned my addr 172.21.15.67:0/2136658264 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.832+0000 7fa3c77fe640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81434d0 0x7fa3c81f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3ccf21640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c81f5330 0x7fa3c806ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c7fff640 1 -- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81434d0 msgr2=0x7fa3c81f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c7fff640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81434d0 0x7fa3c81f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c7fff640 1 -- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c81f5330 msgr2=0x7fa3c806ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c7fff640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c81f5330 0x7fa3c806ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c7fff640 1 -- 172.21.15.67:0/2136658264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c80aaa80 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c7fff640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c80d0ed0 0x7fa3c80aa230 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800c460 tx=0x7fa3b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c4ff9640 1 -- 172.21.15.67:0/2136658264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8017070 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c4ff9640 1 -- 172.21.15.67:0/2136658264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b80052f0 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c80aad60 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3c4ff9640 1 -- 172.21.15.67:0/2136658264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8003830 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.833+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c81f7090 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.834+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3c80b4a00 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.835+0000 7fa3c4ff9640 1 -- 172.21.15.67:0/2136658264 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa3b801a070 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.835+0000 7fa3c4ff9640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3a80744b0 0x7fa3a80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.835+0000 7fa3c4ff9640 1 -- 172.21.15.67:0/2136658264 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fa3b80946e0 con 0x7fa3c80d0ed0 2024-04-25T15:28:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.835+0000 7fa3c77fe640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3a80744b0 0x7fa3a80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:07.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.836+0000 7fa3c77fe640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3a80744b0 0x7fa3a80769c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa3c81f22d0 tx=0x7fa3b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:07.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:07.840+0000 7fa3c4ff9640 1 -- 172.21.15.67:0/2136658264 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3c80b4a00 con 0x7fa3c80d0ed0 2024-04-25T15:28:08.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.025+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3c81f72f0 con 0x7fa3c80d0ed0 2024-04-25T15:28:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.026+0000 7fa3c4ff9640 1 -- 172.21.15.67:0/2136658264 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16632 (secure 0 0 0) 0x7fa3b8065e20 con 0x7fa3c80d0ed0 2024-04-25T15:28:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:08.028 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:48.764450+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3a80744b0 msgr2=0x7fa3a80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3a80744b0 0x7fa3a80769c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa3c81f22d0 tx=0x7fa3b0009040 comp rx=0 tx=0).stop 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c80d0ed0 msgr2=0x7fa3c80aa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c80d0ed0 0x7fa3c80aa230 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800c460 tx=0x7fa3b800c920 comp rx=0 tx=0).stop 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 shutdown_connections 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa3a80744b0 0x7fa3a80769c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c81f5330 0x7fa3c806ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c81434d0 0x7fa3c81f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 --2- 172.21.15.67:0/2136658264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c80d0ed0 0x7fa3c80aa230 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 >> 172.21.15.67:0/2136658264 conn(0x7fa3c80d1ae0 msgr2=0x7fa3c815cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 shutdown_connections 2024-04-25T15:28:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.028+0000 7fa3ce984640 1 -- 172.21.15.67:0/2136658264 wait complete. 2024-04-25T15:28:08.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:27:45.453204+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:27:46.073423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:08.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.154+0000 7fcc31049640 1 Processor -- start 2024-04-25T15:28:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.154+0000 7fcc31049640 1 -- start start 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.155+0000 7fcc31049640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc2c0b9270 0x7fcc2c0d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc31049640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcc2c162a10 0x7fcc2c0b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc31049640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc2c0cf810 con 0x7fcc2c161ef0 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc31049640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcc2c162e20 con 0x7fcc2c0b9270 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc31049640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcc2c0b0690 con 0x7fcc2c162a10 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc2c0b9270 0x7fcc2c0d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc2c161ef0 0x7fcc2c162300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52090/0 (socket says 172.21.15.67:52090) 2024-04-25T15:28:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2a59c640 1 -- 172.21.15.67:0/3916536276 learned_addr learned my addr 172.21.15.67:0/3916536276 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2b59e640 1 --2- 172.21.15.67:0/3916536276 >> v2:172.21.15.176:3300/0 conn(0x7fcc2c162a10 0x7fcc2c0b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2ad9d640 1 -- 172.21.15.67:0/3916536276 >> v2:172.21.15.176:3300/0 conn(0x7fcc2c162a10 msgr2=0x7fcc2c0b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2ad9d640 1 --2- 172.21.15.67:0/3916536276 >> v2:172.21.15.176:3300/0 conn(0x7fcc2c162a10 0x7fcc2c0b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2ad9d640 1 -- 172.21.15.67:0/3916536276 >> v1:172.21.15.67:6789/0 conn(0x7fcc2c161ef0 legacy=0x7fcc2c162300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc2ad9d640 1 -- 172.21.15.67:0/3916536276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c0cfab0 con 0x7fcc2c0b9270 2024-04-25T15:28:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.156+0000 7fcc29d9b640 1 -- 172.21.15.67:0/3916536276 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 605202257 0 0) 0x7fcc2c0cf810 con 0x7fcc2c161ef0 2024-04-25T15:28:08.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.157+0000 7fcc2ad9d640 1 --2- 172.21.15.67:0/3916536276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c0b9270 0x7fcc2c0d15e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c070930 tx=0x7fcc2402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c16c633660255e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.157+0000 7fcc29d9b640 1 -- 172.21.15.67:0/3916536276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24037070 con 0x7fcc2c0b9270 2024-04-25T15:28:08.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.157+0000 7fcc29d9b640 1 -- 172.21.15.67:0/3916536276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc2402add0 con 0x7fcc2c0b9270 2024-04-25T15:28:08.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.157+0000 7fcc29d9b640 1 -- 172.21.15.67:0/3916536276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24032660 con 0x7fcc2c0b9270 2024-04-25T15:28:08.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 -- 172.21.15.67:0/3916536276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c0b9270 msgr2=0x7fcc2c0d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 --2- 172.21.15.67:0/3916536276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c0b9270 0x7fcc2c0d15e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c070930 tx=0x7fcc2402a080 comp rx=0 tx=0).stop 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 -- 172.21.15.67:0/3916536276 shutdown_connections 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 --2- 172.21.15.67:0/3916536276 >> v2:172.21.15.176:3300/0 conn(0x7fcc2c162a10 0x7fcc2c0b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 --2- 172.21.15.67:0/3916536276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c0b9270 0x7fcc2c0d15e0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 -- 172.21.15.67:0/3916536276 >> 172.21.15.67:0/3916536276 conn(0x7fcc2c0d3aa0 msgr2=0x7fcc2c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 -- 172.21.15.67:0/3916536276 shutdown_connections 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.158+0000 7fcc31049640 1 -- 172.21.15.67:0/3916536276 wait complete. 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.160+0000 7fcc31049640 1 Processor -- start 2024-04-25T15:28:08.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.160+0000 7fcc31049640 1 -- start start 2024-04-25T15:28:08.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc31049640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc2c0b9270 0x7fcc2c0cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc31049640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c161ef0 0x7fcc2c0cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc31049640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc2c162a10 0x7fcc2c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc31049640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c0b1530 con 0x7fcc2c161ef0 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc31049640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c0b13b0 con 0x7fcc2c0b9270 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc31049640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2c0b16b0 con 0x7fcc2c162a10 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc2a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c161ef0 0x7fcc2c0cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc2a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c161ef0 0x7fcc2c0cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41328/0 (socket says 172.21.15.67:41328) 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.161+0000 7fcc2a59c640 1 -- 172.21.15.67:0/3111058748 learned_addr learned my addr 172.21.15.67:0/3111058748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2ad9d640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc2c0b9270 0x7fcc2c0cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2b59e640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc2c162a10 0x7fcc2c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2a59c640 1 -- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc2c162a10 msgr2=0x7fcc2c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2a59c640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc2c162a10 0x7fcc2c06e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2a59c640 1 -- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc2c0b9270 msgr2=0x7fcc2c0cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2a59c640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc2c0b9270 0x7fcc2c0cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2a59c640 1 -- 172.21.15.67:0/3111058748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c06f630 con 0x7fcc2c161ef0 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2b59e640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc2c162a10 0x7fcc2c06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc2a59c640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c161ef0 0x7fcc2c0cd720 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400c410 tx=0x7fcc1400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc237fe640 1 -- 172.21.15.67:0/3111058748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14017070 con 0x7fcc2c161ef0 2024-04-25T15:28:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc237fe640 1 -- 172.21.15.67:0/3111058748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc14005ce0 con 0x7fcc2c161ef0 2024-04-25T15:28:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2c1e7fd0 con 0x7fcc2c161ef0 2024-04-25T15:28:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc237fe640 1 -- 172.21.15.67:0/3111058748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc140072f0 con 0x7fcc2c161ef0 2024-04-25T15:28:08.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.162+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2c1e8590 con 0x7fcc2c161ef0 2024-04-25T15:28:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.163+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2c050740 con 0x7fcc2c161ef0 2024-04-25T15:28:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.164+0000 7fcc237fe640 1 -- 172.21.15.67:0/3111058748 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcc1401a070 con 0x7fcc2c161ef0 2024-04-25T15:28:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.164+0000 7fcc237fe640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc0c0744b0 0x7fcc0c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.164+0000 7fcc237fe640 1 -- 172.21.15.67:0/3111058748 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fcc14094570 con 0x7fcc2c161ef0 2024-04-25T15:28:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.165+0000 7fcc2ad9d640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc0c0744b0 0x7fcc0c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.165+0000 7fcc2ad9d640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc0c0744b0 0x7fcc0c0769c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcc24034000 tx=0x7fcc2402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.169+0000 7fcc237fe640 1 -- 172.21.15.67:0/3111058748 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc2c06f630 con 0x7fcc2c161ef0 2024-04-25T15:28:08.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.353+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc2c0d4770 con 0x7fcc2c161ef0 2024-04-25T15:28:08.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.353+0000 7fcc237fe640 1 -- 172.21.15.67:0/3111058748 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) ==== 93+0+10 (secure 0 0 0) 0x7fcc1405ed90 con 0x7fcc2c161ef0 2024-04-25T15:28:08.355 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:08.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc0c0744b0 msgr2=0x7fcc0c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc0c0744b0 0x7fcc0c0769c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcc24034000 tx=0x7fcc2402e000 comp rx=0 tx=0).stop 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c161ef0 msgr2=0x7fcc2c0cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c161ef0 0x7fcc2c0cd720 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400c410 tx=0x7fcc1400c8d0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 shutdown_connections 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc0c0744b0 0x7fcc0c0769c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc2c162a10 0x7fcc2c06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc2c161ef0 0x7fcc2c0cd720 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 --2- 172.21.15.67:0/3111058748 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc2c0b9270 0x7fcc2c0cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 >> 172.21.15.67:0/3111058748 conn(0x7fcc2c0d3aa0 msgr2=0x7fcc2c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 shutdown_connections 2024-04-25T15:28:08.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.355+0000 7fcc31049640 1 -- 172.21.15.67:0/3111058748 wait complete. 2024-04-25T15:28:08.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:28:08.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.481+0000 7f35f496c640 1 Processor -- start 2024-04-25T15:28:08.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.481+0000 7f35f496c640 1 -- start start 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35f496c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35f008dec0 0x7f35f00b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35f496c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35f496c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35f00d21e0 con 0x7f35f008cf20 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35f496c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f35f00b95f0 con 0x7f35f008dec0 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35f496c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f35f00d1940 con 0x7f35f00b1c50 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35ee59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35f008dec0 0x7f35f00b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35ee59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35f008dec0 0x7f35f00b90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41336/0 (socket says 172.21.15.67:41336) 2024-04-25T15:28:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.482+0000 7f35ee59c640 1 -- 172.21.15.67:0/1545878399 learned_addr learned my addr 172.21.15.67:0/1545878399 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.483+0000 7f35eed9d640 1 --2- 172.21.15.67:0/1545878399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.483+0000 7f35ee59c640 1 -- 172.21.15.67:0/1545878399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 msgr2=0x7f35f00ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.483+0000 7f35ee59c640 1 --2- 172.21.15.67:0/1545878399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.483+0000 7f35ee59c640 1 -- 172.21.15.67:0/1545878399 >> v1:172.21.15.176:6789/0 conn(0x7f35f008cf20 legacy=0x7f35f008d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.483+0000 7f35ee59c640 1 -- 172.21.15.67:0/1545878399 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f00d1c20 con 0x7f35f008dec0 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.483+0000 7f35eed9d640 1 --2- 172.21.15.67:0/1545878399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.483+0000 7f35ee59c640 1 --2- 172.21.15.67:0/1545878399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008dec0 0x7f35f00b90b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f35f008cef0 tx=0x7f35e802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21beb2a6d68b7d8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35ed59a640 1 -- 172.21.15.67:0/1545878399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8036070 con 0x7f35f008dec0 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35ed59a640 1 -- 172.21.15.67:0/1545878399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e8031600 con 0x7f35f008dec0 2024-04-25T15:28:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35ed59a640 1 -- 172.21.15.67:0/1545878399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e80319d0 con 0x7f35f008dec0 2024-04-25T15:28:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 -- 172.21.15.67:0/1545878399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008dec0 msgr2=0x7f35f00b90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 --2- 172.21.15.67:0/1545878399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008dec0 0x7f35f00b90b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f35f008cef0 tx=0x7f35e802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 -- 172.21.15.67:0/1545878399 shutdown_connections 2024-04-25T15:28:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 --2- 172.21.15.67:0/1545878399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 --2- 172.21.15.67:0/1545878399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008dec0 0x7f35f00b90b0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 -- 172.21.15.67:0/1545878399 >> 172.21.15.67:0/1545878399 conn(0x7f35f00d0510 msgr2=0x7f35f00d0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 -- 172.21.15.67:0/1545878399 shutdown_connections 2024-04-25T15:28:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.484+0000 7f35f496c640 1 -- 172.21.15.67:0/1545878399 wait complete. 2024-04-25T15:28:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.486+0000 7f35f496c640 1 Processor -- start 2024-04-25T15:28:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.486+0000 7f35f496c640 1 -- start start 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35f496c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008cf20 0x7f35f01592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35f496c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35f008dec0 0x7f35f00c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35f496c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35f496c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35f00d15d0 con 0x7f35f008cf20 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35f496c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f35f00b0720 con 0x7f35f008dec0 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35f496c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f35f00cedc0 con 0x7f35f00b1c50 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008cf20 0x7f35f01592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008cf20 0x7f35f01592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41340/0 (socket says 172.21.15.67:41340) 2024-04-25T15:28:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 -- 172.21.15.67:0/2188003449 learned_addr learned my addr 172.21.15.67:0/2188003449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35eed9d640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35edd9b640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35f008dec0 0x7f35f00c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 -- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 msgr2=0x7f35f00cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 -- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35f008dec0 msgr2=0x7f35f00c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35f008dec0 0x7f35f00c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 -- 172.21.15.67:0/2188003449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f01612a0 con 0x7f35f008cf20 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35eed9d640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35edd9b640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35f008dec0 0x7f35f00c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35ee59c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008cf20 0x7f35f01592a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009f40 tx=0x7f35e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35e6ffd640 1 -- 172.21.15.67:0/2188003449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8041020 con 0x7f35f008cf20 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35e6ffd640 1 -- 172.21.15.67:0/2188003449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e8007800 con 0x7f35f008cf20 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35e6ffd640 1 -- 172.21.15.67:0/2188003449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8036040 con 0x7f35f008cf20 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.488+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f0160a00 con 0x7f35f008cf20 2024-04-25T15:28:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.489+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f0160fc0 con 0x7f35f008cf20 2024-04-25T15:28:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.489+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f0050740 con 0x7f35f008cf20 2024-04-25T15:28:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.490+0000 7f35e6ffd640 1 -- 172.21.15.67:0/2188003449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f35e8044020 con 0x7f35f008cf20 2024-04-25T15:28:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.491+0000 7f35e6ffd640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35d00744e0 0x7f35d00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.491+0000 7f35e6ffd640 1 -- 172.21.15.67:0/2188003449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f35e80b6580 con 0x7f35f008cf20 2024-04-25T15:28:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.491+0000 7f35edd9b640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35d00744e0 0x7f35d00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.492+0000 7f35edd9b640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35d00744e0 0x7f35d00769f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f35f00b18d0 tx=0x7f35d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.495+0000 7f35e6ffd640 1 -- 172.21.15.67:0/2188003449 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35e804b070 con 0x7f35f008cf20 2024-04-25T15:28:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.678+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35f00b49c0 con 0x7f35f008cf20 2024-04-25T15:28:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.679+0000 7f35e6ffd640 1 -- 172.21.15.67:0/2188003449 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) ==== 104+0+11 (secure 0 0 0) 0x7f35e8080da0 con 0x7f35f008cf20 2024-04-25T15:28:08.681 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.680+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35d00744e0 msgr2=0x7f35d00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.680+0000 7f35f496c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35d00744e0 0x7f35d00769f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f35f00b18d0 tx=0x7f35d8008040 comp rx=0 tx=0).stop 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008cf20 msgr2=0x7f35f01592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008cf20 0x7f35f01592a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009f40 tx=0x7f35e8004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 shutdown_connections 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35d00744e0 0x7f35d00769f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35f00b1c50 0x7f35f00cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35f008dec0 0x7f35f00c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 --2- 172.21.15.67:0/2188003449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35f008cf20 0x7f35f01592a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 >> 172.21.15.67:0/2188003449 conn(0x7f35f00d0510 msgr2=0x7f35f0072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 shutdown_connections 2024-04-25T15:28:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.681+0000 7f35f496c640 1 -- 172.21.15.67:0/2188003449 wait complete. 2024-04-25T15:28:08.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:28:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.806+0000 7f9e376b1640 1 Processor -- start 2024-04-25T15:28:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.806+0000 7f9e376b1640 1 -- start start 2024-04-25T15:28:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e376b1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9e3006ecd0 0x7f9e300b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e376b1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e300b3f30 con 0x7f9e30160640 2024-04-25T15:28:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e376b1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e300b4210 con 0x7f9e300d0190 2024-04-25T15:28:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e376b1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9e300b23e0 con 0x7f9e3006ecd0 2024-04-25T15:28:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e35c4e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9e30160640 0x7f9e300b2e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52094/0 (socket says 172.21.15.67:52094) 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e35c4e640 1 -- 172.21.15.67:0/2035462350 learned_addr learned my addr 172.21.15.67:0/2035462350 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e34c4c640 1 --2- 172.21.15.67:0/2035462350 >> v2:172.21.15.176:3300/0 conn(0x7f9e3006ecd0 0x7f9e300b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e27fff640 1 -- 172.21.15.67:0/2035462350 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2681353083 0 0) 0x7f9e300b3f30 con 0x7f9e30160640 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.808+0000 7f9e27fff640 1 -- 172.21.15.67:0/2035462350 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e1c003520 con 0x7f9e30160640 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.809+0000 7f9e34c4c640 1 -- 172.21.15.67:0/2035462350 >> v1:172.21.15.176:6789/0 conn(0x7f9e300d0190 legacy=0x7f9e300d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.809+0000 7f9e27fff640 1 -- 172.21.15.67:0/2035462350 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048830016 0 0) 0x7f9e300b4210 con 0x7f9e300d0190 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.809+0000 7f9e34c4c640 1 -- 172.21.15.67:0/2035462350 >> v1:172.21.15.67:6789/0 conn(0x7f9e30160640 legacy=0x7f9e300b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.809+0000 7f9e34c4c640 1 -- 172.21.15.67:0/2035462350 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e300b2630 con 0x7f9e3006ecd0 2024-04-25T15:28:08.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.809+0000 7f9e34c4c640 1 --2- 172.21.15.67:0/2035462350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e3006ecd0 0x7f9e300b77e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9e200097a0 tx=0x7f9e2002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=399fec35fe1aa804 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e27fff640 1 -- 172.21.15.67:0/2035462350 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20037070 con 0x7f9e3006ecd0 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e27fff640 1 -- 172.21.15.67:0/2035462350 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e20032440 con 0x7f9e3006ecd0 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e27fff640 1 -- 172.21.15.67:0/2035462350 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2002ae60 con 0x7f9e3006ecd0 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e376b1640 1 -- 172.21.15.67:0/2035462350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e3006ecd0 msgr2=0x7f9e300b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e376b1640 1 --2- 172.21.15.67:0/2035462350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e3006ecd0 0x7f9e300b77e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9e200097a0 tx=0x7f9e2002a110 comp rx=0 tx=0).stop 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e376b1640 1 -- 172.21.15.67:0/2035462350 shutdown_connections 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e376b1640 1 --2- 172.21.15.67:0/2035462350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e3006ecd0 0x7f9e300b77e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e376b1640 1 -- 172.21.15.67:0/2035462350 >> 172.21.15.67:0/2035462350 conn(0x7f9e3009b7d0 msgr2=0x7f9e3009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e376b1640 1 -- 172.21.15.67:0/2035462350 shutdown_connections 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.810+0000 7f9e376b1640 1 -- 172.21.15.67:0/2035462350 wait complete. 2024-04-25T15:28:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.812+0000 7f9e376b1640 1 Processor -- start 2024-04-25T15:28:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.812+0000 7f9e376b1640 1 -- start start 2024-04-25T15:28:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e376b1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e3006ecd0 0x7f9e300c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e376b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e300d0190 0x7f9e3014f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e376b1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e30160640 0x7f9e3014fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e376b1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e300b3cd0 con 0x7f9e3006ecd0 2024-04-25T15:28:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e376b1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9e300b3b50 con 0x7f9e30160640 2024-04-25T15:28:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e376b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9e300b3540 con 0x7f9e300d0190 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e3544d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e3006ecd0 0x7f9e300c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e3544d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e3006ecd0 0x7f9e300c9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41342/0 (socket says 172.21.15.67:41342) 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.813+0000 7f9e3544d640 1 -- 172.21.15.67:0/1118395586 learned_addr learned my addr 172.21.15.67:0/1118395586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e34c4c640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e300d0190 0x7f9e3014f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e35c4e640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e30160640 0x7f9e3014fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e3544d640 1 -- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e300d0190 msgr2=0x7f9e3014f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e3544d640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e300d0190 0x7f9e3014f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e3544d640 1 -- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e30160640 msgr2=0x7f9e3014fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e3544d640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e30160640 0x7f9e3014fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e3544d640 1 -- 172.21.15.67:0/1118395586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e300ca5f0 con 0x7f9e3006ecd0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e3544d640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e3006ecd0 0x7f9e300c9b30 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800c3e0 tx=0x7f9e2800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e25ffb640 1 -- 172.21.15.67:0/1118395586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28017070 con 0x7f9e3006ecd0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e25ffb640 1 -- 172.21.15.67:0/1118395586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e280035f0 con 0x7f9e3006ecd0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e25ffb640 1 -- 172.21.15.67:0/1118395586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e280039c0 con 0x7f9e3006ecd0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e300ca8d0 con 0x7f9e3006ecd0 2024-04-25T15:28:08.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.814+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e300cae90 con 0x7f9e3006ecd0 2024-04-25T15:28:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.815+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e30050740 con 0x7f9e3006ecd0 2024-04-25T15:28:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.816+0000 7f9e25ffb640 1 -- 172.21.15.67:0/1118395586 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9e2801a070 con 0x7f9e3006ecd0 2024-04-25T15:28:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.816+0000 7f9e25ffb640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e140744e0 0x7f9e140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:08.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.816+0000 7f9e25ffb640 1 -- 172.21.15.67:0/1118395586 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f9e28094540 con 0x7f9e3006ecd0 2024-04-25T15:28:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.817+0000 7f9e34c4c640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e140744e0 0x7f9e140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:08.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.817+0000 7f9e34c4c640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e140744e0 0x7f9e140769f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9e20034000 tx=0x7f9e2002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:08.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:08.821+0000 7f9e25ffb640 1 -- 172.21.15.67:0/1118395586 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e2805e690 con 0x7f9e3006ecd0 2024-04-25T15:28:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.008+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9e300cf560 con 0x7f9e3006ecd0 2024-04-25T15:28:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.008+0000 7f9e25ffb640 1 -- 172.21.15.67:0/1118395586 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) ==== 100+0+11 (secure 0 0 0) 0x7f9e28062990 con 0x7f9e3006ecd0 2024-04-25T15:28:09.010 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:28:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.010+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e140744e0 msgr2=0x7f9e140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.010+0000 7f9e376b1640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e140744e0 0x7f9e140769f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9e20034000 tx=0x7f9e2002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.010+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e3006ecd0 msgr2=0x7f9e300c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.010+0000 7f9e376b1640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e3006ecd0 0x7f9e300c9b30 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800c3e0 tx=0x7f9e2800c8a0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 shutdown_connections 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9e140744e0 0x7f9e140769f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e30160640 0x7f9e3014fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e300d0190 0x7f9e3014f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 --2- 172.21.15.67:0/1118395586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e3006ecd0 0x7f9e300c9b30 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 >> 172.21.15.67:0/1118395586 conn(0x7f9e3009b7d0 msgr2=0x7f9e300717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 shutdown_connections 2024-04-25T15:28:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.011+0000 7f9e376b1640 1 -- 172.21.15.67:0/1118395586 wait complete. 2024-04-25T15:28:09.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:09.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.134+0000 7f91b3234640 1 Processor -- start 2024-04-25T15:28:09.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.134+0000 7f91b3234640 1 -- start start 2024-04-25T15:28:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91b3234640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91ac0d0e00 0x7f91ac0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91b3234640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91ac0d1850 con 0x7f91ac06c5c0 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91b3234640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91ac160de0 con 0x7f91ac0d4a50 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91b3234640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f91ac1610c0 con 0x7f91ac0d0e00 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91b0fd0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91ac0d0e00 0x7f91ac0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91abfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f91ac06c5c0 0x7f91ac06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52098/0 (socket says 172.21.15.67:52098) 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91b0fd0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91ac0d0e00 0x7f91ac0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41358/0 (socket says 172.21.15.67:41358) 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.135+0000 7f91abfff640 1 -- 172.21.15.67:0/1910971842 learned_addr learned my addr 172.21.15.67:0/1910971842 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.136+0000 7f91b0fd0640 1 -- 172.21.15.67:0/1910971842 >> v1:172.21.15.176:6789/0 conn(0x7f91ac0d4a50 legacy=0x7f91ac0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.136+0000 7f91b0fd0640 1 -- 172.21.15.67:0/1910971842 >> v1:172.21.15.67:6789/0 conn(0x7f91ac06c5c0 legacy=0x7f91ac06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.136+0000 7f91ab7fe640 1 -- 172.21.15.67:0/1910971842 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1796809170 0 0) 0x7f91ac0d1850 con 0x7f91ac06c5c0 2024-04-25T15:28:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.136+0000 7f91b0fd0640 1 -- 172.21.15.67:0/1910971842 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ac1613a0 con 0x7f91ac0d0e00 2024-04-25T15:28:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.136+0000 7f91b0fd0640 1 --2- 172.21.15.67:0/1910971842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d0e00 0x7f91ac0d1210 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f91a00097a0 tx=0x7f91a002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10ee2250c09b4a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91ab7fe640 1 -- 172.21.15.67:0/1910971842 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0037070 con 0x7f91ac0d0e00 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91ab7fe640 1 -- 172.21.15.67:0/1910971842 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a002ae50 con 0x7f91ac0d0e00 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91ab7fe640 1 -- 172.21.15.67:0/1910971842 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0032660 con 0x7f91ac0d0e00 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91b3234640 1 -- 172.21.15.67:0/1910971842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d0e00 msgr2=0x7f91ac0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91b3234640 1 --2- 172.21.15.67:0/1910971842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d0e00 0x7f91ac0d1210 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f91a00097a0 tx=0x7f91a002a140 comp rx=0 tx=0).stop 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91b3234640 1 -- 172.21.15.67:0/1910971842 shutdown_connections 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91b3234640 1 --2- 172.21.15.67:0/1910971842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d0e00 0x7f91ac0d1210 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91b3234640 1 -- 172.21.15.67:0/1910971842 >> 172.21.15.67:0/1910971842 conn(0x7f91ac0d5d90 msgr2=0x7f91ac0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91b3234640 1 -- 172.21.15.67:0/1910971842 shutdown_connections 2024-04-25T15:28:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.137+0000 7f91b3234640 1 -- 172.21.15.67:0/1910971842 wait complete. 2024-04-25T15:28:09.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.139+0000 7f91b3234640 1 Processor -- start 2024-04-25T15:28:09.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.139+0000 7f91b3234640 1 -- start start 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91ac06c5c0 0x7f91ac14e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91ac0d0e00 0x7f91ac1f9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d4a50 0x7f91ac1fa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac164e00 con 0x7f91ac0d4a50 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac0b5320 con 0x7f91ac0d0e00 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f91ac164f80 con 0x7f91ac06c5c0 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d4a50 0x7f91ac1fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d4a50 0x7f91ac1fa470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41366/0 (socket says 172.21.15.67:41366) 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 -- 172.21.15.67:0/3868845606 learned_addr learned my addr 172.21.15.67:0/3868845606 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b0fd0640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91ac06c5c0 0x7f91ac14e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91abfff640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91ac0d0e00 0x7f91ac1f9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 -- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91ac06c5c0 msgr2=0x7f91ac14e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91ac06c5c0 0x7f91ac14e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 -- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91ac0d0e00 msgr2=0x7f91ac1f9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91ac0d0e00 0x7f91ac1f9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 -- 172.21.15.67:0/3868845606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ac0c99d0 con 0x7f91ac0d4a50 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b17d1640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d4a50 0x7f91ac1fa470 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f91a400b970 tx=0x7f91a400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91a97fa640 1 -- 172.21.15.67:0/3868845606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4019070 con 0x7f91ac0d4a50 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91a97fa640 1 -- 172.21.15.67:0/3868845606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a400cb40 con 0x7f91ac0d4a50 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ac0c9cb0 con 0x7f91ac0d4a50 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91a97fa640 1 -- 172.21.15.67:0/3868845606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4014440 con 0x7f91ac0d4a50 2024-04-25T15:28:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.142+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ac0ca270 con 0x7f91ac0d4a50 2024-04-25T15:28:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.143+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ac050740 con 0x7f91ac0d4a50 2024-04-25T15:28:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.143+0000 7f91a97fa640 1 -- 172.21.15.67:0/3868845606 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f91a401c070 con 0x7f91ac0d4a50 2024-04-25T15:28:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.144+0000 7f91a97fa640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91900744b0 0x7f91900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.144+0000 7f91a97fa640 1 -- 172.21.15.67:0/3868845606 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f91a4094e00 con 0x7f91ac0d4a50 2024-04-25T15:28:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.144+0000 7f91b0fd0640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91900744b0 0x7f91900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.144+0000 7f91b0fd0640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91900744b0 0x7f91900769c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91a0034000 tx=0x7f91a002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.149+0000 7f91a97fa640 1 -- 172.21.15.67:0/3868845606 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91a405ef40 con 0x7f91ac0d4a50 2024-04-25T15:28:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.371+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91ac0d4320 con 0x7f91ac0d4a50 2024-04-25T15:28:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.371+0000 7f91a97fa640 1 -- 172.21.15.67:0/3868845606 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5189 (secure 0 0 0) 0x7f91a4063240 con 0x7f91ac0d4a50 2024-04-25T15:28:09.375 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:09.375 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2076054232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2076054232},{"type":"v1","addr":"172.21.15.176:6836","nonce":2076054232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":5537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3930910548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3930910548},{"type":"v1","addr":"172.21.15.176:6837","nonce":3930910548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2322157634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322157634},{"type":"v1","addr":"172.21.15.67:6834","nonce":2322157634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:47.123622+0000","modified":"2024-04-25T15:27:50.154657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6273":{"gid":6273,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3903876028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3903876028},{"type":"v1","addr":"172.21.15.67:6835","nonce":3903876028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6273,"qdb_cluster":[6273]},"id":5}]} 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91900744b0 msgr2=0x7f91900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91900744b0 0x7f91900769c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91a0034000 tx=0x7f91a002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d4a50 msgr2=0x7f91ac1fa470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d4a50 0x7f91ac1fa470 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f91a400b970 tx=0x7f91a400be30 comp rx=0 tx=0).stop 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 shutdown_connections 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91900744b0 0x7f91900769c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac0d4a50 0x7f91ac1fa470 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91ac0d0e00 0x7f91ac1f9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 --2- 172.21.15.67:0/3868845606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91ac06c5c0 0x7f91ac14e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 >> 172.21.15.67:0/3868845606 conn(0x7f91ac0d5d90 msgr2=0x7f91ac1641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 shutdown_connections 2024-04-25T15:28:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.373+0000 7f91b3234640 1 -- 172.21.15.67:0/3868845606 wait complete. 2024-04-25T15:28:09.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.496+0000 7fe7e5eaa640 1 Processor -- start 2024-04-25T15:28:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.496+0000 7fe7e5eaa640 1 -- start start 2024-04-25T15:28:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.497+0000 7fe7e5eaa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7e0164b00 0x7fe7e00b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7e5eaa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7e5eaa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7e01615f0 con 0x7fe7e015df80 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7e5eaa640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e01613b0 con 0x7fe7e0164b00 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7e5eaa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e0164670 con 0x7fe7e0160fa0 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7df7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7e0164b00 0x7fe7e00b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7df7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7e0164b00 0x7fe7e00b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41382/0 (socket says 172.21.15.67:41382) 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7df7fe640 1 -- 172.21.15.67:0/551195739 learned_addr learned my addr 172.21.15.67:0/551195739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7dffff640 1 --2- 172.21.15.67:0/551195739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7df7fe640 1 -- 172.21.15.67:0/551195739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 msgr2=0x7fe7e00b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7df7fe640 1 --2- 172.21.15.67:0/551195739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e00b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7df7fe640 1 -- 172.21.15.67:0/551195739 >> v1:172.21.15.176:6789/0 conn(0x7fe7e015df80 legacy=0x7fe7e0160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.498+0000 7fe7df7fe640 1 -- 172.21.15.67:0/551195739 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e0164020 con 0x7fe7e0164b00 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.499+0000 7fe7df7fe640 1 --2- 172.21.15.67:0/551195739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e00b7b80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dad8f14a463dda3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.499+0000 7fe7de7fc640 1 -- 172.21.15.67:0/551195739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0037070 con 0x7fe7e0164b00 2024-04-25T15:28:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.499+0000 7fe7de7fc640 1 -- 172.21.15.67:0/551195739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d0032440 con 0x7fe7e0164b00 2024-04-25T15:28:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.499+0000 7fe7de7fc640 1 -- 172.21.15.67:0/551195739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0032810 con 0x7fe7e0164b00 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/551195739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 msgr2=0x7fe7e00b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/551195739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e00b7b80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d002a210 comp rx=0 tx=0).stop 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/551195739 shutdown_connections 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/551195739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/551195739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e00b7b80 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/551195739 >> 172.21.15.67:0/551195739 conn(0x7fe7e00d2bc0 msgr2=0x7fe7e00d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/551195739 shutdown_connections 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.500+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/551195739 wait complete. 2024-04-25T15:28:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.501+0000 7fe7e5eaa640 1 Processor -- start 2024-04-25T15:28:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.502+0000 7fe7e5eaa640 1 -- start start 2024-04-25T15:28:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7e5eaa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7e015df80 0x7fe7e014ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7e5eaa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e01f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7e5eaa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7e5eaa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e00b4a00 con 0x7fe7e0164b00 2024-04-25T15:28:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7e5eaa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e00b4880 con 0x7fe7e015df80 2024-04-25T15:28:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7e5eaa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e00b4b80 con 0x7fe7e0160fa0 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e015be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41386/0 (socket says 172.21.15.67:41386) 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 -- 172.21.15.67:0/962961304 learned_addr learned my addr 172.21.15.67:0/962961304 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7df7fe640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7e015df80 0x7fe7e014ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7deffd640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e01f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 -- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 msgr2=0x7fe7e01f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e01f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 -- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7e015df80 msgr2=0x7fe7e014ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7e015df80 0x7fe7e014ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 -- 172.21.15.67:0/962961304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e01fa570 con 0x7fe7e0164b00 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.503+0000 7fe7dffff640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e015be30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8007d70 tx=0x7fe7d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.504+0000 7fe7c7fff640 1 -- 172.21.15.67:0/962961304 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8014420 con 0x7fe7e0164b00 2024-04-25T15:28:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.504+0000 7fe7c7fff640 1 -- 172.21.15.67:0/962961304 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d8019070 con 0x7fe7e0164b00 2024-04-25T15:28:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.504+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e01fa850 con 0x7fe7e0164b00 2024-04-25T15:28:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.504+0000 7fe7c7fff640 1 -- 172.21.15.67:0/962961304 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8014820 con 0x7fe7e0164b00 2024-04-25T15:28:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.504+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e00c9a80 con 0x7fe7e0164b00 2024-04-25T15:28:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.505+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e0050740 con 0x7fe7e0164b00 2024-04-25T15:28:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.505+0000 7fe7c7fff640 1 -- 172.21.15.67:0/962961304 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe7d80149c0 con 0x7fe7e0164b00 2024-04-25T15:28:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.506+0000 7fe7c7fff640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7cc0744b0 0x7fe7cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.506+0000 7fe7c7fff640 1 -- 172.21.15.67:0/962961304 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fe7d8095a00 con 0x7fe7e0164b00 2024-04-25T15:28:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.506+0000 7fe7df7fe640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7cc0744b0 0x7fe7cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.507+0000 7fe7df7fe640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7cc0744b0 0x7fe7cc0769c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0034000 tx=0x7fe7d002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.511+0000 7fe7c7fff640 1 -- 172.21.15.67:0/962961304 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7d805fb40 con 0x7fe7e0164b00 2024-04-25T15:28:09.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.698+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7e00d4170 con 0x7fe7e0164b00 2024-04-25T15:28:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.699+0000 7fe7c7fff640 1 -- 172.21.15.67:0/962961304 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16632 (secure 0 0 0) 0x7fe7d8063e40 con 0x7fe7e0164b00 2024-04-25T15:28:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:09.702 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:27:48.764450+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.701+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7cc0744b0 msgr2=0x7fe7cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.701+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7cc0744b0 0x7fe7cc0769c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0034000 tx=0x7fe7d002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.701+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 msgr2=0x7fe7e015be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.701+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e015be30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8007d70 tx=0x7fe7d800e4b0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 shutdown_connections 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe7cc0744b0 0x7fe7cc0769c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e0164b00 0x7fe7e015be30 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7e0160fa0 0x7fe7e01f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 --2- 172.21.15.67:0/962961304 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7e015df80 0x7fe7e014ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 >> 172.21.15.67:0/962961304 conn(0x7fe7e00d2bc0 msgr2=0x7fe7e00d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 shutdown_connections 2024-04-25T15:28:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.702+0000 7fe7e5eaa640 1 -- 172.21.15.67:0/962961304 wait complete. 2024-04-25T15:28:09.715 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:28:09.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.825+0000 7f5725292640 1 Processor -- start 2024-04-25T15:28:09.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.825+0000 7f5725292640 1 -- start start 2024-04-25T15:28:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f5725292640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5720164b00 0x7f57200b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f5725292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f572015df80 0x7f57200b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f5725292640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57201615f0 con 0x7f5720160890 2024-04-25T15:28:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f5725292640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f57201613b0 con 0x7f5720164b00 2024-04-25T15:28:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f5725292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5720164670 con 0x7f572015df80 2024-04-25T15:28:09.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5720164b00 0x7f57200b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5720164b00 0x7f57200b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41390/0 (socket says 172.21.15.67:41390) 2024-04-25T15:28:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571effd640 1 -- 172.21.15.67:0/518947513 learned_addr learned my addr 172.21.15.67:0/518947513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571e7fc640 1 --2- 172.21.15.67:0/518947513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f572015df80 0x7f57200b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571effd640 1 -- 172.21.15.67:0/518947513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f572015df80 msgr2=0x7f57200b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571effd640 1 --2- 172.21.15.67:0/518947513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f572015df80 0x7f57200b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571effd640 1 -- 172.21.15.67:0/518947513 >> v1:172.21.15.176:6789/0 conn(0x7f5720160890 legacy=0x7f5720160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.827+0000 7f571effd640 1 -- 172.21.15.67:0/518947513 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5720164020 con 0x7f5720164b00 2024-04-25T15:28:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.828+0000 7f571effd640 1 --2- 172.21.15.67:0/518947513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720164b00 0x7f57200b7b80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5718009870 tx=0x7f571802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9559077d3a93c784 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.828+0000 7f571dffb640 1 -- 172.21.15.67:0/518947513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718037070 con 0x7f5720164b00 2024-04-25T15:28:09.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.828+0000 7f571dffb640 1 -- 172.21.15.67:0/518947513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5718032440 con 0x7f5720164b00 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.828+0000 7f571dffb640 1 -- 172.21.15.67:0/518947513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718032810 con 0x7f5720164b00 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 -- 172.21.15.67:0/518947513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720164b00 msgr2=0x7f57200b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 --2- 172.21.15.67:0/518947513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720164b00 0x7f57200b7b80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5718009870 tx=0x7f571802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 -- 172.21.15.67:0/518947513 shutdown_connections 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 --2- 172.21.15.67:0/518947513 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f572015df80 0x7f57200b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 --2- 172.21.15.67:0/518947513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720164b00 0x7f57200b7b80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 -- 172.21.15.67:0/518947513 >> 172.21.15.67:0/518947513 conn(0x7f57200d2bc0 msgr2=0x7f57200d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 -- 172.21.15.67:0/518947513 shutdown_connections 2024-04-25T15:28:09.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.829+0000 7f5725292640 1 -- 172.21.15.67:0/518947513 wait complete. 2024-04-25T15:28:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.831+0000 7f5725292640 1 Processor -- start 2024-04-25T15:28:09.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.831+0000 7f5725292640 1 -- start start 2024-04-25T15:28:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f5725292640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f572015df80 0x7f57201f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f5725292640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720160890 0x7f57201f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f5725292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5720164b00 0x7f5720158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f5725292640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f57200b4a00 con 0x7f5720160890 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f5725292640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f57200b4880 con 0x7f572015df80 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f5725292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f57200b4b80 con 0x7f5720164b00 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f571e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720160890 0x7f57201f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f571e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720160890 0x7f57201f6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41402/0 (socket says 172.21.15.67:41402) 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.832+0000 7f571e7fc640 1 -- 172.21.15.67:0/2639352467 learned_addr learned my addr 172.21.15.67:0/2639352467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571effd640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f572015df80 0x7f57201f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571f7fe640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5720164b00 0x7f5720158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571e7fc640 1 -- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5720164b00 msgr2=0x7f5720158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571e7fc640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5720164b00 0x7f5720158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571e7fc640 1 -- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f572015df80 msgr2=0x7f57201f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571e7fc640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f572015df80 0x7f57201f5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:09.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571e7fc640 1 -- 172.21.15.67:0/2639352467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5720157e80 con 0x7f5720160890 2024-04-25T15:28:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f571e7fc640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720160890 0x7f57201f6400 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f570800d5f0 tx=0x7f570800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f57177fe640 1 -- 172.21.15.67:0/2639352467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57080125f0 con 0x7f5720160890 2024-04-25T15:28:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f57177fe640 1 -- 172.21.15.67:0/2639352467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5708019070 con 0x7f5720160890 2024-04-25T15:28:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5720162f00 con 0x7f5720160890 2024-04-25T15:28:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f57177fe640 1 -- 172.21.15.67:0/2639352467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57080129d0 con 0x7f5720160890 2024-04-25T15:28:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.833+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57201429d0 con 0x7f5720160890 2024-04-25T15:28:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.834+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57201611d0 con 0x7f5720160890 2024-04-25T15:28:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.835+0000 7f57177fe640 1 -- 172.21.15.67:0/2639352467 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5708012b70 con 0x7f5720160890 2024-04-25T15:28:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.835+0000 7f57177fe640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57000744b0 0x7f57000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.835+0000 7f57177fe640 1 -- 172.21.15.67:0/2639352467 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f57080956a0 con 0x7f5720160890 2024-04-25T15:28:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.835+0000 7f571effd640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57000744b0 0x7f57000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:09.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.836+0000 7f571effd640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57000744b0 0x7f57000769c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5718034000 tx=0x7f571802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:09.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:09.840+0000 7f57177fe640 1 -- 172.21.15.67:0/2639352467 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f57201611d0 con 0x7f5720160890 2024-04-25T15:28:10.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.073+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5720050740 con 0x7f5720160890 2024-04-25T15:28:10.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.073+0000 7f57177fe640 1 -- 172.21.15.67:0/2639352467 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5189 (secure 0 0 0) 0x7f570805fdb0 con 0x7f5720160890 2024-04-25T15:28:10.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:10.075 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2076054232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2076054232},{"type":"v1","addr":"172.21.15.176:6836","nonce":2076054232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":5537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3930910548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3930910548},{"type":"v1","addr":"172.21.15.176:6837","nonce":3930910548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2322157634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322157634},{"type":"v1","addr":"172.21.15.67:6834","nonce":2322157634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:47.123622+0000","modified":"2024-04-25T15:27:50.154657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6273":{"gid":6273,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3903876028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3903876028},{"type":"v1","addr":"172.21.15.67:6835","nonce":3903876028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6273,"qdb_cluster":[6273]},"id":5}]} 2024-04-25T15:28:10.075 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57000744b0 msgr2=0x7f57000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57000744b0 0x7f57000769c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5718034000 tx=0x7f571802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720160890 msgr2=0x7f57201f6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720160890 0x7f57201f6400 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f570800d5f0 tx=0x7f570800dab0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 shutdown_connections 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f57000744b0 0x7f57000769c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5720164b00 0x7f5720158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5720160890 0x7f57201f6400 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 --2- 172.21.15.67:0/2639352467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f572015df80 0x7f57201f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 >> 172.21.15.67:0/2639352467 conn(0x7f57200d2bc0 msgr2=0x7f572015ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 shutdown_connections 2024-04-25T15:28:10.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.076+0000 7f5725292640 1 -- 172.21.15.67:0/2639352467 wait complete. 2024-04-25T15:28:10.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.198+0000 7f7f0109f640 1 Processor -- start 2024-04-25T15:28:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.198+0000 7f7f0109f640 1 -- start start 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7f0109f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc0b8290 0x7f7efc160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7f0109f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7efc15df80 0x7f7efc160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7f0109f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7efc1615f0 con 0x7f7efc164b00 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7f0109f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7efc164670 con 0x7f7efc15df80 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7f0109f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7efc164020 con 0x7f7efc0b8290 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efb59e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7efc15df80 0x7f7efc160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efb59e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7efc15df80 0x7f7efc160dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41414/0 (socket says 172.21.15.67:41414) 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efb59e640 1 -- 172.21.15.67:0/2980683811 learned_addr learned my addr 172.21.15.67:0/2980683811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efa59c640 1 --2- 172.21.15.67:0/2980683811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc0b8290 0x7f7efc160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efb59e640 1 -- 172.21.15.67:0/2980683811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc0b8290 msgr2=0x7f7efc160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efb59e640 1 --2- 172.21.15.67:0/2980683811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc0b8290 0x7f7efc160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efb59e640 1 -- 172.21.15.67:0/2980683811 >> v1:172.21.15.176:6789/0 conn(0x7f7efc164b00 legacy=0x7f7efc0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.200+0000 7f7efb59e640 1 -- 172.21.15.67:0/2980683811 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7efc164300 con 0x7f7efc15df80 2024-04-25T15:28:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.201+0000 7f7efb59e640 1 --2- 172.21.15.67:0/2980683811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc15df80 0x7f7efc160dd0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009e50 tx=0x7f7eec02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab07977e52bd14e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.201+0000 7f7ef9d9b640 1 -- 172.21.15.67:0/2980683811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec036070 con 0x7f7efc15df80 2024-04-25T15:28:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.201+0000 7f7ef9d9b640 1 -- 172.21.15.67:0/2980683811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eec031600 con 0x7f7efc15df80 2024-04-25T15:28:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.201+0000 7f7ef9d9b640 1 -- 172.21.15.67:0/2980683811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eec0319d0 con 0x7f7efc15df80 2024-04-25T15:28:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 -- 172.21.15.67:0/2980683811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc15df80 msgr2=0x7f7efc160dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 --2- 172.21.15.67:0/2980683811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc15df80 0x7f7efc160dd0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009e50 tx=0x7f7eec02a710 comp rx=0 tx=0).stop 2024-04-25T15:28:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 -- 172.21.15.67:0/2980683811 shutdown_connections 2024-04-25T15:28:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 --2- 172.21.15.67:0/2980683811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc15df80 0x7f7efc160dd0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 --2- 172.21.15.67:0/2980683811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc0b8290 0x7f7efc160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 -- 172.21.15.67:0/2980683811 >> 172.21.15.67:0/2980683811 conn(0x7f7efc0d2bc0 msgr2=0x7f7efc0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 -- 172.21.15.67:0/2980683811 shutdown_connections 2024-04-25T15:28:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.202+0000 7f7f0109f640 1 -- 172.21.15.67:0/2980683811 wait complete. 2024-04-25T15:28:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.204+0000 7f7f0109f640 1 Processor -- start 2024-04-25T15:28:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.204+0000 7f7f0109f640 1 -- start start 2024-04-25T15:28:10.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7f0109f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc15df80 0x7f7efc0cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:10.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7f0109f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc164b00 0x7f7efc0c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7f0109f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7efc1f9f60 0x7f7efc0caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7f0109f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc0b4a00 con 0x7f7efc164b00 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7f0109f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7efc0b4880 con 0x7f7efc1f9f60 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7f0109f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7efc0b4b80 con 0x7f7efc15df80 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7efa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc164b00 0x7f7efc0c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7efa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc164b00 0x7f7efc0c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41422/0 (socket says 172.21.15.67:41422) 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7efa59c640 1 -- 172.21.15.67:0/1752607162 learned_addr learned my addr 172.21.15.67:0/1752607162 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7efad9d640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc15df80 0x7f7efc0cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.205+0000 7f7efb59e640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7efc1f9f60 0x7f7efc0caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7efa59c640 1 -- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc15df80 msgr2=0x7f7efc0cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7efa59c640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc15df80 0x7f7efc0cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7efa59c640 1 -- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7efc1f9f60 msgr2=0x7f7efc0caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7efa59c640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7efc1f9f60 0x7f7efc0caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7efa59c640 1 -- 172.21.15.67:0/1752607162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7efc1fa6b0 con 0x7f7efc164b00 2024-04-25T15:28:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7efb59e640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7efc1f9f60 0x7f7efc0caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7efa59c640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc164b00 0x7f7efc0c9830 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000d620 tx=0x7f7ef000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7ee37fe640 1 -- 172.21.15.67:0/1752607162 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000b620 con 0x7f7efc164b00 2024-04-25T15:28:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7ee37fe640 1 -- 172.21.15.67:0/1752607162 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef0019070 con 0x7f7efc164b00 2024-04-25T15:28:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7efc1fa990 con 0x7f7efc164b00 2024-04-25T15:28:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7ee37fe640 1 -- 172.21.15.67:0/1752607162 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000ba00 con 0x7f7efc164b00 2024-04-25T15:28:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.206+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7efc1f68f0 con 0x7f7efc164b00 2024-04-25T15:28:10.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.207+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7efc050740 con 0x7f7efc164b00 2024-04-25T15:28:10.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.208+0000 7f7ee37fe640 1 -- 172.21.15.67:0/1752607162 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7ef000bba0 con 0x7f7efc164b00 2024-04-25T15:28:10.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.208+0000 7f7ee37fe640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ee80744b0 0x7f7ee80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:10.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.208+0000 7f7ee37fe640 1 -- 172.21.15.67:0/1752607162 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5767+0+0 (secure 0 0 0) 0x7f7ef00955c0 con 0x7f7efc164b00 2024-04-25T15:28:10.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.208+0000 7f7efad9d640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ee80744b0 0x7f7ee80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:10.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.209+0000 7f7efad9d640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ee80744b0 0x7f7ee80769c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40016e0 tx=0x7f7ef4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:10.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.213+0000 7f7ee37fe640 1 -- 172.21.15.67:0/1752607162 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ef005fdb0 con 0x7f7efc164b00 2024-04-25T15:28:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.435+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7efc0d4170 con 0x7f7efc164b00 2024-04-25T15:28:10.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.436+0000 7f7ee37fe640 1 -- 172.21.15.67:0/1752607162 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5189 (secure 0 0 0) 0x7f7ef005f6b0 con 0x7f7efc164b00 2024-04-25T15:28:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:10.437 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2076054232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2076054232},{"type":"v1","addr":"172.21.15.176:6836","nonce":2076054232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":5537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3930910548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3930910548},{"type":"v1","addr":"172.21.15.176:6837","nonce":3930910548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2322157634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322157634},{"type":"v1","addr":"172.21.15.67:6834","nonce":2322157634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:27:47.123622+0000","modified":"2024-04-25T15:27:50.154657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6273":{"gid":6273,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3903876028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3903876028},{"type":"v1","addr":"172.21.15.67:6835","nonce":3903876028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6273,"qdb_cluster":[6273]},"id":5}]} 2024-04-25T15:28:10.437 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 41 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ee80744b0 msgr2=0x7f7ee80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ee80744b0 0x7f7ee80769c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40016e0 tx=0x7f7ef4009040 comp rx=0 tx=0).stop 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc164b00 msgr2=0x7f7efc0c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc164b00 0x7f7efc0c9830 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000d620 tx=0x7f7ef000dae0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 shutdown_connections 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7ee80744b0 0x7f7ee80769c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7efc1f9f60 0x7f7efc0caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7efc164b00 0x7f7efc0c9830 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 --2- 172.21.15.67:0/1752607162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7efc15df80 0x7f7efc0cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:10.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 >> 172.21.15.67:0/1752607162 conn(0x7f7efc0d2bc0 msgr2=0x7f7efc0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 shutdown_connections 2024-04-25T15:28:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.438+0000 7f7f0109f640 1 -- 172.21.15.67:0/1752607162 wait complete. 2024-04-25T15:28:10.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:10.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.561+0000 7fe5a21c7640 1 Processor -- start 2024-04-25T15:28:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.561+0000 7fe5a21c7640 1 -- start start 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe5a21c7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe59c0d0e00 0x7fe59c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe5a21c7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe59c06c5c0 0x7fe59c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe5a21c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c0b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe5a21c7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe59c06cf10 con 0x7fe59c06c5c0 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe5a21c7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe59c0d1850 con 0x7fe59c0d0e00 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe5a21c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe59c160de0 con 0x7fe59c0b7810 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe59affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe59c06c5c0 0x7fe59c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe59affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe59c06c5c0 0x7fe59c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41434/0 (socket says 172.21.15.67:41434) 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.562+0000 7fe59affd640 1 -- 172.21.15.67:0/2635427388 learned_addr learned my addr 172.21.15.67:0/2635427388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59bfff640 1 --2- 172.21.15.67:0/2635427388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c0b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59b7fe640 1 --2- 172.21.15.67:0/2635427388 >> v2:172.21.15.176:3300/0 conn(0x7fe59c0d0e00 0x7fe59c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59affd640 1 -- 172.21.15.67:0/2635427388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 msgr2=0x7fe59c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59affd640 1 --2- 172.21.15.67:0/2635427388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c0b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59affd640 1 -- 172.21.15.67:0/2635427388 >> v2:172.21.15.176:3300/0 conn(0x7fe59c0d0e00 msgr2=0x7fe59c0d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59affd640 1 --2- 172.21.15.67:0/2635427388 >> v2:172.21.15.176:3300/0 conn(0x7fe59c0d0e00 0x7fe59c0d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59affd640 1 -- 172.21.15.67:0/2635427388 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59c1610c0 con 0x7fe59c06c5c0 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59bfff640 1 --2- 172.21.15.67:0/2635427388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.563+0000 7fe59b7fe640 1 --2- 172.21.15.67:0/2635427388 >> v2:172.21.15.176:3300/0 conn(0x7fe59c0d0e00 0x7fe59c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe59affd640 1 --2- 172.21.15.67:0/2635427388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c06c5c0 0x7fe59c06c9d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0d3e60 tx=0x7fe59002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e4e08e1b1230e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe59a7fc640 1 -- 172.21.15.67:0/2635427388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590037070 con 0x7fe59c06c5c0 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe59a7fc640 1 -- 172.21.15.67:0/2635427388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe590032440 con 0x7fe59c06c5c0 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe59a7fc640 1 -- 172.21.15.67:0/2635427388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590032810 con 0x7fe59c06c5c0 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2635427388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c06c5c0 msgr2=0x7fe59c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2635427388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c06c5c0 0x7fe59c06c9d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0d3e60 tx=0x7fe59002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2635427388 shutdown_connections 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2635427388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c0b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2635427388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c06c5c0 0x7fe59c06c9d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2635427388 >> v2:172.21.15.176:3300/0 conn(0x7fe59c0d0e00 0x7fe59c0d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2635427388 >> 172.21.15.67:0/2635427388 conn(0x7fe59c0d5d90 msgr2=0x7fe59c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.564+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2635427388 shutdown_connections 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.565+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2635427388 wait complete. 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.566+0000 7fe5a21c7640 1 Processor -- start 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.566+0000 7fe5a21c7640 1 -- start start 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe5a21c7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe59c06c5c0 0x7fe59c0ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe5a21c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe5a21c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c0d0e00 0x7fe59c0ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe5a21c7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c0b5320 con 0x7fe59c0d0e00 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe5a21c7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c160ad0 con 0x7fe59c06c5c0 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe5a21c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe59c164e00 con 0x7fe59c0b7810 2024-04-25T15:28:11.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c0d0e00 0x7fe59c0ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c0d0e00 0x7fe59c0ac6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41446/0 (socket says 172.21.15.67:41446) 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 -- 172.21.15.67:0/2785876881 learned_addr learned my addr 172.21.15.67:0/2785876881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59affd640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59b7fe640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe59c06c5c0 0x7fe59c0ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 -- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 msgr2=0x7fe59c15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 -- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe59c06c5c0 msgr2=0x7fe59c0ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe59c06c5c0 0x7fe59c0ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 -- 172.21.15.67:0/2785876881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59c0cee40 con 0x7fe59c0d0e00 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59affd640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:11.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe59bfff640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c0d0e00 0x7fe59c0ac6a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe594007de0 tx=0x7fe59400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.568+0000 7fe583fff640 1 -- 172.21.15.67:0/2785876881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59401a070 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.569+0000 7fe583fff640 1 -- 172.21.15.67:0/2785876881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe594003990 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.569+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe59c0cf120 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.569+0000 7fe583fff640 1 -- 172.21.15.67:0/2785876881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe594003d80 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.569+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe59c0cb160 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.569+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe59c050740 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.570+0000 7fe583fff640 1 -- 172.21.15.67:0/2785876881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe59401d070 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.571+0000 7fe583fff640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe5880744e0 0x7fe5880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.571+0000 7fe583fff640 1 -- 172.21.15.67:0/2785876881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5767+0+0 (secure 0 0 0) 0x7fe594095450 con 0x7fe59c0d0e00 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.571+0000 7fe59b7fe640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe5880744e0 0x7fe5880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.572+0000 7fe59b7fe640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe5880744e0 0x7fe5880769f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe58c006fd0 tx=0x7fe58c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.575+0000 7fe583fff640 1 -- 172.21.15.67:0/2785876881 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe594010070 con 0x7fe59c0d0e00 2024-04-25T15:28:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.763+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe59c0d4320 con 0x7fe59c0d0e00 2024-04-25T15:28:11.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.764+0000 7fe583fff640 1 -- 172.21.15.67:0/2785876881 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16729 (secure 0 0 0) 0x7fe59405f5e0 con 0x7fe59c0d0e00 2024-04-25T15:28:11.189 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:11.189 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":68,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:09.934259+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:27:45.453204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-25T15:27:46.073423+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.9","mappings":[{"from":6,"to":0}]},{"pgid":"11.14","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe5880744e0 msgr2=0x7fe5880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe5880744e0 0x7fe5880769f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe58c006fd0 tx=0x7fe58c008040 comp rx=0 tx=0).stop 2024-04-25T15:28:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c0d0e00 msgr2=0x7fe59c0ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c0d0e00 0x7fe59c0ac6a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe594007de0 tx=0x7fe59400d4a0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 shutdown_connections 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe5880744e0 0x7fe5880769f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe59c0d0e00 0x7fe59c0ac6a0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe59c0b7810 0x7fe59c15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 --2- 172.21.15.67:0/2785876881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe59c06c5c0 0x7fe59c0ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 >> 172.21.15.67:0/2785876881 conn(0x7fe59c0d5d90 msgr2=0x7fe59c1601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 shutdown_connections 2024-04-25T15:28:11.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:10.766+0000 7fe5a21c7640 1 -- 172.21.15.67:0/2785876881 wait complete. 2024-04-25T15:28:11.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:28:11.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.309+0000 7f2859a92640 1 Processor -- start 2024-04-25T15:28:11.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.309+0000 7f2859a92640 1 -- start start 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f2859a92640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28540d4570 0x7f28540b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f2859a92640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f285406e8f0 0x7f28540d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f2859a92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28540b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f2859a92640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f285406e460 con 0x7f28540d4570 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f2859a92640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f285406d870 con 0x7f285406e8f0 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f2859a92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f285406db10 con 0x7f28540d09f0 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f28537fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28540d4570 0x7f28540b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f28537fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28540d4570 0x7f28540b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41460/0 (socket says 172.21.15.67:41460) 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.310+0000 7f28537fe640 1 -- 172.21.15.67:0/3692029695 learned_addr learned my addr 172.21.15.67:0/3692029695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f2853fff640 1 --2- 172.21.15.67:0/3692029695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28540b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f2852ffd640 1 --2- 172.21.15.67:0/3692029695 >> v2:172.21.15.176:3300/0 conn(0x7f285406e8f0 0x7f28540d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f28537fe640 1 -- 172.21.15.67:0/3692029695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 msgr2=0x7f28540b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:11.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f28537fe640 1 --2- 172.21.15.67:0/3692029695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28540b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f28537fe640 1 -- 172.21.15.67:0/3692029695 >> v2:172.21.15.176:3300/0 conn(0x7f285406e8f0 msgr2=0x7f28540d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:11.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f28537fe640 1 --2- 172.21.15.67:0/3692029695 >> v2:172.21.15.176:3300/0 conn(0x7f285406e8f0 0x7f28540d04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f28537fe640 1 -- 172.21.15.67:0/3692029695 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285406e0f0 con 0x7f28540d4570 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.311+0000 7f28537fe640 1 --2- 172.21.15.67:0/3692029695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540d4570 0x7f28540b4830 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f2854070c30 tx=0x7f284402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2943c27f9a6eed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f28527fc640 1 -- 172.21.15.67:0/3692029695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844004030 con 0x7f28540d4570 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f28527fc640 1 -- 172.21.15.67:0/3692029695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28440041d0 con 0x7f28540d4570 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f28527fc640 1 -- 172.21.15.67:0/3692029695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844003610 con 0x7f28540d4570 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 -- 172.21.15.67:0/3692029695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540d4570 msgr2=0x7f28540b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 --2- 172.21.15.67:0/3692029695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540d4570 0x7f28540b4830 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f2854070c30 tx=0x7f284402a130 comp rx=0 tx=0).stop 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 -- 172.21.15.67:0/3692029695 shutdown_connections 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 --2- 172.21.15.67:0/3692029695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28540b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 --2- 172.21.15.67:0/3692029695 >> v2:172.21.15.176:3300/0 conn(0x7f285406e8f0 0x7f28540d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 --2- 172.21.15.67:0/3692029695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28540d4570 0x7f28540b4830 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 -- 172.21.15.67:0/3692029695 >> 172.21.15.67:0/3692029695 conn(0x7f28540d3a50 msgr2=0x7f28540d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 -- 172.21.15.67:0/3692029695 shutdown_connections 2024-04-25T15:28:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.312+0000 7f2859a92640 1 -- 172.21.15.67:0/3692029695 wait complete. 2024-04-25T15:28:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.314+0000 7f2859a92640 1 Processor -- start 2024-04-25T15:28:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.314+0000 7f2859a92640 1 -- start start 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2859a92640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285406e8f0 0x7f2854154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2859a92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28541622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2859a92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28540d4570 0x7f28540d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2859a92640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28540b3300 con 0x7f285406e8f0 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2859a92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f285406ed00 con 0x7f28540d4570 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2859a92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f28540b3480 con 0x7f28540d09f0 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285406e8f0 0x7f2854154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285406e8f0 0x7f2854154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41476/0 (socket says 172.21.15.67:41476) 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 -- 172.21.15.67:0/3977655746 learned_addr learned my addr 172.21.15.67:0/3977655746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2852ffd640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28541622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2853fff640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28540d4570 0x7f28540d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 -- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 msgr2=0x7f28541622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28541622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 -- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28540d4570 msgr2=0x7f28540d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28540d4570 0x7f28540d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 -- 172.21.15.67:0/3977655746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28540b2460 con 0x7f285406e8f0 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f2853fff640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28540d4570 0x7f28540d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.316+0000 7f28537fe640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285406e8f0 0x7f2854154e70 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f284402a170 tx=0x7f28440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.317+0000 7f284bfff640 1 -- 172.21.15.67:0/3977655746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844041020 con 0x7f285406e8f0 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.317+0000 7f284bfff640 1 -- 172.21.15.67:0/3977655746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2844003c50 con 0x7f285406e8f0 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.317+0000 7f284bfff640 1 -- 172.21.15.67:0/3977655746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844004030 con 0x7f285406e8f0 2024-04-25T15:28:11.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.317+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28540b13c0 con 0x7f285406e8f0 2024-04-25T15:28:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.317+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28540b1980 con 0x7f285406e8f0 2024-04-25T15:28:11.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.318+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2854050740 con 0x7f285406e8f0 2024-04-25T15:28:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.318+0000 7f284bfff640 1 -- 172.21.15.67:0/3977655746 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2844044020 con 0x7f285406e8f0 2024-04-25T15:28:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.319+0000 7f284bfff640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28340744e0 0x7f28340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.319+0000 7f284bfff640 1 -- 172.21.15.67:0/3977655746 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5767+0+0 (secure 0 0 0) 0x7f28440b5f80 con 0x7f285406e8f0 2024-04-25T15:28:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.319+0000 7f2852ffd640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28340744e0 0x7f28340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.320+0000 7f2852ffd640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28340744e0 0x7f28340769f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f283c006fd0 tx=0x7f283c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.324+0000 7f284bfff640 1 -- 172.21.15.67:0/3977655746 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f284404b030 con 0x7f285406e8f0 2024-04-25T15:28:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.554+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f28540d37b0 con 0x7f285406e8f0 2024-04-25T15:28:11.555 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:28:11.554+0000 7f3e02b50640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:28:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.556+0000 7f284bfff640 1 -- 172.21.15.67:0/3977655746 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) ==== 161+0+0 (secure 0 0 0) 0x7f28440806d0 con 0x7f285406e8f0 2024-04-25T15:28:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:28:11.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28340744e0 msgr2=0x7f28340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28340744e0 0x7f28340769f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f283c006fd0 tx=0x7f283c008040 comp rx=0 tx=0).stop 2024-04-25T15:28:11.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285406e8f0 msgr2=0x7f2854154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285406e8f0 0x7f2854154e70 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f284402a170 tx=0x7f28440046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 shutdown_connections 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28340744e0 0x7f28340769f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28540d4570 0x7f28540d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28540d09f0 0x7f28541622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 --2- 172.21.15.67:0/3977655746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285406e8f0 0x7f2854154e70 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.558+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 >> 172.21.15.67:0/3977655746 conn(0x7f28540d3a50 msgr2=0x7f28540ce140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.559+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 shutdown_connections 2024-04-25T15:28:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.559+0000 7f2859a92640 1 -- 172.21.15.67:0/3977655746 wait complete. 2024-04-25T15:28:11.563 INFO:tasks.ceph.mds.a.smithi067.stderr: -5706> 2024-04-25T15:27:49.727+0000 7f9fe118f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:28:11.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:28:11.620 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:28:11.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.688+0000 7f70d6796640 1 Processor -- start 2024-04-25T15:28:11.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.688+0000 7f70d6796640 1 -- start start 2024-04-25T15:28:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70d6796640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d006ccd0 0x7f70d00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70d6796640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d00d4a50 0x7f70d00d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70d6796640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70d00d1850 con 0x7f70d00d0e00 2024-04-25T15:28:11.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70d6796640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f70d0160de0 con 0x7f70d00d4a50 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70d6796640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70d01610c0 con 0x7f70d006ccd0 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70d4d33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d00d4a50 0x7f70d00d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70cffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f70d00d0e00 0x7f70d006c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52104/0 (socket says 172.21.15.67:52104) 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.689+0000 7f70cffff640 1 -- 172.21.15.67:0/72229968 learned_addr learned my addr 172.21.15.67:0/72229968 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.690+0000 7f70d4d33640 1 -- 172.21.15.67:0/72229968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d006ccd0 msgr2=0x7f70d00d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.690+0000 7f70d4d33640 1 --2- 172.21.15.67:0/72229968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d006ccd0 0x7f70d00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.690+0000 7f70d4d33640 1 -- 172.21.15.67:0/72229968 >> v1:172.21.15.67:6789/0 conn(0x7f70d00d0e00 legacy=0x7f70d006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.690+0000 7f70d4d33640 1 -- 172.21.15.67:0/72229968 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d01613a0 con 0x7f70d00d4a50 2024-04-25T15:28:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.690+0000 7f70ceffd640 1 -- 172.21.15.67:0/72229968 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3725486010 0 0) 0x7f70d00d1850 con 0x7f70d00d0e00 2024-04-25T15:28:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.690+0000 7f70d4d33640 1 --2- 172.21.15.67:0/72229968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d4a50 0x7f70d00d4e60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009d90 tx=0x7f70c802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55886a4587d1e8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.691+0000 7f70ceffd640 1 -- 172.21.15.67:0/72229968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8007740 con 0x7f70d00d4a50 2024-04-25T15:28:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.691+0000 7f70ceffd640 1 -- 172.21.15.67:0/72229968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c80078e0 con 0x7f70d00d4a50 2024-04-25T15:28:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.691+0000 7f70ceffd640 1 -- 172.21.15.67:0/72229968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8007cb0 con 0x7f70d00d4a50 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 -- 172.21.15.67:0/72229968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d4a50 msgr2=0x7f70d00d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 --2- 172.21.15.67:0/72229968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d4a50 0x7f70d00d4e60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009d90 tx=0x7f70c802a660 comp rx=0 tx=0).stop 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 -- 172.21.15.67:0/72229968 shutdown_connections 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 --2- 172.21.15.67:0/72229968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d4a50 0x7f70d00d4e60 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 --2- 172.21.15.67:0/72229968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d006ccd0 0x7f70d00d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 -- 172.21.15.67:0/72229968 >> 172.21.15.67:0/72229968 conn(0x7f70d00d5d90 msgr2=0x7f70d00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 -- 172.21.15.67:0/72229968 shutdown_connections 2024-04-25T15:28:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.692+0000 7f70d6796640 1 -- 172.21.15.67:0/72229968 wait complete. 2024-04-25T15:28:11.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.694+0000 7f70d6796640 1 Processor -- start 2024-04-25T15:28:11.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.694+0000 7f70d6796640 1 -- start start 2024-04-25T15:28:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70d6796640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d0e00 0x7f70d0154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70d6796640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d00d4a50 0x7f70d00ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70d6796640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70d00cef30 0x7f70d00cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70d6796640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70d0164e00 con 0x7f70d00d0e00 2024-04-25T15:28:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70d6796640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f70d00b5320 con 0x7f70d00cef30 2024-04-25T15:28:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70d6796640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70d0164f80 con 0x7f70d00d4a50 2024-04-25T15:28:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d0e00 0x7f70d0154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d0e00 0x7f70d0154b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41506/0 (socket says 172.21.15.67:41506) 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70cffff640 1 -- 172.21.15.67:0/2098010239 learned_addr learned my addr 172.21.15.67:0/2098010239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.695+0000 7f70cf7fe640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d00d4a50 0x7f70d00ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70d4d33640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70d00cef30 0x7f70d00cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70cffff640 1 -- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d00d4a50 msgr2=0x7f70d00ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70cffff640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d00d4a50 0x7f70d00ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70cffff640 1 -- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70d00cef30 msgr2=0x7f70d00cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70cffff640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70d00cef30 0x7f70d00cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70cffff640 1 -- 172.21.15.67:0/2098010239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d0155050 con 0x7f70d00d0e00 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70cf7fe640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d00d4a50 0x7f70d00ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70cffff640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d0e00 0x7f70d0154b10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c460 tx=0x7f70c000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70ccff9640 1 -- 172.21.15.67:0/2098010239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0018070 con 0x7f70d00d0e00 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70ccff9640 1 -- 172.21.15.67:0/2098010239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c0007500 con 0x7f70d00d0e00 2024-04-25T15:28:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d00b5940 con 0x7f70d00d0e00 2024-04-25T15:28:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70ccff9640 1 -- 172.21.15.67:0/2098010239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0004a20 con 0x7f70d00d0e00 2024-04-25T15:28:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.696+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d00b5f00 con 0x7f70d00d0e00 2024-04-25T15:28:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.697+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d0050740 con 0x7f70d00d0e00 2024-04-25T15:28:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.698+0000 7f70ccff9640 1 -- 172.21.15.67:0/2098010239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f70c001b070 con 0x7f70d00d0e00 2024-04-25T15:28:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.698+0000 7f70ccff9640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70bc0744b0 0x7f70bc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.699+0000 7f70ccff9640 1 -- 172.21.15.67:0/2098010239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5853+0+0 (secure 0 0 0) 0x7f70c0094500 con 0x7f70d00d0e00 2024-04-25T15:28:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.699+0000 7f70cf7fe640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70bc0744b0 0x7f70bc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.699+0000 7f70cf7fe640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70bc0744b0 0x7f70bc0769c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f70d00cc8a0 tx=0x7f70c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.703+0000 7f70ccff9640 1 -- 172.21.15.67:0/2098010239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70c005ec90 con 0x7f70d00d0e00 2024-04-25T15:28:11.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:11.931+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f70d00d4320 con 0x7f70d00d0e00 2024-04-25T15:28:12.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.557+0000 7f70ccff9640 1 -- 172.21.15.67:0/2098010239 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) ==== 104+0+0 (secure 0 0 0) 0x7f70c005e590 con 0x7f70d00d0e00 2024-04-25T15:28:12.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70bc0744b0 msgr2=0x7f70bc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:12.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70bc0744b0 0x7f70bc0769c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f70d00cc8a0 tx=0x7f70c4008040 comp rx=0 tx=0).stop 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d0e00 msgr2=0x7f70d0154b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d0e00 0x7f70d0154b10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c460 tx=0x7f70c000c920 comp rx=0 tx=0).stop 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 shutdown_connections 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f70bc0744b0 0x7f70bc0769c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70d00cef30 0x7f70d00cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70d00d4a50 0x7f70d00ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 --2- 172.21.15.67:0/2098010239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d00d0e00 0x7f70d0154b10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.560+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 >> 172.21.15.67:0/2098010239 conn(0x7f70d00d5d90 msgr2=0x7f70d006c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.561+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 shutdown_connections 2024-04-25T15:28:12.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.561+0000 7f70d6796640 1 -- 172.21.15.67:0/2098010239 wait complete. 2024-04-25T15:28:12.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:28:12.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.697+0000 7f0c8cbce640 1 Processor -- start 2024-04-25T15:28:12.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.697+0000 7f0c8cbce640 1 -- start start 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.698+0000 7f0c8cbce640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c880b2e60 0x7f0c880d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.698+0000 7f0c8cbce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c88164020 0x7f0c88164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.698+0000 7f0c8cbce640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c88164970 con 0x7f0c880d1e20 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.698+0000 7f0c8cbce640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0c880d23f0 con 0x7f0c880b2e60 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.698+0000 7f0c8cbce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0c880d26d0 con 0x7f0c88164020 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c85d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c880b2e60 0x7f0c880d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c85d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c880b2e60 0x7f0c880d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41508/0 (socket says 172.21.15.67:41508) 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c85d9b640 1 -- 172.21.15.67:0/2870340787 learned_addr learned my addr 172.21.15.67:0/2870340787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c86d9d640 1 --2- 172.21.15.67:0/2870340787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c88164020 0x7f0c88164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c85d9b640 1 -- 172.21.15.67:0/2870340787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c88164020 msgr2=0x7f0c88164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c85d9b640 1 --2- 172.21.15.67:0/2870340787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c88164020 0x7f0c88164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c85d9b640 1 -- 172.21.15.67:0/2870340787 >> v1:172.21.15.176:6789/0 conn(0x7f0c880d1e20 legacy=0x7f0c8808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.699+0000 7f0c85d9b640 1 -- 172.21.15.67:0/2870340787 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c880d29b0 con 0x7f0c880b2e60 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.700+0000 7f0c85d9b640 1 --2- 172.21.15.67:0/2870340787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c880d3490 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009870 tx=0x7f0c7c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256ac6f074ae923 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.700+0000 7f0c8559a640 1 -- 172.21.15.67:0/2870340787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c037070 con 0x7f0c880b2e60 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.700+0000 7f0c8559a640 1 -- 172.21.15.67:0/2870340787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c7c032440 con 0x7f0c880b2e60 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.700+0000 7f0c8559a640 1 -- 172.21.15.67:0/2870340787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c032810 con 0x7f0c880b2e60 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.700+0000 7f0c8cbce640 1 -- 172.21.15.67:0/2870340787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 msgr2=0x7f0c880d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.700+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/2870340787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c880d3490 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009870 tx=0x7f0c7c02a2a0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.701+0000 7f0c8cbce640 1 -- 172.21.15.67:0/2870340787 shutdown_connections 2024-04-25T15:28:12.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.701+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/2870340787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c88164020 0x7f0c88164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.701+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/2870340787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c880d3490 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.701+0000 7f0c8cbce640 1 -- 172.21.15.67:0/2870340787 >> 172.21.15.67:0/2870340787 conn(0x7f0c880d0530 msgr2=0x7f0c880d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:12.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.701+0000 7f0c8cbce640 1 -- 172.21.15.67:0/2870340787 shutdown_connections 2024-04-25T15:28:12.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.701+0000 7f0c8cbce640 1 -- 172.21.15.67:0/2870340787 wait complete. 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.703+0000 7f0c8cbce640 1 Processor -- start 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.703+0000 7f0c8cbce640 1 -- start start 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8cbce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c881f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8cbce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c880d1e20 0x7f0c8815e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8cbce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c88164020 0x7f0c880b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8cbce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0c880ce070 con 0x7f0c880b2e60 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8cbce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0c880d2f70 con 0x7f0c88164020 2024-04-25T15:28:12.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8cbce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0c880ce1f0 con 0x7f0c880d1e20 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c881f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c881f60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41524/0 (socket says 172.21.15.67:41524) 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 -- 172.21.15.67:0/1844735060 learned_addr learned my addr 172.21.15.67:0/1844735060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c85d9b640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c880d1e20 0x7f0c8815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c86d9d640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c88164020 0x7f0c880b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 -- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c880d1e20 msgr2=0x7f0c8815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c880d1e20 0x7f0c8815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 -- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c88164020 msgr2=0x7f0c880b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c88164020 0x7f0c880b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c8659c640 1 -- 172.21.15.67:0/1844735060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c880cd4e0 con 0x7f0c880b2e60 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c85d9b640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c880d1e20 0x7f0c8815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.704+0000 7f0c86d9d640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c88164020 0x7f0c880b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.705+0000 7f0c8659c640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c881f60c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800b3f0 tx=0x7f0c7800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.705+0000 7f0c76ffd640 1 -- 172.21.15.67:0/1844735060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78015020 con 0x7f0c880b2e60 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.705+0000 7f0c76ffd640 1 -- 172.21.15.67:0/1844735060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c78011830 con 0x7f0c880b2e60 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.705+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c880cd7c0 con 0x7f0c880b2e60 2024-04-25T15:28:12.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.705+0000 7f0c76ffd640 1 -- 172.21.15.67:0/1844735060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78011c20 con 0x7f0c880b2e60 2024-04-25T15:28:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.705+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c880b7000 con 0x7f0c880b2e60 2024-04-25T15:28:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.706+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c88050740 con 0x7f0c880b2e60 2024-04-25T15:28:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.707+0000 7f0c76ffd640 1 -- 172.21.15.67:0/1844735060 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0c78018020 con 0x7f0c880b2e60 2024-04-25T15:28:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.707+0000 7f0c76ffd640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0c680744b0 0x7f0c680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.707+0000 7f0c76ffd640 1 -- 172.21.15.67:0/1844735060 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5853+0+0 (secure 0 0 0) 0x7f0c78094f30 con 0x7f0c880b2e60 2024-04-25T15:28:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.708+0000 7f0c85d9b640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0c680744b0 0x7f0c680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.708+0000 7f0c85d9b640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0c680744b0 0x7f0c680769c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c034000 tx=0x7f0c7c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:12.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.712+0000 7f0c76ffd640 1 -- 172.21.15.67:0/1844735060 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c7801f030 con 0x7f0c880b2e60 2024-04-25T15:28:12.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:12.899+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0c88071000 con 0x7f0c880b2e60 2024-04-25T15:28:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.564+0000 7f0c76ffd640 1 -- 172.21.15.67:0/1844735060 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v71) ==== 181+0+0 (secure 0 0 0) 0x7f0c7805f630 con 0x7f0c880b2e60 2024-04-25T15:28:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:28:13.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0c680744b0 msgr2=0x7f0c680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:13.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0c680744b0 0x7f0c680769c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c034000 tx=0x7f0c7c02e000 comp rx=0 tx=0).stop 2024-04-25T15:28:13.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 msgr2=0x7f0c881f60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:13.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c881f60c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800b3f0 tx=0x7f0c7800b8b0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 shutdown_connections 2024-04-25T15:28:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0c680744b0 0x7f0c680769c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c88164020 0x7f0c880b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c880d1e20 0x7f0c8815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 --2- 172.21.15.67:0/1844735060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c880b2e60 0x7f0c881f60c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 >> 172.21.15.67:0/1844735060 conn(0x7f0c880d0530 msgr2=0x7f0c880b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 shutdown_connections 2024-04-25T15:28:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.567+0000 7f0c8cbce640 1 -- 172.21.15.67:0/1844735060 wait complete. 2024-04-25T15:28:13.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:28:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.703+0000 7f368717d640 1 Processor -- start 2024-04-25T15:28:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.704+0000 7f368717d640 1 -- start start 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368717d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f36800b2e60 0x7f36800d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368717d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3680164020 0x7f3680164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368717d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3680164970 con 0x7f36800d1e20 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368717d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f36800d23f0 con 0x7f36800b2e60 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368717d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f36800d26d0 con 0x7f3680164020 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368571a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3680164020 0x7f3680164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f367ffff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f36800b2e60 0x7f36800d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368571a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3680164020 0x7f3680164430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:45852/0 (socket says 172.21.15.67:45852) 2024-04-25T15:28:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f367ffff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f36800b2e60 0x7f36800d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:59686/0 (socket says 172.21.15.67:59686) 2024-04-25T15:28:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f3684f19640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f36800d1e20 0x7f368008d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:37540/0 (socket says 172.21.15.67:37540) 2024-04-25T15:28:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.705+0000 7f368571a640 1 -- 172.21.15.67:0/1655271636 learned_addr learned my addr 172.21.15.67:0/1655271636 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.706+0000 7f367ffff640 1 -- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3680164020 msgr2=0x7f3680164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.706+0000 7f367ffff640 1 --2- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3680164020 0x7f3680164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.706+0000 7f367ffff640 1 -- 172.21.15.67:0/1655271636 >> v1:172.21.15.176:6789/0 conn(0x7f36800d1e20 legacy=0x7f368008d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.706+0000 7f367ffff640 1 -- 172.21.15.67:0/1655271636 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36800d29b0 con 0x7f36800b2e60 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.706+0000 7f368571a640 1 --2- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3680164020 0x7f3680164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.706+0000 7f367ffff640 1 --2- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36800b2e60 0x7f36800d3490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f36680097a0 tx=0x7f366802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=188be552aeca1dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f367f7fe640 1 -- 172.21.15.67:0/1655271636 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3668037070 con 0x7f36800b2e60 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f367f7fe640 1 -- 172.21.15.67:0/1655271636 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3668032440 con 0x7f36800b2e60 2024-04-25T15:28:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f367f7fe640 1 -- 172.21.15.67:0/1655271636 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3668032810 con 0x7f36800b2e60 2024-04-25T15:28:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 -- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36800b2e60 msgr2=0x7f36800d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 --2- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36800b2e60 0x7f36800d3490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f36680097a0 tx=0x7f366802a210 comp rx=0 tx=0).stop 2024-04-25T15:28:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 -- 172.21.15.67:0/1655271636 shutdown_connections 2024-04-25T15:28:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 --2- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3680164020 0x7f3680164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 --2- 172.21.15.67:0/1655271636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36800b2e60 0x7f36800d3490 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 -- 172.21.15.67:0/1655271636 >> 172.21.15.67:0/1655271636 conn(0x7f36800d0530 msgr2=0x7f36800d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 -- 172.21.15.67:0/1655271636 shutdown_connections 2024-04-25T15:28:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.707+0000 7f368717d640 1 -- 172.21.15.67:0/1655271636 wait complete. 2024-04-25T15:28:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.709+0000 7f368717d640 1 Processor -- start 2024-04-25T15:28:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.709+0000 7f368717d640 1 -- start start 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f368717d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36800b2e60 0x7f368014d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f368717d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36800d1e20 0x7f36800b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f368717d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3680164020 0x7f36800caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f368717d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f36800ce070 con 0x7f36800b2e60 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f368717d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f36800d2f70 con 0x7f3680164020 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f368717d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f36800ce1f0 con 0x7f36800d1e20 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36800b2e60 0x7f368014d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36800b2e60 0x7f368014d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41526/0 (socket says 172.21.15.67:41526) 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 -- 172.21.15.67:0/1475177804 learned_addr learned my addr 172.21.15.67:0/1475177804 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f367ffff640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36800d1e20 0x7f36800b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f368571a640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3680164020 0x7f36800caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 -- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36800d1e20 msgr2=0x7f36800b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36800d1e20 0x7f36800b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 -- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3680164020 msgr2=0x7f36800caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3680164020 0x7f36800caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 -- 172.21.15.67:0/1475177804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368014d600 con 0x7f36800b2e60 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f3684f19640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36800b2e60 0x7f368014d0c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f367000c420 tx=0x7f367000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.711+0000 7f367d7fa640 1 -- 172.21.15.67:0/1475177804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700074e0 con 0x7f36800b2e60 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.712+0000 7f367d7fa640 1 -- 172.21.15.67:0/1475177804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3670007680 con 0x7f36800b2e60 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.712+0000 7f367d7fa640 1 -- 172.21.15.67:0/1475177804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670004a00 con 0x7f36800b2e60 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.712+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36800b4720 con 0x7f36800b2e60 2024-04-25T15:28:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.712+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36800b4ce0 con 0x7f36800b2e60 2024-04-25T15:28:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.713+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36800713e0 con 0x7f36800b2e60 2024-04-25T15:28:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.713+0000 7f367d7fa640 1 -- 172.21.15.67:0/1475177804 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f367001a020 con 0x7f36800b2e60 2024-04-25T15:28:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.714+0000 7f367d7fa640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f36600744e0 0x7f36600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.714+0000 7f367d7fa640 1 -- 172.21.15.67:0/1475177804 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5415+0+0 (secure 0 0 0) 0x7f3670094770 con 0x7f36800b2e60 2024-04-25T15:28:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.714+0000 7f367ffff640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f36600744e0 0x7f36600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.714+0000 7f367ffff640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f36600744e0 0x7f36600769f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3668034000 tx=0x7f366802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.718+0000 7f367d7fa640 1 -- 172.21.15.67:0/1475177804 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36800713e0 con 0x7f36800b2e60 2024-04-25T15:28:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:13.909+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3680050740 con 0x7f36800b2e60 2024-04-25T15:28:14.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.570+0000 7f367d7fa640 1 -- 172.21.15.67:0/1475177804 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v72) ==== 169+0+0 (secure 0 0 0) 0x7f3680164970 con 0x7f36800b2e60 2024-04-25T15:28:14.571 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:28:14.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.574+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f36600744e0 msgr2=0x7f36600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.574+0000 7f368717d640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f36600744e0 0x7f36600769f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3668034000 tx=0x7f366802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:14.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.574+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36800b2e60 msgr2=0x7f368014d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.574+0000 7f368717d640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36800b2e60 0x7f368014d0c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f367000c420 tx=0x7f367000c8e0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.575+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 shutdown_connections 2024-04-25T15:28:14.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.576+0000 7f368717d640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f36600744e0 0x7f36600769f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.576+0000 7f368717d640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3680164020 0x7f36800caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.576+0000 7f368717d640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36800d1e20 0x7f36800b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.576+0000 7f368717d640 1 --2- 172.21.15.67:0/1475177804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36800b2e60 0x7f368014d0c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.576+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 >> 172.21.15.67:0/1475177804 conn(0x7f36800d0530 msgr2=0x7f36800ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:14.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.577+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 shutdown_connections 2024-04-25T15:28:14.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.577+0000 7f368717d640 1 -- 172.21.15.67:0/1475177804 wait complete. 2024-04-25T15:28:14.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:28:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.711+0000 7f8e52005640 1 Processor -- start 2024-04-25T15:28:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.711+0000 7f8e52005640 1 -- start start 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e52005640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8e4c0b2e60 0x7f8e4c0d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e52005640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4c0d23f0 con 0x7f8e4c164020 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e52005640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4c0d26d0 con 0x7f8e4c0d1e20 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e52005640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8e4c0d29b0 con 0x7f8e4c0b2e60 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e4bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8e4c164020 0x7f8e4c164430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52116/0 (socket says 172.21.15.67:52116) 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e4bfff640 1 -- 172.21.15.67:0/1653566435 learned_addr learned my addr 172.21.15.67:0/1653566435 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e4affd640 1 --2- 172.21.15.67:0/1653566435 >> v2:172.21.15.176:3300/0 conn(0x7f8e4c0b2e60 0x7f8e4c0d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216631977 0 0) 0x7f8e4c0d23f0 con 0x7f8e4c164020 2024-04-25T15:28:14.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.713+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e34003520 con 0x7f8e4c164020 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458720722 0 0) 0x7f8e4c0d26d0 con 0x7f8e4c0d1e20 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e4c0d23f0 con 0x7f8e4c0d1e20 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1970751392 0 0) 0x7f8e34003520 con 0x7f8e4c164020 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 >> v2:172.21.15.176:3300/0 conn(0x7f8e4c0b2e60 msgr2=0x7f8e4c0d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 --2- 172.21.15.67:0/1653566435 >> v2:172.21.15.176:3300/0 conn(0x7f8e4c0b2e60 0x7f8e4c0d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 >> v1:172.21.15.176:6789/0 conn(0x7f8e4c0d1e20 legacy=0x7f8e4c08d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c0d2c90 con 0x7f8e4c164020 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8e440019f0 con 0x7f8e4c164020 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 >> v1:172.21.15.67:6789/0 conn(0x7f8e4c164020 legacy=0x7f8e4c164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e34003b90 0x7f8e34003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e34004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e34005210 0x7f8e3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c0d29b0 con 0x7f8e34004800 2024-04-25T15:28:14.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3400c0f0 con 0x7f8e34003b90 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3400c2f0 con 0x7f8e34005210 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4affd640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e34004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4b7fe640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e34003b90 0x7f8e34003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4bfff640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e34005210 0x7f8e3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4affd640 1 -- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e34005210 msgr2=0x7f8e3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4affd640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e34005210 0x7f8e3400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4affd640 1 -- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e34003b90 msgr2=0x7f8e34003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4affd640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e34003b90 0x7f8e34003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4affd640 1 -- 172.21.15.67:0/1653566435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e34003520 con 0x7f8e34004800 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.714+0000 7f8e4bfff640 1 -- 172.21.15.67:0/1653566435 reap_dead start 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.715+0000 7f8e4affd640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e34004c10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009ac0 tx=0x7f8e34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518ed61faadd5ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.715+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40036070 con 0x7f8e34004800 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.715+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e40031400 con 0x7f8e34004800 2024-04-25T15:28:14.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.715+0000 7f8e4a7fc640 1 -- 172.21.15.67:0/1653566435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40009690 con 0x7f8e34004800 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.716+0000 7f8e52005640 1 -- 172.21.15.67:0/1653566435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 msgr2=0x7f8e34004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.716+0000 7f8e52005640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e34004c10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009ac0 tx=0x7f8e34005830 comp rx=0 tx=0).stop 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.716+0000 7f8e52005640 1 -- 172.21.15.67:0/1653566435 shutdown_connections 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.716+0000 7f8e52005640 1 --2- 172.21.15.67:0/1653566435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e34004c10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.716+0000 7f8e52005640 1 -- 172.21.15.67:0/1653566435 >> 172.21.15.67:0/1653566435 conn(0x7f8e4c0d0530 msgr2=0x7f8e4c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.716+0000 7f8e52005640 1 -- 172.21.15.67:0/1653566435 shutdown_connections 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.716+0000 7f8e52005640 1 -- 172.21.15.67:0/1653566435 wait complete. 2024-04-25T15:28:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.718+0000 7f8e52005640 1 Processor -- start 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.718+0000 7f8e52005640 1 -- start start 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e52005640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e4c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e52005640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e4c15ef00 0x7f8e4c160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e52005640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e4c15d6f0 0x7f8e4c160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e52005640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c0d3610 con 0x7f8e34004800 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e52005640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c0d3490 con 0x7f8e4c15d6f0 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e52005640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4c0d3790 con 0x7f8e4c15ef00 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e4b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e4c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e4b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e4c155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41544/0 (socket says 172.21.15.67:41544) 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.719+0000 7f8e4b7fe640 1 -- 172.21.15.67:0/2910775652 learned_addr learned my addr 172.21.15.67:0/2910775652 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4affd640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e4c15ef00 0x7f8e4c160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4bfff640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e4c15d6f0 0x7f8e4c160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4b7fe640 1 -- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e4c15ef00 msgr2=0x7f8e4c160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4b7fe640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e4c15ef00 0x7f8e4c160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4b7fe640 1 -- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e4c15d6f0 msgr2=0x7f8e4c160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4b7fe640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e4c15d6f0 0x7f8e4c160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4b7fe640 1 -- 172.21.15.67:0/2910775652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c15de60 con 0x7f8e34004800 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4bfff640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e4c15d6f0 0x7f8e4c160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e4b7fe640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e4c155fe0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00f7d0 tx=0x7f8e3c00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e33fff640 1 -- 172.21.15.67:0/2910775652 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c01a070 con 0x7f8e34004800 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e33fff640 1 -- 172.21.15.67:0/2910775652 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3c005320 con 0x7f8e34004800 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4c15e140 con 0x7f8e34004800 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e33fff640 1 -- 172.21.15.67:0/2910775652 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c007270 con 0x7f8e34004800 2024-04-25T15:28:14.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.720+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4c15e700 con 0x7f8e34004800 2024-04-25T15:28:14.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.721+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4c050740 con 0x7f8e34004800 2024-04-25T15:28:14.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.722+0000 7f8e33fff640 1 -- 172.21.15.67:0/2910775652 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8e3c01d070 con 0x7f8e34004800 2024-04-25T15:28:14.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.722+0000 7f8e33fff640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e380744b0 0x7f8e380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:14.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.722+0000 7f8e33fff640 1 -- 172.21.15.67:0/2910775652 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7f8e3c0947d0 con 0x7f8e34004800 2024-04-25T15:28:14.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.722+0000 7f8e4affd640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e380744b0 0x7f8e380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:14.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.723+0000 7f8e4affd640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e380744b0 0x7f8e380769c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8e40033000 tx=0x7f8e40008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:14.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.727+0000 7f8e33fff640 1 -- 172.21.15.67:0/2910775652 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e3c05f260 con 0x7f8e34004800 2024-04-25T15:28:14.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.913+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f8e4c071000 con 0x7f8e34004800 2024-04-25T15:28:14.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.916+0000 7f8e33fff640 1 -- 172.21.15.67:0/2910775652 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7f8e3c05eb60 con 0x7f8e34004800 2024-04-25T15:28:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e380744b0 msgr2=0x7f8e380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e380744b0 0x7f8e380769c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8e40033000 tx=0x7f8e40008030 comp rx=0 tx=0).stop 2024-04-25T15:28:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 msgr2=0x7f8e4c155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e4c155fe0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00f7d0 tx=0x7f8e3c00fc90 comp rx=0 tx=0).stop 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 shutdown_connections 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8e380744b0 0x7f8e380769c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e4c15d6f0 0x7f8e4c160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e4c15ef00 0x7f8e4c160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 --2- 172.21.15.67:0/2910775652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e34004800 0x7f8e4c155fe0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.918+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 >> 172.21.15.67:0/2910775652 conn(0x7f8e4c0d0530 msgr2=0x7f8e4c08ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.919+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 shutdown_connections 2024-04-25T15:28:14.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:14.919+0000 7f8e52005640 1 -- 172.21.15.67:0/2910775652 wait complete. 2024-04-25T15:28:14.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df' 2024-04-25T15:28:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.041+0000 7fe37bf1f640 1 Processor -- start 2024-04-25T15:28:15.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.041+0000 7fe37bf1f640 1 -- start start 2024-04-25T15:28:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe37bf1f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe374164b00 0x7fe3740b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe37bf1f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37415df80 0x7fe3740b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe37bf1f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3741615f0 con 0x7fe374160890 2024-04-25T15:28:15.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe37bf1f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe3741613b0 con 0x7fe374164b00 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe37bf1f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe374164670 con 0x7fe37415df80 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe3794ba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37415df80 0x7fe3740b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe379cbb640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe374164b00 0x7fe3740b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe379cbb640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe374164b00 0x7fe3740b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:59730/0 (socket says 172.21.15.67:59730) 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe379cbb640 1 -- 172.21.15.67:0/883430975 learned_addr learned my addr 172.21.15.67:0/883430975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe3794ba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37415df80 0x7fe3740b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:45906/0 (socket says 172.21.15.67:45906) 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe37a4bc640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe374160890 0x7fe374160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:37552/0 (socket says 172.21.15.67:37552) 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe379cbb640 1 -- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37415df80 msgr2=0x7fe3740b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.043+0000 7fe379cbb640 1 --2- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37415df80 0x7fe3740b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.044+0000 7fe379cbb640 1 -- 172.21.15.67:0/883430975 >> v1:172.21.15.176:6789/0 conn(0x7fe374160890 legacy=0x7fe374160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.044+0000 7fe379cbb640 1 -- 172.21.15.67:0/883430975 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe374164020 con 0x7fe374164b00 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.044+0000 7fe378cb9640 1 -- 172.21.15.67:0/883430975 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2957220002 0 0) 0x7fe3741615f0 con 0x7fe374160890 2024-04-25T15:28:15.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.044+0000 7fe3794ba640 1 --2- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37415df80 0x7fe3740b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.044+0000 7fe379cbb640 1 --2- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe374164b00 0x7fe3740b7b80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009870 tx=0x7fe36c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bacd8cd12e8425b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.044+0000 7fe378cb9640 1 -- 172.21.15.67:0/883430975 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c037070 con 0x7fe374164b00 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe378cb9640 1 -- 172.21.15.67:0/883430975 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c032440 con 0x7fe374164b00 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe378cb9640 1 -- 172.21.15.67:0/883430975 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c032810 con 0x7fe374164b00 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 -- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe374164b00 msgr2=0x7fe3740b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe374164b00 0x7fe3740b7b80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009870 tx=0x7fe36c02a270 comp rx=0 tx=0).stop 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 -- 172.21.15.67:0/883430975 shutdown_connections 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe37415df80 0x7fe3740b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/883430975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe374164b00 0x7fe3740b7b80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 -- 172.21.15.67:0/883430975 >> 172.21.15.67:0/883430975 conn(0x7fe3740d2bc0 msgr2=0x7fe3740d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 -- 172.21.15.67:0/883430975 shutdown_connections 2024-04-25T15:28:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.045+0000 7fe37bf1f640 1 -- 172.21.15.67:0/883430975 wait complete. 2024-04-25T15:28:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.047+0000 7fe37bf1f640 1 Processor -- start 2024-04-25T15:28:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.047+0000 7fe37bf1f640 1 -- start start 2024-04-25T15:28:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.048+0000 7fe37bf1f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe37415df80 0x7fe3741f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.048+0000 7fe37bf1f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe374160890 0x7fe3740b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.048+0000 7fe37bf1f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe374164b00 0x7fe3740b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.048+0000 7fe37bf1f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe3740b4a00 con 0x7fe374160890 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.048+0000 7fe37bf1f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3740b4880 con 0x7fe37415df80 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.048+0000 7fe37bf1f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3740b4b80 con 0x7fe374164b00 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.048+0000 7fe3794ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe374160890 0x7fe3740b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe374160890 0x7fe3740b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41550/0 (socket says 172.21.15.67:41550) 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 -- 172.21.15.67:0/7689225 learned_addr learned my addr 172.21.15.67:0/7689225 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe379cbb640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe37415df80 0x7fe3741f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe37a4bc640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe374164b00 0x7fe3740b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 -- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe374164b00 msgr2=0x7fe3740b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe374164b00 0x7fe3740b5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 -- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe37415df80 msgr2=0x7fe3741f64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe37415df80 0x7fe3741f64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 -- 172.21.15.67:0/7689225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3740b6810 con 0x7fe374160890 2024-04-25T15:28:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3794ba640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe374160890 0x7fe3740b5a50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe36800e990 tx=0x7fe36800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3727fc640 1 -- 172.21.15.67:0/7689225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe368017070 con 0x7fe374160890 2024-04-25T15:28:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3727fc640 1 -- 172.21.15.67:0/7689225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36800c9e0 con 0x7fe374160890 2024-04-25T15:28:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe3727fc640 1 -- 172.21.15.67:0/7689225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36800cdb0 con 0x7fe374160890 2024-04-25T15:28:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3741545f0 con 0x7fe374160890 2024-04-25T15:28:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.049+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe374142eb0 con 0x7fe374160890 2024-04-25T15:28:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.050+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe374050740 con 0x7fe374160890 2024-04-25T15:28:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.051+0000 7fe3727fc640 1 -- 172.21.15.67:0/7689225 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe36801a070 con 0x7fe374160890 2024-04-25T15:28:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.051+0000 7fe3727fc640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3600744e0 0x7fe3600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.052+0000 7fe3727fc640 1 -- 172.21.15.67:0/7689225 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7fe3680950d0 con 0x7fe374160890 2024-04-25T15:28:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.052+0000 7fe379cbb640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3600744e0 0x7fe3600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.052+0000 7fe379cbb640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3600744e0 0x7fe3600769f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe36c034000 tx=0x7fe36c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.056+0000 7fe3727fc640 1 -- 172.21.15.67:0/7689225 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe36805f520 con 0x7fe374160890 2024-04-25T15:28:15.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.319+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]} v 0) -- 0x7fe3740d4170 con 0x7fe374160890 2024-04-25T15:28:15.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.576+0000 7fe3727fc640 1 -- 172.21.15.67:0/7689225 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]}]=0 v181) ==== 134+0+0 (secure 0 0 0) 0x7fe368063820 con 0x7fe374160890 2024-04-25T15:28:15.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.578+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3600744e0 msgr2=0x7fe3600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.578+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3600744e0 0x7fe3600769f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe36c034000 tx=0x7fe36c02e000 comp rx=0 tx=0).stop 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.578+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe374160890 msgr2=0x7fe3740b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.578+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe374160890 0x7fe3740b5a50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe36800e990 tx=0x7fe36800ee50 comp rx=0 tx=0).stop 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 shutdown_connections 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe3600744e0 0x7fe3600769f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe374164b00 0x7fe3740b5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe374160890 0x7fe3740b5a50 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 --2- 172.21.15.67:0/7689225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe37415df80 0x7fe3741f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 >> 172.21.15.67:0/7689225 conn(0x7fe3740d2bc0 msgr2=0x7fe374160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 shutdown_connections 2024-04-25T15:28:15.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.579+0000 7fe37bf1f640 1 -- 172.21.15.67:0/7689225 wait complete. 2024-04-25T15:28:15.591 INFO:tasks.cephfs_test_runner:That when a client modifies the quota on a directory used ... ok 2024-04-25T15:28:15.591 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_getfattr (tasks.cephfs.test_quota.TestQuota) That quota changes made from one client are visible to another 2024-04-25T15:28:15.591 INFO:tasks.cephfs_test_runner:test_remote_update_getfattr (tasks.cephfs.test_quota.TestQuota) 2024-04-25T15:28:15.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:15.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.704+0000 7f56cfa9f640 1 Processor -- start 2024-04-25T15:28:15.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.704+0000 7f56cfa9f640 1 -- start start 2024-04-25T15:28:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cfa9f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56c80cfd40 0x7f56c80b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cfa9f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80b3e00 0x7f56c80b7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cfa9f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f56c80b7d80 0x7f56c80d2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cfa9f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f56c80d27e0 con 0x7f56c80cfd40 2024-04-25T15:28:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cfa9f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f56c80b8190 con 0x7f56c80b7d80 2024-04-25T15:28:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cfa9f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f56c80d1c10 con 0x7f56c80b3e00 2024-04-25T15:28:15.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cd83b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56c80cfd40 0x7f56c80b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cd83b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56c80cfd40 0x7f56c80b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41552/0 (socket says 172.21.15.67:41552) 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.705+0000 7f56cd83b640 1 -- 172.21.15.67:0/4098855870 learned_addr learned my addr 172.21.15.67:0/4098855870 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56ce03c640 1 --2- 172.21.15.67:0/4098855870 >> v2:172.21.15.176:3300/0 conn(0x7f56c80b7d80 0x7f56c80d2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56cd03a640 1 --2- 172.21.15.67:0/4098855870 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80b3e00 0x7f56c80b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56cd83b640 1 -- 172.21.15.67:0/4098855870 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80b3e00 msgr2=0x7f56c80b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56cd83b640 1 --2- 172.21.15.67:0/4098855870 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80b3e00 0x7f56c80b7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56cd83b640 1 -- 172.21.15.67:0/4098855870 >> v2:172.21.15.176:3300/0 conn(0x7f56c80b7d80 msgr2=0x7f56c80d2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56cd83b640 1 --2- 172.21.15.67:0/4098855870 >> v2:172.21.15.176:3300/0 conn(0x7f56c80b7d80 0x7f56c80d2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56cd83b640 1 -- 172.21.15.67:0/4098855870 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c80d1eb0 con 0x7f56c80cfd40 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.706+0000 7f56ce03c640 1 --2- 172.21.15.67:0/4098855870 >> v2:172.21.15.176:3300/0 conn(0x7f56c80b7d80 0x7f56c80d2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:15.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.707+0000 7f56cd83b640 1 --2- 172.21.15.67:0/4098855870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80cfd40 0x7f56c80b21a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f56c80b59b0 tx=0x7f56bc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164009c133812ee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.707+0000 7f56cc839640 1 -- 172.21.15.67:0/4098855870 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc037070 con 0x7f56c80cfd40 2024-04-25T15:28:15.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.707+0000 7f56cc839640 1 -- 172.21.15.67:0/4098855870 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56bc032440 con 0x7f56c80cfd40 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.707+0000 7f56cc839640 1 -- 172.21.15.67:0/4098855870 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc032810 con 0x7f56c80cfd40 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.707+0000 7f56cfa9f640 1 -- 172.21.15.67:0/4098855870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80cfd40 msgr2=0x7f56c80b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.707+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/4098855870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80cfd40 0x7f56c80b21a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f56c80b59b0 tx=0x7f56bc02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.707+0000 7f56cfa9f640 1 -- 172.21.15.67:0/4098855870 shutdown_connections 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.708+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/4098855870 >> v2:172.21.15.176:3300/0 conn(0x7f56c80b7d80 0x7f56c80d2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.708+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/4098855870 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80b3e00 0x7f56c80b7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.708+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/4098855870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80cfd40 0x7f56c80b21a0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.708+0000 7f56cfa9f640 1 -- 172.21.15.67:0/4098855870 >> 172.21.15.67:0/4098855870 conn(0x7f56c809b7d0 msgr2=0x7f56c809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.708+0000 7f56cfa9f640 1 -- 172.21.15.67:0/4098855870 shutdown_connections 2024-04-25T15:28:15.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.708+0000 7f56cfa9f640 1 -- 172.21.15.67:0/4098855870 wait complete. 2024-04-25T15:28:15.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.709+0000 7f56cfa9f640 1 Processor -- start 2024-04-25T15:28:15.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.709+0000 7f56cfa9f640 1 -- start start 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cfa9f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80b3e00 0x7f56c814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cfa9f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56c80b7d80 0x7f56c815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cfa9f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80cfd40 0x7f56c80cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cfa9f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56c80b4210 con 0x7f56c80b3e00 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cfa9f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f56c80d2470 con 0x7f56c80b7d80 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cfa9f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f56c80b4390 con 0x7f56c80cfd40 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80b3e00 0x7f56c814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80b3e00 0x7f56c814e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41554/0 (socket says 172.21.15.67:41554) 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 -- 172.21.15.67:0/510050247 learned_addr learned my addr 172.21.15.67:0/510050247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd03a640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56c80b7d80 0x7f56c815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56ce03c640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80cfd40 0x7f56c80cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 -- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80cfd40 msgr2=0x7f56c80cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80cfd40 0x7f56c80cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 -- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56c80b7d80 msgr2=0x7f56c815b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56c80b7d80 0x7f56c815b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 -- 172.21.15.67:0/510050247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c8162370 con 0x7f56c80b3e00 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56ce03c640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80cfd40 0x7f56c80cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cd83b640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80b3e00 0x7f56c814e830 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f56bc000c00 tx=0x7f56bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56ba7fc640 1 -- 172.21.15.67:0/510050247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc041020 con 0x7f56c80b3e00 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56ba7fc640 1 -- 172.21.15.67:0/510050247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56bc0306c0 con 0x7f56c80b3e00 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c8162630 con 0x7f56c80b3e00 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.711+0000 7f56ba7fc640 1 -- 172.21.15.67:0/510050247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc037040 con 0x7f56c80b3e00 2024-04-25T15:28:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.712+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c8162bf0 con 0x7f56c80b3e00 2024-04-25T15:28:15.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.713+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c8050740 con 0x7f56c80b3e00 2024-04-25T15:28:15.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.713+0000 7f56ba7fc640 1 -- 172.21.15.67:0/510050247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f56bc044020 con 0x7f56c80b3e00 2024-04-25T15:28:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.714+0000 7f56ba7fc640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f56ac0744e0 0x7f56ac0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.714+0000 7f56ba7fc640 1 -- 172.21.15.67:0/510050247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7f56bc0b56a0 con 0x7f56c80b3e00 2024-04-25T15:28:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.714+0000 7f56cd03a640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f56ac0744e0 0x7f56ac0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.715+0000 7f56cd03a640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f56ac0744e0 0x7f56ac0769f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f56c81f9d60 tx=0x7f56c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.719+0000 7f56ba7fc640 1 -- 172.21.15.67:0/510050247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f56bc04b030 con 0x7f56c80b3e00 2024-04-25T15:28:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.902+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56c806c5c0 con 0x7f56c80b3e00 2024-04-25T15:28:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.903+0000 7f56ba7fc640 1 -- 172.21.15.67:0/510050247 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+12985 (secure 0 0 0) 0x7f56bc080130 con 0x7f56c80b3e00 2024-04-25T15:28:15.904 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:15.904 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:14.565879+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/3903876028":"2024-04-26T15:28:11.555227+0000","172.21.15.67:6833/3903876028":"2024-04-26T15:28:11.555227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f56ac0744e0 msgr2=0x7f56ac0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f56ac0744e0 0x7f56ac0769f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f56c81f9d60 tx=0x7f56c4008040 comp rx=0 tx=0).stop 2024-04-25T15:28:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80b3e00 msgr2=0x7f56c814e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80b3e00 0x7f56c814e830 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f56bc000c00 tx=0x7f56bc0046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 shutdown_connections 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f56ac0744e0 0x7f56ac0769f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56c80cfd40 0x7f56c80cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56c80b7d80 0x7f56c815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 --2- 172.21.15.67:0/510050247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56c80b3e00 0x7f56c814e830 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 >> 172.21.15.67:0/510050247 conn(0x7f56c809b7d0 msgr2=0x7f56c81600a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 shutdown_connections 2024-04-25T15:28:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:15.905+0000 7f56cfa9f640 1 -- 172.21.15.67:0/510050247 wait complete. 2024-04-25T15:28:15.917 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:15.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:16.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.030+0000 7efc1b28c640 1 Processor -- start 2024-04-25T15:28:16.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.031+0000 7efc1b28c640 1 -- start start 2024-04-25T15:28:16.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc1b28c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7efc140ce250 0x7efc1415cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc1b28c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc1406d5d0 0x7efc1406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc1b28c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc1406df20 con 0x7efc141458c0 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc1b28c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efc140b4ec0 con 0x7efc1406d5d0 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc1b28c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7efc14145cd0 con 0x7efc140ce250 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19829640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc1406d5d0 0x7efc1406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc18827640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efc141458c0 0x7efc140b47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52130/0 (socket says 172.21.15.67:52130) 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc18827640 1 -- 172.21.15.67:0/4231838956 learned_addr learned my addr 172.21.15.67:0/4231838956 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19829640 1 --2- 172.21.15.67:0/4231838956 >> v2:172.21.15.67:3300/0 conn(0x7efc1406d5d0 0x7efc1406d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41570/0 (socket says 172.21.15.67:41570) 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19028640 1 --2- 172.21.15.67:0/4231838956 >> v2:172.21.15.176:3300/0 conn(0x7efc140ce250 0x7efc1415cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19829640 1 -- 172.21.15.67:0/4231838956 >> v2:172.21.15.176:3300/0 conn(0x7efc140ce250 msgr2=0x7efc1415cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19829640 1 --2- 172.21.15.67:0/4231838956 >> v2:172.21.15.176:3300/0 conn(0x7efc140ce250 0x7efc1415cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19829640 1 -- 172.21.15.67:0/4231838956 >> v1:172.21.15.67:6789/0 conn(0x7efc141458c0 legacy=0x7efc140b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19829640 1 -- 172.21.15.67:0/4231838956 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1415d5c0 con 0x7efc1406d5d0 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc0bfff640 1 -- 172.21.15.67:0/4231838956 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 758606535 0 0) 0x7efc04001110 con 0x7efc141458c0 2024-04-25T15:28:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.032+0000 7efc19028640 1 --2- 172.21.15.67:0/4231838956 >> v2:172.21.15.176:3300/0 conn(0x7efc140ce250 0x7efc1415cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.033+0000 7efc19829640 1 --2- 172.21.15.67:0/4231838956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1406d5d0 0x7efc1406d9e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7efc140b4530 tx=0x7efc0c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65a8f70ad9763f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.033+0000 7efc0bfff640 1 -- 172.21.15.67:0/4231838956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c037070 con 0x7efc1406d5d0 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.033+0000 7efc0bfff640 1 -- 172.21.15.67:0/4231838956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0c032440 con 0x7efc1406d5d0 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.033+0000 7efc0bfff640 1 -- 172.21.15.67:0/4231838956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c032810 con 0x7efc1406d5d0 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 -- 172.21.15.67:0/4231838956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1406d5d0 msgr2=0x7efc1406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 --2- 172.21.15.67:0/4231838956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1406d5d0 0x7efc1406d9e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7efc140b4530 tx=0x7efc0c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 -- 172.21.15.67:0/4231838956 shutdown_connections 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 --2- 172.21.15.67:0/4231838956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1406d5d0 0x7efc1406d9e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 --2- 172.21.15.67:0/4231838956 >> v2:172.21.15.176:3300/0 conn(0x7efc140ce250 0x7efc1415cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 -- 172.21.15.67:0/4231838956 >> 172.21.15.67:0/4231838956 conn(0x7efc14073090 msgr2=0x7efc140734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 -- 172.21.15.67:0/4231838956 shutdown_connections 2024-04-25T15:28:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.034+0000 7efc1b28c640 1 -- 172.21.15.67:0/4231838956 wait complete. 2024-04-25T15:28:16.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.036+0000 7efc1b28c640 1 Processor -- start 2024-04-25T15:28:16.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.036+0000 7efc1b28c640 1 -- start start 2024-04-25T15:28:16.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc1b28c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc1406d5d0 0x7efc140afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc1b28c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc140ce250 0x7efc140b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc1b28c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc141458c0 0x7efc141f0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc1b28c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efc1406f640 con 0x7efc141458c0 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc1b28c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efc1415db80 con 0x7efc1406d5d0 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc1b28c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efc1415de60 con 0x7efc140ce250 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc19829640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc141458c0 0x7efc141f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc19829640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc141458c0 0x7efc141f0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41576/0 (socket says 172.21.15.67:41576) 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.037+0000 7efc19829640 1 -- 172.21.15.67:0/3824733801 learned_addr learned my addr 172.21.15.67:0/3824733801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19028640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc1406d5d0 0x7efc140afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc18827640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc140ce250 0x7efc140b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19829640 1 -- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc140ce250 msgr2=0x7efc140b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19829640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc140ce250 0x7efc140b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19829640 1 -- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc1406d5d0 msgr2=0x7efc140afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19829640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc1406d5d0 0x7efc140afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19829640 1 -- 172.21.15.67:0/3824733801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc140aec00 con 0x7efc141458c0 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19028640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc1406d5d0 0x7efc140afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc19829640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc141458c0 0x7efc141f0830 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7efc0c02a1e0 tx=0x7efc0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc09ffb640 1 -- 172.21.15.67:0/3824733801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c032e60 con 0x7efc141458c0 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc09ffb640 1 -- 172.21.15.67:0/3824733801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0c042070 con 0x7efc141458c0 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc140aeee0 con 0x7efc141458c0 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc09ffb640 1 -- 172.21.15.67:0/3824733801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c037040 con 0x7efc141458c0 2024-04-25T15:28:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.038+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc140af4a0 con 0x7efc141458c0 2024-04-25T15:28:16.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.039+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc14050740 con 0x7efc141458c0 2024-04-25T15:28:16.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.040+0000 7efc09ffb640 1 -- 172.21.15.67:0/3824733801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efc0c030b50 con 0x7efc141458c0 2024-04-25T15:28:16.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.041+0000 7efc09ffb640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efbf80744e0 0x7efbf80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.041+0000 7efc09ffb640 1 -- 172.21.15.67:0/3824733801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7efc0c0b56c0 con 0x7efc141458c0 2024-04-25T15:28:16.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.041+0000 7efc19028640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efbf80744e0 0x7efbf80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.041+0000 7efc19028640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efbf80744e0 0x7efbf80769f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efc140b6c80 tx=0x7efc100074b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:16.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.045+0000 7efc09ffb640 1 -- 172.21.15.67:0/3824733801 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc0c0801e0 con 0x7efc141458c0 2024-04-25T15:28:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.235+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efc14073fe0 con 0x7efc141458c0 2024-04-25T15:28:16.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.236+0000 7efc09ffb640 1 -- 172.21.15.67:0/3824733801 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7efc0c07fae0 con 0x7efc141458c0 2024-04-25T15:28:16.237 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efbf80744e0 msgr2=0x7efbf80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efbf80744e0 0x7efbf80769f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efc140b6c80 tx=0x7efc100074b0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc141458c0 msgr2=0x7efc141f0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc141458c0 0x7efc141f0830 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7efc0c02a1e0 tx=0x7efc0c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 shutdown_connections 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efbf80744e0 0x7efbf80769f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc141458c0 0x7efc141f0830 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc140ce250 0x7efc140b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 --2- 172.21.15.67:0/3824733801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc1406d5d0 0x7efc140afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 >> 172.21.15.67:0/3824733801 conn(0x7efc14073090 msgr2=0x7efc140734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 shutdown_connections 2024-04-25T15:28:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.238+0000 7efc1b28c640 1 -- 172.21.15.67:0/3824733801 wait complete. 2024-04-25T15:28:16.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr' 2024-04-25T15:28:16.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.361+0000 7f536234d640 1 Processor -- start 2024-04-25T15:28:16.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.361+0000 7f536234d640 1 -- start start 2024-04-25T15:28:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.362+0000 7f536234d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 0x7f535c160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.362+0000 7f536234d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f535c1615f0 con 0x7f535c0b8290 2024-04-25T15:28:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.362+0000 7f536234d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f535c164020 con 0x7f535c164b00 2024-04-25T15:28:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.362+0000 7f536234d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f535c164670 con 0x7f535c15df80 2024-04-25T15:28:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.362+0000 7f535b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f535c0b8290 0x7f535c160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52138/0 (socket says 172.21.15.67:52138) 2024-04-25T15:28:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535b7fe640 1 -- 172.21.15.67:0/4275230850 learned_addr learned my addr 172.21.15.67:0/4275230850 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:16.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813801254 0 0) 0x7f535c1615f0 con 0x7f535c0b8290 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f53608ea640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 0x7f535c160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5348003520 con 0x7f535c0b8290 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 567464570 0 0) 0x7f5348003520 con 0x7f535c0b8290 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 msgr2=0x7f535c160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 0x7f535c160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 >> v1:172.21.15.176:6789/0 conn(0x7f535c164b00 legacy=0x7f535c0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535c164300 con 0x7f535c0b8290 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f5344001670 con 0x7f535c0b8290 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 >> v1:172.21.15.67:6789/0 conn(0x7f535c0b8290 legacy=0x7f535c160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53480039e0 0x7f5348003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 0x7f5348004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5348004d40 0x7f5348005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f535c164670 con 0x7f5348004d40 2024-04-25T15:28:16.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f534800baf0 con 0x7f53480039e0 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f534800bcf0 con 0x7f535c15df80 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f53608ea640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5348004d40 0x7f5348005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535bfff640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53480039e0 0x7f5348003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.363+0000 7f535b7fe640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 0x7f5348004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.364+0000 7f53608ea640 1 -- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 msgr2=0x7f5348004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.364+0000 7f53608ea640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c15df80 0x7f5348004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.364+0000 7f53608ea640 1 -- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53480039e0 msgr2=0x7f5348003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.364+0000 7f53608ea640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53480039e0 0x7f5348003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.364+0000 7f53608ea640 1 -- 172.21.15.67:0/4275230850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5348003520 con 0x7f5348004d40 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.364+0000 7f53608ea640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5348004d40 0x7f5348005150 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f535c0d5bb0 tx=0x7f5348005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e271e1dc3b829801 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.364+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354009c10 con 0x7f5348004d40 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5354036070 con 0x7f5348004d40 2024-04-25T15:28:16.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f535affd640 1 -- 172.21.15.67:0/4275230850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53540317f0 con 0x7f5348004d40 2024-04-25T15:28:16.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f536234d640 1 -- 172.21.15.67:0/4275230850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5348004d40 msgr2=0x7f5348005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f536234d640 1 --2- 172.21.15.67:0/4275230850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5348004d40 0x7f5348005150 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f535c0d5bb0 tx=0x7f5348005830 comp rx=0 tx=0).stop 2024-04-25T15:28:16.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f53608ea640 1 -- 172.21.15.67:0/4275230850 reap_dead start 2024-04-25T15:28:16.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f536234d640 1 -- 172.21.15.67:0/4275230850 shutdown_connections 2024-04-25T15:28:16.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f536234d640 1 -- 172.21.15.67:0/4275230850 >> 172.21.15.67:0/4275230850 conn(0x7f535c0d2bc0 msgr2=0x7f535c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:16.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f536234d640 1 -- 172.21.15.67:0/4275230850 shutdown_connections 2024-04-25T15:28:16.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.365+0000 7f536234d640 1 -- 172.21.15.67:0/4275230850 wait complete. 2024-04-25T15:28:16.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.367+0000 7f536234d640 1 Processor -- start 2024-04-25T15:28:16.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.367+0000 7f536234d640 1 -- start start 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.368+0000 7f536234d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5348004d40 0x7f535c151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.368+0000 7f536234d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f535c151a00 0x7f535c162880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.368+0000 7f536234d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c162dc0 0x7f535c1581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.368+0000 7f536234d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1613b0 con 0x7f535c151a00 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.368+0000 7f536234d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f535c161230 con 0x7f5348004d40 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.368+0000 7f536234d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f535c0b4880 con 0x7f535c162dc0 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f535c151a00 0x7f535c162880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f535c151a00 0x7f535c162880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41590/0 (socket says 172.21.15.67:41590) 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 -- 172.21.15.67:0/3303508332 learned_addr learned my addr 172.21.15.67:0/3303508332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535bfff640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5348004d40 0x7f535c151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 -- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c162dc0 msgr2=0x7f535c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c162dc0 0x7f535c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 -- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5348004d40 msgr2=0x7f535c151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5348004d40 0x7f535c151150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 -- 172.21.15.67:0/3303508332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535c1631d0 con 0x7f535c151a00 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f535b7fe640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f535c151a00 0x7f535c162880 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f53440072f0 tx=0x7f53440077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f5358ff9640 1 -- 172.21.15.67:0/3303508332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534400f940 con 0x7f535c151a00 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f5358ff9640 1 -- 172.21.15.67:0/3303508332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5344018070 con 0x7f535c151a00 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f535c1f6460 con 0x7f535c151a00 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f5358ff9640 1 -- 172.21.15.67:0/3303508332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534400fd40 con 0x7f535c151a00 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.369+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f535c1f6a20 con 0x7f535c151a00 2024-04-25T15:28:16.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.370+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f535c050740 con 0x7f535c151a00 2024-04-25T15:28:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.371+0000 7f5358ff9640 1 -- 172.21.15.67:0/3303508332 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f534401d460 con 0x7f535c151a00 2024-04-25T15:28:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.372+0000 7f5358ff9640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f533c0744b0 0x7f533c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.372+0000 7f5358ff9640 1 -- 172.21.15.67:0/3303508332 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7f5344094f90 con 0x7f535c151a00 2024-04-25T15:28:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.372+0000 7f535bfff640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f533c0744b0 0x7f533c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.372+0000 7f535bfff640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f533c0744b0 0x7f533c0769c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f534c005590 tx=0x7f534c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:16.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.376+0000 7f5358ff9640 1 -- 172.21.15.67:0/3303508332 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f535c1631d0 con 0x7f535c151a00 2024-04-25T15:28:16.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:16.634+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]} v 0) -- 0x7f535c152810 con 0x7f535c151a00 2024-04-25T15:28:17.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.585+0000 7f5358ff9640 1 -- 172.21.15.67:0/3303508332 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]}]=0 v183) ==== 143+0+0 (secure 0 0 0) 0x7f534405fa20 con 0x7f535c151a00 2024-04-25T15:28:17.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f533c0744b0 msgr2=0x7f533c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:17.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f533c0744b0 0x7f533c0769c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f534c005590 tx=0x7f534c009040 comp rx=0 tx=0).stop 2024-04-25T15:28:17.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f535c151a00 msgr2=0x7f535c162880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:17.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f535c151a00 0x7f535c162880 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f53440072f0 tx=0x7f53440077b0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 shutdown_connections 2024-04-25T15:28:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f533c0744b0 0x7f533c0769c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f535c162dc0 0x7f535c1581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f535c151a00 0x7f535c162880 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 --2- 172.21.15.67:0/3303508332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5348004d40 0x7f535c151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 >> 172.21.15.67:0/3303508332 conn(0x7f535c0d2bc0 msgr2=0x7f535c0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 shutdown_connections 2024-04-25T15:28:17.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.589+0000 7f536234d640 1 -- 172.21.15.67:0/3303508332 wait complete. 2024-04-25T15:28:17.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:28:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.724+0000 7f7e03e13640 1 Processor -- start 2024-04-25T15:28:17.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.724+0000 7f7e03e13640 1 -- start start 2024-04-25T15:28:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e03e13640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7dfc0b77d0 0x7f7dfc164bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e03e13640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7dfc0d0d60 0x7f7dfc160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e03e13640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dfc0cffa0 con 0x7f7dfc1608a0 2024-04-25T15:28:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e03e13640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7dfc0b7be0 con 0x7f7dfc0b77d0 2024-04-25T15:28:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e03e13640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7dfc165110 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e01baf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7dfc0b77d0 0x7f7dfc164bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e01baf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7dfc0b77d0 0x7f7dfc164bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51904/0 (socket says 172.21.15.67:51904) 2024-04-25T15:28:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e01baf640 1 -- 172.21.15.67:0/1929544655 learned_addr learned my addr 172.21.15.67:0/1929544655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e013ae640 1 --2- 172.21.15.67:0/1929544655 >> v2:172.21.15.176:3300/0 conn(0x7f7dfc0d0d60 0x7f7dfc160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e01baf640 1 -- 172.21.15.67:0/1929544655 >> v2:172.21.15.176:3300/0 conn(0x7f7dfc0d0d60 msgr2=0x7f7dfc160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e01baf640 1 --2- 172.21.15.67:0/1929544655 >> v2:172.21.15.176:3300/0 conn(0x7f7dfc0d0d60 0x7f7dfc160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e01baf640 1 -- 172.21.15.67:0/1929544655 >> v1:172.21.15.176:6789/0 conn(0x7f7dfc1608a0 legacy=0x7f7dfc0b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:17.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.726+0000 7f7e01baf640 1 -- 172.21.15.67:0/1929544655 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dfc0d0280 con 0x7f7dfc0b77d0 2024-04-25T15:28:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.727+0000 7f7e01baf640 1 --2- 172.21.15.67:0/1929544655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0b77d0 0x7f7dfc164bd0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc072d40 tx=0x7f7df402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443fee40f1676e65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.727+0000 7f7e00bad640 1 -- 172.21.15.67:0/1929544655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4037070 con 0x7f7dfc0b77d0 2024-04-25T15:28:17.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.727+0000 7f7e00bad640 1 -- 172.21.15.67:0/1929544655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df4032640 con 0x7f7dfc0b77d0 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.727+0000 7f7e00bad640 1 -- 172.21.15.67:0/1929544655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4032a10 con 0x7f7dfc0b77d0 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 -- 172.21.15.67:0/1929544655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0b77d0 msgr2=0x7f7dfc164bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 --2- 172.21.15.67:0/1929544655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0b77d0 0x7f7dfc164bd0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc072d40 tx=0x7f7df402a300 comp rx=0 tx=0).stop 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 -- 172.21.15.67:0/1929544655 shutdown_connections 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 --2- 172.21.15.67:0/1929544655 >> v2:172.21.15.176:3300/0 conn(0x7f7dfc0d0d60 0x7f7dfc160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 --2- 172.21.15.67:0/1929544655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0b77d0 0x7f7dfc164bd0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 -- 172.21.15.67:0/1929544655 >> 172.21.15.67:0/1929544655 conn(0x7f7dfc09b7d0 msgr2=0x7f7dfc0b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 -- 172.21.15.67:0/1929544655 shutdown_connections 2024-04-25T15:28:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.728+0000 7f7e03e13640 1 -- 172.21.15.67:0/1929544655 wait complete. 2024-04-25T15:28:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.730+0000 7f7e03e13640 1 Processor -- start 2024-04-25T15:28:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.730+0000 7f7e03e13640 1 -- start start 2024-04-25T15:28:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e03e13640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0b77d0 0x7f7dfc15e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e03e13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0d0d60 0x7f7dfc160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e03e13640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc1614e0 0x7f7dfc0ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e03e13640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc0d1170 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e03e13640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc0d08d0 con 0x7f7dfc1614e0 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e03e13640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7dfc0d12f0 con 0x7f7dfc0b77d0 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e013ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0d0d60 0x7f7dfc160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e013ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0d0d60 0x7f7dfc160fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51910/0 (socket says 172.21.15.67:51910) 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e013ae640 1 -- 172.21.15.67:0/948633683 learned_addr learned my addr 172.21.15.67:0/948633683 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e01baf640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0b77d0 0x7f7dfc15e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e023b0640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc1614e0 0x7f7dfc0ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e013ae640 1 -- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0b77d0 msgr2=0x7f7dfc15e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e013ae640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0b77d0 0x7f7dfc15e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e013ae640 1 -- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc1614e0 msgr2=0x7f7dfc0ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.731+0000 7f7e013ae640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc1614e0 0x7f7dfc0ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7e013ae640 1 -- 172.21.15.67:0/948633683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dfc163b50 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7e023b0640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc1614e0 0x7f7dfc0ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7e013ae640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0d0d60 0x7f7dfc160fa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7df000d5f0 tx=0x7f7df000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7dfa7fc640 1 -- 172.21.15.67:0/948633683 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df00125f0 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7dfa7fc640 1 -- 172.21.15.67:0/948633683 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df0019070 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dfc163e30 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7dfa7fc640 1 -- 172.21.15.67:0/948633683 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df00129d0 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.732+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dfc1643f0 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.733+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dfc06e2b0 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.733+0000 7f7dfa7fc640 1 -- 172.21.15.67:0/948633683 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7df0012b70 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.734+0000 7f7dfa7fc640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7de80744b0 0x7f7de80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.734+0000 7f7dfa7fc640 1 -- 172.21.15.67:0/948633683 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7f7df00952a0 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.734+0000 7f7e01baf640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7de80744b0 0x7f7de80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.735+0000 7f7e01baf640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7de80744b0 0x7f7de80769c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7df4034000 tx=0x7f7df402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.739+0000 7f7dfa7fc640 1 -- 172.21.15.67:0/948633683 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7df005fd30 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.925+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f7dfc0d2e30 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.929+0000 7f7dfa7fc640 1 -- 172.21.15.67:0/948633683 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7f7df00950c0 con 0x7f7dfc0d0d60 2024-04-25T15:28:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7de80744b0 msgr2=0x7f7de80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7de80744b0 0x7f7de80769c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7df4034000 tx=0x7f7df402e000 comp rx=0 tx=0).stop 2024-04-25T15:28:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0d0d60 msgr2=0x7f7dfc160fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0d0d60 0x7f7dfc160fa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7df000d5f0 tx=0x7f7df000dab0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 shutdown_connections 2024-04-25T15:28:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7de80744b0 0x7f7de80769c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc1614e0 0x7f7dfc0ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dfc0d0d60 0x7f7dfc160fa0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 --2- 172.21.15.67:0/948633683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0b77d0 0x7f7dfc15e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 >> 172.21.15.67:0/948633683 conn(0x7f7dfc09b7d0 msgr2=0x7f7dfc0b0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 shutdown_connections 2024-04-25T15:28:17.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:17.931+0000 7f7e03e13640 1 -- 172.21.15.67:0/948633683 wait complete. 2024-04-25T15:28:17.943 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:28:17.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:17.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:18.023 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:28:18.023 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:18.024 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:18.041 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:28:18.042 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:18.042 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:18.081 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:28:18.081 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:18.081 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:18.097 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:28:18.097 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:18.097 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:18.138 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:28:18.138 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:28:18.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:18.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.249+0000 7fdbb201a640 1 Processor -- start 2024-04-25T15:28:18.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.249+0000 7fdbb201a640 1 -- start start 2024-04-25T15:28:18.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbb201a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdbac160870 0x7fdbac0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbb201a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbac0d2410 0x7fdbac0b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbb201a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac0cfb30 con 0x7fdbac0cf300 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbb201a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdbac0d00f0 con 0x7fdbac0d2410 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbb201a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdbac0b2470 con 0x7fdbac160870 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbabfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbac0d2410 0x7fdbac0b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbaaffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdbac0cf300 0x7fdbac0d1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56990/0 (socket says 172.21.15.67:56990) 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.250+0000 7fdbaaffd640 1 -- 172.21.15.67:0/44931828 learned_addr learned my addr 172.21.15.67:0/44931828 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbab7fe640 1 --2- 172.21.15.67:0/44931828 >> v2:172.21.15.176:3300/0 conn(0x7fdbac160870 0x7fdbac0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbabfff640 1 -- 172.21.15.67:0/44931828 >> v2:172.21.15.176:3300/0 conn(0x7fdbac160870 msgr2=0x7fdbac0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbabfff640 1 --2- 172.21.15.67:0/44931828 >> v2:172.21.15.176:3300/0 conn(0x7fdbac160870 0x7fdbac0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbabfff640 1 -- 172.21.15.67:0/44931828 >> v1:172.21.15.67:6789/0 conn(0x7fdbac0cf300 legacy=0x7fdbac0d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbabfff640 1 -- 172.21.15.67:0/44931828 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac0cfe10 con 0x7fdbac0d2410 2024-04-25T15:28:18.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbaa7fc640 1 -- 172.21.15.67:0/44931828 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566469077 0 0) 0x7fdbac0cfb30 con 0x7fdbac0cf300 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbabfff640 1 --2- 172.21.15.67:0/44931828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac0b2900 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0d2960 tx=0x7fdba402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77861a672c485b44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.251+0000 7fdbaa7fc640 1 -- 172.21.15.67:0/44931828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba4007740 con 0x7fdbac0d2410 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbaa7fc640 1 -- 172.21.15.67:0/44931828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba40078e0 con 0x7fdbac0d2410 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbaa7fc640 1 -- 172.21.15.67:0/44931828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba4007cb0 con 0x7fdbac0d2410 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 -- 172.21.15.67:0/44931828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 msgr2=0x7fdbac0b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 --2- 172.21.15.67:0/44931828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac0b2900 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0d2960 tx=0x7fdba402a680 comp rx=0 tx=0).stop 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 -- 172.21.15.67:0/44931828 shutdown_connections 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 --2- 172.21.15.67:0/44931828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac0b2900 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 --2- 172.21.15.67:0/44931828 >> v2:172.21.15.176:3300/0 conn(0x7fdbac160870 0x7fdbac0b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 -- 172.21.15.67:0/44931828 >> 172.21.15.67:0/44931828 conn(0x7fdbac09b7d0 msgr2=0x7fdbac09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:18.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 -- 172.21.15.67:0/44931828 shutdown_connections 2024-04-25T15:28:18.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.252+0000 7fdbb201a640 1 -- 172.21.15.67:0/44931828 wait complete. 2024-04-25T15:28:18.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.254+0000 7fdbb201a640 1 Processor -- start 2024-04-25T15:28:18.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.254+0000 7fdbb201a640 1 -- start start 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.255+0000 7fdbb201a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdbac0cf300 0x7fdbac1f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.255+0000 7fdbb201a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac1f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.255+0000 7fdbb201a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdbac160870 0x7fdbac070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.255+0000 7fdbb201a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac0af9d0 con 0x7fdbac0d2410 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbb201a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac0af850 con 0x7fdbac160870 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbb201a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdbac0cf710 con 0x7fdbac0cf300 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac1f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac1f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51914/0 (socket says 172.21.15.67:51914) 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 -- 172.21.15.67:0/1739256198 learned_addr learned my addr 172.21.15.67:0/1739256198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:18.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbab7fe640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdbac0cf300 0x7fdbac1f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbabfff640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdbac160870 0x7fdbac070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 -- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdbac0cf300 msgr2=0x7fdbac1f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdbac0cf300 0x7fdbac1f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 -- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdbac160870 msgr2=0x7fdbac070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdbac160870 0x7fdbac070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 -- 172.21.15.67:0/1739256198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac1e5c70 con 0x7fdbac0d2410 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbab7fe640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdbac0cf300 0x7fdbac1f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbabfff640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdbac160870 0x7fdbac070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbaaffd640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac1f70f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c430 tx=0x7fdba000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdb93fff640 1 -- 172.21.15.67:0/1739256198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba00074e0 con 0x7fdbac0d2410 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdb93fff640 1 -- 172.21.15.67:0/1739256198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba00047e0 con 0x7fdbac0d2410 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdb93fff640 1 -- 172.21.15.67:0/1739256198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0004bd0 con 0x7fdbac0d2410 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbac1e5f50 con 0x7fdbac0d2410 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.256+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbac1e6510 con 0x7fdbac0d2410 2024-04-25T15:28:18.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.257+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbac050740 con 0x7fdbac0d2410 2024-04-25T15:28:18.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.258+0000 7fdb93fff640 1 -- 172.21.15.67:0/1739256198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdba0011070 con 0x7fdbac0d2410 2024-04-25T15:28:18.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.258+0000 7fdb93fff640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb980744e0 0x7fdb980769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.259+0000 7fdb93fff640 1 -- 172.21.15.67:0/1739256198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7fdba0094c40 con 0x7fdbac0d2410 2024-04-25T15:28:18.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.259+0000 7fdbab7fe640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb980744e0 0x7fdb980769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.259+0000 7fdbab7fe640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb980744e0 0x7fdb980769f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdbac073a00 tx=0x7fdb9c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:18.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.263+0000 7fdb93fff640 1 -- 172.21.15.67:0/1739256198 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdba005f650 con 0x7fdbac0d2410 2024-04-25T15:28:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.488+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbac0b37b0 con 0x7fdbac0d2410 2024-04-25T15:28:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.489+0000 7fdb93fff640 1 -- 172.21.15.67:0/1739256198 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+3872 (secure 0 0 0) 0x7fdba005ef50 con 0x7fdbac0d2410 2024-04-25T15:28:18.490 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:18.490 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5534,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2076054232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2076054232},{"type":"v1","addr":"172.21.15.176:6836","nonce":2076054232}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":5537,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3930910548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":3930910548},{"type":"v1","addr":"172.21.15.176:6837","nonce":3930910548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36},{"gid":6270,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2322157634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2322157634},{"type":"v1","addr":"172.21.15.67:6834","nonce":2322157634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35},{"gid":6645,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/582077928","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":582077928},{"type":"v1","addr":"172.21.15.67:6835","nonce":582077928}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[]} 2024-04-25T15:28:18.490 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 43 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb980744e0 msgr2=0x7fdb980769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb980744e0 0x7fdb980769f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdbac073a00 tx=0x7fdb9c008040 comp rx=0 tx=0).stop 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 msgr2=0x7fdbac1f70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac1f70f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c430 tx=0x7fdba000c8f0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 shutdown_connections 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdb980744e0 0x7fdb980769f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdbac160870 0x7fdbac070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbac0d2410 0x7fdbac1f70f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 --2- 172.21.15.67:0/1739256198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdbac0cf300 0x7fdbac1f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 >> 172.21.15.67:0/1739256198 conn(0x7fdbac09b7d0 msgr2=0x7fdbac070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 shutdown_connections 2024-04-25T15:28:18.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.491+0000 7fdbb201a640 1 -- 172.21.15.67:0/1739256198 wait complete. 2024-04-25T15:28:18.503 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:28:18.503 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:28:18.504 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:28:18.504 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:18.504 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:28:18.504 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:28:18.504 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:28:18.504 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:18.504 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:28:18.504 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:28:18.505 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:28:18.505 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:18.505 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:28:18.505 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:28:18.505 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:28:18.505 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:18.514 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:28:18.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:28:18.516 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:28:18.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:28:18.517 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:28:18.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:28:18.518 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:28:18.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:28:18.558 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:28:18.559 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:18.560 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:28:18.560 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:18.561 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:28:18.561 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:18.561 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:28:18.562 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:18.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:28:18.648 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:28:18.649 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:28:18.651 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:28:18.652 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:28:18.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.682+0000 7f7a33ab6640 1 Processor -- start 2024-04-25T15:28:18.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.682+0000 7f7a33ab6640 1 -- start start 2024-04-25T15:28:18.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.683+0000 7f7a33ab6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d0e00 0x7f7a2c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.683+0000 7f7a33ab6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7a2c0d4a50 0x7f7a2c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.683+0000 7f7a33ab6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a2c0d1850 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.683+0000 7f7a33ab6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7a2c160de0 con 0x7f7a2c0d4a50 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.683+0000 7f7a33ab6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c1610c0 con 0x7f7a2c0d0e00 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a31852640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d0e00 0x7f7a2c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a31852640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d0e00 0x7f7a2c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:54944/0 (socket says 172.21.15.67:54944) 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a31852640 1 -- 172.21.15.67:0/1148354710 learned_addr learned my addr 172.21.15.67:0/1148354710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a32053640 1 --2- 172.21.15.67:0/1148354710 >> v2:172.21.15.176:3300/0 conn(0x7f7a2c0d4a50 0x7f7a2c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a31852640 1 -- 172.21.15.67:0/1148354710 >> v2:172.21.15.176:3300/0 conn(0x7f7a2c0d4a50 msgr2=0x7f7a2c0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a31852640 1 --2- 172.21.15.67:0/1148354710 >> v2:172.21.15.176:3300/0 conn(0x7f7a2c0d4a50 0x7f7a2c0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a31852640 1 -- 172.21.15.67:0/1148354710 >> v1:172.21.15.176:6789/0 conn(0x7f7a2c06c5c0 legacy=0x7f7a2c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.684+0000 7f7a31852640 1 -- 172.21.15.67:0/1148354710 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c1613a0 con 0x7f7a2c0d0e00 2024-04-25T15:28:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.685+0000 7f7a31852640 1 --2- 172.21.15.67:0/1148354710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d0e00 0x7f7a2c0d1210 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009870 tx=0x7f7a2802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e86a05cfefa2dea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.685+0000 7f7a30850640 1 -- 172.21.15.67:0/1148354710 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28037070 con 0x7f7a2c0d0e00 2024-04-25T15:28:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.685+0000 7f7a30850640 1 -- 172.21.15.67:0/1148354710 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a28032440 con 0x7f7a2c0d0e00 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.685+0000 7f7a30850640 1 -- 172.21.15.67:0/1148354710 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28032810 con 0x7f7a2c0d0e00 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.685+0000 7f7a33ab6640 1 -- 172.21.15.67:0/1148354710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d0e00 msgr2=0x7f7a2c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.685+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/1148354710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d0e00 0x7f7a2c0d1210 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009870 tx=0x7f7a2802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.686+0000 7f7a33ab6640 1 -- 172.21.15.67:0/1148354710 shutdown_connections 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.686+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/1148354710 >> v2:172.21.15.176:3300/0 conn(0x7f7a2c0d4a50 0x7f7a2c0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.686+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/1148354710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d0e00 0x7f7a2c0d1210 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.686+0000 7f7a33ab6640 1 -- 172.21.15.67:0/1148354710 >> 172.21.15.67:0/1148354710 conn(0x7f7a2c0d5d90 msgr2=0x7f7a2c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.686+0000 7f7a33ab6640 1 -- 172.21.15.67:0/1148354710 shutdown_connections 2024-04-25T15:28:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.686+0000 7f7a33ab6640 1 -- 172.21.15.67:0/1148354710 wait complete. 2024-04-25T15:28:18.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.687+0000 7f7a33ab6640 1 Processor -- start 2024-04-25T15:28:18.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.687+0000 7f7a33ab6640 1 -- start start 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a33ab6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a2c06c5c0 0x7f7a2c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a33ab6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a2c0d0e00 0x7f7a2c0b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a33ab6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d4a50 0x7f7a2c0b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a33ab6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c164e00 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a33ab6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c0b5320 con 0x7f7a2c0d0e00 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a33ab6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2c164f80 con 0x7f7a2c0d4a50 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a2c06c5c0 0x7f7a2c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a2c06c5c0 0x7f7a2c0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51938/0 (socket says 172.21.15.67:51938) 2024-04-25T15:28:18.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 -- 172.21.15.67:0/188292933 learned_addr learned my addr 172.21.15.67:0/188292933 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a32053640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d4a50 0x7f7a2c0b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31051640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a2c0d0e00 0x7f7a2c0b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 -- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d4a50 msgr2=0x7f7a2c0b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d4a50 0x7f7a2c0b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 -- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a2c0d0e00 msgr2=0x7f7a2c0b7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a2c0d0e00 0x7f7a2c0b7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 -- 172.21.15.67:0/188292933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c15b2a0 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31051640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a2c0d0e00 0x7f7a2c0b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.689+0000 7f7a31852640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a2c06c5c0 0x7f7a2c0cf980 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7a2802a2b0 tx=0x7f7a280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.690+0000 7f7a227fc640 1 -- 172.21.15.67:0/188292933 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28037040 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.690+0000 7f7a227fc640 1 -- 172.21.15.67:0/188292933 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a28042070 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.690+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2c15b510 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.690+0000 7f7a227fc640 1 -- 172.21.15.67:0/188292933 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28030880 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.690+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2c15bad0 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.691+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2c050740 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.692+0000 7f7a227fc640 1 -- 172.21.15.67:0/188292933 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7a28030a20 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.692+0000 7f7a227fc640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7a100744b0 0x7f7a100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.692+0000 7f7a227fc640 1 -- 172.21.15.67:0/188292933 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4943+0+0 (secure 0 0 0) 0x7f7a280b6160 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.692+0000 7f7a31051640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7a100744b0 0x7f7a100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:18.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.693+0000 7f7a31051640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7a100744b0 0x7f7a100769c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c1fb620 tx=0x7f7a1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:18.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.696+0000 7f7a227fc640 1 -- 172.21.15.67:0/188292933 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a28080d60 con 0x7f7a2c06c5c0 2024-04-25T15:28:18.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:18.897+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7a2c0b86c0 con 0x7f7a2c06c5c0 2024-04-25T15:28:19.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.602+0000 7f7a227fc640 1 -- 172.21.15.67:0/188292933 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v74) ==== 97+0+0 (secure 0 0 0) 0x7f7a28080660 con 0x7f7a2c06c5c0 2024-04-25T15:28:19.603 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-25T15:28:19.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7a100744b0 msgr2=0x7f7a100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:19.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7a100744b0 0x7f7a100769c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c1fb620 tx=0x7f7a1c009040 comp rx=0 tx=0).stop 2024-04-25T15:28:19.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a2c06c5c0 msgr2=0x7f7a2c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:19.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a2c06c5c0 0x7f7a2c0cf980 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f7a2802a2b0 tx=0x7f7a280046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 shutdown_connections 2024-04-25T15:28:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7a100744b0 0x7f7a100769c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7a2c0d4a50 0x7f7a2c0b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7a2c0d0e00 0x7f7a2c0b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 --2- 172.21.15.67:0/188292933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a2c06c5c0 0x7f7a2c0cf980 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 >> 172.21.15.67:0/188292933 conn(0x7f7a2c0d5d90 msgr2=0x7f7a2c160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 shutdown_connections 2024-04-25T15:28:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.604+0000 7f7a33ab6640 1 -- 172.21.15.67:0/188292933 wait complete. 2024-04-25T15:28:19.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:28:19.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.736+0000 7f1a7ac9f640 1 Processor -- start 2024-04-25T15:28:19.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.736+0000 7f1a7ac9f640 1 -- start start 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a7ac9f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1a740d0e00 0x7f1a740d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a7ac9f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a740d4a50 0x7f1a740d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a7ac9f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a740d1850 con 0x7f1a7406c5c0 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a7ac9f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1a74160de0 con 0x7f1a740d0e00 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a7ac9f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a741610c0 con 0x7f1a740d4a50 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a73fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1a7406c5c0 0x7f1a7406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57012/0 (socket says 172.21.15.67:57012) 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a73fff640 1 -- 172.21.15.67:0/2291968173 learned_addr learned my addr 172.21.15.67:0/2291968173 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a7923c640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a740d4a50 0x7f1a740d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.737+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/2291968173 >> v2:172.21.15.176:3300/0 conn(0x7f1a740d0e00 0x7f1a740d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1630360006 0 0) 0x7f1a740d1850 con 0x7f1a7406c5c0 2024-04-25T15:28:19.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a60003520 con 0x7f1a7406c5c0 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1338390455 0 0) 0x7f1a60003520 con 0x7f1a7406c5c0 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a740d4a50 msgr2=0x7f1a740d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a740d4a50 0x7f1a740d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 >> v2:172.21.15.176:3300/0 conn(0x7f1a740d0e00 msgr2=0x7f1a740d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 --2- 172.21.15.67:0/2291968173 >> v2:172.21.15.176:3300/0 conn(0x7f1a740d0e00 0x7f1a740d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a741613a0 con 0x7f1a7406c5c0 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f1a64001670 con 0x7f1a7406c5c0 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 >> v1:172.21.15.67:6789/0 conn(0x7f1a7406c5c0 legacy=0x7f1a7406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a600039b0 0x7f1a60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a60004800 0x7f1a60004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a60005150 0x7f1a6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74160de0 con 0x7f1a600039b0 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a741610c0 con 0x7f1a60004800 2024-04-25T15:28:19.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a6000bf80 con 0x7f1a60005150 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/2291968173 >> v2:172.21.15.176:3300/0 conn(0x7f1a740d0e00 0x7f1a740d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a7923c640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a740d4a50 0x7f1a740d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a73fff640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a60004800 0x7f1a60004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a600039b0 0x7f1a60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a7923c640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a60005150 0x7f1a6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.738+0000 7f1a78a3b640 1 -- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a60005150 msgr2=0x7f1a6000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a60005150 0x7f1a6000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a78a3b640 1 -- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a60004800 msgr2=0x7f1a60004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a60004800 0x7f1a60004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a78a3b640 1 -- 172.21.15.67:0/2291968173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a60003520 con 0x7f1a600039b0 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a73fff640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a60004800 0x7f1a60004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a600039b0 0x7f1a60003e20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009c70 tx=0x7f1a60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4d666993cb4c22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c009840 con 0x7f1a600039b0 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c004020 con 0x7f1a600039b0 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.739+0000 7f1a737fe640 1 -- 172.21.15.67:0/2291968173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c0043d0 con 0x7f1a600039b0 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.740+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/2291968173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a600039b0 msgr2=0x7f1a60003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.740+0000 7f1a7ac9f640 1 --2- 172.21.15.67:0/2291968173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a600039b0 0x7f1a60003e20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009c70 tx=0x7f1a60005830 comp rx=0 tx=0).stop 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.740+0000 7f1a7923c640 1 -- 172.21.15.67:0/2291968173 reap_dead start 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.740+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/2291968173 shutdown_connections 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.740+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/2291968173 >> 172.21.15.67:0/2291968173 conn(0x7f1a740d5d90 msgr2=0x7f1a740d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.740+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/2291968173 shutdown_connections 2024-04-25T15:28:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.740+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/2291968173 wait complete. 2024-04-25T15:28:19.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.742+0000 7f1a7ac9f640 1 Processor -- start 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.742+0000 7f1a7ac9f640 1 -- start start 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.743+0000 7f1a7ac9f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a600039b0 0x7f1a740ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.743+0000 7f1a7ac9f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a741fa9f0 0x7f1a74148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.743+0000 7f1a7ac9f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a74158c90 0x7f1a740cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.743+0000 7f1a7ac9f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74164e00 con 0x7f1a741fa9f0 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.743+0000 7f1a7ac9f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a740b5320 con 0x7f1a600039b0 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.743+0000 7f1a7ac9f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a74164f80 con 0x7f1a74158c90 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.743+0000 7f1a73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a741fa9f0 0x7f1a74148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a741fa9f0 0x7f1a74148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51968/0 (socket says 172.21.15.67:51968) 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 -- 172.21.15.67:0/3288204200 learned_addr learned my addr 172.21.15.67:0/3288204200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a600039b0 0x7f1a740ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a7923c640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a74158c90 0x7f1a740cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 -- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a74158c90 msgr2=0x7f1a740cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a74158c90 0x7f1a740cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 -- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a600039b0 msgr2=0x7f1a740ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a600039b0 0x7f1a740ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 -- 172.21.15.67:0/3288204200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a74149430 con 0x7f1a741fa9f0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a600039b0 0x7f1a740ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a73fff640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a741fa9f0 0x7f1a74148ef0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1a64007310 tx=0x7f1a64007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a717fa640 1 -- 172.21.15.67:0/3288204200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6400f9c0 con 0x7f1a741fa9f0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a717fa640 1 -- 172.21.15.67:0/3288204200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a64018070 con 0x7f1a741fa9f0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a717fa640 1 -- 172.21.15.67:0/3288204200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6400fd80 con 0x7f1a741fa9f0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a740cfa80 con 0x7f1a741fa9f0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.744+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a740d0040 con 0x7f1a741fa9f0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.745+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a74050740 con 0x7f1a741fa9f0 2024-04-25T15:28:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.746+0000 7f1a717fa640 1 -- 172.21.15.67:0/3288204200 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1a6401d460 con 0x7f1a741fa9f0 2024-04-25T15:28:19.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.746+0000 7f1a717fa640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a540744b0 0x7f1a540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:19.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.747+0000 7f1a717fa640 1 -- 172.21.15.67:0/3288204200 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4857+0+0 (secure 0 0 0) 0x7f1a64095080 con 0x7f1a741fa9f0 2024-04-25T15:28:19.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.747+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a540744b0 0x7f1a540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:19.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.747+0000 7f1a78a3b640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a540744b0 0x7f1a540769c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c032000 tx=0x7f1a5c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:19.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:19.751+0000 7f1a717fa640 1 -- 172.21.15.67:0/3288204200 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a6405fb60 con 0x7f1a741fa9f0 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.008+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1a740cd390 con 0x7f1a741fa9f0 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.008+0000 7f1a717fa640 1 -- 172.21.15.67:0/3288204200 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7f1a6405f460 con 0x7f1a741fa9f0 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-25T15:28:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg==", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA==", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q==", 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw==", 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-25T15:28:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg==", 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw==", 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA==", 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g==", 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA==", 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA==", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ==", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA==", 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q==", 2024-04-25T15:28:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ==", 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:28:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg==", 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA==", 2024-04-25T15:28:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmVIPuGRAAswFIakCxyA7OvvoJIBYUiQ==", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg==", 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag==", 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:28:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:28:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:28:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:28:20.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a540744b0 msgr2=0x7f1a540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a540744b0 0x7f1a540769c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c032000 tx=0x7f1a5c008030 comp rx=0 tx=0).stop 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a741fa9f0 msgr2=0x7f1a74148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a741fa9f0 0x7f1a74148ef0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f1a64007310 tx=0x7f1a64007710 comp rx=0 tx=0).stop 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 shutdown_connections 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a540744b0 0x7f1a540769c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a74158c90 0x7f1a740cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a741fa9f0 0x7f1a74148ef0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 --2- 172.21.15.67:0/3288204200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a600039b0 0x7f1a740ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 >> 172.21.15.67:0/3288204200 conn(0x7f1a740d5d90 msgr2=0x7f1a740b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 shutdown_connections 2024-04-25T15:28:20.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.010+0000 7f1a7ac9f640 1 -- 172.21.15.67:0/3288204200 wait complete. 2024-04-25T15:28:20.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:20.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.136+0000 7f2865613640 1 Processor -- start 2024-04-25T15:28:20.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.136+0000 7f2865613640 1 -- start start 2024-04-25T15:28:20.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.137+0000 7f2865613640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 0x7f286006f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.137+0000 7f2865613640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286006f9e0 con 0x7f286015ecd0 2024-04-25T15:28:20.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.137+0000 7f2865613640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286006c660 con 0x7f28600cf5e0 2024-04-25T15:28:20.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.137+0000 7f2865613640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f286006fef0 con 0x7f286006f090 2024-04-25T15:28:20.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f286015ecd0 0x7f286015f0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57026/0 (socket says 172.21.15.67:57026) 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285f7fe640 1 -- 172.21.15.67:0/4292657801 learned_addr learned my addr 172.21.15.67:0/4292657801 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285e7fc640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 0x7f286006f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946652997 0 0) 0x7f2850001490 con 0x7f286015ecd0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f284c003520 con 0x7f286015ecd0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3249386036 0 0) 0x7f286006c660 con 0x7f28600cf5e0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2850001490 con 0x7f28600cf5e0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2897623266 0 0) 0x7f284c003520 con 0x7f286015ecd0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 msgr2=0x7f286006f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 0x7f286006f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 >> v1:172.21.15.176:6789/0 conn(0x7f28600cf5e0 legacy=0x7f286006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28600cfbc0 con 0x7f286015ecd0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f2850001bc0 con 0x7f286015ecd0 2024-04-25T15:28:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 >> v1:172.21.15.67:6789/0 conn(0x7f286015ecd0 legacy=0x7f286015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f284c003b70 0x7f284c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 0x7f284c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f284c004d40 0x7f284c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f286006fef0 con 0x7f284c004d40 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f284c00bbd0 con 0x7f284c003b70 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.138+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f284c00bdd0 con 0x7f286006f090 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285f7fe640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f284c004d40 0x7f284c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285e7fc640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 0x7f284c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285effd640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f284c003b70 0x7f284c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285f7fe640 1 -- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 msgr2=0x7f284c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285f7fe640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f286006f090 0x7f284c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285f7fe640 1 -- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f284c003b70 msgr2=0x7f284c003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285f7fe640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f284c003b70 0x7f284c003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285f7fe640 1 -- 172.21.15.67:0/4292657801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c003520 con 0x7f284c004d40 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.139+0000 7f285f7fe640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f284c004d40 0x7f284c005190 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f28600d18d0 tx=0x7f284c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709321ef54e71973 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285000a500 con 0x7f284c004d40 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285002f070 con 0x7f284c004d40 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f285dffb640 1 -- 172.21.15.67:0/4292657801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285002bd00 con 0x7f284c004d40 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f2865613640 1 -- 172.21.15.67:0/4292657801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f284c004d40 msgr2=0x7f284c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f2865613640 1 --2- 172.21.15.67:0/4292657801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f284c004d40 0x7f284c005190 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f28600d18d0 tx=0x7f284c005830 comp rx=0 tx=0).stop 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f285f7fe640 1 -- 172.21.15.67:0/4292657801 reap_dead start 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f2865613640 1 -- 172.21.15.67:0/4292657801 shutdown_connections 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f2865613640 1 -- 172.21.15.67:0/4292657801 >> 172.21.15.67:0/4292657801 conn(0x7f28600d1ae0 msgr2=0x7f28600d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f2865613640 1 -- 172.21.15.67:0/4292657801 shutdown_connections 2024-04-25T15:28:20.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.140+0000 7f2865613640 1 -- 172.21.15.67:0/4292657801 wait complete. 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.142+0000 7f2865613640 1 Processor -- start 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.142+0000 7f2865613640 1 -- start start 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f2865613640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600ce480 0x7f286015dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f2865613640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28600ae1f0 0x7f28600afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f2865613640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28600ae600 0x7f28600b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f2865613640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28600cfea0 con 0x7f28600ce480 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f2865613640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f286006cc70 con 0x7f28600ae1f0 2024-04-25T15:28:20.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f2865613640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f28600d0020 con 0x7f28600ae600 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600ce480 0x7f286015dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600ce480 0x7f286015dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51994/0 (socket says 172.21.15.67:51994) 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 -- 172.21.15.67:0/3142945393 learned_addr learned my addr 172.21.15.67:0/3142945393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285f7fe640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28600ae600 0x7f28600b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285e7fc640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28600ae1f0 0x7f28600afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 -- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28600ae600 msgr2=0x7f28600b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28600ae600 0x7f28600b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 -- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28600ae1f0 msgr2=0x7f28600afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28600ae1f0 0x7f28600afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 -- 172.21.15.67:0/3142945393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286015e2b0 con 0x7f28600ce480 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285e7fc640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28600ae1f0 0x7f28600afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285f7fe640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28600ae600 0x7f28600b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f285effd640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600ce480 0x7f286015dd70 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f285800f770 tx=0x7f285800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f28577fe640 1 -- 172.21.15.67:0/3142945393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285801a070 con 0x7f28600ce480 2024-04-25T15:28:20.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f28577fe640 1 -- 172.21.15.67:0/3142945393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285800fe80 con 0x7f28600ce480 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.144+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28600aeb60 con 0x7f28600ce480 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.145+0000 7f28577fe640 1 -- 172.21.15.67:0/3142945393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858007050 con 0x7f28600ce480 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.145+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28600aaa80 con 0x7f28600ce480 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.146+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2860050740 con 0x7f28600ce480 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.146+0000 7f28577fe640 1 -- 172.21.15.67:0/3142945393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f285801d070 con 0x7f28600ce480 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.147+0000 7f28577fe640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28400744b0 0x7f28400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.147+0000 7f28577fe640 1 -- 172.21.15.67:0/3142945393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4857+0+0 (secure 0 0 0) 0x7f28580946b0 con 0x7f28600ce480 2024-04-25T15:28:20.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.147+0000 7f285e7fc640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28400744b0 0x7f28400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.147+0000 7f285e7fc640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28400744b0 0x7f28400769c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2848009ab0 tx=0x7f2848008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.151+0000 7f28577fe640 1 -- 172.21.15.67:0/3142945393 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f285805f2b0 con 0x7f28600ce480 2024-04-25T15:28:20.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.336+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2860070760 con 0x7f28600ce480 2024-04-25T15:28:20.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.337+0000 7f28577fe640 1 -- 172.21.15.67:0/3142945393 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+12856 (secure 0 0 0) 0x7f285805ebb0 con 0x7f28600ce480 2024-04-25T15:28:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:20.339 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:19.614317+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:20.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28400744b0 msgr2=0x7f28400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28400744b0 0x7f28400769c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2848009ab0 tx=0x7f2848008040 comp rx=0 tx=0).stop 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600ce480 msgr2=0x7f286015dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600ce480 0x7f286015dd70 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f285800f770 tx=0x7f285800fc30 comp rx=0 tx=0).stop 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 shutdown_connections 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f28400744b0 0x7f28400769c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28600ae600 0x7f28600b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28600ae1f0 0x7f28600afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 --2- 172.21.15.67:0/3142945393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600ce480 0x7f286015dd70 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 >> 172.21.15.67:0/3142945393 conn(0x7f28600d1ae0 msgr2=0x7f286015f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 shutdown_connections 2024-04-25T15:28:20.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.339+0000 7f2865613640 1 -- 172.21.15.67:0/3142945393 wait complete. 2024-04-25T15:28:20.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:20.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.466+0000 7f2c538aa640 1 Processor -- start 2024-04-25T15:28:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.466+0000 7f2c538aa640 1 -- start start 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c538aa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c4c164730 0x7f2c4c0d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c538aa640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4c0d23f0 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c538aa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4c0d26d0 con 0x7f2c4c0d1e20 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c538aa640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2c4c0d29b0 con 0x7f2c4c164730 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c51e47640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c4c164730 0x7f2c4c0d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c50e45640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2c4c0b2e60 0x7f2c4c164020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57032/0 (socket says 172.21.15.67:57032) 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c50e45640 1 -- 172.21.15.67:0/708646443 learned_addr learned my addr 172.21.15.67:0/708646443 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:20.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c51e47640 1 -- 172.21.15.67:0/708646443 >> v1:172.21.15.176:6789/0 conn(0x7f2c4c0d1e20 legacy=0x7f2c4c08d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c51e47640 1 -- 172.21.15.67:0/708646443 >> v1:172.21.15.67:6789/0 conn(0x7f2c4c0b2e60 legacy=0x7f2c4c164020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c4bfff640 1 -- 172.21.15.67:0/708646443 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150849806 0 0) 0x7f2c4c0d23f0 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.468+0000 7f2c51e47640 1 -- 172.21.15.67:0/708646443 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c0d2c90 con 0x7f2c4c164730 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.469+0000 7f2c51e47640 1 --2- 172.21.15.67:0/708646443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c164730 0x7f2c4c0d3490 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009d90 tx=0x7f2c3c02a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb22b5f8bb61eac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.469+0000 7f2c4bfff640 1 -- 172.21.15.67:0/708646443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c037070 con 0x7f2c4c164730 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.469+0000 7f2c4bfff640 1 -- 172.21.15.67:0/708646443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c3c007740 con 0x7f2c4c164730 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.469+0000 7f2c4bfff640 1 -- 172.21.15.67:0/708646443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c007b10 con 0x7f2c4c164730 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.470+0000 7f2c538aa640 1 -- 172.21.15.67:0/708646443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c164730 msgr2=0x7f2c4c0d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.470+0000 7f2c538aa640 1 --2- 172.21.15.67:0/708646443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c164730 0x7f2c4c0d3490 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009d90 tx=0x7f2c3c02a780 comp rx=0 tx=0).stop 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.470+0000 7f2c538aa640 1 -- 172.21.15.67:0/708646443 shutdown_connections 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.470+0000 7f2c538aa640 1 --2- 172.21.15.67:0/708646443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c164730 0x7f2c4c0d3490 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.470+0000 7f2c538aa640 1 -- 172.21.15.67:0/708646443 >> 172.21.15.67:0/708646443 conn(0x7f2c4c0d0530 msgr2=0x7f2c4c0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:20.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.470+0000 7f2c538aa640 1 -- 172.21.15.67:0/708646443 shutdown_connections 2024-04-25T15:28:20.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.470+0000 7f2c538aa640 1 -- 172.21.15.67:0/708646443 wait complete. 2024-04-25T15:28:20.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.472+0000 7f2c538aa640 1 Processor -- start 2024-04-25T15:28:20.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.472+0000 7f2c538aa640 1 -- start start 2024-04-25T15:28:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c538aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c0b2e60 0x7f2c4c0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c538aa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c0d1e20 0x7f2c4c0b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c538aa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c152e40 0x7f2c4c0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c538aa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c0ce1f0 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c538aa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c0ce070 con 0x7f2c4c152e40 2024-04-25T15:28:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c538aa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2c4c0ce370 con 0x7f2c4c0d1e20 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c0b2e60 0x7f2c4c0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c0b2e60 0x7f2c4c0cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52018/0 (socket says 172.21.15.67:52018) 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 -- 172.21.15.67:0/1014148204 learned_addr learned my addr 172.21.15.67:0/1014148204 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51e47640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c152e40 0x7f2c4c0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c50e45640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c0d1e20 0x7f2c4c0b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 -- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c0d1e20 msgr2=0x7f2c4c0b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c0d1e20 0x7f2c4c0b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 -- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c152e40 msgr2=0x7f2c4c0b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c152e40 0x7f2c4c0b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.473+0000 7f2c51646640 1 -- 172.21.15.67:0/1014148204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c0cc330 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.474+0000 7f2c51646640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c0b2e60 0x7f2c4c0cd3d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400c430 tx=0x7f2c4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.474+0000 7f2c49ffb640 1 -- 172.21.15.67:0/1014148204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44017070 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.474+0000 7f2c49ffb640 1 -- 172.21.15.67:0/1014148204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c440052d0 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.474+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4c0cc610 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.474+0000 7f2c49ffb640 1 -- 172.21.15.67:0/1014148204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44003830 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.474+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c4c0ccbd0 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.475+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c4c050740 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.476+0000 7f2c49ffb640 1 -- 172.21.15.67:0/1014148204 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2c4401a070 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.476+0000 7f2c49ffb640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c380744e0 0x7f2c380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.476+0000 7f2c49ffb640 1 -- 172.21.15.67:0/1014148204 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4857+0+0 (secure 0 0 0) 0x7f2c44094230 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.476+0000 7f2c50e45640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c380744e0 0x7f2c380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.477+0000 7f2c50e45640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c380744e0 0x7f2c380769f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c161560 tx=0x7f2c40007d20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.481+0000 7f2c49ffb640 1 -- 172.21.15.67:0/1014148204 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c4405ed10 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.667+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2c4c071000 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.667+0000 7f2c49ffb640 1 -- 172.21.15.67:0/1014148204 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) ==== 93+0+10 (secure 0 0 0) 0x7f2c4405e610 con 0x7f2c4c0b2e60 2024-04-25T15:28:20.668 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:20.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c380744e0 msgr2=0x7f2c380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c380744e0 0x7f2c380769f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c161560 tx=0x7f2c40007d20 comp rx=0 tx=0).stop 2024-04-25T15:28:20.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c0b2e60 msgr2=0x7f2c4c0cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c0b2e60 0x7f2c4c0cd3d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400c430 tx=0x7f2c4400c8f0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 shutdown_connections 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c380744e0 0x7f2c380769f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c152e40 0x7f2c4c0b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c0d1e20 0x7f2c4c0b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 --2- 172.21.15.67:0/1014148204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c4c0b2e60 0x7f2c4c0cd3d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 >> 172.21.15.67:0/1014148204 conn(0x7f2c4c0d0530 msgr2=0x7f2c4c0cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 shutdown_connections 2024-04-25T15:28:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.669+0000 7f2c538aa640 1 -- 172.21.15.67:0/1014148204 wait complete. 2024-04-25T15:28:20.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:28:20.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.792+0000 7f361d7c5640 1 Processor -- start 2024-04-25T15:28:20.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.792+0000 7f361d7c5640 1 -- start start 2024-04-25T15:28:20.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f361d7c5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f36180d0de0 0x7f36180d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f361d7c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d4a30 0x7f36180d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f361d7c5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36180d1830 con 0x7f361806c5c0 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f361d7c5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3618160dc0 con 0x7f36180d0de0 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f361d7c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f36181610a0 con 0x7f36180d4a30 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f36167fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f361806c5c0 0x7f361806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57034/0 (socket says 172.21.15.67:57034) 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f36167fc640 1 -- 172.21.15.67:0/484859775 learned_addr learned my addr 172.21.15.67:0/484859775 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f3616ffd640 1 --2- 172.21.15.67:0/484859775 >> v2:172.21.15.176:3300/0 conn(0x7f36180d0de0 0x7f36180d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f36177fe640 1 --2- 172.21.15.67:0/484859775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d4a30 0x7f36180d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f3615ffb640 1 -- 172.21.15.67:0/484859775 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969659425 0 0) 0x7f36180d1830 con 0x7f361806c5c0 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.794+0000 7f3615ffb640 1 -- 172.21.15.67:0/484859775 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3604003520 con 0x7f361806c5c0 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.795+0000 7f3615ffb640 1 -- 172.21.15.67:0/484859775 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3357690198 0 0) 0x7f3604003520 con 0x7f361806c5c0 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.795+0000 7f36177fe640 1 -- 172.21.15.67:0/484859775 >> v2:172.21.15.176:3300/0 conn(0x7f36180d0de0 msgr2=0x7f36180d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.795+0000 7f36177fe640 1 --2- 172.21.15.67:0/484859775 >> v2:172.21.15.176:3300/0 conn(0x7f36180d0de0 0x7f36180d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.795+0000 7f36177fe640 1 -- 172.21.15.67:0/484859775 >> v1:172.21.15.67:6789/0 conn(0x7f361806c5c0 legacy=0x7f361806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.795+0000 7f36177fe640 1 -- 172.21.15.67:0/484859775 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618161380 con 0x7f36180d4a30 2024-04-25T15:28:20.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.795+0000 7f3616ffd640 1 --2- 172.21.15.67:0/484859775 >> v2:172.21.15.176:3300/0 conn(0x7f36180d0de0 0x7f36180d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.795+0000 7f36177fe640 1 --2- 172.21.15.67:0/484859775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d4a30 0x7f36180d4e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3608009d90 tx=0x7f360802a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee76faaa50ad7b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f3615ffb640 1 -- 172.21.15.67:0/484859775 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600001670 con 0x7f36180d4a30 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f3615ffb640 1 -- 172.21.15.67:0/484859775 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3608007540 con 0x7f36180d4a30 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f3615ffb640 1 -- 172.21.15.67:0/484859775 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36080078f0 con 0x7f36180d4a30 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 -- 172.21.15.67:0/484859775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d4a30 msgr2=0x7f36180d4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 --2- 172.21.15.67:0/484859775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d4a30 0x7f36180d4e40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3608009d90 tx=0x7f360802a5f0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 -- 172.21.15.67:0/484859775 shutdown_connections 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 --2- 172.21.15.67:0/484859775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d4a30 0x7f36180d4e40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 --2- 172.21.15.67:0/484859775 >> v2:172.21.15.176:3300/0 conn(0x7f36180d0de0 0x7f36180d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 -- 172.21.15.67:0/484859775 >> 172.21.15.67:0/484859775 conn(0x7f36180d5d70 msgr2=0x7f36180d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:20.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 -- 172.21.15.67:0/484859775 shutdown_connections 2024-04-25T15:28:20.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.796+0000 7f361d7c5640 1 -- 172.21.15.67:0/484859775 wait complete. 2024-04-25T15:28:20.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.798+0000 7f361d7c5640 1 Processor -- start 2024-04-25T15:28:20.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.798+0000 7f361d7c5640 1 -- start start 2024-04-25T15:28:20.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.799+0000 7f361d7c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f361806c5c0 0x7f36180cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.799+0000 7f361d7c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d0de0 0x7f36181f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.799+0000 7f361d7c5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36180d4a30 0x7f36181fa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f361d7c5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3618164de0 con 0x7f361806c5c0 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f361d7c5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f36180b5300 con 0x7f36180d4a30 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f361d7c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3618164f60 con 0x7f36180d0de0 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f361806c5c0 0x7f36180cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f361806c5c0 0x7f36180cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52030/0 (socket says 172.21.15.67:52030) 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 -- 172.21.15.67:0/137719887 learned_addr learned my addr 172.21.15.67:0/137719887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f36167fc640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d0de0 0x7f36181f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f36177fe640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36180d4a30 0x7f36181fa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 -- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d0de0 msgr2=0x7f36181f9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d0de0 0x7f36181f9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 -- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36180d4a30 msgr2=0x7f36181fa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:20.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36180d4a30 0x7f36181fa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 -- 172.21.15.67:0/137719887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36180cdff0 con 0x7f361806c5c0 2024-04-25T15:28:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f3616ffd640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f361806c5c0 0x7f36180cd2c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f361000e990 tx=0x7f361000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f360f7fe640 1 -- 172.21.15.67:0/137719887 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610017070 con 0x7f361806c5c0 2024-04-25T15:28:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f360f7fe640 1 -- 172.21.15.67:0/137719887 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f361000c9e0 con 0x7f361806c5c0 2024-04-25T15:28:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36180ce2d0 con 0x7f361806c5c0 2024-04-25T15:28:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f360f7fe640 1 -- 172.21.15.67:0/137719887 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361000cdb0 con 0x7f361806c5c0 2024-04-25T15:28:20.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.800+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36180ce890 con 0x7f361806c5c0 2024-04-25T15:28:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.802+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3618050740 con 0x7f361806c5c0 2024-04-25T15:28:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.802+0000 7f360f7fe640 1 -- 172.21.15.67:0/137719887 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f361001a070 con 0x7f361806c5c0 2024-04-25T15:28:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.802+0000 7f360f7fe640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35f80744b0 0x7f35f80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.803+0000 7f360f7fe640 1 -- 172.21.15.67:0/137719887 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4857+0+0 (secure 0 0 0) 0x7f3610095010 con 0x7f361806c5c0 2024-04-25T15:28:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.803+0000 7f36167fc640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35f80744b0 0x7f35f80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:20.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.803+0000 7f36167fc640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35f80744b0 0x7f35f80769c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f361815e940 tx=0x7f3600009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:20.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.807+0000 7f360f7fe640 1 -- 172.21.15.67:0/137719887 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f361005fc10 con 0x7f361806c5c0 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.997+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f36180d4300 con 0x7f361806c5c0 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.997+0000 7f360f7fe640 1 -- 172.21.15.67:0/137719887 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) ==== 84+0+65 (secure 0 0 0) 0x7f361005f510 con 0x7f361806c5c0 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:20.998 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:28:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35f80744b0 msgr2=0x7f35f80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35f80744b0 0x7f35f80769c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f361815e940 tx=0x7f3600009040 comp rx=0 tx=0).stop 2024-04-25T15:28:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f361806c5c0 msgr2=0x7f36180cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f361806c5c0 0x7f36180cd2c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f361000e990 tx=0x7f361000ee50 comp rx=0 tx=0).stop 2024-04-25T15:28:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 shutdown_connections 2024-04-25T15:28:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f35f80744b0 0x7f35f80769c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36180d4a30 0x7f36181fa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f36180d0de0 0x7f36181f9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 --2- 172.21.15.67:0/137719887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f361806c5c0 0x7f36180cd2c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 >> 172.21.15.67:0/137719887 conn(0x7f36180d5d70 msgr2=0x7f3618162a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 shutdown_connections 2024-04-25T15:28:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:20.999+0000 7f361d7c5640 1 -- 172.21.15.67:0/137719887 wait complete. 2024-04-25T15:28:21.011 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:28:21.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:28:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.126+0000 7f79d8bfe640 1 Processor -- start 2024-04-25T15:28:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.126+0000 7f79d8bfe640 1 -- start start 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.127+0000 7f79d8bfe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d41628e0 0x7f79d4161650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.127+0000 7f79d8bfe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f79d40ce3f0 0x7f79d40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.127+0000 7f79d8bfe640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79d40cef70 con 0x7f79d40b39f0 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.127+0000 7f79d8bfe640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f79d40ced40 con 0x7f79d40ce3f0 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.127+0000 7f79d8bfe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79d40cf250 con 0x7f79d41628e0 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d2d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79d40b39f0 0x7f79d40b3e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57050/0 (socket says 172.21.15.67:57050) 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d2d9d640 1 -- 172.21.15.67:0/3133333823 learned_addr learned my addr 172.21.15.67:0/3133333823 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d1d9b640 1 --2- 172.21.15.67:0/3133333823 >> v2:172.21.15.176:3300/0 conn(0x7f79d40ce3f0 0x7f79d40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d259c640 1 --2- 172.21.15.67:0/3133333823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d41628e0 0x7f79d4161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d159a640 1 -- 172.21.15.67:0/3133333823 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475997991 0 0) 0x7f79d40cef70 con 0x7f79d40b39f0 2024-04-25T15:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d159a640 1 -- 172.21.15.67:0/3133333823 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79bc003520 con 0x7f79d40b39f0 2024-04-25T15:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d159a640 1 -- 172.21.15.67:0/3133333823 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1883082694 0 0) 0x7f79bc003520 con 0x7f79d40b39f0 2024-04-25T15:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d259c640 1 -- 172.21.15.67:0/3133333823 >> v2:172.21.15.176:3300/0 conn(0x7f79d40ce3f0 msgr2=0x7f79d40ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d259c640 1 --2- 172.21.15.67:0/3133333823 >> v2:172.21.15.176:3300/0 conn(0x7f79d40ce3f0 0x7f79d40ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d259c640 1 -- 172.21.15.67:0/3133333823 >> v1:172.21.15.67:6789/0 conn(0x7f79d40b39f0 legacy=0x7f79d40b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.128+0000 7f79d259c640 1 -- 172.21.15.67:0/3133333823 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d4161e00 con 0x7f79d41628e0 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.129+0000 7f79d259c640 1 --2- 172.21.15.67:0/3133333823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d41628e0 0x7f79d4161650 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f79c40097a0 tx=0x7f79c402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1ad1f2853293af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.129+0000 7f79d159a640 1 -- 172.21.15.67:0/3133333823 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4037070 con 0x7f79d41628e0 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.129+0000 7f79d159a640 1 -- 172.21.15.67:0/3133333823 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79c402ab10 con 0x7f79d41628e0 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.129+0000 7f79d159a640 1 -- 172.21.15.67:0/3133333823 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc0019f0 con 0x7f79d41628e0 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3133333823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d41628e0 msgr2=0x7f79d4161650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3133333823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d41628e0 0x7f79d4161650 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f79c40097a0 tx=0x7f79c402a080 comp rx=0 tx=0).stop 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3133333823 shutdown_connections 2024-04-25T15:28:21.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3133333823 >> v2:172.21.15.176:3300/0 conn(0x7f79d40ce3f0 0x7f79d40ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3133333823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d41628e0 0x7f79d4161650 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3133333823 >> 172.21.15.67:0/3133333823 conn(0x7f79d409b7d0 msgr2=0x7f79d409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3133333823 shutdown_connections 2024-04-25T15:28:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.130+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3133333823 wait complete. 2024-04-25T15:28:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.132+0000 7f79d8bfe640 1 Processor -- start 2024-04-25T15:28:21.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.132+0000 7f79d8bfe640 1 -- start start 2024-04-25T15:28:21.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d8bfe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d40b39f0 0x7f79d4155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d8bfe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79d40ce3f0 0x7f79d41428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d8bfe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d41628e0 0x7f79d415f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d8bfe640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79d40d00c0 con 0x7f79d41628e0 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d8bfe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4162450 con 0x7f79d40ce3f0 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d8bfe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79d40d0240 con 0x7f79d40b39f0 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d41628e0 0x7f79d415f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d41628e0 0x7f79d415f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52040/0 (socket says 172.21.15.67:52040) 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.133+0000 7f79d2d9d640 1 -- 172.21.15.67:0/3448202152 learned_addr learned my addr 172.21.15.67:0/3448202152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d259c640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d40b39f0 0x7f79d4155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d1d9b640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79d40ce3f0 0x7f79d41428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d2d9d640 1 -- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d40b39f0 msgr2=0x7f79d4155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d2d9d640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d40b39f0 0x7f79d4155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d2d9d640 1 -- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79d40ce3f0 msgr2=0x7f79d41428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d2d9d640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79d40ce3f0 0x7f79d41428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d2d9d640 1 -- 172.21.15.67:0/3448202152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d4156060 con 0x7f79d41628e0 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d259c640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d40b39f0 0x7f79d4155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d2d9d640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d41628e0 0x7f79d415f8a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00cc30 tx=0x7f79cc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79c2ffd640 1 -- 172.21.15.67:0/3448202152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc017070 con 0x7f79d41628e0 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79c2ffd640 1 -- 172.21.15.67:0/3448202152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79cc00ce40 con 0x7f79d41628e0 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d4157e40 con 0x7f79d41628e0 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79c2ffd640 1 -- 172.21.15.67:0/3448202152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc012600 con 0x7f79d41628e0 2024-04-25T15:28:21.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.134+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d4158400 con 0x7f79d41628e0 2024-04-25T15:28:21.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.135+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79d4050740 con 0x7f79d41628e0 2024-04-25T15:28:21.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.136+0000 7f79c2ffd640 1 -- 172.21.15.67:0/3448202152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f79cc01a070 con 0x7f79d41628e0 2024-04-25T15:28:21.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.136+0000 7f79c2ffd640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79b40744b0 0x7f79b40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.137+0000 7f79c2ffd640 1 -- 172.21.15.67:0/3448202152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4857+0+0 (secure 0 0 0) 0x7f79cc094ca0 con 0x7f79d41628e0 2024-04-25T15:28:21.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.137+0000 7f79d259c640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79b40744b0 0x7f79b40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.137+0000 7f79d259c640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79b40744b0 0x7f79b40769c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f79c4034000 tx=0x7f79c402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:21.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.141+0000 7f79c2ffd640 1 -- 172.21.15.67:0/3448202152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79cc05f130 con 0x7f79d41628e0 2024-04-25T15:28:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.324+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f79d4157bd0 con 0x7f79d41628e0 2024-04-25T15:28:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.608+0000 7f79c2ffd640 1 -- 172.21.15.67:0/3448202152 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) ==== 138+0+0 (secure 0 0 0) 0x7f79cc063430 con 0x7f79d41628e0 2024-04-25T15:28:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79b40744b0 msgr2=0x7f79b40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79b40744b0 0x7f79b40769c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f79c4034000 tx=0x7f79c402e000 comp rx=0 tx=0).stop 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d41628e0 msgr2=0x7f79d415f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d41628e0 0x7f79d415f8a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00cc30 tx=0x7f79cc00a430 comp rx=0 tx=0).stop 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 shutdown_connections 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79b40744b0 0x7f79b40769c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79d41628e0 0x7f79d415f8a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79d40ce3f0 0x7f79d41428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 --2- 172.21.15.67:0/3448202152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79d40b39f0 0x7f79d4155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 >> 172.21.15.67:0/3448202152 conn(0x7f79d409b7d0 msgr2=0x7f79d406d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 shutdown_connections 2024-04-25T15:28:21.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.611+0000 7f79d8bfe640 1 -- 172.21.15.67:0/3448202152 wait complete. 2024-04-25T15:28:21.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:28:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.744+0000 7f0a795c5640 1 Processor -- start 2024-04-25T15:28:21.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.744+0000 7f0a795c5640 1 -- start start 2024-04-25T15:28:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a795c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74164b00 0x7f0a740b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a795c5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0a7415df80 0x7f0a740b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a795c5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a741615f0 con 0x7f0a74160890 2024-04-25T15:28:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a795c5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0a741613b0 con 0x7f0a7415df80 2024-04-25T15:28:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a795c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0a74164670 con 0x7f0a74164b00 2024-04-25T15:28:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a737fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0a74160890 0x7f0a74160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57054/0 (socket says 172.21.15.67:57054) 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a737fe640 1 -- 172.21.15.67:0/1641209539 learned_addr learned my addr 172.21.15.67:0/1641209539 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a727fc640 1 --2- 172.21.15.67:0/1641209539 >> v2:172.21.15.176:3300/0 conn(0x7f0a7415df80 0x7f0a740b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a72ffd640 1 --2- 172.21.15.67:0/1641209539 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74164b00 0x7f0a740b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a71ffb640 1 -- 172.21.15.67:0/1641209539 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1832626653 0 0) 0x7f0a741615f0 con 0x7f0a74160890 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a71ffb640 1 -- 172.21.15.67:0/1641209539 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a60003520 con 0x7f0a74160890 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a72ffd640 1 -- 172.21.15.67:0/1641209539 >> v2:172.21.15.176:3300/0 conn(0x7f0a7415df80 msgr2=0x7f0a740b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a72ffd640 1 --2- 172.21.15.67:0/1641209539 >> v2:172.21.15.176:3300/0 conn(0x7f0a7415df80 0x7f0a740b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a72ffd640 1 -- 172.21.15.67:0/1641209539 >> v1:172.21.15.67:6789/0 conn(0x7f0a74160890 legacy=0x7f0a74160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a72ffd640 1 -- 172.21.15.67:0/1641209539 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a74164020 con 0x7f0a74164b00 2024-04-25T15:28:21.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.746+0000 7f0a727fc640 1 --2- 172.21.15.67:0/1641209539 >> v2:172.21.15.176:3300/0 conn(0x7f0a7415df80 0x7f0a740b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.747+0000 7f0a72ffd640 1 --2- 172.21.15.67:0/1641209539 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74164b00 0x7f0a740b7b80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0097a0 tx=0x7f0a6c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958122ed4c36060d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.747+0000 7f0a71ffb640 1 -- 172.21.15.67:0/1641209539 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c037070 con 0x7f0a74164b00 2024-04-25T15:28:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.747+0000 7f0a71ffb640 1 -- 172.21.15.67:0/1641209539 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6c032440 con 0x7f0a74164b00 2024-04-25T15:28:21.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.747+0000 7f0a71ffb640 1 -- 172.21.15.67:0/1641209539 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c032810 con 0x7f0a74164b00 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 -- 172.21.15.67:0/1641209539 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74164b00 msgr2=0x7f0a740b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1641209539 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74164b00 0x7f0a740b7b80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0097a0 tx=0x7f0a6c02a180 comp rx=0 tx=0).stop 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 -- 172.21.15.67:0/1641209539 shutdown_connections 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1641209539 >> v2:172.21.15.176:3300/0 conn(0x7f0a7415df80 0x7f0a740b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1641209539 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74164b00 0x7f0a740b7b80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 -- 172.21.15.67:0/1641209539 >> 172.21.15.67:0/1641209539 conn(0x7f0a740d2bc0 msgr2=0x7f0a740d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 -- 172.21.15.67:0/1641209539 shutdown_connections 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.748+0000 7f0a795c5640 1 -- 172.21.15.67:0/1641209539 wait complete. 2024-04-25T15:28:21.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.750+0000 7f0a795c5640 1 Processor -- start 2024-04-25T15:28:21.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.750+0000 7f0a795c5640 1 -- start start 2024-04-25T15:28:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a795c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7415df80 0x7f0a74155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a795c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74160890 0x7f0a74147500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a795c5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a74164b00 0x7f0a741f9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a795c5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0a740b4a00 con 0x7f0a7415df80 2024-04-25T15:28:21.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a795c5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0a740b4880 con 0x7f0a74164b00 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a795c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0a740b4b80 con 0x7f0a74160890 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a72ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7415df80 0x7f0a74155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a72ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7415df80 0x7f0a74155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52044/0 (socket says 172.21.15.67:52044) 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.751+0000 7f0a72ffd640 1 -- 172.21.15.67:0/1562574265 learned_addr learned my addr 172.21.15.67:0/1562574265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a727fc640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74160890 0x7f0a74147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a737fe640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a74164b00 0x7f0a741f9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a72ffd640 1 -- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74160890 msgr2=0x7f0a74147500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a72ffd640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74160890 0x7f0a74147500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a72ffd640 1 -- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a74164b00 msgr2=0x7f0a741f9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a72ffd640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a74164b00 0x7f0a741f9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a72ffd640 1 -- 172.21.15.67:0/1562574265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a741563f0 con 0x7f0a7415df80 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a737fe640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a74164b00 0x7f0a741f9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a72ffd640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7415df80 0x7f0a74155eb0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c035040 tx=0x7f0a6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:21.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a6b7fe640 1 -- 172.21.15.67:0/1562574265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c042020 con 0x7f0a7415df80 2024-04-25T15:28:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a6b7fe640 1 -- 172.21.15.67:0/1562574265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6c03d750 con 0x7f0a7415df80 2024-04-25T15:28:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a6b7fe640 1 -- 172.21.15.67:0/1562574265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c037040 con 0x7f0a7415df80 2024-04-25T15:28:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a74159ee0 con 0x7f0a7415df80 2024-04-25T15:28:21.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.752+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7415a4a0 con 0x7f0a7415df80 2024-04-25T15:28:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.753+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a74050740 con 0x7f0a7415df80 2024-04-25T15:28:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.754+0000 7f0a6b7fe640 1 -- 172.21.15.67:0/1562574265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0a6c045020 con 0x7f0a7415df80 2024-04-25T15:28:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.754+0000 7f0a6b7fe640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0a540744b0 0x7f0a540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.755+0000 7f0a6b7fe640 1 -- 172.21.15.67:0/1562574265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5227+0+0 (secure 0 0 0) 0x7f0a6c0b6380 con 0x7f0a7415df80 2024-04-25T15:28:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.755+0000 7f0a727fc640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0a540744b0 0x7f0a540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:21.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.755+0000 7f0a727fc640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0a540744b0 0x7f0a540769c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0a741636f0 tx=0x7f0a5c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.759+0000 7f0a6b7fe640 1 -- 172.21.15.67:0/1562574265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a6c04c030 con 0x7f0a7415df80 2024-04-25T15:28:21.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:21.944+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0a740d4170 con 0x7f0a7415df80 2024-04-25T15:28:22.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.611+0000 7f0a6b7fe640 1 -- 172.21.15.67:0/1562574265 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) ==== 170+0+0 (secure 0 0 0) 0x7f0a6c080e10 con 0x7f0a7415df80 2024-04-25T15:28:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.615+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0a540744b0 msgr2=0x7f0a540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.615+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0a540744b0 0x7f0a540769c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0a741636f0 tx=0x7f0a5c008040 comp rx=0 tx=0).stop 2024-04-25T15:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.615+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7415df80 msgr2=0x7f0a74155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.615+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7415df80 0x7f0a74155eb0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c035040 tx=0x7f0a6c0046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.616+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 shutdown_connections 2024-04-25T15:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.616+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0a540744b0 0x7f0a540769c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.616+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a74164b00 0x7f0a741f9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.616+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a74160890 0x7f0a74147500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.616+0000 7f0a795c5640 1 --2- 172.21.15.67:0/1562574265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7415df80 0x7f0a74155eb0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.616+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 >> 172.21.15.67:0/1562574265 conn(0x7f0a740d2bc0 msgr2=0x7f0a74160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.616+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 shutdown_connections 2024-04-25T15:28:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.618+0000 7f0a795c5640 1 -- 172.21.15.67:0/1562574265 wait complete. 2024-04-25T15:28:22.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:28:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.760+0000 7f769606e640 1 Processor -- start 2024-04-25T15:28:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.760+0000 7f769606e640 1 -- start start 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f769606e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f769015e080 0x7f76901646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f769606e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76901609b0 con 0x7f769006ccd0 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f769606e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7690164cd0 con 0x7f7690161710 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f769606e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7690160c30 con 0x7f769015e080 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f768effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f769006ccd0 0x7f7690163fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57064/0 (socket says 172.21.15.67:57064) 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f768effd640 1 -- 172.21.15.67:0/3096200450 learned_addr learned my addr 172.21.15.67:0/3096200450 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f768ffff640 1 --2- 172.21.15.67:0/3096200450 >> v2:172.21.15.176:3300/0 conn(0x7f769015e080 0x7f76901646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.761+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 197663416 0 0) 0x7f7684001110 con 0x7f769006ccd0 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7678003520 con 0x7f769006ccd0 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1659331214 0 0) 0x7f7678003520 con 0x7f769006ccd0 2024-04-25T15:28:22.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 >> v2:172.21.15.176:3300/0 conn(0x7f769015e080 msgr2=0x7f76901646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 --2- 172.21.15.67:0/3096200450 >> v2:172.21.15.176:3300/0 conn(0x7f769015e080 0x7f76901646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 >> v1:172.21.15.176:6789/0 conn(0x7f7690161710 legacy=0x7f769006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7690160f10 con 0x7f769006ccd0 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f7684001840 con 0x7f769006ccd0 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 >> v1:172.21.15.67:6789/0 conn(0x7f769006ccd0 legacy=0x7f7690163fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76780039b0 0x7f7678003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7678004800 0x7f7678004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7678005250 0x7f767800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7690160c30 con 0x7f76780039b0 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f767800c1e0 con 0x7f7678004800 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f767800c3e0 con 0x7f7678005250 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768f7fe640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76780039b0 0x7f7678003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768effd640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7678004800 0x7f7678004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768ffff640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7678005250 0x7f767800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768f7fe640 1 -- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7678005250 msgr2=0x7f767800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768f7fe640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7678005250 0x7f767800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768f7fe640 1 -- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7678004800 msgr2=0x7f7678004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768f7fe640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7678004800 0x7f7678004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768f7fe640 1 -- 172.21.15.67:0/3096200450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7678003520 con 0x7f76780039b0 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.762+0000 7f768ffff640 1 -- 172.21.15.67:0/3096200450 reap_dead start 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.763+0000 7f768f7fe640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76780039b0 0x7f7678003e20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7680001e20 tx=0x7f7678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babcf986ae5302c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.763+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680001a70 con 0x7f76780039b0 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.763+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768002ae40 con 0x7f76780039b0 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.763+0000 7f768e7fc640 1 -- 172.21.15.67:0/3096200450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680003bb0 con 0x7f76780039b0 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.764+0000 7f769606e640 1 -- 172.21.15.67:0/3096200450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76780039b0 msgr2=0x7f7678003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.764+0000 7f769606e640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76780039b0 0x7f7678003e20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7680001e20 tx=0x7f7678005830 comp rx=0 tx=0).stop 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.764+0000 7f769606e640 1 -- 172.21.15.67:0/3096200450 shutdown_connections 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.764+0000 7f769606e640 1 --2- 172.21.15.67:0/3096200450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76780039b0 0x7f7678003e20 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.764+0000 7f769606e640 1 -- 172.21.15.67:0/3096200450 >> 172.21.15.67:0/3096200450 conn(0x7f76900d5d90 msgr2=0x7f76900d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.764+0000 7f769606e640 1 -- 172.21.15.67:0/3096200450 shutdown_connections 2024-04-25T15:28:22.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.764+0000 7f769606e640 1 -- 172.21.15.67:0/3096200450 wait complete. 2024-04-25T15:28:22.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.766+0000 7f769606e640 1 Processor -- start 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.766+0000 7f769606e640 1 -- start start 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f769606e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76780039b0 0x7f76901f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f769606e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7690159480 0x7f76900d2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f769606e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76900d3150 0x7f76900cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f769606e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f769006c740 con 0x7f7690159480 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f769606e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f769006c5c0 con 0x7f76900d3150 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f769606e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f769006c8c0 con 0x7f76780039b0 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f768effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7690159480 0x7f76900d2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f768effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7690159480 0x7f76900d2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52056/0 (socket says 172.21.15.67:52056) 2024-04-25T15:28:22.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.767+0000 7f768effd640 1 -- 172.21.15.67:0/148229927 learned_addr learned my addr 172.21.15.67:0/148229927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768ffff640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76900d3150 0x7f76900cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768f7fe640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76780039b0 0x7f76901f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768effd640 1 -- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76780039b0 msgr2=0x7f76901f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768effd640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76780039b0 0x7f76901f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768effd640 1 -- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76900d3150 msgr2=0x7f76900cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768effd640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76900d3150 0x7f76900cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768effd640 1 -- 172.21.15.67:0/148229927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76900d3560 con 0x7f7690159480 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f768effd640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7690159480 0x7f76900d2c10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f76840072c0 tx=0x7f76840076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f7677fff640 1 -- 172.21.15.67:0/148229927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7684001230 con 0x7f7690159480 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f7677fff640 1 -- 172.21.15.67:0/148229927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7684018070 con 0x7f7690159480 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f7677fff640 1 -- 172.21.15.67:0/148229927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768400fc20 con 0x7f7690159480 2024-04-25T15:28:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76901f6ea0 con 0x7f7690159480 2024-04-25T15:28:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.768+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76900b68a0 con 0x7f7690159480 2024-04-25T15:28:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.769+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76900b6330 con 0x7f7690159480 2024-04-25T15:28:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.771+0000 7f7677fff640 1 -- 172.21.15.67:0/148229927 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f768401d460 con 0x7f7690159480 2024-04-25T15:28:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.771+0000 7f7677fff640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f767c0744b0 0x7f767c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.771+0000 7f7677fff640 1 -- 172.21.15.67:0/148229927 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5609+0+0 (secure 0 0 0) 0x7f7684095af0 con 0x7f7690159480 2024-04-25T15:28:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.772+0000 7f768f7fe640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f767c0744b0 0x7f767c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:22.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.772+0000 7f768f7fe640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f767c0744b0 0x7f767c0769c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7680032000 tx=0x7f768002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:22.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:22.775+0000 7f7677fff640 1 -- 172.21.15.67:0/148229927 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76900d3560 con 0x7f7690159480 2024-04-25T15:28:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.006+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7690050740 con 0x7f7690159480 2024-04-25T15:28:23.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.632+0000 7f7677fff640 1 -- 172.21.15.67:0/148229927 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v46) ==== 174+0+0 (secure 0 0 0) 0x7f76901609b0 con 0x7f7690159480 2024-04-25T15:28:23.641 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 12 and data pool 13 2024-04-25T15:28:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.648+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f767c0744b0 msgr2=0x7f767c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.648+0000 7f769606e640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f767c0744b0 0x7f767c0769c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7680032000 tx=0x7f768002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.648+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7690159480 msgr2=0x7f76900d2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.648+0000 7f769606e640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7690159480 0x7f76900d2c10 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f76840072c0 tx=0x7f76840076c0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.650+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 shutdown_connections 2024-04-25T15:28:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.650+0000 7f769606e640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f767c0744b0 0x7f767c0769c0 secure :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7680032000 tx=0x7f768002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.650+0000 7f769606e640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76900d3150 0x7f76900cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.650+0000 7f769606e640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7690159480 0x7f76900d2c10 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.650+0000 7f769606e640 1 --2- 172.21.15.67:0/148229927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76780039b0 0x7f76901f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.650+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 >> 172.21.15.67:0/148229927 conn(0x7f76900d5d90 msgr2=0x7f769015cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.652+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 shutdown_connections 2024-04-25T15:28:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.654+0000 7f769606e640 1 -- 172.21.15.67:0/148229927 wait complete. 2024-04-25T15:28:23.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.789+0000 7f4221a92640 1 Processor -- start 2024-04-25T15:28:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.789+0000 7f4221a92640 1 -- start start 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f4221a92640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f421c0d1bc0 0x7f421c0b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f4221a92640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f421c0b0770 con 0x7f421c0ce800 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f4221a92640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f421c0b0a50 con 0x7f421c0d1620 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f4221a92640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f421c0b10a0 con 0x7f421c0d1bc0 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f421bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f421c0d1bc0 0x7f421c0b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f421b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f421c0ce800 0x7f421c162a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57070/0 (socket says 172.21.15.67:57070) 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f421bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f421c0d1bc0 0x7f421c0b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52072/0 (socket says 172.21.15.67:52072) 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.790+0000 7f421b7fe640 1 -- 172.21.15.67:0/1994222953 learned_addr learned my addr 172.21.15.67:0/1994222953 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.791+0000 7f421bfff640 1 -- 172.21.15.67:0/1994222953 >> v1:172.21.15.176:6789/0 conn(0x7f421c0d1620 legacy=0x7f421c163130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.791+0000 7f421bfff640 1 -- 172.21.15.67:0/1994222953 >> v1:172.21.15.67:6789/0 conn(0x7f421c0ce800 legacy=0x7f421c162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.791+0000 7f421bfff640 1 -- 172.21.15.67:0/1994222953 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c0b0d30 con 0x7f421c0d1bc0 2024-04-25T15:28:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.791+0000 7f421a7fc640 1 -- 172.21.15.67:0/1994222953 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999760368 0 0) 0x7f421c0b0770 con 0x7f421c0ce800 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.791+0000 7f421bfff640 1 --2- 172.21.15.67:0/1994222953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1bc0 0x7f421c0b3c40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f421c0b1220 tx=0x7f421402a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62c6d46a4c864f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f421a7fc640 1 -- 172.21.15.67:0/1994222953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4214007740 con 0x7f421c0d1bc0 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f421a7fc640 1 -- 172.21.15.67:0/1994222953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42140078e0 con 0x7f421c0d1bc0 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f421a7fc640 1 -- 172.21.15.67:0/1994222953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4214007cb0 con 0x7f421c0d1bc0 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f4221a92640 1 -- 172.21.15.67:0/1994222953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1bc0 msgr2=0x7f421c0b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f4221a92640 1 --2- 172.21.15.67:0/1994222953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1bc0 0x7f421c0b3c40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f421c0b1220 tx=0x7f421402a570 comp rx=0 tx=0).stop 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f4221a92640 1 -- 172.21.15.67:0/1994222953 shutdown_connections 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f4221a92640 1 --2- 172.21.15.67:0/1994222953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1bc0 0x7f421c0b3c40 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f4221a92640 1 -- 172.21.15.67:0/1994222953 >> 172.21.15.67:0/1994222953 conn(0x7f421c0d3a50 msgr2=0x7f421c0d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f4221a92640 1 -- 172.21.15.67:0/1994222953 shutdown_connections 2024-04-25T15:28:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.792+0000 7f4221a92640 1 -- 172.21.15.67:0/1994222953 wait complete. 2024-04-25T15:28:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.794+0000 7f4221a92640 1 Processor -- start 2024-04-25T15:28:23.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.794+0000 7f4221a92640 1 -- start start 2024-04-25T15:28:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f4221a92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f421c0ce800 0x7f421c1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f4221a92640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1620 0x7f421c156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f4221a92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0d1bc0 0x7f421c15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f4221a92640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f421c0b4180 con 0x7f421c0d1620 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f4221a92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f421c0d1fd0 con 0x7f421c0d1bc0 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f4221a92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f421c0b4300 con 0x7f421c0ce800 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1620 0x7f421c156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1620 0x7f421c156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52084/0 (socket says 172.21.15.67:52084) 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 -- 172.21.15.67:0/3427847281 learned_addr learned my addr 172.21.15.67:0/3427847281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421bfff640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0d1bc0 0x7f421c15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421b7fe640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f421c0ce800 0x7f421c1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 -- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f421c0ce800 msgr2=0x7f421c1585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f421c0ce800 0x7f421c1585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 -- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0d1bc0 msgr2=0x7f421c15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0d1bc0 0x7f421c15bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 -- 172.21.15.67:0/3427847281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c0b2190 con 0x7f421c0d1620 2024-04-25T15:28:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.796+0000 7f421affd640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1620 0x7f421c156cf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f420c009fd0 tx=0x7f420c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.797+0000 7f4203fff640 1 -- 172.21.15.67:0/3427847281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c015020 con 0x7f421c0d1620 2024-04-25T15:28:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.797+0000 7f4203fff640 1 -- 172.21.15.67:0/3427847281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f420c011830 con 0x7f421c0d1620 2024-04-25T15:28:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.797+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421c0b2470 con 0x7f421c0d1620 2024-04-25T15:28:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.797+0000 7f4203fff640 1 -- 172.21.15.67:0/3427847281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c011c00 con 0x7f421c0d1620 2024-04-25T15:28:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.797+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f421c0b2a30 con 0x7f421c0d1620 2024-04-25T15:28:23.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.798+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f421c050740 con 0x7f421c0d1620 2024-04-25T15:28:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.798+0000 7f4203fff640 1 -- 172.21.15.67:0/3427847281 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f420c018020 con 0x7f421c0d1620 2024-04-25T15:28:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.799+0000 7f4203fff640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f42080744b0 0x7f42080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.799+0000 7f4203fff640 1 -- 172.21.15.67:0/3427847281 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5709+0+0 (secure 0 0 0) 0x7f420c094ff0 con 0x7f421c0d1620 2024-04-25T15:28:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.799+0000 7f421b7fe640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f42080744b0 0x7f42080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.800+0000 7f421b7fe640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f42080744b0 0x7f42080769c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f42100010e0 tx=0x7f4210009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:23.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.804+0000 7f4203fff640 1 -- 172.21.15.67:0/3427847281 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f420c01f020 con 0x7f421c0d1620 2024-04-25T15:28:23.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.989+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f421c0d1220 con 0x7f421c0d1620 2024-04-25T15:28:23.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.990+0000 7f4203fff640 1 -- 172.21.15.67:0/3427847281 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16637 (secure 0 0 0) 0x7f420c05f130 con 0x7f421c0d1620 2024-04-25T15:28:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:23.992 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":79,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:23.008085+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f42080744b0 msgr2=0x7f42080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f42080744b0 0x7f42080769c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f42100010e0 tx=0x7f4210009040 comp rx=0 tx=0).stop 2024-04-25T15:28:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1620 msgr2=0x7f421c156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:23.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1620 0x7f421c156cf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f420c009fd0 tx=0x7f420c00b870 comp rx=0 tx=0).stop 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 shutdown_connections 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f42080744b0 0x7f42080769c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0d1bc0 0x7f421c15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f421c0d1620 0x7f421c156cf0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 --2- 172.21.15.67:0/3427847281 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f421c0ce800 0x7f421c1585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 >> 172.21.15.67:0/3427847281 conn(0x7f421c0d3a50 msgr2=0x7f421c0d09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 shutdown_connections 2024-04-25T15:28:23.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:23.992+0000 7f4221a92640 1 -- 172.21.15.67:0/3427847281 wait complete. 2024-04-25T15:28:24.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:24.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.116+0000 7f7ce53c5640 1 Processor -- start 2024-04-25T15:28:24.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.116+0000 7f7ce53c5640 1 -- start start 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.117+0000 7f7ce53c5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7ce00d4570 0x7f7ce00b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.117+0000 7f7ce53c5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ce00b2dc0 0x7f7ce00b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.117+0000 7f7ce53c5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce006d870 con 0x7f7ce006e8f0 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.117+0000 7f7ce53c5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce006e460 con 0x7f7ce00b2dc0 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.117+0000 7f7ce53c5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce006db10 con 0x7f7ce00d4570 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.117+0000 7f7cdf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ce00b2dc0 0x7f7ce00b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.118+0000 7f7cdf7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ce00b2dc0 0x7f7ce00b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52100/0 (socket says 172.21.15.67:52100) 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.118+0000 7f7cdf7fe640 1 -- 172.21.15.67:0/1004276947 learned_addr learned my addr 172.21.15.67:0/1004276947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.118+0000 7f7cdeffd640 1 --2- 172.21.15.67:0/1004276947 >> v2:172.21.15.176:3300/0 conn(0x7f7ce00d4570 0x7f7ce00b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.118+0000 7f7cdf7fe640 1 -- 172.21.15.67:0/1004276947 >> v2:172.21.15.176:3300/0 conn(0x7f7ce00d4570 msgr2=0x7f7ce00b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.118+0000 7f7cdf7fe640 1 --2- 172.21.15.67:0/1004276947 >> v2:172.21.15.176:3300/0 conn(0x7f7ce00d4570 0x7f7ce00b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.118+0000 7f7cdf7fe640 1 -- 172.21.15.67:0/1004276947 >> v1:172.21.15.176:6789/0 conn(0x7f7ce006e8f0 legacy=0x7f7ce00d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.118+0000 7f7cdf7fe640 1 -- 172.21.15.67:0/1004276947 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce006e0f0 con 0x7f7ce00b2dc0 2024-04-25T15:28:24.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7cdf7fe640 1 --2- 172.21.15.67:0/1004276947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce00b2dc0 0x7f7ce00b31d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7ce006e5e0 tx=0x7f7cd802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319532edae39f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:24.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7cde7fc640 1 -- 172.21.15.67:0/1004276947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8036070 con 0x7f7ce00b2dc0 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7cde7fc640 1 -- 172.21.15.67:0/1004276947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd8031600 con 0x7f7ce00b2dc0 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7cde7fc640 1 -- 172.21.15.67:0/1004276947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80319d0 con 0x7f7ce00b2dc0 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1004276947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce00b2dc0 msgr2=0x7f7ce00b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1004276947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce00b2dc0 0x7f7ce00b31d0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7ce006e5e0 tx=0x7f7cd802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1004276947 shutdown_connections 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1004276947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce00b2dc0 0x7f7ce00b31d0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1004276947 >> v2:172.21.15.176:3300/0 conn(0x7f7ce00d4570 0x7f7ce00b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.119+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1004276947 >> 172.21.15.67:0/1004276947 conn(0x7f7ce00d3a50 msgr2=0x7f7ce00d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.120+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1004276947 shutdown_connections 2024-04-25T15:28:24.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.120+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1004276947 wait complete. 2024-04-25T15:28:24.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.121+0000 7f7ce53c5640 1 Processor -- start 2024-04-25T15:28:24.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.121+0000 7f7ce53c5640 1 -- start start 2024-04-25T15:28:24.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.122+0000 7f7ce53c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce006e8f0 0x7f7ce0163a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.122+0000 7f7ce53c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ce00b2dc0 0x7f7ce0156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7ce53c5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ce00d4570 0x7f7ce01f9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7ce53c5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce006ed00 con 0x7f7ce006e8f0 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7ce53c5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce00d0fb0 con 0x7f7ce00d4570 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7ce53c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce00b3710 con 0x7f7ce00b2dc0 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce006e8f0 0x7f7ce0163a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce006e8f0 0x7f7ce0163a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52110/0 (socket says 172.21.15.67:52110) 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 -- 172.21.15.67:0/1239844061 learned_addr learned my addr 172.21.15.67:0/1239844061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cd7fff640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ce00b2dc0 0x7f7ce0156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdf7fe640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ce00d4570 0x7f7ce01f9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 -- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ce00b2dc0 msgr2=0x7f7ce0156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ce00b2dc0 0x7f7ce0156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 -- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ce00d4570 msgr2=0x7f7ce01f9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ce00d4570 0x7f7ce01f9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 -- 172.21.15.67:0/1239844061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce009e440 con 0x7f7ce006e8f0 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cd7fff640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ce00b2dc0 0x7f7ce0156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdf7fe640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ce00d4570 0x7f7ce01f9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cdeffd640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce006e8f0 0x7f7ce0163a90 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7cd000e8a0 tx=0x7f7cd000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cd77fe640 1 -- 172.21.15.67:0/1239844061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0017070 con 0x7f7ce006e8f0 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cd77fe640 1 -- 172.21.15.67:0/1239844061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd000c940 con 0x7f7ce006e8f0 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce009e720 con 0x7f7ce006e8f0 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7cd77fe640 1 -- 172.21.15.67:0/1239844061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd000ccf0 con 0x7f7ce006e8f0 2024-04-25T15:28:24.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.123+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce009e130 con 0x7f7ce006e8f0 2024-04-25T15:28:24.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.124+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce0050740 con 0x7f7ce006e8f0 2024-04-25T15:28:24.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.125+0000 7f7cd77fe640 1 -- 172.21.15.67:0/1239844061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7cd001a070 con 0x7f7ce006e8f0 2024-04-25T15:28:24.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.126+0000 7f7cd77fe640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7cc00744e0 0x7f7cc00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.126+0000 7f7cd77fe640 1 -- 172.21.15.67:0/1239844061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5709+0+0 (secure 0 0 0) 0x7f7cd0095600 con 0x7f7ce006e8f0 2024-04-25T15:28:24.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.126+0000 7f7cd7fff640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7cc00744e0 0x7f7cc00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.126+0000 7f7cd7fff640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7cc00744e0 0x7f7cc00769f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00cdec0 tx=0x7f7cc8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:24.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.130+0000 7f7cd77fe640 1 -- 172.21.15.67:0/1239844061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7cd005fd90 con 0x7f7ce006e8f0 2024-04-25T15:28:24.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.316+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ce00d1220 con 0x7f7ce006e8f0 2024-04-25T15:28:24.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.317+0000 7f7cd77fe640 1 -- 172.21.15.67:0/1239844061 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+16637 (secure 0 0 0) 0x7f7cd005f690 con 0x7f7ce006e8f0 2024-04-25T15:28:24.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:24.318 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":79,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:23.008085+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7cc00744e0 msgr2=0x7f7cc00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7cc00744e0 0x7f7cc00769f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00cdec0 tx=0x7f7cc8009040 comp rx=0 tx=0).stop 2024-04-25T15:28:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce006e8f0 msgr2=0x7f7ce0163a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce006e8f0 0x7f7ce0163a90 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7cd000e8a0 tx=0x7f7cd000ed60 comp rx=0 tx=0).stop 2024-04-25T15:28:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 shutdown_connections 2024-04-25T15:28:24.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7cc00744e0 0x7f7cc00769f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ce00d4570 0x7f7ce01f9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ce00b2dc0 0x7f7ce0156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 --2- 172.21.15.67:0/1239844061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ce006e8f0 0x7f7ce0163a90 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 >> 172.21.15.67:0/1239844061 conn(0x7f7ce00d3a50 msgr2=0x7f7ce0072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:24.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 shutdown_connections 2024-04-25T15:28:24.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.319+0000 7f7ce53c5640 1 -- 172.21.15.67:0/1239844061 wait complete. 2024-04-25T15:28:24.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:28:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.445+0000 7f20e9049640 1 Processor -- start 2024-04-25T15:28:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.445+0000 7f20e9049640 1 -- start start 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.446+0000 7f20e9049640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20e40ce250 0x7f20e415cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.446+0000 7f20e9049640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20e41458c0 0x7f20e40b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.446+0000 7f20e9049640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e4145cd0 con 0x7f20e406d5d0 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.446+0000 7f20e9049640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f20e415d5c0 con 0x7f20e40ce250 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.446+0000 7f20e9049640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f20e415d8a0 con 0x7f20e41458c0 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.446+0000 7f20e2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20e40ce250 0x7f20e415cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.446+0000 7f20e2d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20e40ce250 0x7f20e415cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52120/0 (socket says 172.21.15.67:52120) 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.447+0000 7f20e2d9d640 1 -- 172.21.15.67:0/3035133872 learned_addr learned my addr 172.21.15.67:0/3035133872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.447+0000 7f20dbfff640 1 --2- 172.21.15.67:0/3035133872 >> v2:172.21.15.176:3300/0 conn(0x7f20e41458c0 0x7f20e40b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.447+0000 7f20e2d9d640 1 -- 172.21.15.67:0/3035133872 >> v2:172.21.15.176:3300/0 conn(0x7f20e41458c0 msgr2=0x7f20e40b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.447+0000 7f20e2d9d640 1 --2- 172.21.15.67:0/3035133872 >> v2:172.21.15.176:3300/0 conn(0x7f20e41458c0 0x7f20e40b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.447+0000 7f20e2d9d640 1 -- 172.21.15.67:0/3035133872 >> v1:172.21.15.176:6789/0 conn(0x7f20e406d5d0 legacy=0x7f20e406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.447+0000 7f20e2d9d640 1 -- 172.21.15.67:0/3035133872 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e415db80 con 0x7f20e40ce250 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e2d9d640 1 --2- 172.21.15.67:0/3035133872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e415cfa0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f20e40cf7d0 tx=0x7f20d402c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32620a42bb97f759 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e259c640 1 -- 172.21.15.67:0/3035133872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4036020 con 0x7f20e40ce250 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e259c640 1 -- 172.21.15.67:0/3035133872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d4032640 con 0x7f20e40ce250 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e259c640 1 -- 172.21.15.67:0/3035133872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4032a10 con 0x7f20e40ce250 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e9049640 1 -- 172.21.15.67:0/3035133872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 msgr2=0x7f20e415cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e9049640 1 --2- 172.21.15.67:0/3035133872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e415cfa0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f20e40cf7d0 tx=0x7f20d402c940 comp rx=0 tx=0).stop 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e9049640 1 -- 172.21.15.67:0/3035133872 shutdown_connections 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e9049640 1 --2- 172.21.15.67:0/3035133872 >> v2:172.21.15.176:3300/0 conn(0x7f20e41458c0 0x7f20e40b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e9049640 1 --2- 172.21.15.67:0/3035133872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e415cfa0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.448+0000 7f20e9049640 1 -- 172.21.15.67:0/3035133872 >> 172.21.15.67:0/3035133872 conn(0x7f20e4073090 msgr2=0x7f20e40734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.449+0000 7f20e9049640 1 -- 172.21.15.67:0/3035133872 shutdown_connections 2024-04-25T15:28:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.449+0000 7f20e9049640 1 -- 172.21.15.67:0/3035133872 wait complete. 2024-04-25T15:28:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.450+0000 7f20e9049640 1 Processor -- start 2024-04-25T15:28:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.450+0000 7f20e9049640 1 -- start start 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e9049640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20e406d5d0 0x7f20e40b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e9049640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e4139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e9049640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20e41458c0 0x7f20e413f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e9049640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20e4071310 con 0x7f20e40ce250 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e9049640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f20e415de60 con 0x7f20e41458c0 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e9049640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f20e40b4cf0 con 0x7f20e406d5d0 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e4139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e4139240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52122/0 (socket says 172.21.15.67:52122) 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 -- 172.21.15.67:0/1506600484 learned_addr learned my addr 172.21.15.67:0/1506600484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e2d9d640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20e406d5d0 0x7f20e40b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20e359e640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20e41458c0 0x7f20e413f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 -- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20e406d5d0 msgr2=0x7f20e40b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20e406d5d0 0x7f20e40b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 -- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20e41458c0 msgr2=0x7f20e413f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20e41458c0 0x7f20e413f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 -- 172.21.15.67:0/1506600484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e40b2080 con 0x7f20e40ce250 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.452+0000 7f20dbfff640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e4139240 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00e8a0 tx=0x7f20cc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.453+0000 7f20db7fe640 1 -- 172.21.15.67:0/1506600484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc017070 con 0x7f20e40ce250 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.453+0000 7f20db7fe640 1 -- 172.21.15.67:0/1506600484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20cc00c940 con 0x7f20e40ce250 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.453+0000 7f20db7fe640 1 -- 172.21.15.67:0/1506600484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc00cd30 con 0x7f20e40ce250 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.453+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e40b2360 con 0x7f20e40ce250 2024-04-25T15:28:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.453+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e40b2920 con 0x7f20e40ce250 2024-04-25T15:28:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.454+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e4050740 con 0x7f20e40ce250 2024-04-25T15:28:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.454+0000 7f20db7fe640 1 -- 172.21.15.67:0/1506600484 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f20cc01a070 con 0x7f20e40ce250 2024-04-25T15:28:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.455+0000 7f20db7fe640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f20c40744b0 0x7f20c40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.455+0000 7f20db7fe640 1 -- 172.21.15.67:0/1506600484 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5709+0+0 (secure 0 0 0) 0x7f20cc095450 con 0x7f20e40ce250 2024-04-25T15:28:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.455+0000 7f20e2d9d640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f20c40744b0 0x7f20c40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.456+0000 7f20e2d9d640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f20c40744b0 0x7f20c40769c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f20d4037000 tx=0x7f20d4032000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:24.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.460+0000 7f20db7fe640 1 -- 172.21.15.67:0/1506600484 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20cc05f590 con 0x7f20e40ce250 2024-04-25T15:28:24.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:24.684+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f20e4141d80 con 0x7f20e40ce250 2024-04-25T15:28:25.612 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-04-25T15:28:25.611+0000 7f4ef1a9f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:28:25.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.651+0000 7f20db7fe640 1 -- 172.21.15.67:0/1506600484 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) ==== 118+0+0 (secure 0 0 0) 0x7f20cc063890 con 0x7f20e40ce250 2024-04-25T15:28:25.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f20c40744b0 msgr2=0x7f20c40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:25.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f20c40744b0 0x7f20c40769c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f20d4037000 tx=0x7f20d4032000 comp rx=0 tx=0).stop 2024-04-25T15:28:25.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 msgr2=0x7f20e4139240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:25.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e4139240 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00e8a0 tx=0x7f20cc00ed60 comp rx=0 tx=0).stop 2024-04-25T15:28:25.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 shutdown_connections 2024-04-25T15:28:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f20c40744b0 0x7f20c40769c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20e41458c0 0x7f20e413f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20e40ce250 0x7f20e4139240 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 --2- 172.21.15.67:0/1506600484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20e406d5d0 0x7f20e40b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 >> 172.21.15.67:0/1506600484 conn(0x7f20e4073090 msgr2=0x7f20e40b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 shutdown_connections 2024-04-25T15:28:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.654+0000 7f20e9049640 1 -- 172.21.15.67:0/1506600484 wait complete. 2024-04-25T15:28:25.669 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:28:25.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:28:25.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.790+0000 7f58c4e0c640 1 Processor -- start 2024-04-25T15:28:25.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.790+0000 7f58c4e0c640 1 -- start start 2024-04-25T15:28:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58c4e0c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f58c00b2e60 0x7f58c00d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58c4e0c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c00d23f0 con 0x7f58c0164020 2024-04-25T15:28:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58c4e0c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c00d26d0 con 0x7f58c00d1e20 2024-04-25T15:28:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58c4e0c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f58c00d29b0 con 0x7f58c00b2e60 2024-04-25T15:28:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f58c0164020 0x7f58c0164430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57074/0 (socket says 172.21.15.67:57074) 2024-04-25T15:28:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bed9d640 1 -- 172.21.15.67:0/4173457683 learned_addr learned my addr 172.21.15.67:0/4173457683 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:25.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bdd9b640 1 --2- 172.21.15.67:0/4173457683 >> v2:172.21.15.176:3300/0 conn(0x7f58c00b2e60 0x7f58c00d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276855090 0 0) 0x7f58c00d23f0 con 0x7f58c0164020 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58a8003520 con 0x7f58c0164020 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2160168201 0 0) 0x7f58c00d26d0 con 0x7f58c00d1e20 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58c00d23f0 con 0x7f58c00d1e20 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3408665101 0 0) 0x7f58a8003520 con 0x7f58c0164020 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 >> v2:172.21.15.176:3300/0 conn(0x7f58c00b2e60 msgr2=0x7f58c00d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 --2- 172.21.15.67:0/4173457683 >> v2:172.21.15.176:3300/0 conn(0x7f58c00b2e60 0x7f58c00d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 >> v1:172.21.15.176:6789/0 conn(0x7f58c00d1e20 legacy=0x7f58c008d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.792+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c00d2c90 con 0x7f58c0164020 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f58b00019f0 con 0x7f58c0164020 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 >> v1:172.21.15.67:6789/0 conn(0x7f58c0164020 legacy=0x7f58c0164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a8003b40 0x7f58a8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8004800 0x7f58a8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58a80051a0 0x7f58a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58c00d29b0 con 0x7f58a8003b40 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f58a800c090 con 0x7f58a8004800 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58a800c290 con 0x7f58a80051a0 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58be59c640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a8003b40 0x7f58a8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bdd9b640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8004800 0x7f58a8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bed9d640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58a80051a0 0x7f58a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58be59c640 1 -- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58a80051a0 msgr2=0x7f58a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58be59c640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58a80051a0 0x7f58a800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58be59c640 1 -- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8004800 msgr2=0x7f58a8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58be59c640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8004800 0x7f58a8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58be59c640 1 -- 172.21.15.67:0/4173457683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a8003520 con 0x7f58a8003b40 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.793+0000 7f58bed9d640 1 -- 172.21.15.67:0/4173457683 reap_dead start 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.794+0000 7f58be59c640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a8003b40 0x7f58a8003f90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f58b80018a0 tx=0x7f58a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb98920bccdf925 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.794+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b80014f0 con 0x7f58a8003b40 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.794+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b80052d0 con 0x7f58a8003b40 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.794+0000 7f58bd59a640 1 -- 172.21.15.67:0/4173457683 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b802ac40 con 0x7f58a8003b40 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.795+0000 7f58c4e0c640 1 -- 172.21.15.67:0/4173457683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a8003b40 msgr2=0x7f58a8003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:25.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.795+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a8003b40 0x7f58a8003f90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f58b80018a0 tx=0x7f58a8005830 comp rx=0 tx=0).stop 2024-04-25T15:28:25.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.795+0000 7f58c4e0c640 1 -- 172.21.15.67:0/4173457683 shutdown_connections 2024-04-25T15:28:25.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.795+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/4173457683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58a8003b40 0x7f58a8003f90 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.795+0000 7f58c4e0c640 1 -- 172.21.15.67:0/4173457683 >> 172.21.15.67:0/4173457683 conn(0x7f58c00d0530 msgr2=0x7f58c00d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:25.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.795+0000 7f58c4e0c640 1 -- 172.21.15.67:0/4173457683 shutdown_connections 2024-04-25T15:28:25.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.795+0000 7f58c4e0c640 1 -- 172.21.15.67:0/4173457683 wait complete. 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.797+0000 7f58c4e0c640 1 Processor -- start 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.797+0000 7f58c4e0c640 1 -- start start 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58c4e0c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8003b40 0x7f58c015e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58c4e0c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58c01f6cb0 0x7f58c0162b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58c4e0c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c0163070 0x7f58c00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58c4e0c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58c00d3610 con 0x7f58c0163070 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58c4e0c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f58c00d3490 con 0x7f58a8003b40 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58c4e0c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58c00d3790 con 0x7f58c01f6cb0 2024-04-25T15:28:25.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58bed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c0163070 0x7f58c00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58bed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c0163070 0x7f58c00cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52136/0 (socket says 172.21.15.67:52136) 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58bed9d640 1 -- 172.21.15.67:0/2069607044 learned_addr learned my addr 172.21.15.67:0/2069607044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58bdd9b640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58c01f6cb0 0x7f58c0162b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.798+0000 7f58be59c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8003b40 0x7f58c015e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58bed9d640 1 -- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58c01f6cb0 msgr2=0x7f58c0162b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58bed9d640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58c01f6cb0 0x7f58c0162b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58bed9d640 1 -- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8003b40 msgr2=0x7f58c015e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58bed9d640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8003b40 0x7f58c015e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58bed9d640 1 -- 172.21.15.67:0/2069607044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c00cb680 con 0x7f58c0163070 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58bdd9b640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58c01f6cb0 0x7f58c0162b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58bed9d640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c0163070 0x7f58c00cc440 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f58b0000f80 tx=0x7f58b0007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58aeffd640 1 -- 172.21.15.67:0/2069607044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b000eb00 con 0x7f58c0163070 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58aeffd640 1 -- 172.21.15.67:0/2069607044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b0018070 con 0x7f58c0163070 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c00cb960 con 0x7f58c0163070 2024-04-25T15:28:25.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58aeffd640 1 -- 172.21.15.67:0/2069607044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0013420 con 0x7f58c0163070 2024-04-25T15:28:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.799+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c00cbf20 con 0x7f58c0163070 2024-04-25T15:28:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.800+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c00b80d0 con 0x7f58c0163070 2024-04-25T15:28:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.801+0000 7f58aeffd640 1 -- 172.21.15.67:0/2069607044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f58b00135c0 con 0x7f58c0163070 2024-04-25T15:28:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.801+0000 7f58aeffd640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58a00744b0 0x7f58a00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.801+0000 7f58aeffd640 1 -- 172.21.15.67:0/2069607044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f58b00953f0 con 0x7f58c0163070 2024-04-25T15:28:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.802+0000 7f58be59c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58a00744b0 0x7f58a00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.802+0000 7f58be59c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58a00744b0 0x7f58a00769c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f58b8032000 tx=0x7f58b8009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:25.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:25.806+0000 7f58aeffd640 1 -- 172.21.15.67:0/2069607044 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58b005fca0 con 0x7f58c0163070 2024-04-25T15:28:26.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.031+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f58c0050740 con 0x7f58c0163070 2024-04-25T15:28:26.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.657+0000 7f58aeffd640 1 -- 172.21.15.67:0/2069607044 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v49) ==== 122+0+0 (secure 0 0 0) 0x7f58b005f5a0 con 0x7f58c0163070 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58a00744b0 msgr2=0x7f58a00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58a00744b0 0x7f58a00769c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f58b8032000 tx=0x7f58b8009030 comp rx=0 tx=0).stop 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c0163070 msgr2=0x7f58c00cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c0163070 0x7f58c00cc440 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f58b0000f80 tx=0x7f58b0007bc0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 shutdown_connections 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f58a00744b0 0x7f58a00769c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58c0163070 0x7f58c00cc440 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58c01f6cb0 0x7f58c0162b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 --2- 172.21.15.67:0/2069607044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58a8003b40 0x7f58c015e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.659+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 >> 172.21.15.67:0/2069607044 conn(0x7f58c00d0530 msgr2=0x7f58c00cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.660+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 shutdown_connections 2024-04-25T15:28:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.660+0000 7f58c4e0c640 1 -- 172.21.15.67:0/2069607044 wait complete. 2024-04-25T15:28:26.674 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:28:26.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.791+0000 7ff08752e640 1 Processor -- start 2024-04-25T15:28:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.791+0000 7ff08752e640 1 -- start start 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff08752e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 0x7ff0800d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff08752e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff08006d7e0 0x7ff08006dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff08752e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff08006e130 con 0x7ff0800cf610 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff08752e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff0800b3af0 con 0x7ff08006d7e0 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff08752e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff080164320 con 0x7ff0800b7750 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff084ac9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff0800cf610 0x7ff080164800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57088/0 (socket says 172.21.15.67:57088) 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff084ac9640 1 -- 172.21.15.67:0/1125278886 learned_addr learned my addr 172.21.15.67:0/1125278886 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff085acb640 1 --2- 172.21.15.67:0/1125278886 >> v2:172.21.15.176:3300/0 conn(0x7ff08006d7e0 0x7ff08006dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff0852ca640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 0x7ff0800d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3884886381 0 0) 0x7ff074001110 con 0x7ff0800cf610 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff068003520 con 0x7ff0800cf610 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4106860210 0 0) 0x7ff068003520 con 0x7ff0800cf610 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 msgr2=0x7ff0800d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 0x7ff0800d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 >> v2:172.21.15.176:3300/0 conn(0x7ff08006d7e0 msgr2=0x7ff08006dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 --2- 172.21.15.67:0/1125278886 >> v2:172.21.15.176:3300/0 conn(0x7ff08006d7e0 0x7ff08006dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.793+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0800b2f70 con 0x7ff0800cf610 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7ff074001840 con 0x7ff0800cf610 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff085acb640 1 --2- 172.21.15.67:0/1125278886 >> v2:172.21.15.176:3300/0 conn(0x7ff08006d7e0 0x7ff08006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 >> v1:172.21.15.67:6789/0 conn(0x7ff0800cf610 legacy=0x7ff080164800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0680039b0 0x7ff068003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff068004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 0x7ff068005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0800b3af0 con 0x7ff068004800 2024-04-25T15:28:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff080164320 con 0x7ff0680039b0 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff06800ba70 con 0x7ff0800b7750 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff084ac9640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff068004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff0852ca640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0680039b0 0x7ff068003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff085acb640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 0x7ff068005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff084ac9640 1 -- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 msgr2=0x7ff068005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff084ac9640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0800b7750 0x7ff068005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff084ac9640 1 -- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0680039b0 msgr2=0x7ff068003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff084ac9640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0680039b0 0x7ff068003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff084ac9640 1 -- 172.21.15.67:0/1125278886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff068003520 con 0x7ff068004800 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.794+0000 7ff0852ca640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0680039b0 0x7ff068003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.795+0000 7ff084ac9640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff068004c10 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff07400c490 tx=0x7ff068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5fa65ee3c905e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.795+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074032020 con 0x7ff068004800 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.795+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff074005ad0 con 0x7ff068004800 2024-04-25T15:28:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.795+0000 7ff07ffff640 1 -- 172.21.15.67:0/1125278886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07400c0e0 con 0x7ff068004800 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.796+0000 7ff08752e640 1 -- 172.21.15.67:0/1125278886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 msgr2=0x7ff068004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.796+0000 7ff08752e640 1 --2- 172.21.15.67:0/1125278886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff068004c10 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7ff07400c490 tx=0x7ff068005830 comp rx=0 tx=0).stop 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.796+0000 7ff085acb640 1 -- 172.21.15.67:0/1125278886 reap_dead start 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.796+0000 7ff08752e640 1 -- 172.21.15.67:0/1125278886 shutdown_connections 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.796+0000 7ff08752e640 1 -- 172.21.15.67:0/1125278886 >> 172.21.15.67:0/1125278886 conn(0x7ff08009b7d0 msgr2=0x7ff0800b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.796+0000 7ff08752e640 1 -- 172.21.15.67:0/1125278886 shutdown_connections 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.796+0000 7ff08752e640 1 -- 172.21.15.67:0/1125278886 wait complete. 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.798+0000 7ff08752e640 1 Processor -- start 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.798+0000 7ff08752e640 1 -- start start 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff08752e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff08015f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff08752e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0801f7ae0 0x7ff08009f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff08752e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff08006c5c0 0x7ff08006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff08752e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0800b3810 con 0x7ff068004800 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff08752e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff0800b3530 con 0x7ff0801f7ae0 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff08752e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff0800cfa20 con 0x7ff08006c5c0 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff08015f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff08015f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52152/0 (socket says 172.21.15.67:52152) 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 -- 172.21.15.67:0/2544769502 learned_addr learned my addr 172.21.15.67:0/2544769502 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff084ac9640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0801f7ae0 0x7ff08009f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff085acb640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff08006c5c0 0x7ff08006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 -- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff08006c5c0 msgr2=0x7ff08006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff08006c5c0 0x7ff08006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 -- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0801f7ae0 msgr2=0x7ff08009f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0801f7ae0 0x7ff08009f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff0852ca640 1 -- 172.21.15.67:0/2544769502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08006cf10 con 0x7ff068004800 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.799+0000 7ff084ac9640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0801f7ae0 0x7ff08009f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.800+0000 7ff0852ca640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff08015f7d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff078006c00 tx=0x7ff078011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.800+0000 7ff07dffb640 1 -- 172.21.15.67:0/2544769502 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07800fbb0 con 0x7ff068004800 2024-04-25T15:28:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.800+0000 7ff07dffb640 1 -- 172.21.15.67:0/2544769502 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff07800fd50 con 0x7ff068004800 2024-04-25T15:28:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.800+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08009f6e0 con 0x7ff068004800 2024-04-25T15:28:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.800+0000 7ff07dffb640 1 -- 172.21.15.67:0/2544769502 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff078014680 con 0x7ff068004800 2024-04-25T15:28:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.800+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0801f7000 con 0x7ff068004800 2024-04-25T15:28:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.801+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff080050740 con 0x7ff068004800 2024-04-25T15:28:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.802+0000 7ff07dffb640 1 -- 172.21.15.67:0/2544769502 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff078012040 con 0x7ff068004800 2024-04-25T15:28:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.802+0000 7ff07dffb640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff06c0744b0 0x7ff06c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.802+0000 7ff07dffb640 1 -- 172.21.15.67:0/2544769502 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7ff078095960 con 0x7ff068004800 2024-04-25T15:28:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.802+0000 7ff084ac9640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff06c0744b0 0x7ff06c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.803+0000 7ff084ac9640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff06c0744b0 0x7ff06c0769c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff074033000 tx=0x7ff074009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:26.807+0000 7ff07dffb640 1 -- 172.21.15.67:0/2544769502 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff08006cf10 con 0x7ff068004800 2024-04-25T15:28:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.031+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0800d1420 con 0x7ff068004800 2024-04-25T15:28:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.032+0000 7ff07dffb640 1 -- 172.21.15.67:0/2544769502 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5186 (secure 0 0 0) 0x7ff078019030 con 0x7ff068004800 2024-04-25T15:28:27.033 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:27.033 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/445614904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":445614904},{"type":"v1","addr":"172.21.15.67:6833","nonce":445614904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":6322,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3018066741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3018066741},{"type":"v1","addr":"172.21.15.67:6835","nonce":3018066741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2045345118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2045345118},{"type":"v1","addr":"172.21.15.176:6836","nonce":2045345118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:23.008043+0000","modified":"2024-04-25T15:28:26.033114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6328":{"gid":6328,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6837/2484404027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2484404027},{"type":"v1","addr":"172.21.15.176:6837","nonce":2484404027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6328,"qdb_cluster":[6328]},"id":6}]} 2024-04-25T15:28:27.033 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 49 2024-04-25T15:28:27.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff06c0744b0 msgr2=0x7ff06c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff06c0744b0 0x7ff06c0769c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff074033000 tx=0x7ff074009030 comp rx=0 tx=0).stop 2024-04-25T15:28:27.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 msgr2=0x7ff08015f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff08015f7d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff078006c00 tx=0x7ff078011ee0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 shutdown_connections 2024-04-25T15:28:27.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff06c0744b0 0x7ff06c0769c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff08006c5c0 0x7ff08006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0801f7ae0 0x7ff08009f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 --2- 172.21.15.67:0/2544769502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff068004800 0x7ff08015f7d0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 >> 172.21.15.67:0/2544769502 conn(0x7ff08009b7d0 msgr2=0x7ff080070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:27.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 shutdown_connections 2024-04-25T15:28:27.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.034+0000 7ff08752e640 1 -- 172.21.15.67:0/2544769502 wait complete. 2024-04-25T15:28:27.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:27.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.157+0000 7f76568d9640 1 Processor -- start 2024-04-25T15:28:27.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.157+0000 7f76568d9640 1 -- start start 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f76568d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500d4570 0x7f76500b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f76568d9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76500b2dc0 0x7f76500b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f76568d9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765006d870 con 0x7f765006e8f0 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f76568d9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f765006e460 con 0x7f76500b2dc0 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f76568d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f765006db10 con 0x7f76500d4570 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f7654e76640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76500b2dc0 0x7f76500b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f7654e76640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76500b2dc0 0x7f76500b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52164/0 (socket says 172.21.15.67:52164) 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f7654e76640 1 -- 172.21.15.67:0/678037650 learned_addr learned my addr 172.21.15.67:0/678037650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f764ffff640 1 --2- 172.21.15.67:0/678037650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500d4570 0x7f76500b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f7654e76640 1 -- 172.21.15.67:0/678037650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500d4570 msgr2=0x7f76500b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f7654e76640 1 --2- 172.21.15.67:0/678037650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500d4570 0x7f76500b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f7654e76640 1 -- 172.21.15.67:0/678037650 >> v1:172.21.15.176:6789/0 conn(0x7f765006e8f0 legacy=0x7f76500d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.159+0000 7f7654e76640 1 -- 172.21.15.67:0/678037650 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765006e0f0 con 0x7f76500b2dc0 2024-04-25T15:28:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.160+0000 7f7654e76640 1 --2- 172.21.15.67:0/678037650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76500b2dc0 0x7f76500b31d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f765006e5e0 tx=0x7f764802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d74edfcb9965689 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.160+0000 7f764effd640 1 -- 172.21.15.67:0/678037650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648036070 con 0x7f76500b2dc0 2024-04-25T15:28:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.160+0000 7f764effd640 1 -- 172.21.15.67:0/678037650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7648031600 con 0x7f76500b2dc0 2024-04-25T15:28:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.160+0000 7f764effd640 1 -- 172.21.15.67:0/678037650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76480319d0 con 0x7f76500b2dc0 2024-04-25T15:28:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 -- 172.21.15.67:0/678037650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76500b2dc0 msgr2=0x7f76500b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 --2- 172.21.15.67:0/678037650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76500b2dc0 0x7f76500b31d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f765006e5e0 tx=0x7f764802a680 comp rx=0 tx=0).stop 2024-04-25T15:28:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 -- 172.21.15.67:0/678037650 shutdown_connections 2024-04-25T15:28:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 --2- 172.21.15.67:0/678037650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76500b2dc0 0x7f76500b31d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 --2- 172.21.15.67:0/678037650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500d4570 0x7f76500b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 -- 172.21.15.67:0/678037650 >> 172.21.15.67:0/678037650 conn(0x7f76500d3a50 msgr2=0x7f76500d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 -- 172.21.15.67:0/678037650 shutdown_connections 2024-04-25T15:28:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.161+0000 7f76568d9640 1 -- 172.21.15.67:0/678037650 wait complete. 2024-04-25T15:28:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.162+0000 7f76568d9640 1 Processor -- start 2024-04-25T15:28:27.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.163+0000 7f76568d9640 1 -- start start 2024-04-25T15:28:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f76568d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f765006e8f0 0x7f765009fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f76568d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500b2dc0 0x7f76500d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f76568d9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76500d4570 0x7f76501f9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f76568d9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f765006ed00 con 0x7f765006e8f0 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f76568d9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f76500d0fb0 con 0x7f76500d4570 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f76568d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f76500b3710 con 0x7f76500b2dc0 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f764ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f765006e8f0 0x7f765009fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f764ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f765006e8f0 0x7f765009fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52170/0 (socket says 172.21.15.67:52170) 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f764ffff640 1 -- 172.21.15.67:0/1025214134 learned_addr learned my addr 172.21.15.67:0/1025214134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f764f7fe640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500b2dc0 0x7f76500d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.164+0000 7f7654e76640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76500d4570 0x7f76501f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764ffff640 1 -- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500b2dc0 msgr2=0x7f76500d2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764ffff640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500b2dc0 0x7f76500d2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764ffff640 1 -- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76500d4570 msgr2=0x7f76501f9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764ffff640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76500d4570 0x7f76501f9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764ffff640 1 -- 172.21.15.67:0/1025214134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76500a0880 con 0x7f765006e8f0 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764ffff640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f765006e8f0 0x7f765009fd30 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f763800d7d0 tx=0x7f763800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764cff9640 1 -- 172.21.15.67:0/1025214134 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638018020 con 0x7f765006e8f0 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764cff9640 1 -- 172.21.15.67:0/1025214134 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76380096c0 con 0x7f765006e8f0 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76500a0b60 con 0x7f765006e8f0 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f764cff9640 1 -- 172.21.15.67:0/1025214134 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638009ab0 con 0x7f765006e8f0 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.165+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76500a1120 con 0x7f765006e8f0 2024-04-25T15:28:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.166+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76500a1560 con 0x7f765006e8f0 2024-04-25T15:28:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.167+0000 7f764cff9640 1 -- 172.21.15.67:0/1025214134 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f763801b020 con 0x7f765006e8f0 2024-04-25T15:28:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.167+0000 7f764cff9640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76300744b0 0x7f76300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.167+0000 7f764cff9640 1 -- 172.21.15.67:0/1025214134 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f7638095760 con 0x7f765006e8f0 2024-04-25T15:28:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.168+0000 7f764f7fe640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76300744b0 0x7f76300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.168+0000 7f764f7fe640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76300744b0 0x7f76300769c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7650072260 tx=0x7f7640009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.172+0000 7f764cff9640 1 -- 172.21.15.67:0/1025214134 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7638022020 con 0x7f765006e8f0 2024-04-25T15:28:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.359+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7650050740 con 0x7f765006e8f0 2024-04-25T15:28:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.360+0000 7f764cff9640 1 -- 172.21.15.67:0/1025214134 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16624 (secure 0 0 0) 0x7f763805f930 con 0x7f765006e8f0 2024-04-25T15:28:27.361 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:27.361 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:24.632893+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76300744b0 msgr2=0x7f76300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76300744b0 0x7f76300769c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7650072260 tx=0x7f7640009040 comp rx=0 tx=0).stop 2024-04-25T15:28:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f765006e8f0 msgr2=0x7f765009fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f765006e8f0 0x7f765009fd30 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f763800d7d0 tx=0x7f763800dc90 comp rx=0 tx=0).stop 2024-04-25T15:28:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 shutdown_connections 2024-04-25T15:28:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76300744b0 0x7f76300769c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76500d4570 0x7f76501f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76500b2dc0 0x7f76500d2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 --2- 172.21.15.67:0/1025214134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f765006e8f0 0x7f765009fd30 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 >> 172.21.15.67:0/1025214134 conn(0x7f76500d3a50 msgr2=0x7f7650164030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 shutdown_connections 2024-04-25T15:28:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.362+0000 7f76568d9640 1 -- 172.21.15.67:0/1025214134 wait complete. 2024-04-25T15:28:27.375 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:28:27.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:28:27.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.486+0000 7f3b682bd640 1 Processor -- start 2024-04-25T15:28:27.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.486+0000 7f3b682bd640 1 -- start start 2024-04-25T15:28:27.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b682bd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3b6015dca0 0x7f3b600d05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b682bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b600d0b30 0x7f3b6006fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b682bd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b600d1390 con 0x7f3b600b50d0 2024-04-25T15:28:27.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b682bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3b600d1670 con 0x7f3b600d0b30 2024-04-25T15:28:27.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b682bd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3b6006f130 con 0x7f3b6015dca0 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b6685a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b600d0b30 0x7f3b6006fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b6685a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b600d0b30 0x7f3b6006fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52182/0 (socket says 172.21.15.67:52182) 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b6685a640 1 -- 172.21.15.67:0/4030082569 learned_addr learned my addr 172.21.15.67:0/4030082569 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b65858640 1 --2- 172.21.15.67:0/4030082569 >> v2:172.21.15.176:3300/0 conn(0x7f3b6015dca0 0x7f3b600d05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b6685a640 1 -- 172.21.15.67:0/4030082569 >> v2:172.21.15.176:3300/0 conn(0x7f3b6015dca0 msgr2=0x7f3b600d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b6685a640 1 --2- 172.21.15.67:0/4030082569 >> v2:172.21.15.176:3300/0 conn(0x7f3b6015dca0 0x7f3b600d05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b6685a640 1 -- 172.21.15.67:0/4030082569 >> v1:172.21.15.176:6789/0 conn(0x7f3b600b50d0 legacy=0x7f3b6015d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.488+0000 7f3b6685a640 1 -- 172.21.15.67:0/4030082569 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6006f410 con 0x7f3b600d0b30 2024-04-25T15:28:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.489+0000 7f3b6685a640 1 --2- 172.21.15.67:0/4030082569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b600d0b30 0x7f3b6006fc10 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f3b60072540 tx=0x7f3b5002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4277b9b37aa3d32a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.489+0000 7f3b65057640 1 -- 172.21.15.67:0/4030082569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50036070 con 0x7f3b600d0b30 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.489+0000 7f3b65057640 1 -- 172.21.15.67:0/4030082569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b50031600 con 0x7f3b600d0b30 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.489+0000 7f3b65057640 1 -- 172.21.15.67:0/4030082569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b500319d0 con 0x7f3b600d0b30 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 -- 172.21.15.67:0/4030082569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b600d0b30 msgr2=0x7f3b6006fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 --2- 172.21.15.67:0/4030082569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b600d0b30 0x7f3b6006fc10 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f3b60072540 tx=0x7f3b5002a730 comp rx=0 tx=0).stop 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 -- 172.21.15.67:0/4030082569 shutdown_connections 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 --2- 172.21.15.67:0/4030082569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b600d0b30 0x7f3b6006fc10 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 --2- 172.21.15.67:0/4030082569 >> v2:172.21.15.176:3300/0 conn(0x7f3b6015dca0 0x7f3b600d05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 -- 172.21.15.67:0/4030082569 >> 172.21.15.67:0/4030082569 conn(0x7f3b600d1ae0 msgr2=0x7f3b600d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 -- 172.21.15.67:0/4030082569 shutdown_connections 2024-04-25T15:28:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.490+0000 7f3b682bd640 1 -- 172.21.15.67:0/4030082569 wait complete. 2024-04-25T15:28:27.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.492+0000 7f3b682bd640 1 Processor -- start 2024-04-25T15:28:27.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.492+0000 7f3b682bd640 1 -- start start 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b682bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b600b50d0 0x7f3b6013f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b682bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b600d0b30 0x7f3b601e69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b682bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6015dca0 0x7f3b601f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b682bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3b600d10c0 con 0x7f3b6015dca0 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b682bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3b600d0f40 con 0x7f3b600d0b30 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b682bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3b60143660 con 0x7f3b600b50d0 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6015dca0 0x7f3b601f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6015dca0 0x7f3b601f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52196/0 (socket says 172.21.15.67:52196) 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 -- 172.21.15.67:0/1573235737 learned_addr learned my addr 172.21.15.67:0/1573235737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b66059640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b600b50d0 0x7f3b6013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b65858640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b600d0b30 0x7f3b601e69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 -- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b600b50d0 msgr2=0x7f3b6013f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b600b50d0 0x7f3b6013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 -- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b600d0b30 msgr2=0x7f3b601e69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b600d0b30 0x7f3b601e69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 -- 172.21.15.67:0/1573235737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b601f4450 con 0x7f3b6015dca0 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b65858640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b600d0b30 0x7f3b601e69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.493+0000 7f3b6685a640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6015dca0 0x7f3b601f7a40 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3b5002ad20 tx=0x7f3b50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.494+0000 7f3b5effd640 1 -- 172.21.15.67:0/1573235737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5003f020 con 0x7f3b6015dca0 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.494+0000 7f3b5effd640 1 -- 172.21.15.67:0/1573235737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b50037040 con 0x7f3b6015dca0 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.494+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b601f46c0 con 0x7f3b6015dca0 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.494+0000 7f3b5effd640 1 -- 172.21.15.67:0/1573235737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50036070 con 0x7f3b6015dca0 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.494+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b601f4c80 con 0x7f3b6015dca0 2024-04-25T15:28:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.495+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b60068f70 con 0x7f3b6015dca0 2024-04-25T15:28:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.495+0000 7f3b5effd640 1 -- 172.21.15.67:0/1573235737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3b50042020 con 0x7f3b6015dca0 2024-04-25T15:28:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.496+0000 7f3b5effd640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b4c0744e0 0x7f3b4c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.496+0000 7f3b5effd640 1 -- 172.21.15.67:0/1573235737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f3b500b5900 con 0x7f3b6015dca0 2024-04-25T15:28:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.496+0000 7f3b66059640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b4c0744e0 0x7f3b4c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.496+0000 7f3b66059640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b4c0744e0 0x7f3b4c0769f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006ce20 tx=0x7f3b58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.500+0000 7f3b5effd640 1 -- 172.21.15.67:0/1573235737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b60068f70 con 0x7f3b6015dca0 2024-04-25T15:28:27.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.680+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b60050740 con 0x7f3b4c0744e0 2024-04-25T15:28:27.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.687+0000 7f3b5effd640 1 -- 172.21.15.67:0/1573235737 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250157 (secure 0 0 0) 0x7f3b60050740 con 0x7f3b4c0744e0 2024-04-25T15:28:27.691 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.690+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b4c0744e0 msgr2=0x7f3b4c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.690+0000 7f3b682bd640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b4c0744e0 0x7f3b4c0769f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006ce20 tx=0x7f3b58009040 comp rx=0 tx=0).stop 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.690+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6015dca0 msgr2=0x7f3b601f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.690+0000 7f3b682bd640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6015dca0 0x7f3b601f7a40 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3b5002ad20 tx=0x7f3b50004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 shutdown_connections 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3b4c0744e0 0x7f3b4c0769f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b6015dca0 0x7f3b601f7a40 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b600d0b30 0x7f3b601e69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 --2- 172.21.15.67:0/1573235737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b600b50d0 0x7f3b6013f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 >> 172.21.15.67:0/1573235737 conn(0x7f3b600d1ae0 msgr2=0x7f3b600d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 shutdown_connections 2024-04-25T15:28:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.691+0000 7f3b682bd640 1 -- 172.21.15.67:0/1573235737 wait complete. 2024-04-25T15:28:27.703 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-25T15:28:26.838930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233380,"kb_used_data":8868,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754741340,"statfs":{"total":773094113280,"available":772855132160,"internally_reserved":0,"allocated":9080832,"data_stored":4501190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.224231"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282682+0000","last_change":"2024-04-25T15:28:23.652205+0000","last_active":"2024-04-25T15:28:25.282682+0000","last_peered":"2024-04-25T15:28:25.282682+0000","last_clean":"2024-04-25T15:28:25.282682+0000","last_became_active":"2024-04-25T15:28:23.651726+0000","last_became_peered":"2024-04-25T15:28:23.651726+0000","last_unstale":"2024-04-25T15:28:25.282682+0000","last_undegraded":"2024-04-25T15:28:25.282682+0000","last_fullsized":"2024-04-25T15:28:25.282682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:46.911832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648103+0000","last_change":"2024-04-25T15:28:23.649010+0000","last_active":"2024-04-25T15:28:24.648103+0000","last_peered":"2024-04-25T15:28:24.648103+0000","last_clean":"2024-04-25T15:28:24.648103+0000","last_became_active":"2024-04-25T15:28:23.648637+0000","last_became_peered":"2024-04-25T15:28:23.648637+0000","last_unstale":"2024-04-25T15:28:24.648103+0000","last_undegraded":"2024-04-25T15:28:24.648103+0000","last_fullsized":"2024-04-25T15:28:24.648103+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:28.422017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281638+0000","last_change":"2024-04-25T15:28:23.643910+0000","last_active":"2024-04-25T15:28:25.281638+0000","last_peered":"2024-04-25T15:28:25.281638+0000","last_clean":"2024-04-25T15:28:25.281638+0000","last_became_active":"2024-04-25T15:28:23.643580+0000","last_became_peered":"2024-04-25T15:28:23.643580+0000","last_unstale":"2024-04-25T15:28:25.281638+0000","last_undegraded":"2024-04-25T15:28:25.281638+0000","last_fullsized":"2024-04-25T15:28:25.281638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281992+0000","last_change":"2024-04-25T15:28:23.649272+0000","last_active":"2024-04-25T15:28:25.281992+0000","last_peered":"2024-04-25T15:28:25.281992+0000","last_clean":"2024-04-25T15:28:25.281992+0000","last_became_active":"2024-04-25T15:28:23.647269+0000","last_became_peered":"2024-04-25T15:28:23.647269+0000","last_unstale":"2024-04-25T15:28:25.281992+0000","last_undegraded":"2024-04-25T15:28:25.281992+0000","last_fullsized":"2024-04-25T15:28:25.281992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:27.187638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281710+0000","last_change":"2024-04-25T15:28:23.652480+0000","last_active":"2024-04-25T15:28:25.281710+0000","last_peered":"2024-04-25T15:28:25.281710+0000","last_clean":"2024-04-25T15:28:25.281710+0000","last_became_active":"2024-04-25T15:28:23.652042+0000","last_became_peered":"2024-04-25T15:28:23.652042+0000","last_unstale":"2024-04-25T15:28:25.281710+0000","last_undegraded":"2024-04-25T15:28:25.281710+0000","last_fullsized":"2024-04-25T15:28:25.281710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:26.656523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281499+0000","last_change":"2024-04-25T15:28:23.645012+0000","last_active":"2024-04-25T15:28:25.281499+0000","last_peered":"2024-04-25T15:28:25.281499+0000","last_clean":"2024-04-25T15:28:25.281499+0000","last_became_active":"2024-04-25T15:28:23.644620+0000","last_became_peered":"2024-04-25T15:28:23.644620+0000","last_unstale":"2024-04-25T15:28:25.281499+0000","last_undegraded":"2024-04-25T15:28:25.281499+0000","last_fullsized":"2024-04-25T15:28:25.281499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:25.372903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281525+0000","last_change":"2024-04-25T15:28:23.645167+0000","last_active":"2024-04-25T15:28:25.281525+0000","last_peered":"2024-04-25T15:28:25.281525+0000","last_clean":"2024-04-25T15:28:25.281525+0000","last_became_active":"2024-04-25T15:28:23.644781+0000","last_became_peered":"2024-04-25T15:28:23.644781+0000","last_unstale":"2024-04-25T15:28:25.281525+0000","last_undegraded":"2024-04-25T15:28:25.281525+0000","last_fullsized":"2024-04-25T15:28:25.281525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:25.989787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282012+0000","last_change":"2024-04-25T15:28:23.658433+0000","last_active":"2024-04-25T15:28:25.282012+0000","last_peered":"2024-04-25T15:28:25.282012+0000","last_clean":"2024-04-25T15:28:25.282012+0000","last_became_active":"2024-04-25T15:28:23.658026+0000","last_became_peered":"2024-04-25T15:28:23.658026+0000","last_unstale":"2024-04-25T15:28:25.282012+0000","last_undegraded":"2024-04-25T15:28:25.282012+0000","last_fullsized":"2024-04-25T15:28:25.282012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282414+0000","last_change":"2024-04-25T15:28:23.643202+0000","last_active":"2024-04-25T15:28:25.282414+0000","last_peered":"2024-04-25T15:28:25.282414+0000","last_clean":"2024-04-25T15:28:25.282414+0000","last_became_active":"2024-04-25T15:28:23.642274+0000","last_became_peered":"2024-04-25T15:28:23.642274+0000","last_unstale":"2024-04-25T15:28:25.282414+0000","last_undegraded":"2024-04-25T15:28:25.282414+0000","last_fullsized":"2024-04-25T15:28:25.282414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.639963+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.639463+0000","last_became_peered":"2024-04-25T15:28:23.639463+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:18.521086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282830+0000","last_change":"2024-04-25T15:28:23.649937+0000","last_active":"2024-04-25T15:28:25.282830+0000","last_peered":"2024-04-25T15:28:25.282830+0000","last_clean":"2024-04-25T15:28:25.282830+0000","last_became_active":"2024-04-25T15:28:23.649189+0000","last_became_peered":"2024-04-25T15:28:23.649189+0000","last_unstale":"2024-04-25T15:28:25.282830+0000","last_undegraded":"2024-04-25T15:28:25.282830+0000","last_fullsized":"2024-04-25T15:28:25.282830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281903+0000","last_change":"2024-04-25T15:28:23.644231+0000","last_active":"2024-04-25T15:28:25.281903+0000","last_peered":"2024-04-25T15:28:25.281903+0000","last_clean":"2024-04-25T15:28:25.281903+0000","last_became_active":"2024-04-25T15:28:23.643912+0000","last_became_peered":"2024-04-25T15:28:23.643912+0000","last_unstale":"2024-04-25T15:28:25.281903+0000","last_undegraded":"2024-04-25T15:28:25.281903+0000","last_fullsized":"2024-04-25T15:28:25.281903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:11.975463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282212+0000","last_change":"2024-04-25T15:28:23.650840+0000","last_active":"2024-04-25T15:28:25.282212+0000","last_peered":"2024-04-25T15:28:25.282212+0000","last_clean":"2024-04-25T15:28:25.282212+0000","last_became_active":"2024-04-25T15:28:23.650019+0000","last_became_peered":"2024-04-25T15:28:23.650019+0000","last_unstale":"2024-04-25T15:28:25.282212+0000","last_undegraded":"2024-04-25T15:28:25.282212+0000","last_fullsized":"2024-04-25T15:28:25.282212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:31.696277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.639188+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.638896+0000","last_became_peered":"2024-04-25T15:28:23.638896+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:12.900679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282156+0000","last_change":"2024-04-25T15:28:23.648256+0000","last_active":"2024-04-25T15:28:25.282156+0000","last_peered":"2024-04-25T15:28:25.282156+0000","last_clean":"2024-04-25T15:28:25.282156+0000","last_became_active":"2024-04-25T15:28:23.647895+0000","last_became_peered":"2024-04-25T15:28:23.647895+0000","last_unstale":"2024-04-25T15:28:25.282156+0000","last_undegraded":"2024-04-25T15:28:25.282156+0000","last_fullsized":"2024-04-25T15:28:25.282156+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:31.405763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648935+0000","last_change":"2024-04-25T15:28:23.635903+0000","last_active":"2024-04-25T15:28:24.648935+0000","last_peered":"2024-04-25T15:28:24.648935+0000","last_clean":"2024-04-25T15:28:24.648935+0000","last_became_active":"2024-04-25T15:28:23.635506+0000","last_became_peered":"2024-04-25T15:28:23.635506+0000","last_unstale":"2024-04-25T15:28:24.648935+0000","last_undegraded":"2024-04-25T15:28:24.648935+0000","last_fullsized":"2024-04-25T15:28:24.648935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282313+0000","last_change":"2024-04-25T15:28:23.655054+0000","last_active":"2024-04-25T15:28:25.282313+0000","last_peered":"2024-04-25T15:28:25.282313+0000","last_clean":"2024-04-25T15:28:25.282313+0000","last_became_active":"2024-04-25T15:28:23.654675+0000","last_became_peered":"2024-04-25T15:28:23.654675+0000","last_unstale":"2024-04-25T15:28:25.282313+0000","last_undegraded":"2024-04-25T15:28:25.282313+0000","last_fullsized":"2024-04-25T15:28:25.282313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:29.272131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282209+0000","last_change":"2024-04-25T15:28:23.654074+0000","last_active":"2024-04-25T15:28:25.282209+0000","last_peered":"2024-04-25T15:28:25.282209+0000","last_clean":"2024-04-25T15:28:25.282209+0000","last_became_active":"2024-04-25T15:28:23.653761+0000","last_became_peered":"2024-04-25T15:28:23.653761+0000","last_unstale":"2024-04-25T15:28:25.282209+0000","last_undegraded":"2024-04-25T15:28:25.282209+0000","last_fullsized":"2024-04-25T15:28:25.282209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:28.122216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.637208+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.636801+0000","last_became_peered":"2024-04-25T15:28:23.636801+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:14.403235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282697+0000","last_change":"2024-04-25T15:28:23.646595+0000","last_active":"2024-04-25T15:28:25.282697+0000","last_peered":"2024-04-25T15:28:25.282697+0000","last_clean":"2024-04-25T15:28:25.282697+0000","last_became_active":"2024-04-25T15:28:23.646279+0000","last_became_peered":"2024-04-25T15:28:23.646279+0000","last_unstale":"2024-04-25T15:28:25.282697+0000","last_undegraded":"2024-04-25T15:28:25.282697+0000","last_fullsized":"2024-04-25T15:28:25.282697+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:28.717546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281566+0000","last_change":"2024-04-25T15:28:23.655097+0000","last_active":"2024-04-25T15:28:25.281566+0000","last_peered":"2024-04-25T15:28:25.281566+0000","last_clean":"2024-04-25T15:28:25.281566+0000","last_became_active":"2024-04-25T15:28:23.654517+0000","last_became_peered":"2024-04-25T15:28:23.654517+0000","last_unstale":"2024-04-25T15:28:25.281566+0000","last_undegraded":"2024-04-25T15:28:25.281566+0000","last_fullsized":"2024-04-25T15:28:25.281566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:43:00.233658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647616+0000","last_change":"2024-04-25T15:28:23.648614+0000","last_active":"2024-04-25T15:28:24.647616+0000","last_peered":"2024-04-25T15:28:24.647616+0000","last_clean":"2024-04-25T15:28:24.647616+0000","last_became_active":"2024-04-25T15:28:23.648286+0000","last_became_peered":"2024-04-25T15:28:23.648286+0000","last_unstale":"2024-04-25T15:28:24.647616+0000","last_undegraded":"2024-04-25T15:28:24.647616+0000","last_fullsized":"2024-04-25T15:28:24.647616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:09.027121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282242+0000","last_change":"2024-04-25T15:28:23.639343+0000","last_active":"2024-04-25T15:28:25.282242+0000","last_peered":"2024-04-25T15:28:25.282242+0000","last_clean":"2024-04-25T15:28:25.282242+0000","last_became_active":"2024-04-25T15:28:23.639037+0000","last_became_peered":"2024-04-25T15:28:23.639037+0000","last_unstale":"2024-04-25T15:28:25.282242+0000","last_undegraded":"2024-04-25T15:28:25.282242+0000","last_fullsized":"2024-04-25T15:28:25.282242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:41.056274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282367+0000","last_change":"2024-04-25T15:28:23.647159+0000","last_active":"2024-04-25T15:28:25.282367+0000","last_peered":"2024-04-25T15:28:25.282367+0000","last_clean":"2024-04-25T15:28:25.282367+0000","last_became_active":"2024-04-25T15:28:23.646103+0000","last_became_peered":"2024-04-25T15:28:23.646103+0000","last_unstale":"2024-04-25T15:28:25.282367+0000","last_undegraded":"2024-04-25T15:28:25.282367+0000","last_fullsized":"2024-04-25T15:28:25.282367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648380+0000","last_change":"2024-04-25T15:28:23.644393+0000","last_active":"2024-04-25T15:28:24.648380+0000","last_peered":"2024-04-25T15:28:24.648380+0000","last_clean":"2024-04-25T15:28:24.648380+0000","last_became_active":"2024-04-25T15:28:23.644073+0000","last_became_peered":"2024-04-25T15:28:23.644073+0000","last_unstale":"2024-04-25T15:28:24.648380+0000","last_undegraded":"2024-04-25T15:28:24.648380+0000","last_fullsized":"2024-04-25T15:28:24.648380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:54.453734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647936+0000","last_change":"2024-04-25T15:28:23.654484+0000","last_active":"2024-04-25T15:28:24.647936+0000","last_peered":"2024-04-25T15:28:24.647936+0000","last_clean":"2024-04-25T15:28:24.647936+0000","last_became_active":"2024-04-25T15:28:23.654191+0000","last_became_peered":"2024-04-25T15:28:23.654191+0000","last_unstale":"2024-04-25T15:28:24.647936+0000","last_undegraded":"2024-04-25T15:28:24.647936+0000","last_fullsized":"2024-04-25T15:28:24.647936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:51.690159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.643192+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.642727+0000","last_became_peered":"2024-04-25T15:28:23.642727+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282773+0000","last_change":"2024-04-25T15:28:22.628837+0000","last_active":"2024-04-25T15:28:25.282773+0000","last_peered":"2024-04-25T15:28:25.282773+0000","last_clean":"2024-04-25T15:28:25.282773+0000","last_became_active":"2024-04-25T15:28:22.628231+0000","last_became_peered":"2024-04-25T15:28:22.628231+0000","last_unstale":"2024-04-25T15:28:25.282773+0000","last_undegraded":"2024-04-25T15:28:25.282773+0000","last_fullsized":"2024-04-25T15:28:25.282773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:53.446196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282653+0000","last_change":"2024-04-25T15:28:23.653258+0000","last_active":"2024-04-25T15:28:25.282653+0000","last_peered":"2024-04-25T15:28:25.282653+0000","last_clean":"2024-04-25T15:28:25.282653+0000","last_became_active":"2024-04-25T15:28:23.652929+0000","last_became_peered":"2024-04-25T15:28:23.652929+0000","last_unstale":"2024-04-25T15:28:25.282653+0000","last_undegraded":"2024-04-25T15:28:25.282653+0000","last_fullsized":"2024-04-25T15:28:25.282653+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:55.065573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.617466+0000","last_change":"2024-04-25T15:28:21.617466+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.617466+0000","last_undegraded":"2024-04-25T15:28:21.617466+0000","last_fullsized":"2024-04-25T15:28:21.617466+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282579+0000","last_change":"2024-04-25T15:28:23.647366+0000","last_active":"2024-04-25T15:28:25.282579+0000","last_peered":"2024-04-25T15:28:25.282579+0000","last_clean":"2024-04-25T15:28:25.282579+0000","last_became_active":"2024-04-25T15:28:23.647027+0000","last_became_peered":"2024-04-25T15:28:23.647027+0000","last_unstale":"2024-04-25T15:28:25.282579+0000","last_undegraded":"2024-04-25T15:28:25.282579+0000","last_fullsized":"2024-04-25T15:28:25.282579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648998+0000","last_change":"2024-04-25T15:28:22.627683+0000","last_active":"2024-04-25T15:28:24.648998+0000","last_peered":"2024-04-25T15:28:24.648998+0000","last_clean":"2024-04-25T15:28:24.648998+0000","last_became_active":"2024-04-25T15:28:22.627379+0000","last_became_peered":"2024-04-25T15:28:22.627379+0000","last_unstale":"2024-04-25T15:28:24.648998+0000","last_undegraded":"2024-04-25T15:28:24.648998+0000","last_fullsized":"2024-04-25T15:28:24.648998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:30.340419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281914+0000","last_change":"2024-04-25T15:28:23.655579+0000","last_active":"2024-04-25T15:28:25.281914+0000","last_peered":"2024-04-25T15:28:25.281914+0000","last_clean":"2024-04-25T15:28:25.281914+0000","last_became_active":"2024-04-25T15:28:23.655206+0000","last_became_peered":"2024-04-25T15:28:23.655206+0000","last_unstale":"2024-04-25T15:28:25.281914+0000","last_undegraded":"2024-04-25T15:28:25.281914+0000","last_fullsized":"2024-04-25T15:28:25.281914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:16.785360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.629301+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628962+0000","last_became_peered":"2024-04-25T15:28:22.628962+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:24.982249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648121+0000","last_change":"2024-04-25T15:28:23.654406+0000","last_active":"2024-04-25T15:28:24.648121+0000","last_peered":"2024-04-25T15:28:24.648121+0000","last_clean":"2024-04-25T15:28:24.648121+0000","last_became_active":"2024-04-25T15:28:23.652920+0000","last_became_peered":"2024-04-25T15:28:23.652920+0000","last_unstale":"2024-04-25T15:28:24.648121+0000","last_undegraded":"2024-04-25T15:28:24.648121+0000","last_fullsized":"2024-04-25T15:28:24.648121+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:31.162871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.617457+0000","last_change":"2024-04-25T15:28:21.617457+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.617457+0000","last_undegraded":"2024-04-25T15:28:21.617457+0000","last_fullsized":"2024-04-25T15:28:21.617457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.616871+0000","last_change":"2024-04-25T15:28:21.616871+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.616871+0000","last_undegraded":"2024-04-25T15:28:21.616871+0000","last_fullsized":"2024-04-25T15:28:21.616871+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281419+0000","last_change":"2024-04-25T15:28:23.639119+0000","last_active":"2024-04-25T15:28:25.281419+0000","last_peered":"2024-04-25T15:28:25.281419+0000","last_clean":"2024-04-25T15:28:25.281419+0000","last_became_active":"2024-04-25T15:28:23.638754+0000","last_became_peered":"2024-04-25T15:28:23.638754+0000","last_unstale":"2024-04-25T15:28:25.281419+0000","last_undegraded":"2024-04-25T15:28:25.281419+0000","last_fullsized":"2024-04-25T15:28:25.281419+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648777+0000","last_change":"2024-04-25T15:28:22.631835+0000","last_active":"2024-04-25T15:28:24.648777+0000","last_peered":"2024-04-25T15:28:24.648777+0000","last_clean":"2024-04-25T15:28:24.648777+0000","last_became_active":"2024-04-25T15:28:22.631539+0000","last_became_peered":"2024-04-25T15:28:22.631539+0000","last_unstale":"2024-04-25T15:28:24.648777+0000","last_undegraded":"2024-04-25T15:28:24.648777+0000","last_fullsized":"2024-04-25T15:28:24.648777+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:18.322027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281369+0000","last_change":"2024-04-25T15:28:23.638974+0000","last_active":"2024-04-25T15:28:25.281369+0000","last_peered":"2024-04-25T15:28:25.281369+0000","last_clean":"2024-04-25T15:28:25.281369+0000","last_became_active":"2024-04-25T15:28:23.638642+0000","last_became_peered":"2024-04-25T15:28:23.638642+0000","last_unstale":"2024-04-25T15:28:25.281369+0000","last_undegraded":"2024-04-25T15:28:25.281369+0000","last_fullsized":"2024-04-25T15:28:25.281369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:06.035044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647693+0000","last_change":"2024-04-25T15:28:22.634256+0000","last_active":"2024-04-25T15:28:24.647693+0000","last_peered":"2024-04-25T15:28:24.647693+0000","last_clean":"2024-04-25T15:28:24.647693+0000","last_became_active":"2024-04-25T15:28:22.633942+0000","last_became_peered":"2024-04-25T15:28:22.633942+0000","last_unstale":"2024-04-25T15:28:24.647693+0000","last_undegraded":"2024-04-25T15:28:24.647693+0000","last_fullsized":"2024-04-25T15:28:24.647693+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:10.420589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282150+0000","last_change":"2024-04-25T15:28:23.639749+0000","last_active":"2024-04-25T15:28:25.282150+0000","last_peered":"2024-04-25T15:28:25.282150+0000","last_clean":"2024-04-25T15:28:25.282150+0000","last_became_active":"2024-04-25T15:28:23.639415+0000","last_became_peered":"2024-04-25T15:28:23.639415+0000","last_unstale":"2024-04-25T15:28:25.282150+0000","last_undegraded":"2024-04-25T15:28:25.282150+0000","last_fullsized":"2024-04-25T15:28:25.282150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:38:27.203252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282928+0000","last_change":"2024-04-25T15:28:22.628552+0000","last_active":"2024-04-25T15:28:25.282928+0000","last_peered":"2024-04-25T15:28:25.282928+0000","last_clean":"2024-04-25T15:28:25.282928+0000","last_became_active":"2024-04-25T15:28:22.627193+0000","last_became_peered":"2024-04-25T15:28:22.627193+0000","last_unstale":"2024-04-25T15:28:25.282928+0000","last_undegraded":"2024-04-25T15:28:25.282928+0000","last_fullsized":"2024-04-25T15:28:25.282928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282633+0000","last_change":"2024-04-25T15:28:23.651525+0000","last_active":"2024-04-25T15:28:25.282633+0000","last_peered":"2024-04-25T15:28:25.282633+0000","last_clean":"2024-04-25T15:28:25.282633+0000","last_became_active":"2024-04-25T15:28:23.651179+0000","last_became_peered":"2024-04-25T15:28:23.651179+0000","last_unstale":"2024-04-25T15:28:25.282633+0000","last_undegraded":"2024-04-25T15:28:25.282633+0000","last_fullsized":"2024-04-25T15:28:25.282633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282316+0000","last_change":"2024-04-25T15:28:22.631506+0000","last_active":"2024-04-25T15:28:25.282316+0000","last_peered":"2024-04-25T15:28:25.282316+0000","last_clean":"2024-04-25T15:28:25.282316+0000","last_became_active":"2024-04-25T15:28:22.631161+0000","last_became_peered":"2024-04-25T15:28:22.631161+0000","last_unstale":"2024-04-25T15:28:25.282316+0000","last_undegraded":"2024-04-25T15:28:25.282316+0000","last_fullsized":"2024-04-25T15:28:25.282316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.414479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649203+0000","last_change":"2024-04-25T15:28:23.636915+0000","last_active":"2024-04-25T15:28:24.649203+0000","last_peered":"2024-04-25T15:28:24.649203+0000","last_clean":"2024-04-25T15:28:24.649203+0000","last_became_active":"2024-04-25T15:28:23.636599+0000","last_became_peered":"2024-04-25T15:28:23.636599+0000","last_unstale":"2024-04-25T15:28:24.649203+0000","last_undegraded":"2024-04-25T15:28:24.649203+0000","last_fullsized":"2024-04-25T15:28:24.649203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281907+0000","last_change":"2024-04-25T15:28:22.628373+0000","last_active":"2024-04-25T15:28:25.281907+0000","last_peered":"2024-04-25T15:28:25.281907+0000","last_clean":"2024-04-25T15:28:25.281907+0000","last_became_active":"2024-04-25T15:28:22.628063+0000","last_became_peered":"2024-04-25T15:28:22.628063+0000","last_unstale":"2024-04-25T15:28:25.281907+0000","last_undegraded":"2024-04-25T15:28:25.281907+0000","last_fullsized":"2024-04-25T15:28:25.281907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:17.513548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282803+0000","last_change":"2024-04-25T15:28:22.627717+0000","last_active":"2024-04-25T15:28:25.282803+0000","last_peered":"2024-04-25T15:28:25.282803+0000","last_clean":"2024-04-25T15:28:25.282803+0000","last_became_active":"2024-04-25T15:28:22.627402+0000","last_became_peered":"2024-04-25T15:28:22.627402+0000","last_unstale":"2024-04-25T15:28:25.282803+0000","last_undegraded":"2024-04-25T15:28:25.282803+0000","last_fullsized":"2024-04-25T15:28:25.282803+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648678+0000","last_change":"2024-04-25T15:28:23.630065+0000","last_active":"2024-04-25T15:28:24.648678+0000","last_peered":"2024-04-25T15:28:24.648678+0000","last_clean":"2024-04-25T15:28:24.648678+0000","last_became_active":"2024-04-25T15:28:23.629649+0000","last_became_peered":"2024-04-25T15:28:23.629649+0000","last_unstale":"2024-04-25T15:28:24.648678+0000","last_undegraded":"2024-04-25T15:28:24.648678+0000","last_fullsized":"2024-04-25T15:28:24.648678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648438+0000","last_change":"2024-04-25T15:28:22.632102+0000","last_active":"2024-04-25T15:28:24.648438+0000","last_peered":"2024-04-25T15:28:24.648438+0000","last_clean":"2024-04-25T15:28:24.648438+0000","last_became_active":"2024-04-25T15:28:22.631723+0000","last_became_peered":"2024-04-25T15:28:22.631723+0000","last_unstale":"2024-04-25T15:28:24.648438+0000","last_undegraded":"2024-04-25T15:28:24.648438+0000","last_fullsized":"2024-04-25T15:28:24.648438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:06.295529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282145+0000","last_change":"2024-04-25T15:28:23.639139+0000","last_active":"2024-04-25T15:28:25.282145+0000","last_peered":"2024-04-25T15:28:25.282145+0000","last_clean":"2024-04-25T15:28:25.282145+0000","last_became_active":"2024-04-25T15:28:23.638812+0000","last_became_peered":"2024-04-25T15:28:23.638812+0000","last_unstale":"2024-04-25T15:28:25.282145+0000","last_undegraded":"2024-04-25T15:28:25.282145+0000","last_fullsized":"2024-04-25T15:28:25.282145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:56.852295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281527+0000","last_change":"2024-04-25T15:28:22.631778+0000","last_active":"2024-04-25T15:28:25.281527+0000","last_peered":"2024-04-25T15:28:25.281527+0000","last_clean":"2024-04-25T15:28:25.281527+0000","last_became_active":"2024-04-25T15:28:22.631460+0000","last_became_peered":"2024-04-25T15:28:22.631460+0000","last_unstale":"2024-04-25T15:28:25.281527+0000","last_undegraded":"2024-04-25T15:28:25.281527+0000","last_fullsized":"2024-04-25T15:28:25.281527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.777821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647614+0000","last_change":"2024-04-25T15:28:23.652363+0000","last_active":"2024-04-25T15:28:24.647614+0000","last_peered":"2024-04-25T15:28:24.647614+0000","last_clean":"2024-04-25T15:28:24.647614+0000","last_became_active":"2024-04-25T15:28:23.651744+0000","last_became_peered":"2024-04-25T15:28:23.651744+0000","last_unstale":"2024-04-25T15:28:24.647614+0000","last_undegraded":"2024-04-25T15:28:24.647614+0000","last_fullsized":"2024-04-25T15:28:24.647614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:17.116922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281587+0000","last_change":"2024-04-25T15:28:22.636713+0000","last_active":"2024-04-25T15:28:25.281587+0000","last_peered":"2024-04-25T15:28:25.281587+0000","last_clean":"2024-04-25T15:28:25.281587+0000","last_became_active":"2024-04-25T15:28:22.636389+0000","last_became_peered":"2024-04-25T15:28:22.636389+0000","last_unstale":"2024-04-25T15:28:25.281587+0000","last_undegraded":"2024-04-25T15:28:25.281587+0000","last_fullsized":"2024-04-25T15:28:25.281587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647465+0000","last_change":"2024-04-25T15:28:23.647463+0000","last_active":"2024-04-25T15:28:24.647465+0000","last_peered":"2024-04-25T15:28:24.647465+0000","last_clean":"2024-04-25T15:28:24.647465+0000","last_became_active":"2024-04-25T15:28:23.647133+0000","last_became_peered":"2024-04-25T15:28:23.647133+0000","last_unstale":"2024-04-25T15:28:24.647465+0000","last_undegraded":"2024-04-25T15:28:24.647465+0000","last_fullsized":"2024-04-25T15:28:24.647465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:14.765406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.628461+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628160+0000","last_became_peered":"2024-04-25T15:28:22.628160+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282411+0000","last_change":"2024-04-25T15:28:23.649571+0000","last_active":"2024-04-25T15:28:25.282411+0000","last_peered":"2024-04-25T15:28:25.282411+0000","last_clean":"2024-04-25T15:28:25.282411+0000","last_became_active":"2024-04-25T15:28:23.649214+0000","last_became_peered":"2024-04-25T15:28:23.649214+0000","last_unstale":"2024-04-25T15:28:25.282411+0000","last_undegraded":"2024-04-25T15:28:25.282411+0000","last_fullsized":"2024-04-25T15:28:25.282411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:22.598027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283048+0000","last_change":"2024-04-25T15:28:22.631820+0000","last_active":"2024-04-25T15:28:25.283048+0000","last_peered":"2024-04-25T15:28:25.283048+0000","last_clean":"2024-04-25T15:28:25.283048+0000","last_became_active":"2024-04-25T15:28:22.631501+0000","last_became_peered":"2024-04-25T15:28:22.631501+0000","last_unstale":"2024-04-25T15:28:25.283048+0000","last_undegraded":"2024-04-25T15:28:25.283048+0000","last_fullsized":"2024-04-25T15:28:25.283048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282947+0000","last_change":"2024-04-25T15:28:23.646728+0000","last_active":"2024-04-25T15:28:25.282947+0000","last_peered":"2024-04-25T15:28:25.282947+0000","last_clean":"2024-04-25T15:28:25.282947+0000","last_became_active":"2024-04-25T15:28:23.646234+0000","last_became_peered":"2024-04-25T15:28:23.646234+0000","last_unstale":"2024-04-25T15:28:25.282947+0000","last_undegraded":"2024-04-25T15:28:25.282947+0000","last_fullsized":"2024-04-25T15:28:25.282947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648950+0000","last_change":"2024-04-25T15:28:22.632918+0000","last_active":"2024-04-25T15:28:24.648950+0000","last_peered":"2024-04-25T15:28:24.648950+0000","last_clean":"2024-04-25T15:28:24.648950+0000","last_became_active":"2024-04-25T15:28:22.632551+0000","last_became_peered":"2024-04-25T15:28:22.632551+0000","last_unstale":"2024-04-25T15:28:24.648950+0000","last_undegraded":"2024-04-25T15:28:24.648950+0000","last_fullsized":"2024-04-25T15:28:24.648950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:25.998670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649243+0000","last_change":"2024-04-25T15:28:22.631396+0000","last_active":"2024-04-25T15:28:24.649243+0000","last_peered":"2024-04-25T15:28:24.649243+0000","last_clean":"2024-04-25T15:28:24.649243+0000","last_became_active":"2024-04-25T15:28:22.630989+0000","last_became_peered":"2024-04-25T15:28:22.630989+0000","last_unstale":"2024-04-25T15:28:24.649243+0000","last_undegraded":"2024-04-25T15:28:24.649243+0000","last_fullsized":"2024-04-25T15:28:24.649243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:25.648985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282440+0000","last_change":"2024-04-25T15:28:23.650548+0000","last_active":"2024-04-25T15:28:25.282440+0000","last_peered":"2024-04-25T15:28:25.282440+0000","last_clean":"2024-04-25T15:28:25.282440+0000","last_became_active":"2024-04-25T15:28:23.649636+0000","last_became_peered":"2024-04-25T15:28:23.649636+0000","last_unstale":"2024-04-25T15:28:25.282440+0000","last_undegraded":"2024-04-25T15:28:25.282440+0000","last_fullsized":"2024-04-25T15:28:25.282440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:11.428127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281645+0000","last_change":"2024-04-25T15:28:22.631144+0000","last_active":"2024-04-25T15:28:25.281645+0000","last_peered":"2024-04-25T15:28:25.281645+0000","last_clean":"2024-04-25T15:28:25.281645+0000","last_became_active":"2024-04-25T15:28:22.630762+0000","last_became_peered":"2024-04-25T15:28:22.630762+0000","last_unstale":"2024-04-25T15:28:25.281645+0000","last_undegraded":"2024-04-25T15:28:25.281645+0000","last_fullsized":"2024-04-25T15:28:25.281645+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:10.967925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283204+0000","last_change":"2024-04-25T15:28:23.647208+0000","last_active":"2024-04-25T15:28:25.283204+0000","last_peered":"2024-04-25T15:28:25.283204+0000","last_clean":"2024-04-25T15:28:25.283204+0000","last_became_active":"2024-04-25T15:28:23.646871+0000","last_became_peered":"2024-04-25T15:28:23.646871+0000","last_unstale":"2024-04-25T15:28:25.283204+0000","last_undegraded":"2024-04-25T15:28:25.283204+0000","last_fullsized":"2024-04-25T15:28:25.283204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282312+0000","last_change":"2024-04-25T15:28:22.623948+0000","last_active":"2024-04-25T15:28:25.282312+0000","last_peered":"2024-04-25T15:28:25.282312+0000","last_clean":"2024-04-25T15:28:25.282312+0000","last_became_active":"2024-04-25T15:28:22.623604+0000","last_became_peered":"2024-04-25T15:28:22.623604+0000","last_unstale":"2024-04-25T15:28:25.282312+0000","last_undegraded":"2024-04-25T15:28:25.282312+0000","last_fullsized":"2024-04-25T15:28:25.282312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:13.369418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:28:23.648910+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:28:23.648524+0000","last_became_peered":"2024-04-25T15:28:23.648524+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.853221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281681+0000","last_change":"2024-04-25T15:28:22.635528+0000","last_active":"2024-04-25T15:28:25.281681+0000","last_peered":"2024-04-25T15:28:25.281681+0000","last_clean":"2024-04-25T15:28:25.281681+0000","last_became_active":"2024-04-25T15:28:22.635320+0000","last_became_peered":"2024-04-25T15:28:22.635320+0000","last_unstale":"2024-04-25T15:28:25.281681+0000","last_undegraded":"2024-04-25T15:28:25.281681+0000","last_fullsized":"2024-04-25T15:28:25.281681+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:37.967626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281434+0000","last_change":"2024-04-25T15:28:23.658467+0000","last_active":"2024-04-25T15:28:25.281434+0000","last_peered":"2024-04-25T15:28:25.281434+0000","last_clean":"2024-04-25T15:28:25.281434+0000","last_became_active":"2024-04-25T15:28:23.658051+0000","last_became_peered":"2024-04-25T15:28:23.658051+0000","last_unstale":"2024-04-25T15:28:25.281434+0000","last_undegraded":"2024-04-25T15:28:25.281434+0000","last_fullsized":"2024-04-25T15:28:25.281434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:27.006208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281183+0000","last_change":"2024-04-25T15:28:22.627923+0000","last_active":"2024-04-25T15:28:25.281183+0000","last_peered":"2024-04-25T15:28:25.281183+0000","last_clean":"2024-04-25T15:28:25.281183+0000","last_became_active":"2024-04-25T15:28:22.627595+0000","last_became_peered":"2024-04-25T15:28:22.627595+0000","last_unstale":"2024-04-25T15:28:25.281183+0000","last_undegraded":"2024-04-25T15:28:25.281183+0000","last_fullsized":"2024-04-25T15:28:25.281183+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:04.050831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647686+0000","last_change":"2024-04-25T15:28:22.629989+0000","last_active":"2024-04-25T15:28:24.647686+0000","last_peered":"2024-04-25T15:28:24.647686+0000","last_clean":"2024-04-25T15:28:24.647686+0000","last_became_active":"2024-04-25T15:28:22.629681+0000","last_became_peered":"2024-04-25T15:28:22.629681+0000","last_unstale":"2024-04-25T15:28:24.647686+0000","last_undegraded":"2024-04-25T15:28:24.647686+0000","last_fullsized":"2024-04-25T15:28:24.647686+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:36.386426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281853+0000","last_change":"2024-04-25T15:28:23.643948+0000","last_active":"2024-04-25T15:28:25.281853+0000","last_peered":"2024-04-25T15:28:25.281853+0000","last_clean":"2024-04-25T15:28:25.281853+0000","last_became_active":"2024-04-25T15:28:23.643600+0000","last_became_peered":"2024-04-25T15:28:23.643600+0000","last_unstale":"2024-04-25T15:28:25.281853+0000","last_undegraded":"2024-04-25T15:28:25.281853+0000","last_fullsized":"2024-04-25T15:28:25.281853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648158+0000","last_change":"2024-04-25T15:28:22.630713+0000","last_active":"2024-04-25T15:28:24.648158+0000","last_peered":"2024-04-25T15:28:24.648158+0000","last_clean":"2024-04-25T15:28:24.648158+0000","last_became_active":"2024-04-25T15:28:22.627937+0000","last_became_peered":"2024-04-25T15:28:22.627937+0000","last_unstale":"2024-04-25T15:28:24.648158+0000","last_undegraded":"2024-04-25T15:28:24.648158+0000","last_fullsized":"2024-04-25T15:28:24.648158+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:34.438583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282846+0000","last_change":"2024-04-25T15:28:23.650199+0000","last_active":"2024-04-25T15:28:25.282846+0000","last_peered":"2024-04-25T15:28:25.282846+0000","last_clean":"2024-04-25T15:28:25.282846+0000","last_became_active":"2024-04-25T15:28:23.649599+0000","last_became_peered":"2024-04-25T15:28:23.649599+0000","last_unstale":"2024-04-25T15:28:25.282846+0000","last_undegraded":"2024-04-25T15:28:25.282846+0000","last_fullsized":"2024-04-25T15:28:25.282846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282154+0000","last_change":"2024-04-25T15:28:22.626743+0000","last_active":"2024-04-25T15:28:25.282154+0000","last_peered":"2024-04-25T15:28:25.282154+0000","last_clean":"2024-04-25T15:28:25.282154+0000","last_became_active":"2024-04-25T15:28:22.626434+0000","last_became_peered":"2024-04-25T15:28:22.626434+0000","last_unstale":"2024-04-25T15:28:25.282154+0000","last_undegraded":"2024-04-25T15:28:25.282154+0000","last_fullsized":"2024-04-25T15:28:25.282154+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:13.395697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282910+0000","last_change":"2024-04-25T15:28:22.630470+0000","last_active":"2024-04-25T15:28:25.282910+0000","last_peered":"2024-04-25T15:28:25.282910+0000","last_clean":"2024-04-25T15:28:25.282910+0000","last_became_active":"2024-04-25T15:28:22.630069+0000","last_became_peered":"2024-04-25T15:28:22.630069+0000","last_unstale":"2024-04-25T15:28:25.282910+0000","last_undegraded":"2024-04-25T15:28:25.282910+0000","last_fullsized":"2024-04-25T15:28:25.282910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:30.155333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282356+0000","last_change":"2024-04-25T15:28:23.648809+0000","last_active":"2024-04-25T15:28:25.282356+0000","last_peered":"2024-04-25T15:28:25.282356+0000","last_clean":"2024-04-25T15:28:25.282356+0000","last_became_active":"2024-04-25T15:28:23.648507+0000","last_became_peered":"2024-04-25T15:28:23.648507+0000","last_unstale":"2024-04-25T15:28:25.282356+0000","last_undegraded":"2024-04-25T15:28:25.282356+0000","last_fullsized":"2024-04-25T15:28:25.282356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648878+0000","last_change":"2024-04-25T15:28:22.628170+0000","last_active":"2024-04-25T15:28:24.648878+0000","last_peered":"2024-04-25T15:28:24.648878+0000","last_clean":"2024-04-25T15:28:24.648878+0000","last_became_active":"2024-04-25T15:28:22.627191+0000","last_became_peered":"2024-04-25T15:28:22.627191+0000","last_unstale":"2024-04-25T15:28:24.648878+0000","last_undegraded":"2024-04-25T15:28:24.648878+0000","last_fullsized":"2024-04-25T15:28:24.648878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:31.302274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648932+0000","last_change":"2024-04-25T15:28:23.630876+0000","last_active":"2024-04-25T15:28:24.648932+0000","last_peered":"2024-04-25T15:28:24.648932+0000","last_clean":"2024-04-25T15:28:24.648932+0000","last_became_active":"2024-04-25T15:28:23.630558+0000","last_became_peered":"2024-04-25T15:28:23.630558+0000","last_unstale":"2024-04-25T15:28:24.648932+0000","last_undegraded":"2024-04-25T15:28:24.648932+0000","last_fullsized":"2024-04-25T15:28:24.648932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.616307+0000","last_change":"2024-04-25T15:28:21.616307+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.616307+0000","last_undegraded":"2024-04-25T15:28:21.616307+0000","last_fullsized":"2024-04-25T15:28:21.616307+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281519+0000","last_change":"2024-04-25T15:28:23.640269+0000","last_active":"2024-04-25T15:28:25.281519+0000","last_peered":"2024-04-25T15:28:25.281519+0000","last_clean":"2024-04-25T15:28:25.281519+0000","last_became_active":"2024-04-25T15:28:23.639944+0000","last_became_peered":"2024-04-25T15:28:23.639944+0000","last_unstale":"2024-04-25T15:28:25.281519+0000","last_undegraded":"2024-04-25T15:28:25.281519+0000","last_fullsized":"2024-04-25T15:28:25.281519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282853+0000","last_change":"2024-04-25T15:28:22.626538+0000","last_active":"2024-04-25T15:28:25.282853+0000","last_peered":"2024-04-25T15:28:25.282853+0000","last_clean":"2024-04-25T15:28:25.282853+0000","last_became_active":"2024-04-25T15:28:22.626207+0000","last_became_peered":"2024-04-25T15:28:22.626207+0000","last_unstale":"2024-04-25T15:28:25.282853+0000","last_undegraded":"2024-04-25T15:28:25.282853+0000","last_fullsized":"2024-04-25T15:28:25.282853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:50.682621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282764+0000","last_change":"2024-04-25T15:28:23.646030+0000","last_active":"2024-04-25T15:28:25.282764+0000","last_peered":"2024-04-25T15:28:25.282764+0000","last_clean":"2024-04-25T15:28:25.282764+0000","last_became_active":"2024-04-25T15:28:23.645733+0000","last_became_peered":"2024-04-25T15:28:23.645733+0000","last_unstale":"2024-04-25T15:28:25.282764+0000","last_undegraded":"2024-04-25T15:28:25.282764+0000","last_fullsized":"2024-04-25T15:28:25.282764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.615544+0000","last_change":"2024-04-25T15:28:21.615544+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.615544+0000","last_undegraded":"2024-04-25T15:28:21.615544+0000","last_fullsized":"2024-04-25T15:28:21.615544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648545+0000","last_change":"2024-04-25T15:28:23.640057+0000","last_active":"2024-04-25T15:28:24.648545+0000","last_peered":"2024-04-25T15:28:24.648545+0000","last_clean":"2024-04-25T15:28:24.648545+0000","last_became_active":"2024-04-25T15:28:23.639747+0000","last_became_peered":"2024-04-25T15:28:23.639747+0000","last_unstale":"2024-04-25T15:28:24.648545+0000","last_undegraded":"2024-04-25T15:28:24.648545+0000","last_fullsized":"2024-04-25T15:28:24.648545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:06.046345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281905+0000","last_change":"2024-04-25T15:28:22.632108+0000","last_active":"2024-04-25T15:28:25.281905+0000","last_peered":"2024-04-25T15:28:25.281905+0000","last_clean":"2024-04-25T15:28:25.281905+0000","last_became_active":"2024-04-25T15:28:22.631768+0000","last_became_peered":"2024-04-25T15:28:22.631768+0000","last_unstale":"2024-04-25T15:28:25.281905+0000","last_undegraded":"2024-04-25T15:28:25.281905+0000","last_fullsized":"2024-04-25T15:28:25.281905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":218,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1581056,"data_stored":1007311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":418507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":397523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":418485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":417455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1581056,"data_stored":1007275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:28:27.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:28:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.853+0000 7f6b8810d640 1 Processor -- start 2024-04-25T15:28:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.854+0000 7f6b8810d640 1 -- start start 2024-04-25T15:28:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b8810d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b800d4130 0x7f6b800d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b8810d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6b8006dcf0 0x7f6b800d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b8810d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b800b3c10 con 0x7f6b8006d8e0 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b8810d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6b800b9520 con 0x7f6b8006dcf0 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b8810d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6b800b4820 con 0x7f6b800d4130 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b866aa640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6b8006dcf0 0x7f6b800d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b856a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b800d4130 0x7f6b800d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b866aa640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6b8006dcf0 0x7f6b800d22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:55684/0 (socket says 172.21.15.67:55684) 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b85ea9640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f6b8006d8e0 0x7f6b800d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:52858/0 (socket says 172.21.15.67:52858) 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b856a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b800d4130 0x7f6b800d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:58868/0 (socket says 172.21.15.67:58868) 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.855+0000 7f6b866aa640 1 -- 172.21.15.67:0/727564751 learned_addr learned my addr 172.21.15.67:0/727564751 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.856+0000 7f6b866aa640 1 -- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b800d4130 msgr2=0x7f6b800d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.856+0000 7f6b866aa640 1 --2- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b800d4130 0x7f6b800d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.856+0000 7f6b866aa640 1 -- 172.21.15.67:0/727564751 >> v1:172.21.15.176:6789/0 conn(0x7f6b8006d8e0 legacy=0x7f6b800d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.856+0000 7f6b866aa640 1 -- 172.21.15.67:0/727564751 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b800b3ef0 con 0x7f6b8006dcf0 2024-04-25T15:28:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.856+0000 7f6b84ea7640 1 -- 172.21.15.67:0/727564751 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214330570 0 0) 0x7f6b800b3c10 con 0x7f6b8006d8e0 2024-04-25T15:28:27.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.856+0000 7f6b856a8640 1 --2- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b800d4130 0x7f6b800d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.856+0000 7f6b866aa640 1 --2- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 0x7f6b800d22a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009e50 tx=0x7f6b7c02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56464bb6e911e6fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b84ea7640 1 -- 172.21.15.67:0/727564751 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c036070 con 0x7f6b8006dcf0 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b84ea7640 1 -- 172.21.15.67:0/727564751 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c031600 con 0x7f6b8006dcf0 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b84ea7640 1 -- 172.21.15.67:0/727564751 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0319d0 con 0x7f6b8006dcf0 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 -- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 msgr2=0x7f6b800d22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 --2- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 0x7f6b800d22a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009e50 tx=0x7f6b7c02a680 comp rx=0 tx=0).stop 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 -- 172.21.15.67:0/727564751 shutdown_connections 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 --2- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 0x7f6b800d22a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 --2- 172.21.15.67:0/727564751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b800d4130 0x7f6b800d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 -- 172.21.15.67:0/727564751 >> 172.21.15.67:0/727564751 conn(0x7f6b8008d110 msgr2=0x7f6b8008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 -- 172.21.15.67:0/727564751 shutdown_connections 2024-04-25T15:28:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.857+0000 7f6b8810d640 1 -- 172.21.15.67:0/727564751 wait complete. 2024-04-25T15:28:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.859+0000 7f6b8810d640 1 Processor -- start 2024-04-25T15:28:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.859+0000 7f6b8810d640 1 -- start start 2024-04-25T15:28:27.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.860+0000 7f6b8810d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b8006d8e0 0x7f6b8015d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.860+0000 7f6b8810d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 0x7f6b8009f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.860+0000 7f6b8810d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b801f9be0 0x7f6b8009fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.860+0000 7f6b8810d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b800b38e0 con 0x7f6b801f9be0 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.860+0000 7f6b8810d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6b800b44b0 con 0x7f6b8006dcf0 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.860+0000 7f6b8810d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6b800d27e0 con 0x7f6b8006d8e0 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b801f9be0 0x7f6b8009fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b801f9be0 0x7f6b8009fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56956/0 (socket says 172.21.15.67:56956) 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 -- 172.21.15.67:0/1686731552 learned_addr learned my addr 172.21.15.67:0/1686731552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b85ea9640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b8006d8e0 0x7f6b8015d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b856a8640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 0x7f6b8009f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 -- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b8006d8e0 msgr2=0x7f6b8015d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b8006d8e0 0x7f6b8015d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 -- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 msgr2=0x7f6b8009f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:27.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 0x7f6b8009f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 -- 172.21.15.67:0/1686731552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b801fa130 con 0x7f6b801f9be0 2024-04-25T15:28:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b866aa640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b801f9be0 0x7f6b8009fd10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009f70 tx=0x7f6b7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b767fc640 1 -- 172.21.15.67:0/1686731552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c036050 con 0x7f6b801f9be0 2024-04-25T15:28:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b767fc640 1 -- 172.21.15.67:0/1686731552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c042070 con 0x7f6b801f9be0 2024-04-25T15:28:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8009dae0 con 0x7f6b801f9be0 2024-04-25T15:28:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b767fc640 1 -- 172.21.15.67:0/1686731552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c009510 con 0x7f6b801f9be0 2024-04-25T15:28:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.861+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8009e060 con 0x7f6b801f9be0 2024-04-25T15:28:27.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.863+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b80050740 con 0x7f6b801f9be0 2024-04-25T15:28:27.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.863+0000 7f6b767fc640 1 -- 172.21.15.67:0/1686731552 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6b7c007bf0 con 0x7f6b801f9be0 2024-04-25T15:28:27.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.864+0000 7f6b767fc640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b640744e0 0x7f6b640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:27.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.864+0000 7f6b767fc640 1 -- 172.21.15.67:0/1686731552 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f6b7c0b5e20 con 0x7f6b801f9be0 2024-04-25T15:28:27.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.864+0000 7f6b85ea9640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b640744e0 0x7f6b640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:27.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.865+0000 7f6b85ea9640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b640744e0 0x7f6b640769f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6b78007fd0 tx=0x7f6b78007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:27.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:27.868+0000 7f6b767fc640 1 -- 172.21.15.67:0/1686731552 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b8009dae0 con 0x7f6b801f9be0 2024-04-25T15:28:28.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.048+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6b8009d820 con 0x7f6b640744e0 2024-04-25T15:28:28.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.055+0000 7f6b767fc640 1 -- 172.21.15.67:0/1686731552 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250157 (secure 0 0 0) 0x7f6b8009d820 con 0x7f6b640744e0 2024-04-25T15:28:28.058 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b640744e0 msgr2=0x7f6b640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b640744e0 0x7f6b640769f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6b78007fd0 tx=0x7f6b78007de0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b801f9be0 msgr2=0x7f6b8009fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b801f9be0 0x7f6b8009fd10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c009f70 tx=0x7f6b7c004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 shutdown_connections 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b640744e0 0x7f6b640769f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b801f9be0 0x7f6b8009fd10 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b8006dcf0 0x7f6b8009f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 --2- 172.21.15.67:0/1686731552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b8006d8e0 0x7f6b8015d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.057+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 >> 172.21.15.67:0/1686731552 conn(0x7f6b8008d110 msgr2=0x7f6b80071f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.058+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 shutdown_connections 2024-04-25T15:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.058+0000 7f6b8810d640 1 -- 172.21.15.67:0/1686731552 wait complete. 2024-04-25T15:28:28.069 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-25T15:28:26.838930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233380,"kb_used_data":8868,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754741340,"statfs":{"total":773094113280,"available":772855132160,"internally_reserved":0,"allocated":9080832,"data_stored":4501190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.224231"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282682+0000","last_change":"2024-04-25T15:28:23.652205+0000","last_active":"2024-04-25T15:28:25.282682+0000","last_peered":"2024-04-25T15:28:25.282682+0000","last_clean":"2024-04-25T15:28:25.282682+0000","last_became_active":"2024-04-25T15:28:23.651726+0000","last_became_peered":"2024-04-25T15:28:23.651726+0000","last_unstale":"2024-04-25T15:28:25.282682+0000","last_undegraded":"2024-04-25T15:28:25.282682+0000","last_fullsized":"2024-04-25T15:28:25.282682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:46.911832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648103+0000","last_change":"2024-04-25T15:28:23.649010+0000","last_active":"2024-04-25T15:28:24.648103+0000","last_peered":"2024-04-25T15:28:24.648103+0000","last_clean":"2024-04-25T15:28:24.648103+0000","last_became_active":"2024-04-25T15:28:23.648637+0000","last_became_peered":"2024-04-25T15:28:23.648637+0000","last_unstale":"2024-04-25T15:28:24.648103+0000","last_undegraded":"2024-04-25T15:28:24.648103+0000","last_fullsized":"2024-04-25T15:28:24.648103+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:28.422017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281638+0000","last_change":"2024-04-25T15:28:23.643910+0000","last_active":"2024-04-25T15:28:25.281638+0000","last_peered":"2024-04-25T15:28:25.281638+0000","last_clean":"2024-04-25T15:28:25.281638+0000","last_became_active":"2024-04-25T15:28:23.643580+0000","last_became_peered":"2024-04-25T15:28:23.643580+0000","last_unstale":"2024-04-25T15:28:25.281638+0000","last_undegraded":"2024-04-25T15:28:25.281638+0000","last_fullsized":"2024-04-25T15:28:25.281638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281992+0000","last_change":"2024-04-25T15:28:23.649272+0000","last_active":"2024-04-25T15:28:25.281992+0000","last_peered":"2024-04-25T15:28:25.281992+0000","last_clean":"2024-04-25T15:28:25.281992+0000","last_became_active":"2024-04-25T15:28:23.647269+0000","last_became_peered":"2024-04-25T15:28:23.647269+0000","last_unstale":"2024-04-25T15:28:25.281992+0000","last_undegraded":"2024-04-25T15:28:25.281992+0000","last_fullsized":"2024-04-25T15:28:25.281992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:27.187638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281710+0000","last_change":"2024-04-25T15:28:23.652480+0000","last_active":"2024-04-25T15:28:25.281710+0000","last_peered":"2024-04-25T15:28:25.281710+0000","last_clean":"2024-04-25T15:28:25.281710+0000","last_became_active":"2024-04-25T15:28:23.652042+0000","last_became_peered":"2024-04-25T15:28:23.652042+0000","last_unstale":"2024-04-25T15:28:25.281710+0000","last_undegraded":"2024-04-25T15:28:25.281710+0000","last_fullsized":"2024-04-25T15:28:25.281710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:26.656523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281499+0000","last_change":"2024-04-25T15:28:23.645012+0000","last_active":"2024-04-25T15:28:25.281499+0000","last_peered":"2024-04-25T15:28:25.281499+0000","last_clean":"2024-04-25T15:28:25.281499+0000","last_became_active":"2024-04-25T15:28:23.644620+0000","last_became_peered":"2024-04-25T15:28:23.644620+0000","last_unstale":"2024-04-25T15:28:25.281499+0000","last_undegraded":"2024-04-25T15:28:25.281499+0000","last_fullsized":"2024-04-25T15:28:25.281499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:25.372903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281525+0000","last_change":"2024-04-25T15:28:23.645167+0000","last_active":"2024-04-25T15:28:25.281525+0000","last_peered":"2024-04-25T15:28:25.281525+0000","last_clean":"2024-04-25T15:28:25.281525+0000","last_became_active":"2024-04-25T15:28:23.644781+0000","last_became_peered":"2024-04-25T15:28:23.644781+0000","last_unstale":"2024-04-25T15:28:25.281525+0000","last_undegraded":"2024-04-25T15:28:25.281525+0000","last_fullsized":"2024-04-25T15:28:25.281525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:25.989787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282012+0000","last_change":"2024-04-25T15:28:23.658433+0000","last_active":"2024-04-25T15:28:25.282012+0000","last_peered":"2024-04-25T15:28:25.282012+0000","last_clean":"2024-04-25T15:28:25.282012+0000","last_became_active":"2024-04-25T15:28:23.658026+0000","last_became_peered":"2024-04-25T15:28:23.658026+0000","last_unstale":"2024-04-25T15:28:25.282012+0000","last_undegraded":"2024-04-25T15:28:25.282012+0000","last_fullsized":"2024-04-25T15:28:25.282012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282414+0000","last_change":"2024-04-25T15:28:23.643202+0000","last_active":"2024-04-25T15:28:25.282414+0000","last_peered":"2024-04-25T15:28:25.282414+0000","last_clean":"2024-04-25T15:28:25.282414+0000","last_became_active":"2024-04-25T15:28:23.642274+0000","last_became_peered":"2024-04-25T15:28:23.642274+0000","last_unstale":"2024-04-25T15:28:25.282414+0000","last_undegraded":"2024-04-25T15:28:25.282414+0000","last_fullsized":"2024-04-25T15:28:25.282414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.639963+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.639463+0000","last_became_peered":"2024-04-25T15:28:23.639463+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:18.521086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282830+0000","last_change":"2024-04-25T15:28:23.649937+0000","last_active":"2024-04-25T15:28:25.282830+0000","last_peered":"2024-04-25T15:28:25.282830+0000","last_clean":"2024-04-25T15:28:25.282830+0000","last_became_active":"2024-04-25T15:28:23.649189+0000","last_became_peered":"2024-04-25T15:28:23.649189+0000","last_unstale":"2024-04-25T15:28:25.282830+0000","last_undegraded":"2024-04-25T15:28:25.282830+0000","last_fullsized":"2024-04-25T15:28:25.282830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281903+0000","last_change":"2024-04-25T15:28:23.644231+0000","last_active":"2024-04-25T15:28:25.281903+0000","last_peered":"2024-04-25T15:28:25.281903+0000","last_clean":"2024-04-25T15:28:25.281903+0000","last_became_active":"2024-04-25T15:28:23.643912+0000","last_became_peered":"2024-04-25T15:28:23.643912+0000","last_unstale":"2024-04-25T15:28:25.281903+0000","last_undegraded":"2024-04-25T15:28:25.281903+0000","last_fullsized":"2024-04-25T15:28:25.281903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:11.975463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282212+0000","last_change":"2024-04-25T15:28:23.650840+0000","last_active":"2024-04-25T15:28:25.282212+0000","last_peered":"2024-04-25T15:28:25.282212+0000","last_clean":"2024-04-25T15:28:25.282212+0000","last_became_active":"2024-04-25T15:28:23.650019+0000","last_became_peered":"2024-04-25T15:28:23.650019+0000","last_unstale":"2024-04-25T15:28:25.282212+0000","last_undegraded":"2024-04-25T15:28:25.282212+0000","last_fullsized":"2024-04-25T15:28:25.282212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:31.696277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.639188+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.638896+0000","last_became_peered":"2024-04-25T15:28:23.638896+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:12.900679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282156+0000","last_change":"2024-04-25T15:28:23.648256+0000","last_active":"2024-04-25T15:28:25.282156+0000","last_peered":"2024-04-25T15:28:25.282156+0000","last_clean":"2024-04-25T15:28:25.282156+0000","last_became_active":"2024-04-25T15:28:23.647895+0000","last_became_peered":"2024-04-25T15:28:23.647895+0000","last_unstale":"2024-04-25T15:28:25.282156+0000","last_undegraded":"2024-04-25T15:28:25.282156+0000","last_fullsized":"2024-04-25T15:28:25.282156+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:31.405763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648935+0000","last_change":"2024-04-25T15:28:23.635903+0000","last_active":"2024-04-25T15:28:24.648935+0000","last_peered":"2024-04-25T15:28:24.648935+0000","last_clean":"2024-04-25T15:28:24.648935+0000","last_became_active":"2024-04-25T15:28:23.635506+0000","last_became_peered":"2024-04-25T15:28:23.635506+0000","last_unstale":"2024-04-25T15:28:24.648935+0000","last_undegraded":"2024-04-25T15:28:24.648935+0000","last_fullsized":"2024-04-25T15:28:24.648935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282313+0000","last_change":"2024-04-25T15:28:23.655054+0000","last_active":"2024-04-25T15:28:25.282313+0000","last_peered":"2024-04-25T15:28:25.282313+0000","last_clean":"2024-04-25T15:28:25.282313+0000","last_became_active":"2024-04-25T15:28:23.654675+0000","last_became_peered":"2024-04-25T15:28:23.654675+0000","last_unstale":"2024-04-25T15:28:25.282313+0000","last_undegraded":"2024-04-25T15:28:25.282313+0000","last_fullsized":"2024-04-25T15:28:25.282313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:29.272131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282209+0000","last_change":"2024-04-25T15:28:23.654074+0000","last_active":"2024-04-25T15:28:25.282209+0000","last_peered":"2024-04-25T15:28:25.282209+0000","last_clean":"2024-04-25T15:28:25.282209+0000","last_became_active":"2024-04-25T15:28:23.653761+0000","last_became_peered":"2024-04-25T15:28:23.653761+0000","last_unstale":"2024-04-25T15:28:25.282209+0000","last_undegraded":"2024-04-25T15:28:25.282209+0000","last_fullsized":"2024-04-25T15:28:25.282209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:28.122216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.637208+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.636801+0000","last_became_peered":"2024-04-25T15:28:23.636801+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:14.403235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282697+0000","last_change":"2024-04-25T15:28:23.646595+0000","last_active":"2024-04-25T15:28:25.282697+0000","last_peered":"2024-04-25T15:28:25.282697+0000","last_clean":"2024-04-25T15:28:25.282697+0000","last_became_active":"2024-04-25T15:28:23.646279+0000","last_became_peered":"2024-04-25T15:28:23.646279+0000","last_unstale":"2024-04-25T15:28:25.282697+0000","last_undegraded":"2024-04-25T15:28:25.282697+0000","last_fullsized":"2024-04-25T15:28:25.282697+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:28.717546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281566+0000","last_change":"2024-04-25T15:28:23.655097+0000","last_active":"2024-04-25T15:28:25.281566+0000","last_peered":"2024-04-25T15:28:25.281566+0000","last_clean":"2024-04-25T15:28:25.281566+0000","last_became_active":"2024-04-25T15:28:23.654517+0000","last_became_peered":"2024-04-25T15:28:23.654517+0000","last_unstale":"2024-04-25T15:28:25.281566+0000","last_undegraded":"2024-04-25T15:28:25.281566+0000","last_fullsized":"2024-04-25T15:28:25.281566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:43:00.233658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647616+0000","last_change":"2024-04-25T15:28:23.648614+0000","last_active":"2024-04-25T15:28:24.647616+0000","last_peered":"2024-04-25T15:28:24.647616+0000","last_clean":"2024-04-25T15:28:24.647616+0000","last_became_active":"2024-04-25T15:28:23.648286+0000","last_became_peered":"2024-04-25T15:28:23.648286+0000","last_unstale":"2024-04-25T15:28:24.647616+0000","last_undegraded":"2024-04-25T15:28:24.647616+0000","last_fullsized":"2024-04-25T15:28:24.647616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:09.027121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282242+0000","last_change":"2024-04-25T15:28:23.639343+0000","last_active":"2024-04-25T15:28:25.282242+0000","last_peered":"2024-04-25T15:28:25.282242+0000","last_clean":"2024-04-25T15:28:25.282242+0000","last_became_active":"2024-04-25T15:28:23.639037+0000","last_became_peered":"2024-04-25T15:28:23.639037+0000","last_unstale":"2024-04-25T15:28:25.282242+0000","last_undegraded":"2024-04-25T15:28:25.282242+0000","last_fullsized":"2024-04-25T15:28:25.282242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:41.056274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282367+0000","last_change":"2024-04-25T15:28:23.647159+0000","last_active":"2024-04-25T15:28:25.282367+0000","last_peered":"2024-04-25T15:28:25.282367+0000","last_clean":"2024-04-25T15:28:25.282367+0000","last_became_active":"2024-04-25T15:28:23.646103+0000","last_became_peered":"2024-04-25T15:28:23.646103+0000","last_unstale":"2024-04-25T15:28:25.282367+0000","last_undegraded":"2024-04-25T15:28:25.282367+0000","last_fullsized":"2024-04-25T15:28:25.282367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648380+0000","last_change":"2024-04-25T15:28:23.644393+0000","last_active":"2024-04-25T15:28:24.648380+0000","last_peered":"2024-04-25T15:28:24.648380+0000","last_clean":"2024-04-25T15:28:24.648380+0000","last_became_active":"2024-04-25T15:28:23.644073+0000","last_became_peered":"2024-04-25T15:28:23.644073+0000","last_unstale":"2024-04-25T15:28:24.648380+0000","last_undegraded":"2024-04-25T15:28:24.648380+0000","last_fullsized":"2024-04-25T15:28:24.648380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:54.453734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647936+0000","last_change":"2024-04-25T15:28:23.654484+0000","last_active":"2024-04-25T15:28:24.647936+0000","last_peered":"2024-04-25T15:28:24.647936+0000","last_clean":"2024-04-25T15:28:24.647936+0000","last_became_active":"2024-04-25T15:28:23.654191+0000","last_became_peered":"2024-04-25T15:28:23.654191+0000","last_unstale":"2024-04-25T15:28:24.647936+0000","last_undegraded":"2024-04-25T15:28:24.647936+0000","last_fullsized":"2024-04-25T15:28:24.647936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:51.690159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.643192+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.642727+0000","last_became_peered":"2024-04-25T15:28:23.642727+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282773+0000","last_change":"2024-04-25T15:28:22.628837+0000","last_active":"2024-04-25T15:28:25.282773+0000","last_peered":"2024-04-25T15:28:25.282773+0000","last_clean":"2024-04-25T15:28:25.282773+0000","last_became_active":"2024-04-25T15:28:22.628231+0000","last_became_peered":"2024-04-25T15:28:22.628231+0000","last_unstale":"2024-04-25T15:28:25.282773+0000","last_undegraded":"2024-04-25T15:28:25.282773+0000","last_fullsized":"2024-04-25T15:28:25.282773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:53.446196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282653+0000","last_change":"2024-04-25T15:28:23.653258+0000","last_active":"2024-04-25T15:28:25.282653+0000","last_peered":"2024-04-25T15:28:25.282653+0000","last_clean":"2024-04-25T15:28:25.282653+0000","last_became_active":"2024-04-25T15:28:23.652929+0000","last_became_peered":"2024-04-25T15:28:23.652929+0000","last_unstale":"2024-04-25T15:28:25.282653+0000","last_undegraded":"2024-04-25T15:28:25.282653+0000","last_fullsized":"2024-04-25T15:28:25.282653+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:55.065573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.617466+0000","last_change":"2024-04-25T15:28:21.617466+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.617466+0000","last_undegraded":"2024-04-25T15:28:21.617466+0000","last_fullsized":"2024-04-25T15:28:21.617466+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282579+0000","last_change":"2024-04-25T15:28:23.647366+0000","last_active":"2024-04-25T15:28:25.282579+0000","last_peered":"2024-04-25T15:28:25.282579+0000","last_clean":"2024-04-25T15:28:25.282579+0000","last_became_active":"2024-04-25T15:28:23.647027+0000","last_became_peered":"2024-04-25T15:28:23.647027+0000","last_unstale":"2024-04-25T15:28:25.282579+0000","last_undegraded":"2024-04-25T15:28:25.282579+0000","last_fullsized":"2024-04-25T15:28:25.282579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648998+0000","last_change":"2024-04-25T15:28:22.627683+0000","last_active":"2024-04-25T15:28:24.648998+0000","last_peered":"2024-04-25T15:28:24.648998+0000","last_clean":"2024-04-25T15:28:24.648998+0000","last_became_active":"2024-04-25T15:28:22.627379+0000","last_became_peered":"2024-04-25T15:28:22.627379+0000","last_unstale":"2024-04-25T15:28:24.648998+0000","last_undegraded":"2024-04-25T15:28:24.648998+0000","last_fullsized":"2024-04-25T15:28:24.648998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:30.340419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281914+0000","last_change":"2024-04-25T15:28:23.655579+0000","last_active":"2024-04-25T15:28:25.281914+0000","last_peered":"2024-04-25T15:28:25.281914+0000","last_clean":"2024-04-25T15:28:25.281914+0000","last_became_active":"2024-04-25T15:28:23.655206+0000","last_became_peered":"2024-04-25T15:28:23.655206+0000","last_unstale":"2024-04-25T15:28:25.281914+0000","last_undegraded":"2024-04-25T15:28:25.281914+0000","last_fullsized":"2024-04-25T15:28:25.281914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:16.785360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.629301+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628962+0000","last_became_peered":"2024-04-25T15:28:22.628962+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:24.982249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648121+0000","last_change":"2024-04-25T15:28:23.654406+0000","last_active":"2024-04-25T15:28:24.648121+0000","last_peered":"2024-04-25T15:28:24.648121+0000","last_clean":"2024-04-25T15:28:24.648121+0000","last_became_active":"2024-04-25T15:28:23.652920+0000","last_became_peered":"2024-04-25T15:28:23.652920+0000","last_unstale":"2024-04-25T15:28:24.648121+0000","last_undegraded":"2024-04-25T15:28:24.648121+0000","last_fullsized":"2024-04-25T15:28:24.648121+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:31.162871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.617457+0000","last_change":"2024-04-25T15:28:21.617457+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.617457+0000","last_undegraded":"2024-04-25T15:28:21.617457+0000","last_fullsized":"2024-04-25T15:28:21.617457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.616871+0000","last_change":"2024-04-25T15:28:21.616871+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.616871+0000","last_undegraded":"2024-04-25T15:28:21.616871+0000","last_fullsized":"2024-04-25T15:28:21.616871+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281419+0000","last_change":"2024-04-25T15:28:23.639119+0000","last_active":"2024-04-25T15:28:25.281419+0000","last_peered":"2024-04-25T15:28:25.281419+0000","last_clean":"2024-04-25T15:28:25.281419+0000","last_became_active":"2024-04-25T15:28:23.638754+0000","last_became_peered":"2024-04-25T15:28:23.638754+0000","last_unstale":"2024-04-25T15:28:25.281419+0000","last_undegraded":"2024-04-25T15:28:25.281419+0000","last_fullsized":"2024-04-25T15:28:25.281419+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648777+0000","last_change":"2024-04-25T15:28:22.631835+0000","last_active":"2024-04-25T15:28:24.648777+0000","last_peered":"2024-04-25T15:28:24.648777+0000","last_clean":"2024-04-25T15:28:24.648777+0000","last_became_active":"2024-04-25T15:28:22.631539+0000","last_became_peered":"2024-04-25T15:28:22.631539+0000","last_unstale":"2024-04-25T15:28:24.648777+0000","last_undegraded":"2024-04-25T15:28:24.648777+0000","last_fullsized":"2024-04-25T15:28:24.648777+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:18.322027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281369+0000","last_change":"2024-04-25T15:28:23.638974+0000","last_active":"2024-04-25T15:28:25.281369+0000","last_peered":"2024-04-25T15:28:25.281369+0000","last_clean":"2024-04-25T15:28:25.281369+0000","last_became_active":"2024-04-25T15:28:23.638642+0000","last_became_peered":"2024-04-25T15:28:23.638642+0000","last_unstale":"2024-04-25T15:28:25.281369+0000","last_undegraded":"2024-04-25T15:28:25.281369+0000","last_fullsized":"2024-04-25T15:28:25.281369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:06.035044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647693+0000","last_change":"2024-04-25T15:28:22.634256+0000","last_active":"2024-04-25T15:28:24.647693+0000","last_peered":"2024-04-25T15:28:24.647693+0000","last_clean":"2024-04-25T15:28:24.647693+0000","last_became_active":"2024-04-25T15:28:22.633942+0000","last_became_peered":"2024-04-25T15:28:22.633942+0000","last_unstale":"2024-04-25T15:28:24.647693+0000","last_undegraded":"2024-04-25T15:28:24.647693+0000","last_fullsized":"2024-04-25T15:28:24.647693+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:10.420589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282150+0000","last_change":"2024-04-25T15:28:23.639749+0000","last_active":"2024-04-25T15:28:25.282150+0000","last_peered":"2024-04-25T15:28:25.282150+0000","last_clean":"2024-04-25T15:28:25.282150+0000","last_became_active":"2024-04-25T15:28:23.639415+0000","last_became_peered":"2024-04-25T15:28:23.639415+0000","last_unstale":"2024-04-25T15:28:25.282150+0000","last_undegraded":"2024-04-25T15:28:25.282150+0000","last_fullsized":"2024-04-25T15:28:25.282150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:38:27.203252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282928+0000","last_change":"2024-04-25T15:28:22.628552+0000","last_active":"2024-04-25T15:28:25.282928+0000","last_peered":"2024-04-25T15:28:25.282928+0000","last_clean":"2024-04-25T15:28:25.282928+0000","last_became_active":"2024-04-25T15:28:22.627193+0000","last_became_peered":"2024-04-25T15:28:22.627193+0000","last_unstale":"2024-04-25T15:28:25.282928+0000","last_undegraded":"2024-04-25T15:28:25.282928+0000","last_fullsized":"2024-04-25T15:28:25.282928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282633+0000","last_change":"2024-04-25T15:28:23.651525+0000","last_active":"2024-04-25T15:28:25.282633+0000","last_peered":"2024-04-25T15:28:25.282633+0000","last_clean":"2024-04-25T15:28:25.282633+0000","last_became_active":"2024-04-25T15:28:23.651179+0000","last_became_peered":"2024-04-25T15:28:23.651179+0000","last_unstale":"2024-04-25T15:28:25.282633+0000","last_undegraded":"2024-04-25T15:28:25.282633+0000","last_fullsized":"2024-04-25T15:28:25.282633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282316+0000","last_change":"2024-04-25T15:28:22.631506+0000","last_active":"2024-04-25T15:28:25.282316+0000","last_peered":"2024-04-25T15:28:25.282316+0000","last_clean":"2024-04-25T15:28:25.282316+0000","last_became_active":"2024-04-25T15:28:22.631161+0000","last_became_peered":"2024-04-25T15:28:22.631161+0000","last_unstale":"2024-04-25T15:28:25.282316+0000","last_undegraded":"2024-04-25T15:28:25.282316+0000","last_fullsized":"2024-04-25T15:28:25.282316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.414479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649203+0000","last_change":"2024-04-25T15:28:23.636915+0000","last_active":"2024-04-25T15:28:24.649203+0000","last_peered":"2024-04-25T15:28:24.649203+0000","last_clean":"2024-04-25T15:28:24.649203+0000","last_became_active":"2024-04-25T15:28:23.636599+0000","last_became_peered":"2024-04-25T15:28:23.636599+0000","last_unstale":"2024-04-25T15:28:24.649203+0000","last_undegraded":"2024-04-25T15:28:24.649203+0000","last_fullsized":"2024-04-25T15:28:24.649203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281907+0000","last_change":"2024-04-25T15:28:22.628373+0000","last_active":"2024-04-25T15:28:25.281907+0000","last_peered":"2024-04-25T15:28:25.281907+0000","last_clean":"2024-04-25T15:28:25.281907+0000","last_became_active":"2024-04-25T15:28:22.628063+0000","last_became_peered":"2024-04-25T15:28:22.628063+0000","last_unstale":"2024-04-25T15:28:25.281907+0000","last_undegraded":"2024-04-25T15:28:25.281907+0000","last_fullsized":"2024-04-25T15:28:25.281907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:17.513548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282803+0000","last_change":"2024-04-25T15:28:22.627717+0000","last_active":"2024-04-25T15:28:25.282803+0000","last_peered":"2024-04-25T15:28:25.282803+0000","last_clean":"2024-04-25T15:28:25.282803+0000","last_became_active":"2024-04-25T15:28:22.627402+0000","last_became_peered":"2024-04-25T15:28:22.627402+0000","last_unstale":"2024-04-25T15:28:25.282803+0000","last_undegraded":"2024-04-25T15:28:25.282803+0000","last_fullsized":"2024-04-25T15:28:25.282803+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648678+0000","last_change":"2024-04-25T15:28:23.630065+0000","last_active":"2024-04-25T15:28:24.648678+0000","last_peered":"2024-04-25T15:28:24.648678+0000","last_clean":"2024-04-25T15:28:24.648678+0000","last_became_active":"2024-04-25T15:28:23.629649+0000","last_became_peered":"2024-04-25T15:28:23.629649+0000","last_unstale":"2024-04-25T15:28:24.648678+0000","last_undegraded":"2024-04-25T15:28:24.648678+0000","last_fullsized":"2024-04-25T15:28:24.648678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648438+0000","last_change":"2024-04-25T15:28:22.632102+0000","last_active":"2024-04-25T15:28:24.648438+0000","last_peered":"2024-04-25T15:28:24.648438+0000","last_clean":"2024-04-25T15:28:24.648438+0000","last_became_active":"2024-04-25T15:28:22.631723+0000","last_became_peered":"2024-04-25T15:28:22.631723+0000","last_unstale":"2024-04-25T15:28:24.648438+0000","last_undegraded":"2024-04-25T15:28:24.648438+0000","last_fullsized":"2024-04-25T15:28:24.648438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:06.295529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282145+0000","last_change":"2024-04-25T15:28:23.639139+0000","last_active":"2024-04-25T15:28:25.282145+0000","last_peered":"2024-04-25T15:28:25.282145+0000","last_clean":"2024-04-25T15:28:25.282145+0000","last_became_active":"2024-04-25T15:28:23.638812+0000","last_became_peered":"2024-04-25T15:28:23.638812+0000","last_unstale":"2024-04-25T15:28:25.282145+0000","last_undegraded":"2024-04-25T15:28:25.282145+0000","last_fullsized":"2024-04-25T15:28:25.282145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:56.852295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281527+0000","last_change":"2024-04-25T15:28:22.631778+0000","last_active":"2024-04-25T15:28:25.281527+0000","last_peered":"2024-04-25T15:28:25.281527+0000","last_clean":"2024-04-25T15:28:25.281527+0000","last_became_active":"2024-04-25T15:28:22.631460+0000","last_became_peered":"2024-04-25T15:28:22.631460+0000","last_unstale":"2024-04-25T15:28:25.281527+0000","last_undegraded":"2024-04-25T15:28:25.281527+0000","last_fullsized":"2024-04-25T15:28:25.281527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.777821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647614+0000","last_change":"2024-04-25T15:28:23.652363+0000","last_active":"2024-04-25T15:28:24.647614+0000","last_peered":"2024-04-25T15:28:24.647614+0000","last_clean":"2024-04-25T15:28:24.647614+0000","last_became_active":"2024-04-25T15:28:23.651744+0000","last_became_peered":"2024-04-25T15:28:23.651744+0000","last_unstale":"2024-04-25T15:28:24.647614+0000","last_undegraded":"2024-04-25T15:28:24.647614+0000","last_fullsized":"2024-04-25T15:28:24.647614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:17.116922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281587+0000","last_change":"2024-04-25T15:28:22.636713+0000","last_active":"2024-04-25T15:28:25.281587+0000","last_peered":"2024-04-25T15:28:25.281587+0000","last_clean":"2024-04-25T15:28:25.281587+0000","last_became_active":"2024-04-25T15:28:22.636389+0000","last_became_peered":"2024-04-25T15:28:22.636389+0000","last_unstale":"2024-04-25T15:28:25.281587+0000","last_undegraded":"2024-04-25T15:28:25.281587+0000","last_fullsized":"2024-04-25T15:28:25.281587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647465+0000","last_change":"2024-04-25T15:28:23.647463+0000","last_active":"2024-04-25T15:28:24.647465+0000","last_peered":"2024-04-25T15:28:24.647465+0000","last_clean":"2024-04-25T15:28:24.647465+0000","last_became_active":"2024-04-25T15:28:23.647133+0000","last_became_peered":"2024-04-25T15:28:23.647133+0000","last_unstale":"2024-04-25T15:28:24.647465+0000","last_undegraded":"2024-04-25T15:28:24.647465+0000","last_fullsized":"2024-04-25T15:28:24.647465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:14.765406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.628461+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628160+0000","last_became_peered":"2024-04-25T15:28:22.628160+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282411+0000","last_change":"2024-04-25T15:28:23.649571+0000","last_active":"2024-04-25T15:28:25.282411+0000","last_peered":"2024-04-25T15:28:25.282411+0000","last_clean":"2024-04-25T15:28:25.282411+0000","last_became_active":"2024-04-25T15:28:23.649214+0000","last_became_peered":"2024-04-25T15:28:23.649214+0000","last_unstale":"2024-04-25T15:28:25.282411+0000","last_undegraded":"2024-04-25T15:28:25.282411+0000","last_fullsized":"2024-04-25T15:28:25.282411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:22.598027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283048+0000","last_change":"2024-04-25T15:28:22.631820+0000","last_active":"2024-04-25T15:28:25.283048+0000","last_peered":"2024-04-25T15:28:25.283048+0000","last_clean":"2024-04-25T15:28:25.283048+0000","last_became_active":"2024-04-25T15:28:22.631501+0000","last_became_peered":"2024-04-25T15:28:22.631501+0000","last_unstale":"2024-04-25T15:28:25.283048+0000","last_undegraded":"2024-04-25T15:28:25.283048+0000","last_fullsized":"2024-04-25T15:28:25.283048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282947+0000","last_change":"2024-04-25T15:28:23.646728+0000","last_active":"2024-04-25T15:28:25.282947+0000","last_peered":"2024-04-25T15:28:25.282947+0000","last_clean":"2024-04-25T15:28:25.282947+0000","last_became_active":"2024-04-25T15:28:23.646234+0000","last_became_peered":"2024-04-25T15:28:23.646234+0000","last_unstale":"2024-04-25T15:28:25.282947+0000","last_undegraded":"2024-04-25T15:28:25.282947+0000","last_fullsized":"2024-04-25T15:28:25.282947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648950+0000","last_change":"2024-04-25T15:28:22.632918+0000","last_active":"2024-04-25T15:28:24.648950+0000","last_peered":"2024-04-25T15:28:24.648950+0000","last_clean":"2024-04-25T15:28:24.648950+0000","last_became_active":"2024-04-25T15:28:22.632551+0000","last_became_peered":"2024-04-25T15:28:22.632551+0000","last_unstale":"2024-04-25T15:28:24.648950+0000","last_undegraded":"2024-04-25T15:28:24.648950+0000","last_fullsized":"2024-04-25T15:28:24.648950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:25.998670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649243+0000","last_change":"2024-04-25T15:28:22.631396+0000","last_active":"2024-04-25T15:28:24.649243+0000","last_peered":"2024-04-25T15:28:24.649243+0000","last_clean":"2024-04-25T15:28:24.649243+0000","last_became_active":"2024-04-25T15:28:22.630989+0000","last_became_peered":"2024-04-25T15:28:22.630989+0000","last_unstale":"2024-04-25T15:28:24.649243+0000","last_undegraded":"2024-04-25T15:28:24.649243+0000","last_fullsized":"2024-04-25T15:28:24.649243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:25.648985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282440+0000","last_change":"2024-04-25T15:28:23.650548+0000","last_active":"2024-04-25T15:28:25.282440+0000","last_peered":"2024-04-25T15:28:25.282440+0000","last_clean":"2024-04-25T15:28:25.282440+0000","last_became_active":"2024-04-25T15:28:23.649636+0000","last_became_peered":"2024-04-25T15:28:23.649636+0000","last_unstale":"2024-04-25T15:28:25.282440+0000","last_undegraded":"2024-04-25T15:28:25.282440+0000","last_fullsized":"2024-04-25T15:28:25.282440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:11.428127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281645+0000","last_change":"2024-04-25T15:28:22.631144+0000","last_active":"2024-04-25T15:28:25.281645+0000","last_peered":"2024-04-25T15:28:25.281645+0000","last_clean":"2024-04-25T15:28:25.281645+0000","last_became_active":"2024-04-25T15:28:22.630762+0000","last_became_peered":"2024-04-25T15:28:22.630762+0000","last_unstale":"2024-04-25T15:28:25.281645+0000","last_undegraded":"2024-04-25T15:28:25.281645+0000","last_fullsized":"2024-04-25T15:28:25.281645+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:10.967925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283204+0000","last_change":"2024-04-25T15:28:23.647208+0000","last_active":"2024-04-25T15:28:25.283204+0000","last_peered":"2024-04-25T15:28:25.283204+0000","last_clean":"2024-04-25T15:28:25.283204+0000","last_became_active":"2024-04-25T15:28:23.646871+0000","last_became_peered":"2024-04-25T15:28:23.646871+0000","last_unstale":"2024-04-25T15:28:25.283204+0000","last_undegraded":"2024-04-25T15:28:25.283204+0000","last_fullsized":"2024-04-25T15:28:25.283204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282312+0000","last_change":"2024-04-25T15:28:22.623948+0000","last_active":"2024-04-25T15:28:25.282312+0000","last_peered":"2024-04-25T15:28:25.282312+0000","last_clean":"2024-04-25T15:28:25.282312+0000","last_became_active":"2024-04-25T15:28:22.623604+0000","last_became_peered":"2024-04-25T15:28:22.623604+0000","last_unstale":"2024-04-25T15:28:25.282312+0000","last_undegraded":"2024-04-25T15:28:25.282312+0000","last_fullsized":"2024-04-25T15:28:25.282312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:13.369418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:28:23.648910+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:28:23.648524+0000","last_became_peered":"2024-04-25T15:28:23.648524+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.853221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281681+0000","last_change":"2024-04-25T15:28:22.635528+0000","last_active":"2024-04-25T15:28:25.281681+0000","last_peered":"2024-04-25T15:28:25.281681+0000","last_clean":"2024-04-25T15:28:25.281681+0000","last_became_active":"2024-04-25T15:28:22.635320+0000","last_became_peered":"2024-04-25T15:28:22.635320+0000","last_unstale":"2024-04-25T15:28:25.281681+0000","last_undegraded":"2024-04-25T15:28:25.281681+0000","last_fullsized":"2024-04-25T15:28:25.281681+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:37.967626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281434+0000","last_change":"2024-04-25T15:28:23.658467+0000","last_active":"2024-04-25T15:28:25.281434+0000","last_peered":"2024-04-25T15:28:25.281434+0000","last_clean":"2024-04-25T15:28:25.281434+0000","last_became_active":"2024-04-25T15:28:23.658051+0000","last_became_peered":"2024-04-25T15:28:23.658051+0000","last_unstale":"2024-04-25T15:28:25.281434+0000","last_undegraded":"2024-04-25T15:28:25.281434+0000","last_fullsized":"2024-04-25T15:28:25.281434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:27.006208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281183+0000","last_change":"2024-04-25T15:28:22.627923+0000","last_active":"2024-04-25T15:28:25.281183+0000","last_peered":"2024-04-25T15:28:25.281183+0000","last_clean":"2024-04-25T15:28:25.281183+0000","last_became_active":"2024-04-25T15:28:22.627595+0000","last_became_peered":"2024-04-25T15:28:22.627595+0000","last_unstale":"2024-04-25T15:28:25.281183+0000","last_undegraded":"2024-04-25T15:28:25.281183+0000","last_fullsized":"2024-04-25T15:28:25.281183+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:04.050831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647686+0000","last_change":"2024-04-25T15:28:22.629989+0000","last_active":"2024-04-25T15:28:24.647686+0000","last_peered":"2024-04-25T15:28:24.647686+0000","last_clean":"2024-04-25T15:28:24.647686+0000","last_became_active":"2024-04-25T15:28:22.629681+0000","last_became_peered":"2024-04-25T15:28:22.629681+0000","last_unstale":"2024-04-25T15:28:24.647686+0000","last_undegraded":"2024-04-25T15:28:24.647686+0000","last_fullsized":"2024-04-25T15:28:24.647686+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:36.386426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281853+0000","last_change":"2024-04-25T15:28:23.643948+0000","last_active":"2024-04-25T15:28:25.281853+0000","last_peered":"2024-04-25T15:28:25.281853+0000","last_clean":"2024-04-25T15:28:25.281853+0000","last_became_active":"2024-04-25T15:28:23.643600+0000","last_became_peered":"2024-04-25T15:28:23.643600+0000","last_unstale":"2024-04-25T15:28:25.281853+0000","last_undegraded":"2024-04-25T15:28:25.281853+0000","last_fullsized":"2024-04-25T15:28:25.281853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648158+0000","last_change":"2024-04-25T15:28:22.630713+0000","last_active":"2024-04-25T15:28:24.648158+0000","last_peered":"2024-04-25T15:28:24.648158+0000","last_clean":"2024-04-25T15:28:24.648158+0000","last_became_active":"2024-04-25T15:28:22.627937+0000","last_became_peered":"2024-04-25T15:28:22.627937+0000","last_unstale":"2024-04-25T15:28:24.648158+0000","last_undegraded":"2024-04-25T15:28:24.648158+0000","last_fullsized":"2024-04-25T15:28:24.648158+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:34.438583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282846+0000","last_change":"2024-04-25T15:28:23.650199+0000","last_active":"2024-04-25T15:28:25.282846+0000","last_peered":"2024-04-25T15:28:25.282846+0000","last_clean":"2024-04-25T15:28:25.282846+0000","last_became_active":"2024-04-25T15:28:23.649599+0000","last_became_peered":"2024-04-25T15:28:23.649599+0000","last_unstale":"2024-04-25T15:28:25.282846+0000","last_undegraded":"2024-04-25T15:28:25.282846+0000","last_fullsized":"2024-04-25T15:28:25.282846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282154+0000","last_change":"2024-04-25T15:28:22.626743+0000","last_active":"2024-04-25T15:28:25.282154+0000","last_peered":"2024-04-25T15:28:25.282154+0000","last_clean":"2024-04-25T15:28:25.282154+0000","last_became_active":"2024-04-25T15:28:22.626434+0000","last_became_peered":"2024-04-25T15:28:22.626434+0000","last_unstale":"2024-04-25T15:28:25.282154+0000","last_undegraded":"2024-04-25T15:28:25.282154+0000","last_fullsized":"2024-04-25T15:28:25.282154+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:13.395697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282910+0000","last_change":"2024-04-25T15:28:22.630470+0000","last_active":"2024-04-25T15:28:25.282910+0000","last_peered":"2024-04-25T15:28:25.282910+0000","last_clean":"2024-04-25T15:28:25.282910+0000","last_became_active":"2024-04-25T15:28:22.630069+0000","last_became_peered":"2024-04-25T15:28:22.630069+0000","last_unstale":"2024-04-25T15:28:25.282910+0000","last_undegraded":"2024-04-25T15:28:25.282910+0000","last_fullsized":"2024-04-25T15:28:25.282910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:30.155333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282356+0000","last_change":"2024-04-25T15:28:23.648809+0000","last_active":"2024-04-25T15:28:25.282356+0000","last_peered":"2024-04-25T15:28:25.282356+0000","last_clean":"2024-04-25T15:28:25.282356+0000","last_became_active":"2024-04-25T15:28:23.648507+0000","last_became_peered":"2024-04-25T15:28:23.648507+0000","last_unstale":"2024-04-25T15:28:25.282356+0000","last_undegraded":"2024-04-25T15:28:25.282356+0000","last_fullsized":"2024-04-25T15:28:25.282356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648878+0000","last_change":"2024-04-25T15:28:22.628170+0000","last_active":"2024-04-25T15:28:24.648878+0000","last_peered":"2024-04-25T15:28:24.648878+0000","last_clean":"2024-04-25T15:28:24.648878+0000","last_became_active":"2024-04-25T15:28:22.627191+0000","last_became_peered":"2024-04-25T15:28:22.627191+0000","last_unstale":"2024-04-25T15:28:24.648878+0000","last_undegraded":"2024-04-25T15:28:24.648878+0000","last_fullsized":"2024-04-25T15:28:24.648878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:31.302274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648932+0000","last_change":"2024-04-25T15:28:23.630876+0000","last_active":"2024-04-25T15:28:24.648932+0000","last_peered":"2024-04-25T15:28:24.648932+0000","last_clean":"2024-04-25T15:28:24.648932+0000","last_became_active":"2024-04-25T15:28:23.630558+0000","last_became_peered":"2024-04-25T15:28:23.630558+0000","last_unstale":"2024-04-25T15:28:24.648932+0000","last_undegraded":"2024-04-25T15:28:24.648932+0000","last_fullsized":"2024-04-25T15:28:24.648932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.616307+0000","last_change":"2024-04-25T15:28:21.616307+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.616307+0000","last_undegraded":"2024-04-25T15:28:21.616307+0000","last_fullsized":"2024-04-25T15:28:21.616307+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281519+0000","last_change":"2024-04-25T15:28:23.640269+0000","last_active":"2024-04-25T15:28:25.281519+0000","last_peered":"2024-04-25T15:28:25.281519+0000","last_clean":"2024-04-25T15:28:25.281519+0000","last_became_active":"2024-04-25T15:28:23.639944+0000","last_became_peered":"2024-04-25T15:28:23.639944+0000","last_unstale":"2024-04-25T15:28:25.281519+0000","last_undegraded":"2024-04-25T15:28:25.281519+0000","last_fullsized":"2024-04-25T15:28:25.281519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282853+0000","last_change":"2024-04-25T15:28:22.626538+0000","last_active":"2024-04-25T15:28:25.282853+0000","last_peered":"2024-04-25T15:28:25.282853+0000","last_clean":"2024-04-25T15:28:25.282853+0000","last_became_active":"2024-04-25T15:28:22.626207+0000","last_became_peered":"2024-04-25T15:28:22.626207+0000","last_unstale":"2024-04-25T15:28:25.282853+0000","last_undegraded":"2024-04-25T15:28:25.282853+0000","last_fullsized":"2024-04-25T15:28:25.282853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:50.682621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282764+0000","last_change":"2024-04-25T15:28:23.646030+0000","last_active":"2024-04-25T15:28:25.282764+0000","last_peered":"2024-04-25T15:28:25.282764+0000","last_clean":"2024-04-25T15:28:25.282764+0000","last_became_active":"2024-04-25T15:28:23.645733+0000","last_became_peered":"2024-04-25T15:28:23.645733+0000","last_unstale":"2024-04-25T15:28:25.282764+0000","last_undegraded":"2024-04-25T15:28:25.282764+0000","last_fullsized":"2024-04-25T15:28:25.282764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.615544+0000","last_change":"2024-04-25T15:28:21.615544+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.615544+0000","last_undegraded":"2024-04-25T15:28:21.615544+0000","last_fullsized":"2024-04-25T15:28:21.615544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648545+0000","last_change":"2024-04-25T15:28:23.640057+0000","last_active":"2024-04-25T15:28:24.648545+0000","last_peered":"2024-04-25T15:28:24.648545+0000","last_clean":"2024-04-25T15:28:24.648545+0000","last_became_active":"2024-04-25T15:28:23.639747+0000","last_became_peered":"2024-04-25T15:28:23.639747+0000","last_unstale":"2024-04-25T15:28:24.648545+0000","last_undegraded":"2024-04-25T15:28:24.648545+0000","last_fullsized":"2024-04-25T15:28:24.648545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:06.046345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281905+0000","last_change":"2024-04-25T15:28:22.632108+0000","last_active":"2024-04-25T15:28:25.281905+0000","last_peered":"2024-04-25T15:28:25.281905+0000","last_clean":"2024-04-25T15:28:25.281905+0000","last_became_active":"2024-04-25T15:28:22.631768+0000","last_became_peered":"2024-04-25T15:28:22.631768+0000","last_unstale":"2024-04-25T15:28:25.281905+0000","last_undegraded":"2024-04-25T15:28:25.281905+0000","last_fullsized":"2024-04-25T15:28:25.281905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":218,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1581056,"data_stored":1007311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":418507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":397523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":418485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":417455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1581056,"data_stored":1007275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:28:28.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:28:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.221+0000 7f46c5c30640 1 Processor -- start 2024-04-25T15:28:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.221+0000 7f46c5c30640 1 -- start start 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.222+0000 7f46c5c30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c0164b00 0x7f46c00b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.222+0000 7f46c5c30640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f46c0160fa0 0x7f46c00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.222+0000 7f46c5c30640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c01615f0 con 0x7f46c015df80 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.222+0000 7f46c5c30640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f46c01613b0 con 0x7f46c0160fa0 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.222+0000 7f46c5c30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f46c0164670 con 0x7f46c0164b00 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.222+0000 7f46bf7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c0164b00 0x7f46c00b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46bffff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f46c0160fa0 0x7f46c00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46beffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f46c015df80 0x7f46c0160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:52860/0 (socket says 172.21.15.67:52860) 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46beffd640 1 -- 172.21.15.67:0/2975459093 learned_addr learned my addr 172.21.15.67:0/2975459093 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46bf7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c0164b00 0x7f46c00b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:58890/0 (socket says 172.21.15.67:58890) 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46bf7fe640 1 -- 172.21.15.67:0/2975459093 >> v2:172.21.15.176:3300/0 conn(0x7f46c0160fa0 msgr2=0x7f46c00b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46bf7fe640 1 --2- 172.21.15.67:0/2975459093 >> v2:172.21.15.176:3300/0 conn(0x7f46c0160fa0 0x7f46c00b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46bf7fe640 1 -- 172.21.15.67:0/2975459093 >> v1:172.21.15.176:6789/0 conn(0x7f46c015df80 legacy=0x7f46c0160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46bf7fe640 1 -- 172.21.15.67:0/2975459093 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c0164020 con 0x7f46c0164b00 2024-04-25T15:28:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.223+0000 7f46bffff640 1 --2- 172.21.15.67:0/2975459093 >> v2:172.21.15.176:3300/0 conn(0x7f46c0160fa0 0x7f46c00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.224+0000 7f46bf7fe640 1 --2- 172.21.15.67:0/2975459093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c0164b00 0x7f46c00b7b80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f46a80097a0 tx=0x7f46a802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aaf218194c6ba05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.224+0000 7f46be7fc640 1 -- 172.21.15.67:0/2975459093 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8037070 con 0x7f46c0164b00 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.224+0000 7f46be7fc640 1 -- 172.21.15.67:0/2975459093 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a8032440 con 0x7f46c0164b00 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.224+0000 7f46be7fc640 1 -- 172.21.15.67:0/2975459093 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a8032810 con 0x7f46c0164b00 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 -- 172.21.15.67:0/2975459093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c0164b00 msgr2=0x7f46c00b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 --2- 172.21.15.67:0/2975459093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c0164b00 0x7f46c00b7b80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f46a80097a0 tx=0x7f46a802a180 comp rx=0 tx=0).stop 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 -- 172.21.15.67:0/2975459093 shutdown_connections 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 --2- 172.21.15.67:0/2975459093 >> v2:172.21.15.176:3300/0 conn(0x7f46c0160fa0 0x7f46c00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 --2- 172.21.15.67:0/2975459093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c0164b00 0x7f46c00b7b80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 -- 172.21.15.67:0/2975459093 >> 172.21.15.67:0/2975459093 conn(0x7f46c00d2bc0 msgr2=0x7f46c00d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 -- 172.21.15.67:0/2975459093 shutdown_connections 2024-04-25T15:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.225+0000 7f46c5c30640 1 -- 172.21.15.67:0/2975459093 wait complete. 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.226+0000 7f46c5c30640 1 Processor -- start 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.227+0000 7f46c5c30640 1 -- start start 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46c5c30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c015df80 0x7f46c01f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46c5c30640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46c0160fa0 0x7f46c0144a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46c5c30640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c0164b00 0x7f46c0144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46c5c30640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f46c00b4a00 con 0x7f46c0164b00 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46c5c30640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f46c00b4880 con 0x7f46c0160fa0 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46c5c30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f46c00b4b80 con 0x7f46c015df80 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c0164b00 0x7f46c0144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c0164b00 0x7f46c0144fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56970/0 (socket says 172.21.15.67:56970) 2024-04-25T15:28:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 -- 172.21.15.67:0/810869152 learned_addr learned my addr 172.21.15.67:0/810869152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bf7fe640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c015df80 0x7f46c01f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46beffd640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46c0160fa0 0x7f46c0144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 -- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c015df80 msgr2=0x7f46c01f6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c015df80 0x7f46c01f6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 -- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46c0160fa0 msgr2=0x7f46c0144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46c0160fa0 0x7f46c0144a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.228+0000 7f46bffff640 1 -- 172.21.15.67:0/810869152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c015b810 con 0x7f46c0164b00 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.229+0000 7f46bffff640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c0164b00 0x7f46c0144fa0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009e00 tx=0x7f46b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.229+0000 7f46b7fff640 1 -- 172.21.15.67:0/810869152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8018020 con 0x7f46c0164b00 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.229+0000 7f46b7fff640 1 -- 172.21.15.67:0/810869152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b8014480 con 0x7f46c0164b00 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.229+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c015b530 con 0x7f46c0164b00 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.229+0000 7f46b7fff640 1 -- 172.21.15.67:0/810869152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8014870 con 0x7f46c0164b00 2024-04-25T15:28:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.229+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c0157eb0 con 0x7f46c0164b00 2024-04-25T15:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.230+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c0050740 con 0x7f46c0164b00 2024-04-25T15:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.231+0000 7f46b7fff640 1 -- 172.21.15.67:0/810869152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f46b8005d10 con 0x7f46c0164b00 2024-04-25T15:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.231+0000 7f46b7fff640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f46a00744b0 0x7f46a00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.231+0000 7f46b7fff640 1 -- 172.21.15.67:0/810869152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f46b8095db0 con 0x7f46c0164b00 2024-04-25T15:28:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.231+0000 7f46bf7fe640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f46a00744b0 0x7f46a00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.232+0000 7f46bf7fe640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f46a00744b0 0x7f46a00769c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f46a8034000 tx=0x7f46a802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.236+0000 7f46b7fff640 1 -- 172.21.15.67:0/810869152 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f46b8022030 con 0x7f46c0164b00 2024-04-25T15:28:28.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.418+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f46c00d4170 con 0x7f46a00744b0 2024-04-25T15:28:28.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.424+0000 7f46b7fff640 1 -- 172.21.15.67:0/810869152 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250157 (secure 0 0 0) 0x7f46c00d4170 con 0x7f46a00744b0 2024-04-25T15:28:28.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.427+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f46a00744b0 msgr2=0x7f46a00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.427+0000 7f46c5c30640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f46a00744b0 0x7f46a00769c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f46a8034000 tx=0x7f46a802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.427+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c0164b00 msgr2=0x7f46c0144fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.427+0000 7f46c5c30640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c0164b00 0x7f46c0144fa0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009e00 tx=0x7f46b800c490 comp rx=0 tx=0).stop 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 shutdown_connections 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f46a00744b0 0x7f46a00769c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46c0164b00 0x7f46c0144fa0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46c0160fa0 0x7f46c0144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 --2- 172.21.15.67:0/810869152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c015df80 0x7f46c01f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 >> 172.21.15.67:0/810869152 conn(0x7f46c00d2bc0 msgr2=0x7f46c00d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 shutdown_connections 2024-04-25T15:28:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.428+0000 7f46c5c30640 1 -- 172.21.15.67:0/810869152 wait complete. 2024-04-25T15:28:28.440 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-04-25T15:28:26.838930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233380,"kb_used_data":8868,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754741340,"statfs":{"total":773094113280,"available":772855132160,"internally_reserved":0,"allocated":9080832,"data_stored":4501190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.224231"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282682+0000","last_change":"2024-04-25T15:28:23.652205+0000","last_active":"2024-04-25T15:28:25.282682+0000","last_peered":"2024-04-25T15:28:25.282682+0000","last_clean":"2024-04-25T15:28:25.282682+0000","last_became_active":"2024-04-25T15:28:23.651726+0000","last_became_peered":"2024-04-25T15:28:23.651726+0000","last_unstale":"2024-04-25T15:28:25.282682+0000","last_undegraded":"2024-04-25T15:28:25.282682+0000","last_fullsized":"2024-04-25T15:28:25.282682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:46.911832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648103+0000","last_change":"2024-04-25T15:28:23.649010+0000","last_active":"2024-04-25T15:28:24.648103+0000","last_peered":"2024-04-25T15:28:24.648103+0000","last_clean":"2024-04-25T15:28:24.648103+0000","last_became_active":"2024-04-25T15:28:23.648637+0000","last_became_peered":"2024-04-25T15:28:23.648637+0000","last_unstale":"2024-04-25T15:28:24.648103+0000","last_undegraded":"2024-04-25T15:28:24.648103+0000","last_fullsized":"2024-04-25T15:28:24.648103+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:28.422017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281638+0000","last_change":"2024-04-25T15:28:23.643910+0000","last_active":"2024-04-25T15:28:25.281638+0000","last_peered":"2024-04-25T15:28:25.281638+0000","last_clean":"2024-04-25T15:28:25.281638+0000","last_became_active":"2024-04-25T15:28:23.643580+0000","last_became_peered":"2024-04-25T15:28:23.643580+0000","last_unstale":"2024-04-25T15:28:25.281638+0000","last_undegraded":"2024-04-25T15:28:25.281638+0000","last_fullsized":"2024-04-25T15:28:25.281638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281992+0000","last_change":"2024-04-25T15:28:23.649272+0000","last_active":"2024-04-25T15:28:25.281992+0000","last_peered":"2024-04-25T15:28:25.281992+0000","last_clean":"2024-04-25T15:28:25.281992+0000","last_became_active":"2024-04-25T15:28:23.647269+0000","last_became_peered":"2024-04-25T15:28:23.647269+0000","last_unstale":"2024-04-25T15:28:25.281992+0000","last_undegraded":"2024-04-25T15:28:25.281992+0000","last_fullsized":"2024-04-25T15:28:25.281992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:27.187638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281710+0000","last_change":"2024-04-25T15:28:23.652480+0000","last_active":"2024-04-25T15:28:25.281710+0000","last_peered":"2024-04-25T15:28:25.281710+0000","last_clean":"2024-04-25T15:28:25.281710+0000","last_became_active":"2024-04-25T15:28:23.652042+0000","last_became_peered":"2024-04-25T15:28:23.652042+0000","last_unstale":"2024-04-25T15:28:25.281710+0000","last_undegraded":"2024-04-25T15:28:25.281710+0000","last_fullsized":"2024-04-25T15:28:25.281710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:26.656523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281499+0000","last_change":"2024-04-25T15:28:23.645012+0000","last_active":"2024-04-25T15:28:25.281499+0000","last_peered":"2024-04-25T15:28:25.281499+0000","last_clean":"2024-04-25T15:28:25.281499+0000","last_became_active":"2024-04-25T15:28:23.644620+0000","last_became_peered":"2024-04-25T15:28:23.644620+0000","last_unstale":"2024-04-25T15:28:25.281499+0000","last_undegraded":"2024-04-25T15:28:25.281499+0000","last_fullsized":"2024-04-25T15:28:25.281499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:25.372903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281525+0000","last_change":"2024-04-25T15:28:23.645167+0000","last_active":"2024-04-25T15:28:25.281525+0000","last_peered":"2024-04-25T15:28:25.281525+0000","last_clean":"2024-04-25T15:28:25.281525+0000","last_became_active":"2024-04-25T15:28:23.644781+0000","last_became_peered":"2024-04-25T15:28:23.644781+0000","last_unstale":"2024-04-25T15:28:25.281525+0000","last_undegraded":"2024-04-25T15:28:25.281525+0000","last_fullsized":"2024-04-25T15:28:25.281525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:25.989787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282012+0000","last_change":"2024-04-25T15:28:23.658433+0000","last_active":"2024-04-25T15:28:25.282012+0000","last_peered":"2024-04-25T15:28:25.282012+0000","last_clean":"2024-04-25T15:28:25.282012+0000","last_became_active":"2024-04-25T15:28:23.658026+0000","last_became_peered":"2024-04-25T15:28:23.658026+0000","last_unstale":"2024-04-25T15:28:25.282012+0000","last_undegraded":"2024-04-25T15:28:25.282012+0000","last_fullsized":"2024-04-25T15:28:25.282012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282414+0000","last_change":"2024-04-25T15:28:23.643202+0000","last_active":"2024-04-25T15:28:25.282414+0000","last_peered":"2024-04-25T15:28:25.282414+0000","last_clean":"2024-04-25T15:28:25.282414+0000","last_became_active":"2024-04-25T15:28:23.642274+0000","last_became_peered":"2024-04-25T15:28:23.642274+0000","last_unstale":"2024-04-25T15:28:25.282414+0000","last_undegraded":"2024-04-25T15:28:25.282414+0000","last_fullsized":"2024-04-25T15:28:25.282414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.639963+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.639463+0000","last_became_peered":"2024-04-25T15:28:23.639463+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:18.521086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282830+0000","last_change":"2024-04-25T15:28:23.649937+0000","last_active":"2024-04-25T15:28:25.282830+0000","last_peered":"2024-04-25T15:28:25.282830+0000","last_clean":"2024-04-25T15:28:25.282830+0000","last_became_active":"2024-04-25T15:28:23.649189+0000","last_became_peered":"2024-04-25T15:28:23.649189+0000","last_unstale":"2024-04-25T15:28:25.282830+0000","last_undegraded":"2024-04-25T15:28:25.282830+0000","last_fullsized":"2024-04-25T15:28:25.282830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281903+0000","last_change":"2024-04-25T15:28:23.644231+0000","last_active":"2024-04-25T15:28:25.281903+0000","last_peered":"2024-04-25T15:28:25.281903+0000","last_clean":"2024-04-25T15:28:25.281903+0000","last_became_active":"2024-04-25T15:28:23.643912+0000","last_became_peered":"2024-04-25T15:28:23.643912+0000","last_unstale":"2024-04-25T15:28:25.281903+0000","last_undegraded":"2024-04-25T15:28:25.281903+0000","last_fullsized":"2024-04-25T15:28:25.281903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:11.975463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282212+0000","last_change":"2024-04-25T15:28:23.650840+0000","last_active":"2024-04-25T15:28:25.282212+0000","last_peered":"2024-04-25T15:28:25.282212+0000","last_clean":"2024-04-25T15:28:25.282212+0000","last_became_active":"2024-04-25T15:28:23.650019+0000","last_became_peered":"2024-04-25T15:28:23.650019+0000","last_unstale":"2024-04-25T15:28:25.282212+0000","last_undegraded":"2024-04-25T15:28:25.282212+0000","last_fullsized":"2024-04-25T15:28:25.282212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:31.696277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.639188+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.638896+0000","last_became_peered":"2024-04-25T15:28:23.638896+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:12.900679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282156+0000","last_change":"2024-04-25T15:28:23.648256+0000","last_active":"2024-04-25T15:28:25.282156+0000","last_peered":"2024-04-25T15:28:25.282156+0000","last_clean":"2024-04-25T15:28:25.282156+0000","last_became_active":"2024-04-25T15:28:23.647895+0000","last_became_peered":"2024-04-25T15:28:23.647895+0000","last_unstale":"2024-04-25T15:28:25.282156+0000","last_undegraded":"2024-04-25T15:28:25.282156+0000","last_fullsized":"2024-04-25T15:28:25.282156+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:31.405763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648935+0000","last_change":"2024-04-25T15:28:23.635903+0000","last_active":"2024-04-25T15:28:24.648935+0000","last_peered":"2024-04-25T15:28:24.648935+0000","last_clean":"2024-04-25T15:28:24.648935+0000","last_became_active":"2024-04-25T15:28:23.635506+0000","last_became_peered":"2024-04-25T15:28:23.635506+0000","last_unstale":"2024-04-25T15:28:24.648935+0000","last_undegraded":"2024-04-25T15:28:24.648935+0000","last_fullsized":"2024-04-25T15:28:24.648935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282313+0000","last_change":"2024-04-25T15:28:23.655054+0000","last_active":"2024-04-25T15:28:25.282313+0000","last_peered":"2024-04-25T15:28:25.282313+0000","last_clean":"2024-04-25T15:28:25.282313+0000","last_became_active":"2024-04-25T15:28:23.654675+0000","last_became_peered":"2024-04-25T15:28:23.654675+0000","last_unstale":"2024-04-25T15:28:25.282313+0000","last_undegraded":"2024-04-25T15:28:25.282313+0000","last_fullsized":"2024-04-25T15:28:25.282313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:29.272131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282209+0000","last_change":"2024-04-25T15:28:23.654074+0000","last_active":"2024-04-25T15:28:25.282209+0000","last_peered":"2024-04-25T15:28:25.282209+0000","last_clean":"2024-04-25T15:28:25.282209+0000","last_became_active":"2024-04-25T15:28:23.653761+0000","last_became_peered":"2024-04-25T15:28:23.653761+0000","last_unstale":"2024-04-25T15:28:25.282209+0000","last_undegraded":"2024-04-25T15:28:25.282209+0000","last_fullsized":"2024-04-25T15:28:25.282209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:28.122216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.637208+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.636801+0000","last_became_peered":"2024-04-25T15:28:23.636801+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:14.403235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282697+0000","last_change":"2024-04-25T15:28:23.646595+0000","last_active":"2024-04-25T15:28:25.282697+0000","last_peered":"2024-04-25T15:28:25.282697+0000","last_clean":"2024-04-25T15:28:25.282697+0000","last_became_active":"2024-04-25T15:28:23.646279+0000","last_became_peered":"2024-04-25T15:28:23.646279+0000","last_unstale":"2024-04-25T15:28:25.282697+0000","last_undegraded":"2024-04-25T15:28:25.282697+0000","last_fullsized":"2024-04-25T15:28:25.282697+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:28.717546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281566+0000","last_change":"2024-04-25T15:28:23.655097+0000","last_active":"2024-04-25T15:28:25.281566+0000","last_peered":"2024-04-25T15:28:25.281566+0000","last_clean":"2024-04-25T15:28:25.281566+0000","last_became_active":"2024-04-25T15:28:23.654517+0000","last_became_peered":"2024-04-25T15:28:23.654517+0000","last_unstale":"2024-04-25T15:28:25.281566+0000","last_undegraded":"2024-04-25T15:28:25.281566+0000","last_fullsized":"2024-04-25T15:28:25.281566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:43:00.233658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647616+0000","last_change":"2024-04-25T15:28:23.648614+0000","last_active":"2024-04-25T15:28:24.647616+0000","last_peered":"2024-04-25T15:28:24.647616+0000","last_clean":"2024-04-25T15:28:24.647616+0000","last_became_active":"2024-04-25T15:28:23.648286+0000","last_became_peered":"2024-04-25T15:28:23.648286+0000","last_unstale":"2024-04-25T15:28:24.647616+0000","last_undegraded":"2024-04-25T15:28:24.647616+0000","last_fullsized":"2024-04-25T15:28:24.647616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:09.027121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282242+0000","last_change":"2024-04-25T15:28:23.639343+0000","last_active":"2024-04-25T15:28:25.282242+0000","last_peered":"2024-04-25T15:28:25.282242+0000","last_clean":"2024-04-25T15:28:25.282242+0000","last_became_active":"2024-04-25T15:28:23.639037+0000","last_became_peered":"2024-04-25T15:28:23.639037+0000","last_unstale":"2024-04-25T15:28:25.282242+0000","last_undegraded":"2024-04-25T15:28:25.282242+0000","last_fullsized":"2024-04-25T15:28:25.282242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:41.056274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282367+0000","last_change":"2024-04-25T15:28:23.647159+0000","last_active":"2024-04-25T15:28:25.282367+0000","last_peered":"2024-04-25T15:28:25.282367+0000","last_clean":"2024-04-25T15:28:25.282367+0000","last_became_active":"2024-04-25T15:28:23.646103+0000","last_became_peered":"2024-04-25T15:28:23.646103+0000","last_unstale":"2024-04-25T15:28:25.282367+0000","last_undegraded":"2024-04-25T15:28:25.282367+0000","last_fullsized":"2024-04-25T15:28:25.282367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648380+0000","last_change":"2024-04-25T15:28:23.644393+0000","last_active":"2024-04-25T15:28:24.648380+0000","last_peered":"2024-04-25T15:28:24.648380+0000","last_clean":"2024-04-25T15:28:24.648380+0000","last_became_active":"2024-04-25T15:28:23.644073+0000","last_became_peered":"2024-04-25T15:28:23.644073+0000","last_unstale":"2024-04-25T15:28:24.648380+0000","last_undegraded":"2024-04-25T15:28:24.648380+0000","last_fullsized":"2024-04-25T15:28:24.648380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:54.453734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647936+0000","last_change":"2024-04-25T15:28:23.654484+0000","last_active":"2024-04-25T15:28:24.647936+0000","last_peered":"2024-04-25T15:28:24.647936+0000","last_clean":"2024-04-25T15:28:24.647936+0000","last_became_active":"2024-04-25T15:28:23.654191+0000","last_became_peered":"2024-04-25T15:28:23.654191+0000","last_unstale":"2024-04-25T15:28:24.647936+0000","last_undegraded":"2024-04-25T15:28:24.647936+0000","last_fullsized":"2024-04-25T15:28:24.647936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:51.690159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.643192+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.642727+0000","last_became_peered":"2024-04-25T15:28:23.642727+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282773+0000","last_change":"2024-04-25T15:28:22.628837+0000","last_active":"2024-04-25T15:28:25.282773+0000","last_peered":"2024-04-25T15:28:25.282773+0000","last_clean":"2024-04-25T15:28:25.282773+0000","last_became_active":"2024-04-25T15:28:22.628231+0000","last_became_peered":"2024-04-25T15:28:22.628231+0000","last_unstale":"2024-04-25T15:28:25.282773+0000","last_undegraded":"2024-04-25T15:28:25.282773+0000","last_fullsized":"2024-04-25T15:28:25.282773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:53.446196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282653+0000","last_change":"2024-04-25T15:28:23.653258+0000","last_active":"2024-04-25T15:28:25.282653+0000","last_peered":"2024-04-25T15:28:25.282653+0000","last_clean":"2024-04-25T15:28:25.282653+0000","last_became_active":"2024-04-25T15:28:23.652929+0000","last_became_peered":"2024-04-25T15:28:23.652929+0000","last_unstale":"2024-04-25T15:28:25.282653+0000","last_undegraded":"2024-04-25T15:28:25.282653+0000","last_fullsized":"2024-04-25T15:28:25.282653+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:55.065573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.617466+0000","last_change":"2024-04-25T15:28:21.617466+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.617466+0000","last_undegraded":"2024-04-25T15:28:21.617466+0000","last_fullsized":"2024-04-25T15:28:21.617466+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282579+0000","last_change":"2024-04-25T15:28:23.647366+0000","last_active":"2024-04-25T15:28:25.282579+0000","last_peered":"2024-04-25T15:28:25.282579+0000","last_clean":"2024-04-25T15:28:25.282579+0000","last_became_active":"2024-04-25T15:28:23.647027+0000","last_became_peered":"2024-04-25T15:28:23.647027+0000","last_unstale":"2024-04-25T15:28:25.282579+0000","last_undegraded":"2024-04-25T15:28:25.282579+0000","last_fullsized":"2024-04-25T15:28:25.282579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648998+0000","last_change":"2024-04-25T15:28:22.627683+0000","last_active":"2024-04-25T15:28:24.648998+0000","last_peered":"2024-04-25T15:28:24.648998+0000","last_clean":"2024-04-25T15:28:24.648998+0000","last_became_active":"2024-04-25T15:28:22.627379+0000","last_became_peered":"2024-04-25T15:28:22.627379+0000","last_unstale":"2024-04-25T15:28:24.648998+0000","last_undegraded":"2024-04-25T15:28:24.648998+0000","last_fullsized":"2024-04-25T15:28:24.648998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:30.340419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281914+0000","last_change":"2024-04-25T15:28:23.655579+0000","last_active":"2024-04-25T15:28:25.281914+0000","last_peered":"2024-04-25T15:28:25.281914+0000","last_clean":"2024-04-25T15:28:25.281914+0000","last_became_active":"2024-04-25T15:28:23.655206+0000","last_became_peered":"2024-04-25T15:28:23.655206+0000","last_unstale":"2024-04-25T15:28:25.281914+0000","last_undegraded":"2024-04-25T15:28:25.281914+0000","last_fullsized":"2024-04-25T15:28:25.281914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:16.785360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.629301+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628962+0000","last_became_peered":"2024-04-25T15:28:22.628962+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:24.982249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648121+0000","last_change":"2024-04-25T15:28:23.654406+0000","last_active":"2024-04-25T15:28:24.648121+0000","last_peered":"2024-04-25T15:28:24.648121+0000","last_clean":"2024-04-25T15:28:24.648121+0000","last_became_active":"2024-04-25T15:28:23.652920+0000","last_became_peered":"2024-04-25T15:28:23.652920+0000","last_unstale":"2024-04-25T15:28:24.648121+0000","last_undegraded":"2024-04-25T15:28:24.648121+0000","last_fullsized":"2024-04-25T15:28:24.648121+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:31.162871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.617457+0000","last_change":"2024-04-25T15:28:21.617457+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.617457+0000","last_undegraded":"2024-04-25T15:28:21.617457+0000","last_fullsized":"2024-04-25T15:28:21.617457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.616871+0000","last_change":"2024-04-25T15:28:21.616871+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.616871+0000","last_undegraded":"2024-04-25T15:28:21.616871+0000","last_fullsized":"2024-04-25T15:28:21.616871+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281419+0000","last_change":"2024-04-25T15:28:23.639119+0000","last_active":"2024-04-25T15:28:25.281419+0000","last_peered":"2024-04-25T15:28:25.281419+0000","last_clean":"2024-04-25T15:28:25.281419+0000","last_became_active":"2024-04-25T15:28:23.638754+0000","last_became_peered":"2024-04-25T15:28:23.638754+0000","last_unstale":"2024-04-25T15:28:25.281419+0000","last_undegraded":"2024-04-25T15:28:25.281419+0000","last_fullsized":"2024-04-25T15:28:25.281419+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648777+0000","last_change":"2024-04-25T15:28:22.631835+0000","last_active":"2024-04-25T15:28:24.648777+0000","last_peered":"2024-04-25T15:28:24.648777+0000","last_clean":"2024-04-25T15:28:24.648777+0000","last_became_active":"2024-04-25T15:28:22.631539+0000","last_became_peered":"2024-04-25T15:28:22.631539+0000","last_unstale":"2024-04-25T15:28:24.648777+0000","last_undegraded":"2024-04-25T15:28:24.648777+0000","last_fullsized":"2024-04-25T15:28:24.648777+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:18.322027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281369+0000","last_change":"2024-04-25T15:28:23.638974+0000","last_active":"2024-04-25T15:28:25.281369+0000","last_peered":"2024-04-25T15:28:25.281369+0000","last_clean":"2024-04-25T15:28:25.281369+0000","last_became_active":"2024-04-25T15:28:23.638642+0000","last_became_peered":"2024-04-25T15:28:23.638642+0000","last_unstale":"2024-04-25T15:28:25.281369+0000","last_undegraded":"2024-04-25T15:28:25.281369+0000","last_fullsized":"2024-04-25T15:28:25.281369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:06.035044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647693+0000","last_change":"2024-04-25T15:28:22.634256+0000","last_active":"2024-04-25T15:28:24.647693+0000","last_peered":"2024-04-25T15:28:24.647693+0000","last_clean":"2024-04-25T15:28:24.647693+0000","last_became_active":"2024-04-25T15:28:22.633942+0000","last_became_peered":"2024-04-25T15:28:22.633942+0000","last_unstale":"2024-04-25T15:28:24.647693+0000","last_undegraded":"2024-04-25T15:28:24.647693+0000","last_fullsized":"2024-04-25T15:28:24.647693+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:10.420589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282150+0000","last_change":"2024-04-25T15:28:23.639749+0000","last_active":"2024-04-25T15:28:25.282150+0000","last_peered":"2024-04-25T15:28:25.282150+0000","last_clean":"2024-04-25T15:28:25.282150+0000","last_became_active":"2024-04-25T15:28:23.639415+0000","last_became_peered":"2024-04-25T15:28:23.639415+0000","last_unstale":"2024-04-25T15:28:25.282150+0000","last_undegraded":"2024-04-25T15:28:25.282150+0000","last_fullsized":"2024-04-25T15:28:25.282150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:38:27.203252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282928+0000","last_change":"2024-04-25T15:28:22.628552+0000","last_active":"2024-04-25T15:28:25.282928+0000","last_peered":"2024-04-25T15:28:25.282928+0000","last_clean":"2024-04-25T15:28:25.282928+0000","last_became_active":"2024-04-25T15:28:22.627193+0000","last_became_peered":"2024-04-25T15:28:22.627193+0000","last_unstale":"2024-04-25T15:28:25.282928+0000","last_undegraded":"2024-04-25T15:28:25.282928+0000","last_fullsized":"2024-04-25T15:28:25.282928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282633+0000","last_change":"2024-04-25T15:28:23.651525+0000","last_active":"2024-04-25T15:28:25.282633+0000","last_peered":"2024-04-25T15:28:25.282633+0000","last_clean":"2024-04-25T15:28:25.282633+0000","last_became_active":"2024-04-25T15:28:23.651179+0000","last_became_peered":"2024-04-25T15:28:23.651179+0000","last_unstale":"2024-04-25T15:28:25.282633+0000","last_undegraded":"2024-04-25T15:28:25.282633+0000","last_fullsized":"2024-04-25T15:28:25.282633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282316+0000","last_change":"2024-04-25T15:28:22.631506+0000","last_active":"2024-04-25T15:28:25.282316+0000","last_peered":"2024-04-25T15:28:25.282316+0000","last_clean":"2024-04-25T15:28:25.282316+0000","last_became_active":"2024-04-25T15:28:22.631161+0000","last_became_peered":"2024-04-25T15:28:22.631161+0000","last_unstale":"2024-04-25T15:28:25.282316+0000","last_undegraded":"2024-04-25T15:28:25.282316+0000","last_fullsized":"2024-04-25T15:28:25.282316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.414479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649203+0000","last_change":"2024-04-25T15:28:23.636915+0000","last_active":"2024-04-25T15:28:24.649203+0000","last_peered":"2024-04-25T15:28:24.649203+0000","last_clean":"2024-04-25T15:28:24.649203+0000","last_became_active":"2024-04-25T15:28:23.636599+0000","last_became_peered":"2024-04-25T15:28:23.636599+0000","last_unstale":"2024-04-25T15:28:24.649203+0000","last_undegraded":"2024-04-25T15:28:24.649203+0000","last_fullsized":"2024-04-25T15:28:24.649203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281907+0000","last_change":"2024-04-25T15:28:22.628373+0000","last_active":"2024-04-25T15:28:25.281907+0000","last_peered":"2024-04-25T15:28:25.281907+0000","last_clean":"2024-04-25T15:28:25.281907+0000","last_became_active":"2024-04-25T15:28:22.628063+0000","last_became_peered":"2024-04-25T15:28:22.628063+0000","last_unstale":"2024-04-25T15:28:25.281907+0000","last_undegraded":"2024-04-25T15:28:25.281907+0000","last_fullsized":"2024-04-25T15:28:25.281907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:17.513548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282803+0000","last_change":"2024-04-25T15:28:22.627717+0000","last_active":"2024-04-25T15:28:25.282803+0000","last_peered":"2024-04-25T15:28:25.282803+0000","last_clean":"2024-04-25T15:28:25.282803+0000","last_became_active":"2024-04-25T15:28:22.627402+0000","last_became_peered":"2024-04-25T15:28:22.627402+0000","last_unstale":"2024-04-25T15:28:25.282803+0000","last_undegraded":"2024-04-25T15:28:25.282803+0000","last_fullsized":"2024-04-25T15:28:25.282803+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648678+0000","last_change":"2024-04-25T15:28:23.630065+0000","last_active":"2024-04-25T15:28:24.648678+0000","last_peered":"2024-04-25T15:28:24.648678+0000","last_clean":"2024-04-25T15:28:24.648678+0000","last_became_active":"2024-04-25T15:28:23.629649+0000","last_became_peered":"2024-04-25T15:28:23.629649+0000","last_unstale":"2024-04-25T15:28:24.648678+0000","last_undegraded":"2024-04-25T15:28:24.648678+0000","last_fullsized":"2024-04-25T15:28:24.648678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648438+0000","last_change":"2024-04-25T15:28:22.632102+0000","last_active":"2024-04-25T15:28:24.648438+0000","last_peered":"2024-04-25T15:28:24.648438+0000","last_clean":"2024-04-25T15:28:24.648438+0000","last_became_active":"2024-04-25T15:28:22.631723+0000","last_became_peered":"2024-04-25T15:28:22.631723+0000","last_unstale":"2024-04-25T15:28:24.648438+0000","last_undegraded":"2024-04-25T15:28:24.648438+0000","last_fullsized":"2024-04-25T15:28:24.648438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:06.295529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282145+0000","last_change":"2024-04-25T15:28:23.639139+0000","last_active":"2024-04-25T15:28:25.282145+0000","last_peered":"2024-04-25T15:28:25.282145+0000","last_clean":"2024-04-25T15:28:25.282145+0000","last_became_active":"2024-04-25T15:28:23.638812+0000","last_became_peered":"2024-04-25T15:28:23.638812+0000","last_unstale":"2024-04-25T15:28:25.282145+0000","last_undegraded":"2024-04-25T15:28:25.282145+0000","last_fullsized":"2024-04-25T15:28:25.282145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:56.852295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281527+0000","last_change":"2024-04-25T15:28:22.631778+0000","last_active":"2024-04-25T15:28:25.281527+0000","last_peered":"2024-04-25T15:28:25.281527+0000","last_clean":"2024-04-25T15:28:25.281527+0000","last_became_active":"2024-04-25T15:28:22.631460+0000","last_became_peered":"2024-04-25T15:28:22.631460+0000","last_unstale":"2024-04-25T15:28:25.281527+0000","last_undegraded":"2024-04-25T15:28:25.281527+0000","last_fullsized":"2024-04-25T15:28:25.281527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.777821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647614+0000","last_change":"2024-04-25T15:28:23.652363+0000","last_active":"2024-04-25T15:28:24.647614+0000","last_peered":"2024-04-25T15:28:24.647614+0000","last_clean":"2024-04-25T15:28:24.647614+0000","last_became_active":"2024-04-25T15:28:23.651744+0000","last_became_peered":"2024-04-25T15:28:23.651744+0000","last_unstale":"2024-04-25T15:28:24.647614+0000","last_undegraded":"2024-04-25T15:28:24.647614+0000","last_fullsized":"2024-04-25T15:28:24.647614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:17.116922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281587+0000","last_change":"2024-04-25T15:28:22.636713+0000","last_active":"2024-04-25T15:28:25.281587+0000","last_peered":"2024-04-25T15:28:25.281587+0000","last_clean":"2024-04-25T15:28:25.281587+0000","last_became_active":"2024-04-25T15:28:22.636389+0000","last_became_peered":"2024-04-25T15:28:22.636389+0000","last_unstale":"2024-04-25T15:28:25.281587+0000","last_undegraded":"2024-04-25T15:28:25.281587+0000","last_fullsized":"2024-04-25T15:28:25.281587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647465+0000","last_change":"2024-04-25T15:28:23.647463+0000","last_active":"2024-04-25T15:28:24.647465+0000","last_peered":"2024-04-25T15:28:24.647465+0000","last_clean":"2024-04-25T15:28:24.647465+0000","last_became_active":"2024-04-25T15:28:23.647133+0000","last_became_peered":"2024-04-25T15:28:23.647133+0000","last_unstale":"2024-04-25T15:28:24.647465+0000","last_undegraded":"2024-04-25T15:28:24.647465+0000","last_fullsized":"2024-04-25T15:28:24.647465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:14.765406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.628461+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628160+0000","last_became_peered":"2024-04-25T15:28:22.628160+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282411+0000","last_change":"2024-04-25T15:28:23.649571+0000","last_active":"2024-04-25T15:28:25.282411+0000","last_peered":"2024-04-25T15:28:25.282411+0000","last_clean":"2024-04-25T15:28:25.282411+0000","last_became_active":"2024-04-25T15:28:23.649214+0000","last_became_peered":"2024-04-25T15:28:23.649214+0000","last_unstale":"2024-04-25T15:28:25.282411+0000","last_undegraded":"2024-04-25T15:28:25.282411+0000","last_fullsized":"2024-04-25T15:28:25.282411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:22.598027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283048+0000","last_change":"2024-04-25T15:28:22.631820+0000","last_active":"2024-04-25T15:28:25.283048+0000","last_peered":"2024-04-25T15:28:25.283048+0000","last_clean":"2024-04-25T15:28:25.283048+0000","last_became_active":"2024-04-25T15:28:22.631501+0000","last_became_peered":"2024-04-25T15:28:22.631501+0000","last_unstale":"2024-04-25T15:28:25.283048+0000","last_undegraded":"2024-04-25T15:28:25.283048+0000","last_fullsized":"2024-04-25T15:28:25.283048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282947+0000","last_change":"2024-04-25T15:28:23.646728+0000","last_active":"2024-04-25T15:28:25.282947+0000","last_peered":"2024-04-25T15:28:25.282947+0000","last_clean":"2024-04-25T15:28:25.282947+0000","last_became_active":"2024-04-25T15:28:23.646234+0000","last_became_peered":"2024-04-25T15:28:23.646234+0000","last_unstale":"2024-04-25T15:28:25.282947+0000","last_undegraded":"2024-04-25T15:28:25.282947+0000","last_fullsized":"2024-04-25T15:28:25.282947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648950+0000","last_change":"2024-04-25T15:28:22.632918+0000","last_active":"2024-04-25T15:28:24.648950+0000","last_peered":"2024-04-25T15:28:24.648950+0000","last_clean":"2024-04-25T15:28:24.648950+0000","last_became_active":"2024-04-25T15:28:22.632551+0000","last_became_peered":"2024-04-25T15:28:22.632551+0000","last_unstale":"2024-04-25T15:28:24.648950+0000","last_undegraded":"2024-04-25T15:28:24.648950+0000","last_fullsized":"2024-04-25T15:28:24.648950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:25.998670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649243+0000","last_change":"2024-04-25T15:28:22.631396+0000","last_active":"2024-04-25T15:28:24.649243+0000","last_peered":"2024-04-25T15:28:24.649243+0000","last_clean":"2024-04-25T15:28:24.649243+0000","last_became_active":"2024-04-25T15:28:22.630989+0000","last_became_peered":"2024-04-25T15:28:22.630989+0000","last_unstale":"2024-04-25T15:28:24.649243+0000","last_undegraded":"2024-04-25T15:28:24.649243+0000","last_fullsized":"2024-04-25T15:28:24.649243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:25.648985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282440+0000","last_change":"2024-04-25T15:28:23.650548+0000","last_active":"2024-04-25T15:28:25.282440+0000","last_peered":"2024-04-25T15:28:25.282440+0000","last_clean":"2024-04-25T15:28:25.282440+0000","last_became_active":"2024-04-25T15:28:23.649636+0000","last_became_peered":"2024-04-25T15:28:23.649636+0000","last_unstale":"2024-04-25T15:28:25.282440+0000","last_undegraded":"2024-04-25T15:28:25.282440+0000","last_fullsized":"2024-04-25T15:28:25.282440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:11.428127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281645+0000","last_change":"2024-04-25T15:28:22.631144+0000","last_active":"2024-04-25T15:28:25.281645+0000","last_peered":"2024-04-25T15:28:25.281645+0000","last_clean":"2024-04-25T15:28:25.281645+0000","last_became_active":"2024-04-25T15:28:22.630762+0000","last_became_peered":"2024-04-25T15:28:22.630762+0000","last_unstale":"2024-04-25T15:28:25.281645+0000","last_undegraded":"2024-04-25T15:28:25.281645+0000","last_fullsized":"2024-04-25T15:28:25.281645+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:10.967925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283204+0000","last_change":"2024-04-25T15:28:23.647208+0000","last_active":"2024-04-25T15:28:25.283204+0000","last_peered":"2024-04-25T15:28:25.283204+0000","last_clean":"2024-04-25T15:28:25.283204+0000","last_became_active":"2024-04-25T15:28:23.646871+0000","last_became_peered":"2024-04-25T15:28:23.646871+0000","last_unstale":"2024-04-25T15:28:25.283204+0000","last_undegraded":"2024-04-25T15:28:25.283204+0000","last_fullsized":"2024-04-25T15:28:25.283204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282312+0000","last_change":"2024-04-25T15:28:22.623948+0000","last_active":"2024-04-25T15:28:25.282312+0000","last_peered":"2024-04-25T15:28:25.282312+0000","last_clean":"2024-04-25T15:28:25.282312+0000","last_became_active":"2024-04-25T15:28:22.623604+0000","last_became_peered":"2024-04-25T15:28:22.623604+0000","last_unstale":"2024-04-25T15:28:25.282312+0000","last_undegraded":"2024-04-25T15:28:25.282312+0000","last_fullsized":"2024-04-25T15:28:25.282312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:13.369418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:28:23.648910+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:28:23.648524+0000","last_became_peered":"2024-04-25T15:28:23.648524+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.853221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281681+0000","last_change":"2024-04-25T15:28:22.635528+0000","last_active":"2024-04-25T15:28:25.281681+0000","last_peered":"2024-04-25T15:28:25.281681+0000","last_clean":"2024-04-25T15:28:25.281681+0000","last_became_active":"2024-04-25T15:28:22.635320+0000","last_became_peered":"2024-04-25T15:28:22.635320+0000","last_unstale":"2024-04-25T15:28:25.281681+0000","last_undegraded":"2024-04-25T15:28:25.281681+0000","last_fullsized":"2024-04-25T15:28:25.281681+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:37.967626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281434+0000","last_change":"2024-04-25T15:28:23.658467+0000","last_active":"2024-04-25T15:28:25.281434+0000","last_peered":"2024-04-25T15:28:25.281434+0000","last_clean":"2024-04-25T15:28:25.281434+0000","last_became_active":"2024-04-25T15:28:23.658051+0000","last_became_peered":"2024-04-25T15:28:23.658051+0000","last_unstale":"2024-04-25T15:28:25.281434+0000","last_undegraded":"2024-04-25T15:28:25.281434+0000","last_fullsized":"2024-04-25T15:28:25.281434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:27.006208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281183+0000","last_change":"2024-04-25T15:28:22.627923+0000","last_active":"2024-04-25T15:28:25.281183+0000","last_peered":"2024-04-25T15:28:25.281183+0000","last_clean":"2024-04-25T15:28:25.281183+0000","last_became_active":"2024-04-25T15:28:22.627595+0000","last_became_peered":"2024-04-25T15:28:22.627595+0000","last_unstale":"2024-04-25T15:28:25.281183+0000","last_undegraded":"2024-04-25T15:28:25.281183+0000","last_fullsized":"2024-04-25T15:28:25.281183+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:04.050831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647686+0000","last_change":"2024-04-25T15:28:22.629989+0000","last_active":"2024-04-25T15:28:24.647686+0000","last_peered":"2024-04-25T15:28:24.647686+0000","last_clean":"2024-04-25T15:28:24.647686+0000","last_became_active":"2024-04-25T15:28:22.629681+0000","last_became_peered":"2024-04-25T15:28:22.629681+0000","last_unstale":"2024-04-25T15:28:24.647686+0000","last_undegraded":"2024-04-25T15:28:24.647686+0000","last_fullsized":"2024-04-25T15:28:24.647686+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:36.386426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281853+0000","last_change":"2024-04-25T15:28:23.643948+0000","last_active":"2024-04-25T15:28:25.281853+0000","last_peered":"2024-04-25T15:28:25.281853+0000","last_clean":"2024-04-25T15:28:25.281853+0000","last_became_active":"2024-04-25T15:28:23.643600+0000","last_became_peered":"2024-04-25T15:28:23.643600+0000","last_unstale":"2024-04-25T15:28:25.281853+0000","last_undegraded":"2024-04-25T15:28:25.281853+0000","last_fullsized":"2024-04-25T15:28:25.281853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648158+0000","last_change":"2024-04-25T15:28:22.630713+0000","last_active":"2024-04-25T15:28:24.648158+0000","last_peered":"2024-04-25T15:28:24.648158+0000","last_clean":"2024-04-25T15:28:24.648158+0000","last_became_active":"2024-04-25T15:28:22.627937+0000","last_became_peered":"2024-04-25T15:28:22.627937+0000","last_unstale":"2024-04-25T15:28:24.648158+0000","last_undegraded":"2024-04-25T15:28:24.648158+0000","last_fullsized":"2024-04-25T15:28:24.648158+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:34.438583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282846+0000","last_change":"2024-04-25T15:28:23.650199+0000","last_active":"2024-04-25T15:28:25.282846+0000","last_peered":"2024-04-25T15:28:25.282846+0000","last_clean":"2024-04-25T15:28:25.282846+0000","last_became_active":"2024-04-25T15:28:23.649599+0000","last_became_peered":"2024-04-25T15:28:23.649599+0000","last_unstale":"2024-04-25T15:28:25.282846+0000","last_undegraded":"2024-04-25T15:28:25.282846+0000","last_fullsized":"2024-04-25T15:28:25.282846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282154+0000","last_change":"2024-04-25T15:28:22.626743+0000","last_active":"2024-04-25T15:28:25.282154+0000","last_peered":"2024-04-25T15:28:25.282154+0000","last_clean":"2024-04-25T15:28:25.282154+0000","last_became_active":"2024-04-25T15:28:22.626434+0000","last_became_peered":"2024-04-25T15:28:22.626434+0000","last_unstale":"2024-04-25T15:28:25.282154+0000","last_undegraded":"2024-04-25T15:28:25.282154+0000","last_fullsized":"2024-04-25T15:28:25.282154+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:13.395697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T15:28:22.609329+0000","last_change":"2024-04-25T15:28:22.609329+0000","last_active":"2024-04-25T15:28:22.609329+0000","last_peered":"2024-04-25T15:28:22.609329+0000","last_clean":"2024-04-25T15:28:22.609329+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:22.609329+0000","last_undegraded":"2024-04-25T15:28:22.609329+0000","last_fullsized":"2024-04-25T15:28:22.609329+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282910+0000","last_change":"2024-04-25T15:28:22.630470+0000","last_active":"2024-04-25T15:28:25.282910+0000","last_peered":"2024-04-25T15:28:25.282910+0000","last_clean":"2024-04-25T15:28:25.282910+0000","last_became_active":"2024-04-25T15:28:22.630069+0000","last_became_peered":"2024-04-25T15:28:22.630069+0000","last_unstale":"2024-04-25T15:28:25.282910+0000","last_undegraded":"2024-04-25T15:28:25.282910+0000","last_fullsized":"2024-04-25T15:28:25.282910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:30.155333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282356+0000","last_change":"2024-04-25T15:28:23.648809+0000","last_active":"2024-04-25T15:28:25.282356+0000","last_peered":"2024-04-25T15:28:25.282356+0000","last_clean":"2024-04-25T15:28:25.282356+0000","last_became_active":"2024-04-25T15:28:23.648507+0000","last_became_peered":"2024-04-25T15:28:23.648507+0000","last_unstale":"2024-04-25T15:28:25.282356+0000","last_undegraded":"2024-04-25T15:28:25.282356+0000","last_fullsized":"2024-04-25T15:28:25.282356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648878+0000","last_change":"2024-04-25T15:28:22.628170+0000","last_active":"2024-04-25T15:28:24.648878+0000","last_peered":"2024-04-25T15:28:24.648878+0000","last_clean":"2024-04-25T15:28:24.648878+0000","last_became_active":"2024-04-25T15:28:22.627191+0000","last_became_peered":"2024-04-25T15:28:22.627191+0000","last_unstale":"2024-04-25T15:28:24.648878+0000","last_undegraded":"2024-04-25T15:28:24.648878+0000","last_fullsized":"2024-04-25T15:28:24.648878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:31.302274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648932+0000","last_change":"2024-04-25T15:28:23.630876+0000","last_active":"2024-04-25T15:28:24.648932+0000","last_peered":"2024-04-25T15:28:24.648932+0000","last_clean":"2024-04-25T15:28:24.648932+0000","last_became_active":"2024-04-25T15:28:23.630558+0000","last_became_peered":"2024-04-25T15:28:23.630558+0000","last_unstale":"2024-04-25T15:28:24.648932+0000","last_undegraded":"2024-04-25T15:28:24.648932+0000","last_fullsized":"2024-04-25T15:28:24.648932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.616307+0000","last_change":"2024-04-25T15:28:21.616307+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.616307+0000","last_undegraded":"2024-04-25T15:28:21.616307+0000","last_fullsized":"2024-04-25T15:28:21.616307+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281519+0000","last_change":"2024-04-25T15:28:23.640269+0000","last_active":"2024-04-25T15:28:25.281519+0000","last_peered":"2024-04-25T15:28:25.281519+0000","last_clean":"2024-04-25T15:28:25.281519+0000","last_became_active":"2024-04-25T15:28:23.639944+0000","last_became_peered":"2024-04-25T15:28:23.639944+0000","last_unstale":"2024-04-25T15:28:25.281519+0000","last_undegraded":"2024-04-25T15:28:25.281519+0000","last_fullsized":"2024-04-25T15:28:25.281519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282853+0000","last_change":"2024-04-25T15:28:22.626538+0000","last_active":"2024-04-25T15:28:25.282853+0000","last_peered":"2024-04-25T15:28:25.282853+0000","last_clean":"2024-04-25T15:28:25.282853+0000","last_became_active":"2024-04-25T15:28:22.626207+0000","last_became_peered":"2024-04-25T15:28:22.626207+0000","last_unstale":"2024-04-25T15:28:25.282853+0000","last_undegraded":"2024-04-25T15:28:25.282853+0000","last_fullsized":"2024-04-25T15:28:25.282853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:50.682621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282764+0000","last_change":"2024-04-25T15:28:23.646030+0000","last_active":"2024-04-25T15:28:25.282764+0000","last_peered":"2024-04-25T15:28:25.282764+0000","last_clean":"2024-04-25T15:28:25.282764+0000","last_became_active":"2024-04-25T15:28:23.645733+0000","last_became_peered":"2024-04-25T15:28:23.645733+0000","last_unstale":"2024-04-25T15:28:25.282764+0000","last_undegraded":"2024-04-25T15:28:25.282764+0000","last_fullsized":"2024-04-25T15:28:25.282764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":77,"state":"creating+peering","last_fresh":"2024-04-25T15:28:21.615544+0000","last_change":"2024-04-25T15:28:21.615544+0000","last_active":"2024-04-25T15:28:21.601791+0000","last_peered":"2024-04-25T15:28:21.601791+0000","last_clean":"2024-04-25T15:28:21.601791+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:21.615544+0000","last_undegraded":"2024-04-25T15:28:21.615544+0000","last_fullsized":"2024-04-25T15:28:21.615544+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648545+0000","last_change":"2024-04-25T15:28:23.640057+0000","last_active":"2024-04-25T15:28:24.648545+0000","last_peered":"2024-04-25T15:28:24.648545+0000","last_clean":"2024-04-25T15:28:24.648545+0000","last_became_active":"2024-04-25T15:28:23.639747+0000","last_became_peered":"2024-04-25T15:28:23.639747+0000","last_unstale":"2024-04-25T15:28:24.648545+0000","last_undegraded":"2024-04-25T15:28:24.648545+0000","last_fullsized":"2024-04-25T15:28:24.648545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:06.046345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281905+0000","last_change":"2024-04-25T15:28:22.632108+0000","last_active":"2024-04-25T15:28:25.281905+0000","last_peered":"2024-04-25T15:28:25.281905+0000","last_clean":"2024-04-25T15:28:25.281905+0000","last_became_active":"2024-04-25T15:28:22.631768+0000","last_became_peered":"2024-04-25T15:28:22.631768+0000","last_unstale":"2024-04-25T15:28:25.281905+0000","last_undegraded":"2024-04-25T15:28:25.281905+0000","last_fullsized":"2024-04-25T15:28:25.281905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":218,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1581056,"data_stored":1007311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":418507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":946176,"data_stored":397523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":418485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":417455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1581056,"data_stored":1007275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:28:28.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3b is not active+clean 2024-04-25T15:28:28.443 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.443 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.30 is not active+clean 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2f is not active+clean 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.25 is not active+clean 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.21 is not active+clean 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1e is not active+clean 2024-04-25T15:28:28.444 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:21.617466+0000', 'last_change': '2024-04-25T15:28:21.617466+0000', 'last_active': '2024-04-25T15:28:21.601791+0000', 'last_peered': '2024-04-25T15:28:21.601791+0000', 'last_clean': '2024-04-25T15:28:21.601791+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:21.617466+0000', 'last_undegraded': '2024-04-25T15:28:21.617466+0000', 'last_fullsized': '2024-04-25T15:28:21.617466+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1b is not active+clean 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:21.617457+0000', 'last_change': '2024-04-25T15:28:21.617457+0000', 'last_active': '2024-04-25T15:28:21.601791+0000', 'last_peered': '2024-04-25T15:28:21.601791+0000', 'last_clean': '2024-04-25T15:28:21.601791+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:21.617457+0000', 'last_undegraded': '2024-04-25T15:28:21.617457+0000', 'last_fullsized': '2024-04-25T15:28:21.617457+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1a is not active+clean 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:21.616871+0000', 'last_change': '2024-04-25T15:28:21.616871+0000', 'last_active': '2024-04-25T15:28:21.601791+0000', 'last_peered': '2024-04-25T15:28:21.601791+0000', 'last_clean': '2024-04-25T15:28:21.601791+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:21.616871+0000', 'last_undegraded': '2024-04-25T15:28:21.616871+0000', 'last_fullsized': '2024-04-25T15:28:21.616871+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-04-25T15:28:28.445 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T15:28:22.609329+0000', 'last_change': '2024-04-25T15:28:22.609329+0000', 'last_active': '2024-04-25T15:28:22.609329+0000', 'last_peered': '2024-04-25T15:28:22.609329+0000', 'last_clean': '2024-04-25T15:28:22.609329+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:22.609329+0000', 'last_undegraded': '2024-04-25T15:28:22.609329+0000', 'last_fullsized': '2024-04-25T15:28:22.609329+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:22.609329+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2 is not active+clean 2024-04-25T15:28:28.446 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:21.616307+0000', 'last_change': '2024-04-25T15:28:21.616307+0000', 'last_active': '2024-04-25T15:28:21.601791+0000', 'last_peered': '2024-04-25T15:28:21.601791+0000', 'last_clean': '2024-04-25T15:28:21.601791+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:21.616307+0000', 'last_undegraded': '2024-04-25T15:28:21.616307+0000', 'last_fullsized': '2024-04-25T15:28:21.616307+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:28:28.447 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.0 is not active+clean 2024-04-25T15:28:28.447 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 77, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:21.615544+0000', 'last_change': '2024-04-25T15:28:21.615544+0000', 'last_active': '2024-04-25T15:28:21.601791+0000', 'last_peered': '2024-04-25T15:28:21.601791+0000', 'last_clean': '2024-04-25T15:28:21.601791+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:21.615544+0000', 'last_undegraded': '2024-04-25T15:28:21.615544+0000', 'last_fullsized': '2024-04-25T15:28:21.615544+0000', 'mapping_epoch': 77, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 77, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:21.601791+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:28:28.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-25T15:28:28.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.589+0000 7f84264bd640 1 Processor -- start 2024-04-25T15:28:28.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.589+0000 7f84264bd640 1 -- start start 2024-04-25T15:28:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f84264bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84200b4830 0x7f842006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f84264bd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f842006d870 con 0x7f84200d4570 2024-04-25T15:28:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f84264bd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f842006db10 con 0x7f84200b2dc0 2024-04-25T15:28:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f84264bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f84200b4c40 con 0x7f84200b4830 2024-04-25T15:28:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f841f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84200b4830 0x7f842006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f841ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f84200d4570 0x7f84200d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60460/0 (socket says 172.21.15.67:60460) 2024-04-25T15:28:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f841ffff640 1 -- 172.21.15.67:0/2392368835 learned_addr learned my addr 172.21.15.67:0/2392368835 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f841f7fe640 1 -- 172.21.15.67:0/2392368835 >> v1:172.21.15.176:6789/0 conn(0x7f84200b2dc0 legacy=0x7f84200b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f841f7fe640 1 -- 172.21.15.67:0/2392368835 >> v1:172.21.15.67:6789/0 conn(0x7f84200d4570 legacy=0x7f84200d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.591+0000 7f841f7fe640 1 -- 172.21.15.67:0/2392368835 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842006e0f0 con 0x7f84200b4830 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.592+0000 7f841f7fe640 1 --2- 172.21.15.67:0/2392368835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f842006e8f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f84140097a0 tx=0x7f841402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e9e367fcfa9a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.592+0000 7f841effd640 1 -- 172.21.15.67:0/2392368835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414037070 con 0x7f84200b4830 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.592+0000 7f841effd640 1 -- 172.21.15.67:0/2392368835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8414032440 con 0x7f84200b4830 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.592+0000 7f841effd640 1 -- 172.21.15.67:0/2392368835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414032810 con 0x7f84200b4830 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.593+0000 7f84264bd640 1 -- 172.21.15.67:0/2392368835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 msgr2=0x7f842006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.593+0000 7f84264bd640 1 --2- 172.21.15.67:0/2392368835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f842006e8f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f84140097a0 tx=0x7f841402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.593+0000 7f84264bd640 1 -- 172.21.15.67:0/2392368835 shutdown_connections 2024-04-25T15:28:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.593+0000 7f84264bd640 1 --2- 172.21.15.67:0/2392368835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f842006e8f0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.593+0000 7f84264bd640 1 -- 172.21.15.67:0/2392368835 >> 172.21.15.67:0/2392368835 conn(0x7f84200d3a50 msgr2=0x7f84200d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:28.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.593+0000 7f84264bd640 1 -- 172.21.15.67:0/2392368835 shutdown_connections 2024-04-25T15:28:28.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.593+0000 7f84264bd640 1 -- 172.21.15.67:0/2392368835 wait complete. 2024-04-25T15:28:28.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.595+0000 7f84264bd640 1 Processor -- start 2024-04-25T15:28:28.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.595+0000 7f84264bd640 1 -- start start 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f84264bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84200b2dc0 0x7f8420158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f84264bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f84200a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f84264bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84200d4570 0x7f84200b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f84264bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f84200d0fb0 con 0x7f84200b4830 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f84264bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f842006e460 con 0x7f84200d4570 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f84264bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f84200b38e0 con 0x7f84200b2dc0 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f841f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f84200a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f841f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f84200a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56986/0 (socket says 172.21.15.67:56986) 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.596+0000 7f841f7fe640 1 -- 172.21.15.67:0/2245089903 learned_addr learned my addr 172.21.15.67:0/2245089903 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841f7fe640 1 -- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84200b2dc0 msgr2=0x7f8420158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841ffff640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84200b2dc0 0x7f8420158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f8424a5a640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84200d4570 0x7f84200b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841f7fe640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84200b2dc0 0x7f8420158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841f7fe640 1 -- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84200d4570 msgr2=0x7f84200b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841f7fe640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84200d4570 0x7f84200b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841f7fe640 1 -- 172.21.15.67:0/2245089903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84201f9e80 con 0x7f84200b4830 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841f7fe640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f84200a0d20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f84140098c0 tx=0x7f84140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841cff9640 1 -- 172.21.15.67:0/2245089903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414037040 con 0x7f84200b4830 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841cff9640 1 -- 172.21.15.67:0/2245089903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8414030770 con 0x7f84200b4830 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f841cff9640 1 -- 172.21.15.67:0/2245089903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414030b40 con 0x7f84200b4830 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84201f7d60 con 0x7f84200b4830 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.597+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84201f8320 con 0x7f84200b4830 2024-04-25T15:28:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.598+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8420050740 con 0x7f84200b4830 2024-04-25T15:28:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.599+0000 7f841cff9640 1 -- 172.21.15.67:0/2245089903 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8414044020 con 0x7f84200b4830 2024-04-25T15:28:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.599+0000 7f841cff9640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f840c0744e0 0x7f840c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.599+0000 7f841cff9640 1 -- 172.21.15.67:0/2245089903 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f84140b5880 con 0x7f84200b4830 2024-04-25T15:28:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.600+0000 7f841ffff640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f840c0744e0 0x7f840c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.600+0000 7f841ffff640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f840c0744e0 0x7f840c0769f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f84100010e0 tx=0x7f8410009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.604+0000 7f841cff9640 1 -- 172.21.15.67:0/2245089903 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8414032e20 con 0x7f84200b4830 2024-04-25T15:28:28.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.860+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f84200d1220 con 0x7f84200b4830 2024-04-25T15:28:28.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.861+0000 7f841cff9640 1 -- 172.21.15.67:0/2245089903 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1594 (secure 0 0 0) 0x7f8414080010 con 0x7f84200b4830 2024-04-25T15:28:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:28.862 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":204,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":80,"num_osds":8,"num_up_osds":8,"osd_up_since":1714058715,"num_in_osds":8,"osd_in_since":1714058706,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":81},{"state_name":"unknown","count":11},{"state_name":"creating+peering","count":5}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":238981120,"bytes_avail":772855132160,"bytes_total":773094113280,"unknown_pgs_ratio":0.11340206116437912,"inactive_pgs_ratio":0.051546391099691391,"write_bytes_sec":2744,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":49,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"d","status":"up:active","gid":6328}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-04-25T15:27:06.818678+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f840c0744e0 msgr2=0x7f840c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f840c0744e0 0x7f840c0769f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f84100010e0 tx=0x7f8410009040 comp rx=0 tx=0).stop 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 msgr2=0x7f84200a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f84200a0d20 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f84140098c0 tx=0x7f84140046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 shutdown_connections 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f840c0744e0 0x7f840c0769f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84200d4570 0x7f84200b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84200b4830 0x7f84200a0d20 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 --2- 172.21.15.67:0/2245089903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f84200b2dc0 0x7f8420158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 >> 172.21.15.67:0/2245089903 conn(0x7f84200d3a50 msgr2=0x7f8420162a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:28.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 shutdown_connections 2024-04-25T15:28:28.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.863+0000 7f84264bd640 1 -- 172.21.15.67:0/2245089903 wait complete. 2024-04-25T15:28:28.875 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-25T15:28:28.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:28:28.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.986+0000 7f31187f3640 1 Processor -- start 2024-04-25T15:28:28.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.986+0000 7f31187f3640 1 -- start start 2024-04-25T15:28:28.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f31187f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31100b8290 0x7f3110160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f31187f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f311015df80 0x7f3110160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f31187f3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31101615f0 con 0x7f3110164b00 2024-04-25T15:28:28.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f31187f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3110164670 con 0x7f311015df80 2024-04-25T15:28:28.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f31187f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3110164020 con 0x7f31100b8290 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f3116d90640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f311015df80 0x7f3110160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f311658f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3110164b00 0x7f31100b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60462/0 (socket says 172.21.15.67:60462) 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f311658f640 1 -- 172.21.15.67:0/2164769170 learned_addr learned my addr 172.21.15.67:0/2164769170 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f3115d8e640 1 --2- 172.21.15.67:0/2164769170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31100b8290 0x7f3110160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f3116d90640 1 -- 172.21.15.67:0/2164769170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31100b8290 msgr2=0x7f3110160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f3116d90640 1 --2- 172.21.15.67:0/2164769170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31100b8290 0x7f3110160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f3116d90640 1 -- 172.21.15.67:0/2164769170 >> v1:172.21.15.67:6789/0 conn(0x7f3110164b00 legacy=0x7f31100b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f311558d640 1 -- 172.21.15.67:0/2164769170 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994243356 0 0) 0x7f31101615f0 con 0x7f3110164b00 2024-04-25T15:28:28.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.988+0000 7f3116d90640 1 -- 172.21.15.67:0/2164769170 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3110164300 con 0x7f311015df80 2024-04-25T15:28:28.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.989+0000 7f3116d90640 1 --2- 172.21.15.67:0/2164769170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110160dd0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3100009d90 tx=0x7f310002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c5262ec588d388 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.989+0000 7f311558d640 1 -- 172.21.15.67:0/2164769170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100007740 con 0x7f311015df80 2024-04-25T15:28:28.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.989+0000 7f311558d640 1 -- 172.21.15.67:0/2164769170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31000078e0 con 0x7f311015df80 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.989+0000 7f311558d640 1 -- 172.21.15.67:0/2164769170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100007cb0 con 0x7f311015df80 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 -- 172.21.15.67:0/2164769170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 msgr2=0x7f3110160dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 --2- 172.21.15.67:0/2164769170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110160dd0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3100009d90 tx=0x7f310002a660 comp rx=0 tx=0).stop 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 -- 172.21.15.67:0/2164769170 shutdown_connections 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 --2- 172.21.15.67:0/2164769170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110160dd0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 --2- 172.21.15.67:0/2164769170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31100b8290 0x7f3110160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 -- 172.21.15.67:0/2164769170 >> 172.21.15.67:0/2164769170 conn(0x7f31100d2bc0 msgr2=0x7f31100d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 -- 172.21.15.67:0/2164769170 shutdown_connections 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.990+0000 7f31187f3640 1 -- 172.21.15.67:0/2164769170 wait complete. 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.992+0000 7f31187f3640 1 Processor -- start 2024-04-25T15:28:28.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.992+0000 7f31187f3640 1 -- start start 2024-04-25T15:28:28.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f31187f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f31187f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3110164b00 0x7f3110154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f31187f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3110154c40 0x7f3110162640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f31187f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31100b4a00 con 0x7f311015df80 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f31187f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f31100b4880 con 0x7f3110164b00 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f31187f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f31100b4b80 con 0x7f3110154c40 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f311658f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f311658f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57006/0 (socket says 172.21.15.67:57006) 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f311658f640 1 -- 172.21.15.67:0/3430068577 learned_addr learned my addr 172.21.15.67:0/3430068577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f3116d90640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3110154c40 0x7f3110162640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.993+0000 7f3115d8e640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3110164b00 0x7f3110154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f311658f640 1 -- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3110154c40 msgr2=0x7f3110162640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f311658f640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3110154c40 0x7f3110162640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f311658f640 1 -- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3110164b00 msgr2=0x7f3110154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f311658f640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3110164b00 0x7f3110154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f311658f640 1 -- 172.21.15.67:0/3430068577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3110152a50 con 0x7f311015df80 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f3115d8e640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3110164b00 0x7f3110154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f311658f640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110152510 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f310800c460 tx=0x7f310800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f310effd640 1 -- 172.21.15.67:0/3430068577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108018070 con 0x7f311015df80 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f310effd640 1 -- 172.21.15.67:0/3430068577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3108007500 con 0x7f311015df80 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3110153180 con 0x7f311015df80 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f310effd640 1 -- 172.21.15.67:0/3430068577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108004a20 con 0x7f311015df80 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.994+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3110153740 con 0x7f311015df80 2024-04-25T15:28:28.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.995+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3110050740 con 0x7f311015df80 2024-04-25T15:28:28.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.996+0000 7f310effd640 1 -- 172.21.15.67:0/3430068577 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f310801b070 con 0x7f311015df80 2024-04-25T15:28:28.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.996+0000 7f310effd640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30fc0744b0 0x7f30fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:28.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.996+0000 7f310effd640 1 -- 172.21.15.67:0/3430068577 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f31080943c0 con 0x7f311015df80 2024-04-25T15:28:28.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.996+0000 7f3115d8e640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30fc0744b0 0x7f30fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:28.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:28.997+0000 7f3115d8e640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30fc0744b0 0x7f30fc0769c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3104006fd0 tx=0x7f3104008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:29.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.001+0000 7f310effd640 1 -- 172.21.15.67:0/3430068577 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3110152a50 con 0x7f311015df80 2024-04-25T15:28:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.181+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f31100d3f10 con 0x7f30fc0744b0 2024-04-25T15:28:29.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.187+0000 7f310effd640 1 -- 172.21.15.67:0/3430068577 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252062 (secure 0 0 0) 0x7f31100d3f10 con 0x7f30fc0744b0 2024-04-25T15:28:29.191 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:29.192 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:28:29.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30fc0744b0 msgr2=0x7f30fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30fc0744b0 0x7f30fc0769c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3104006fd0 tx=0x7f3104008040 comp rx=0 tx=0).stop 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 msgr2=0x7f3110152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110152510 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f310800c460 tx=0x7f310800c920 comp rx=0 tx=0).stop 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 shutdown_connections 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f30fc0744b0 0x7f30fc0769c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3110154c40 0x7f3110162640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3110164b00 0x7f3110154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 --2- 172.21.15.67:0/3430068577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f311015df80 0x7f3110152510 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 >> 172.21.15.67:0/3430068577 conn(0x7f31100d2bc0 msgr2=0x7f311015cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 shutdown_connections 2024-04-25T15:28:29.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:29.190+0000 7f31187f3640 1 -- 172.21.15.67:0/3430068577 wait complete. 2024-04-25T15:28:29.202 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-04-25T15:28:28.839579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233416,"kb_used_data":8904,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754741304,"statfs":{"total":773094113280,"available":772855095296,"internally_reserved":0,"allocated":9117696,"data_stored":4520540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.226793"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282682+0000","last_change":"2024-04-25T15:28:23.652205+0000","last_active":"2024-04-25T15:28:25.282682+0000","last_peered":"2024-04-25T15:28:25.282682+0000","last_clean":"2024-04-25T15:28:25.282682+0000","last_became_active":"2024-04-25T15:28:23.651726+0000","last_became_peered":"2024-04-25T15:28:23.651726+0000","last_unstale":"2024-04-25T15:28:25.282682+0000","last_undegraded":"2024-04-25T15:28:25.282682+0000","last_fullsized":"2024-04-25T15:28:25.282682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:46.911832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648103+0000","last_change":"2024-04-25T15:28:23.649010+0000","last_active":"2024-04-25T15:28:24.648103+0000","last_peered":"2024-04-25T15:28:24.648103+0000","last_clean":"2024-04-25T15:28:24.648103+0000","last_became_active":"2024-04-25T15:28:23.648637+0000","last_became_peered":"2024-04-25T15:28:23.648637+0000","last_unstale":"2024-04-25T15:28:24.648103+0000","last_undegraded":"2024-04-25T15:28:24.648103+0000","last_fullsized":"2024-04-25T15:28:24.648103+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:28.422017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281638+0000","last_change":"2024-04-25T15:28:23.643910+0000","last_active":"2024-04-25T15:28:25.281638+0000","last_peered":"2024-04-25T15:28:25.281638+0000","last_clean":"2024-04-25T15:28:25.281638+0000","last_became_active":"2024-04-25T15:28:23.643580+0000","last_became_peered":"2024-04-25T15:28:23.643580+0000","last_unstale":"2024-04-25T15:28:25.281638+0000","last_undegraded":"2024-04-25T15:28:25.281638+0000","last_fullsized":"2024-04-25T15:28:25.281638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281992+0000","last_change":"2024-04-25T15:28:23.649272+0000","last_active":"2024-04-25T15:28:25.281992+0000","last_peered":"2024-04-25T15:28:25.281992+0000","last_clean":"2024-04-25T15:28:25.281992+0000","last_became_active":"2024-04-25T15:28:23.647269+0000","last_became_peered":"2024-04-25T15:28:23.647269+0000","last_unstale":"2024-04-25T15:28:25.281992+0000","last_undegraded":"2024-04-25T15:28:25.281992+0000","last_fullsized":"2024-04-25T15:28:25.281992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:27.187638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649087+0000","last_change":"2024-04-25T15:28:23.641251+0000","last_active":"2024-04-25T15:28:24.649087+0000","last_peered":"2024-04-25T15:28:24.649087+0000","last_clean":"2024-04-25T15:28:24.649087+0000","last_became_active":"2024-04-25T15:28:23.640943+0000","last_became_peered":"2024-04-25T15:28:23.640943+0000","last_unstale":"2024-04-25T15:28:24.649087+0000","last_undegraded":"2024-04-25T15:28:24.649087+0000","last_fullsized":"2024-04-25T15:28:24.649087+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:34.342354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281710+0000","last_change":"2024-04-25T15:28:23.652480+0000","last_active":"2024-04-25T15:28:25.281710+0000","last_peered":"2024-04-25T15:28:25.281710+0000","last_clean":"2024-04-25T15:28:25.281710+0000","last_became_active":"2024-04-25T15:28:23.652042+0000","last_became_peered":"2024-04-25T15:28:23.652042+0000","last_unstale":"2024-04-25T15:28:25.281710+0000","last_undegraded":"2024-04-25T15:28:25.281710+0000","last_fullsized":"2024-04-25T15:28:25.281710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:26.656523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281499+0000","last_change":"2024-04-25T15:28:23.645012+0000","last_active":"2024-04-25T15:28:25.281499+0000","last_peered":"2024-04-25T15:28:25.281499+0000","last_clean":"2024-04-25T15:28:25.281499+0000","last_became_active":"2024-04-25T15:28:23.644620+0000","last_became_peered":"2024-04-25T15:28:23.644620+0000","last_unstale":"2024-04-25T15:28:25.281499+0000","last_undegraded":"2024-04-25T15:28:25.281499+0000","last_fullsized":"2024-04-25T15:28:25.281499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:25.372903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281525+0000","last_change":"2024-04-25T15:28:23.645167+0000","last_active":"2024-04-25T15:28:25.281525+0000","last_peered":"2024-04-25T15:28:25.281525+0000","last_clean":"2024-04-25T15:28:25.281525+0000","last_became_active":"2024-04-25T15:28:23.644781+0000","last_became_peered":"2024-04-25T15:28:23.644781+0000","last_unstale":"2024-04-25T15:28:25.281525+0000","last_undegraded":"2024-04-25T15:28:25.281525+0000","last_fullsized":"2024-04-25T15:28:25.281525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:25.989787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282012+0000","last_change":"2024-04-25T15:28:23.658433+0000","last_active":"2024-04-25T15:28:25.282012+0000","last_peered":"2024-04-25T15:28:25.282012+0000","last_clean":"2024-04-25T15:28:25.282012+0000","last_became_active":"2024-04-25T15:28:23.658026+0000","last_became_peered":"2024-04-25T15:28:23.658026+0000","last_unstale":"2024-04-25T15:28:25.282012+0000","last_undegraded":"2024-04-25T15:28:25.282012+0000","last_fullsized":"2024-04-25T15:28:25.282012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282414+0000","last_change":"2024-04-25T15:28:23.643202+0000","last_active":"2024-04-25T15:28:25.282414+0000","last_peered":"2024-04-25T15:28:25.282414+0000","last_clean":"2024-04-25T15:28:25.282414+0000","last_became_active":"2024-04-25T15:28:23.642274+0000","last_became_peered":"2024-04-25T15:28:23.642274+0000","last_unstale":"2024-04-25T15:28:25.282414+0000","last_undegraded":"2024-04-25T15:28:25.282414+0000","last_fullsized":"2024-04-25T15:28:25.282414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.639963+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.639463+0000","last_became_peered":"2024-04-25T15:28:23.639463+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:18.521086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282830+0000","last_change":"2024-04-25T15:28:23.649937+0000","last_active":"2024-04-25T15:28:25.282830+0000","last_peered":"2024-04-25T15:28:25.282830+0000","last_clean":"2024-04-25T15:28:25.282830+0000","last_became_active":"2024-04-25T15:28:23.649189+0000","last_became_peered":"2024-04-25T15:28:23.649189+0000","last_unstale":"2024-04-25T15:28:25.282830+0000","last_undegraded":"2024-04-25T15:28:25.282830+0000","last_fullsized":"2024-04-25T15:28:25.282830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281903+0000","last_change":"2024-04-25T15:28:23.644231+0000","last_active":"2024-04-25T15:28:25.281903+0000","last_peered":"2024-04-25T15:28:25.281903+0000","last_clean":"2024-04-25T15:28:25.281903+0000","last_became_active":"2024-04-25T15:28:23.643912+0000","last_became_peered":"2024-04-25T15:28:23.643912+0000","last_unstale":"2024-04-25T15:28:25.281903+0000","last_undegraded":"2024-04-25T15:28:25.281903+0000","last_fullsized":"2024-04-25T15:28:25.281903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:11.975463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282212+0000","last_change":"2024-04-25T15:28:23.650840+0000","last_active":"2024-04-25T15:28:25.282212+0000","last_peered":"2024-04-25T15:28:25.282212+0000","last_clean":"2024-04-25T15:28:25.282212+0000","last_became_active":"2024-04-25T15:28:23.650019+0000","last_became_peered":"2024-04-25T15:28:23.650019+0000","last_unstale":"2024-04-25T15:28:25.282212+0000","last_undegraded":"2024-04-25T15:28:25.282212+0000","last_fullsized":"2024-04-25T15:28:25.282212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:31.696277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.639188+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.638896+0000","last_became_peered":"2024-04-25T15:28:23.638896+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:12.900679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648853+0000","last_change":"2024-04-25T15:28:23.637226+0000","last_active":"2024-04-25T15:28:24.648853+0000","last_peered":"2024-04-25T15:28:24.648853+0000","last_clean":"2024-04-25T15:28:24.648853+0000","last_became_active":"2024-04-25T15:28:23.636807+0000","last_became_peered":"2024-04-25T15:28:23.636807+0000","last_unstale":"2024-04-25T15:28:24.648853+0000","last_undegraded":"2024-04-25T15:28:24.648853+0000","last_fullsized":"2024-04-25T15:28:24.648853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:21.505582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649294+0000","last_change":"2024-04-25T15:28:23.649387+0000","last_active":"2024-04-25T15:28:24.649294+0000","last_peered":"2024-04-25T15:28:24.649294+0000","last_clean":"2024-04-25T15:28:24.649294+0000","last_became_active":"2024-04-25T15:28:23.648633+0000","last_became_peered":"2024-04-25T15:28:23.648633+0000","last_unstale":"2024-04-25T15:28:24.649294+0000","last_undegraded":"2024-04-25T15:28:24.649294+0000","last_fullsized":"2024-04-25T15:28:24.649294+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:26.656523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282156+0000","last_change":"2024-04-25T15:28:23.648256+0000","last_active":"2024-04-25T15:28:25.282156+0000","last_peered":"2024-04-25T15:28:25.282156+0000","last_clean":"2024-04-25T15:28:25.282156+0000","last_became_active":"2024-04-25T15:28:23.647895+0000","last_became_peered":"2024-04-25T15:28:23.647895+0000","last_unstale":"2024-04-25T15:28:25.282156+0000","last_undegraded":"2024-04-25T15:28:25.282156+0000","last_fullsized":"2024-04-25T15:28:25.282156+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:31.405763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648935+0000","last_change":"2024-04-25T15:28:23.635903+0000","last_active":"2024-04-25T15:28:24.648935+0000","last_peered":"2024-04-25T15:28:24.648935+0000","last_clean":"2024-04-25T15:28:24.648935+0000","last_became_active":"2024-04-25T15:28:23.635506+0000","last_became_peered":"2024-04-25T15:28:23.635506+0000","last_unstale":"2024-04-25T15:28:24.648935+0000","last_undegraded":"2024-04-25T15:28:24.648935+0000","last_fullsized":"2024-04-25T15:28:24.648935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282313+0000","last_change":"2024-04-25T15:28:23.655054+0000","last_active":"2024-04-25T15:28:25.282313+0000","last_peered":"2024-04-25T15:28:25.282313+0000","last_clean":"2024-04-25T15:28:25.282313+0000","last_became_active":"2024-04-25T15:28:23.654675+0000","last_became_peered":"2024-04-25T15:28:23.654675+0000","last_unstale":"2024-04-25T15:28:25.282313+0000","last_undegraded":"2024-04-25T15:28:25.282313+0000","last_fullsized":"2024-04-25T15:28:25.282313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:29.272131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282209+0000","last_change":"2024-04-25T15:28:23.654074+0000","last_active":"2024-04-25T15:28:25.282209+0000","last_peered":"2024-04-25T15:28:25.282209+0000","last_clean":"2024-04-25T15:28:25.282209+0000","last_became_active":"2024-04-25T15:28:23.653761+0000","last_became_peered":"2024-04-25T15:28:23.653761+0000","last_unstale":"2024-04-25T15:28:25.282209+0000","last_undegraded":"2024-04-25T15:28:25.282209+0000","last_fullsized":"2024-04-25T15:28:25.282209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:28.122216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.637208+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.636801+0000","last_became_peered":"2024-04-25T15:28:23.636801+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:14.403235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282697+0000","last_change":"2024-04-25T15:28:23.646595+0000","last_active":"2024-04-25T15:28:25.282697+0000","last_peered":"2024-04-25T15:28:25.282697+0000","last_clean":"2024-04-25T15:28:25.282697+0000","last_became_active":"2024-04-25T15:28:23.646279+0000","last_became_peered":"2024-04-25T15:28:23.646279+0000","last_unstale":"2024-04-25T15:28:25.282697+0000","last_undegraded":"2024-04-25T15:28:25.282697+0000","last_fullsized":"2024-04-25T15:28:25.282697+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:28.717546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281566+0000","last_change":"2024-04-25T15:28:23.655097+0000","last_active":"2024-04-25T15:28:25.281566+0000","last_peered":"2024-04-25T15:28:25.281566+0000","last_clean":"2024-04-25T15:28:25.281566+0000","last_became_active":"2024-04-25T15:28:23.654517+0000","last_became_peered":"2024-04-25T15:28:23.654517+0000","last_unstale":"2024-04-25T15:28:25.281566+0000","last_undegraded":"2024-04-25T15:28:25.281566+0000","last_fullsized":"2024-04-25T15:28:25.281566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:43:00.233658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647616+0000","last_change":"2024-04-25T15:28:23.648614+0000","last_active":"2024-04-25T15:28:24.647616+0000","last_peered":"2024-04-25T15:28:24.647616+0000","last_clean":"2024-04-25T15:28:24.647616+0000","last_became_active":"2024-04-25T15:28:23.648286+0000","last_became_peered":"2024-04-25T15:28:23.648286+0000","last_unstale":"2024-04-25T15:28:24.647616+0000","last_undegraded":"2024-04-25T15:28:24.647616+0000","last_fullsized":"2024-04-25T15:28:24.647616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:09.027121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282242+0000","last_change":"2024-04-25T15:28:23.639343+0000","last_active":"2024-04-25T15:28:25.282242+0000","last_peered":"2024-04-25T15:28:25.282242+0000","last_clean":"2024-04-25T15:28:25.282242+0000","last_became_active":"2024-04-25T15:28:23.639037+0000","last_became_peered":"2024-04-25T15:28:23.639037+0000","last_unstale":"2024-04-25T15:28:25.282242+0000","last_undegraded":"2024-04-25T15:28:25.282242+0000","last_fullsized":"2024-04-25T15:28:25.282242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:41.056274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649139+0000","last_change":"2024-04-25T15:28:23.642254+0000","last_active":"2024-04-25T15:28:24.649139+0000","last_peered":"2024-04-25T15:28:24.649139+0000","last_clean":"2024-04-25T15:28:24.649139+0000","last_became_active":"2024-04-25T15:28:23.641906+0000","last_became_peered":"2024-04-25T15:28:23.641906+0000","last_unstale":"2024-04-25T15:28:24.649139+0000","last_undegraded":"2024-04-25T15:28:24.649139+0000","last_fullsized":"2024-04-25T15:28:24.649139+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:27.006208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282367+0000","last_change":"2024-04-25T15:28:23.647159+0000","last_active":"2024-04-25T15:28:25.282367+0000","last_peered":"2024-04-25T15:28:25.282367+0000","last_clean":"2024-04-25T15:28:25.282367+0000","last_became_active":"2024-04-25T15:28:23.646103+0000","last_became_peered":"2024-04-25T15:28:23.646103+0000","last_unstale":"2024-04-25T15:28:25.282367+0000","last_undegraded":"2024-04-25T15:28:25.282367+0000","last_fullsized":"2024-04-25T15:28:25.282367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648380+0000","last_change":"2024-04-25T15:28:23.644393+0000","last_active":"2024-04-25T15:28:24.648380+0000","last_peered":"2024-04-25T15:28:24.648380+0000","last_clean":"2024-04-25T15:28:24.648380+0000","last_became_active":"2024-04-25T15:28:23.644073+0000","last_became_peered":"2024-04-25T15:28:23.644073+0000","last_unstale":"2024-04-25T15:28:24.648380+0000","last_undegraded":"2024-04-25T15:28:24.648380+0000","last_fullsized":"2024-04-25T15:28:24.648380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:54.453734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647936+0000","last_change":"2024-04-25T15:28:23.654484+0000","last_active":"2024-04-25T15:28:24.647936+0000","last_peered":"2024-04-25T15:28:24.647936+0000","last_clean":"2024-04-25T15:28:24.647936+0000","last_became_active":"2024-04-25T15:28:23.654191+0000","last_became_peered":"2024-04-25T15:28:23.654191+0000","last_unstale":"2024-04-25T15:28:24.647936+0000","last_undegraded":"2024-04-25T15:28:24.647936+0000","last_fullsized":"2024-04-25T15:28:24.647936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:51.690159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648920+0000","last_change":"2024-04-25T15:28:23.633612+0000","last_active":"2024-04-25T15:28:24.648920+0000","last_peered":"2024-04-25T15:28:24.648920+0000","last_clean":"2024-04-25T15:28:24.648920+0000","last_became_active":"2024-04-25T15:28:23.633288+0000","last_became_peered":"2024-04-25T15:28:23.633288+0000","last_unstale":"2024-04-25T15:28:24.648920+0000","last_undegraded":"2024-04-25T15:28:24.648920+0000","last_fullsized":"2024-04-25T15:28:24.648920+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:42.312490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.643192+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.642727+0000","last_became_peered":"2024-04-25T15:28:23.642727+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649633+0000","last_change":"2024-04-25T15:28:23.643113+0000","last_active":"2024-04-25T15:28:24.649633+0000","last_peered":"2024-04-25T15:28:24.649633+0000","last_clean":"2024-04-25T15:28:24.649633+0000","last_became_active":"2024-04-25T15:28:23.637718+0000","last_became_peered":"2024-04-25T15:28:23.637718+0000","last_unstale":"2024-04-25T15:28:24.649633+0000","last_undegraded":"2024-04-25T15:28:24.649633+0000","last_fullsized":"2024-04-25T15:28:24.649633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282773+0000","last_change":"2024-04-25T15:28:22.628837+0000","last_active":"2024-04-25T15:28:25.282773+0000","last_peered":"2024-04-25T15:28:25.282773+0000","last_clean":"2024-04-25T15:28:25.282773+0000","last_became_active":"2024-04-25T15:28:22.628231+0000","last_became_peered":"2024-04-25T15:28:22.628231+0000","last_unstale":"2024-04-25T15:28:25.282773+0000","last_undegraded":"2024-04-25T15:28:25.282773+0000","last_fullsized":"2024-04-25T15:28:25.282773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:53.446196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282653+0000","last_change":"2024-04-25T15:28:23.653258+0000","last_active":"2024-04-25T15:28:25.282653+0000","last_peered":"2024-04-25T15:28:25.282653+0000","last_clean":"2024-04-25T15:28:25.282653+0000","last_became_active":"2024-04-25T15:28:23.652929+0000","last_became_peered":"2024-04-25T15:28:23.652929+0000","last_unstale":"2024-04-25T15:28:25.282653+0000","last_undegraded":"2024-04-25T15:28:25.282653+0000","last_fullsized":"2024-04-25T15:28:25.282653+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:55.065573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649356+0000","last_change":"2024-04-25T15:28:22.639382+0000","last_active":"2024-04-25T15:28:24.649356+0000","last_peered":"2024-04-25T15:28:24.649356+0000","last_clean":"2024-04-25T15:28:24.649356+0000","last_became_active":"2024-04-25T15:28:22.639130+0000","last_became_peered":"2024-04-25T15:28:22.639130+0000","last_unstale":"2024-04-25T15:28:24.649356+0000","last_undegraded":"2024-04-25T15:28:24.649356+0000","last_fullsized":"2024-04-25T15:28:24.649356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:17.513548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282579+0000","last_change":"2024-04-25T15:28:23.647366+0000","last_active":"2024-04-25T15:28:25.282579+0000","last_peered":"2024-04-25T15:28:25.282579+0000","last_clean":"2024-04-25T15:28:25.282579+0000","last_became_active":"2024-04-25T15:28:23.647027+0000","last_became_peered":"2024-04-25T15:28:23.647027+0000","last_unstale":"2024-04-25T15:28:25.282579+0000","last_undegraded":"2024-04-25T15:28:25.282579+0000","last_fullsized":"2024-04-25T15:28:25.282579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648998+0000","last_change":"2024-04-25T15:28:22.627683+0000","last_active":"2024-04-25T15:28:24.648998+0000","last_peered":"2024-04-25T15:28:24.648998+0000","last_clean":"2024-04-25T15:28:24.648998+0000","last_became_active":"2024-04-25T15:28:22.627379+0000","last_became_peered":"2024-04-25T15:28:22.627379+0000","last_unstale":"2024-04-25T15:28:24.648998+0000","last_undegraded":"2024-04-25T15:28:24.648998+0000","last_fullsized":"2024-04-25T15:28:24.648998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:30.340419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281914+0000","last_change":"2024-04-25T15:28:23.655579+0000","last_active":"2024-04-25T15:28:25.281914+0000","last_peered":"2024-04-25T15:28:25.281914+0000","last_clean":"2024-04-25T15:28:25.281914+0000","last_became_active":"2024-04-25T15:28:23.655206+0000","last_became_peered":"2024-04-25T15:28:23.655206+0000","last_unstale":"2024-04-25T15:28:25.281914+0000","last_undegraded":"2024-04-25T15:28:25.281914+0000","last_fullsized":"2024-04-25T15:28:25.281914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:16.785360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.629301+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628962+0000","last_became_peered":"2024-04-25T15:28:22.628962+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:24.982249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648121+0000","last_change":"2024-04-25T15:28:23.654406+0000","last_active":"2024-04-25T15:28:24.648121+0000","last_peered":"2024-04-25T15:28:24.648121+0000","last_clean":"2024-04-25T15:28:24.648121+0000","last_became_active":"2024-04-25T15:28:23.652920+0000","last_became_peered":"2024-04-25T15:28:23.652920+0000","last_unstale":"2024-04-25T15:28:24.648121+0000","last_undegraded":"2024-04-25T15:28:24.648121+0000","last_fullsized":"2024-04-25T15:28:24.648121+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:31.162871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649152+0000","last_change":"2024-04-25T15:28:22.633397+0000","last_active":"2024-04-25T15:28:24.649152+0000","last_peered":"2024-04-25T15:28:24.649152+0000","last_clean":"2024-04-25T15:28:24.649152+0000","last_became_active":"2024-04-25T15:28:22.633041+0000","last_became_peered":"2024-04-25T15:28:22.633041+0000","last_unstale":"2024-04-25T15:28:24.649152+0000","last_undegraded":"2024-04-25T15:28:24.649152+0000","last_fullsized":"2024-04-25T15:28:24.649152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:08.019583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648965+0000","last_change":"2024-04-25T15:28:23.645615+0000","last_active":"2024-04-25T15:28:24.648965+0000","last_peered":"2024-04-25T15:28:24.648965+0000","last_clean":"2024-04-25T15:28:24.648965+0000","last_became_active":"2024-04-25T15:28:23.645277+0000","last_became_peered":"2024-04-25T15:28:23.645277+0000","last_unstale":"2024-04-25T15:28:24.648965+0000","last_undegraded":"2024-04-25T15:28:24.648965+0000","last_fullsized":"2024-04-25T15:28:24.648965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:19.329565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648771+0000","last_change":"2024-04-25T15:28:22.639411+0000","last_active":"2024-04-25T15:28:24.648771+0000","last_peered":"2024-04-25T15:28:24.648771+0000","last_clean":"2024-04-25T15:28:24.648771+0000","last_became_active":"2024-04-25T15:28:22.639122+0000","last_became_peered":"2024-04-25T15:28:22.639122+0000","last_unstale":"2024-04-25T15:28:24.648771+0000","last_undegraded":"2024-04-25T15:28:24.648771+0000","last_fullsized":"2024-04-25T15:28:24.648771+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:00.247649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281419+0000","last_change":"2024-04-25T15:28:23.639119+0000","last_active":"2024-04-25T15:28:25.281419+0000","last_peered":"2024-04-25T15:28:25.281419+0000","last_clean":"2024-04-25T15:28:25.281419+0000","last_became_active":"2024-04-25T15:28:23.638754+0000","last_became_peered":"2024-04-25T15:28:23.638754+0000","last_unstale":"2024-04-25T15:28:25.281419+0000","last_undegraded":"2024-04-25T15:28:25.281419+0000","last_fullsized":"2024-04-25T15:28:25.281419+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648777+0000","last_change":"2024-04-25T15:28:22.631835+0000","last_active":"2024-04-25T15:28:24.648777+0000","last_peered":"2024-04-25T15:28:24.648777+0000","last_clean":"2024-04-25T15:28:24.648777+0000","last_became_active":"2024-04-25T15:28:22.631539+0000","last_became_peered":"2024-04-25T15:28:22.631539+0000","last_unstale":"2024-04-25T15:28:24.648777+0000","last_undegraded":"2024-04-25T15:28:24.648777+0000","last_fullsized":"2024-04-25T15:28:24.648777+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:18.322027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281369+0000","last_change":"2024-04-25T15:28:23.638974+0000","last_active":"2024-04-25T15:28:25.281369+0000","last_peered":"2024-04-25T15:28:25.281369+0000","last_clean":"2024-04-25T15:28:25.281369+0000","last_became_active":"2024-04-25T15:28:23.638642+0000","last_became_peered":"2024-04-25T15:28:23.638642+0000","last_unstale":"2024-04-25T15:28:25.281369+0000","last_undegraded":"2024-04-25T15:28:25.281369+0000","last_fullsized":"2024-04-25T15:28:25.281369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:06.035044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647693+0000","last_change":"2024-04-25T15:28:22.634256+0000","last_active":"2024-04-25T15:28:24.647693+0000","last_peered":"2024-04-25T15:28:24.647693+0000","last_clean":"2024-04-25T15:28:24.647693+0000","last_became_active":"2024-04-25T15:28:22.633942+0000","last_became_peered":"2024-04-25T15:28:22.633942+0000","last_unstale":"2024-04-25T15:28:24.647693+0000","last_undegraded":"2024-04-25T15:28:24.647693+0000","last_fullsized":"2024-04-25T15:28:24.647693+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:10.420589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282150+0000","last_change":"2024-04-25T15:28:23.639749+0000","last_active":"2024-04-25T15:28:25.282150+0000","last_peered":"2024-04-25T15:28:25.282150+0000","last_clean":"2024-04-25T15:28:25.282150+0000","last_became_active":"2024-04-25T15:28:23.639415+0000","last_became_peered":"2024-04-25T15:28:23.639415+0000","last_unstale":"2024-04-25T15:28:25.282150+0000","last_undegraded":"2024-04-25T15:28:25.282150+0000","last_fullsized":"2024-04-25T15:28:25.282150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:38:27.203252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282928+0000","last_change":"2024-04-25T15:28:22.628552+0000","last_active":"2024-04-25T15:28:25.282928+0000","last_peered":"2024-04-25T15:28:25.282928+0000","last_clean":"2024-04-25T15:28:25.282928+0000","last_became_active":"2024-04-25T15:28:22.627193+0000","last_became_peered":"2024-04-25T15:28:22.627193+0000","last_unstale":"2024-04-25T15:28:25.282928+0000","last_undegraded":"2024-04-25T15:28:25.282928+0000","last_fullsized":"2024-04-25T15:28:25.282928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282633+0000","last_change":"2024-04-25T15:28:23.651525+0000","last_active":"2024-04-25T15:28:25.282633+0000","last_peered":"2024-04-25T15:28:25.282633+0000","last_clean":"2024-04-25T15:28:25.282633+0000","last_became_active":"2024-04-25T15:28:23.651179+0000","last_became_peered":"2024-04-25T15:28:23.651179+0000","last_unstale":"2024-04-25T15:28:25.282633+0000","last_undegraded":"2024-04-25T15:28:25.282633+0000","last_fullsized":"2024-04-25T15:28:25.282633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282316+0000","last_change":"2024-04-25T15:28:22.631506+0000","last_active":"2024-04-25T15:28:25.282316+0000","last_peered":"2024-04-25T15:28:25.282316+0000","last_clean":"2024-04-25T15:28:25.282316+0000","last_became_active":"2024-04-25T15:28:22.631161+0000","last_became_peered":"2024-04-25T15:28:22.631161+0000","last_unstale":"2024-04-25T15:28:25.282316+0000","last_undegraded":"2024-04-25T15:28:25.282316+0000","last_fullsized":"2024-04-25T15:28:25.282316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.414479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649203+0000","last_change":"2024-04-25T15:28:23.636915+0000","last_active":"2024-04-25T15:28:24.649203+0000","last_peered":"2024-04-25T15:28:24.649203+0000","last_clean":"2024-04-25T15:28:24.649203+0000","last_became_active":"2024-04-25T15:28:23.636599+0000","last_became_peered":"2024-04-25T15:28:23.636599+0000","last_unstale":"2024-04-25T15:28:24.649203+0000","last_undegraded":"2024-04-25T15:28:24.649203+0000","last_fullsized":"2024-04-25T15:28:24.649203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281907+0000","last_change":"2024-04-25T15:28:22.628373+0000","last_active":"2024-04-25T15:28:25.281907+0000","last_peered":"2024-04-25T15:28:25.281907+0000","last_clean":"2024-04-25T15:28:25.281907+0000","last_became_active":"2024-04-25T15:28:22.628063+0000","last_became_peered":"2024-04-25T15:28:22.628063+0000","last_unstale":"2024-04-25T15:28:25.281907+0000","last_undegraded":"2024-04-25T15:28:25.281907+0000","last_fullsized":"2024-04-25T15:28:25.281907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:17.513548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649303+0000","last_change":"2024-04-25T15:28:23.640559+0000","last_active":"2024-04-25T15:28:24.649303+0000","last_peered":"2024-04-25T15:28:24.649303+0000","last_clean":"2024-04-25T15:28:24.649303+0000","last_became_active":"2024-04-25T15:28:23.638096+0000","last_became_peered":"2024-04-25T15:28:23.638096+0000","last_unstale":"2024-04-25T15:28:24.649303+0000","last_undegraded":"2024-04-25T15:28:24.649303+0000","last_fullsized":"2024-04-25T15:28:24.649303+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282803+0000","last_change":"2024-04-25T15:28:22.627717+0000","last_active":"2024-04-25T15:28:25.282803+0000","last_peered":"2024-04-25T15:28:25.282803+0000","last_clean":"2024-04-25T15:28:25.282803+0000","last_became_active":"2024-04-25T15:28:22.627402+0000","last_became_peered":"2024-04-25T15:28:22.627402+0000","last_unstale":"2024-04-25T15:28:25.282803+0000","last_undegraded":"2024-04-25T15:28:25.282803+0000","last_fullsized":"2024-04-25T15:28:25.282803+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648678+0000","last_change":"2024-04-25T15:28:23.630065+0000","last_active":"2024-04-25T15:28:24.648678+0000","last_peered":"2024-04-25T15:28:24.648678+0000","last_clean":"2024-04-25T15:28:24.648678+0000","last_became_active":"2024-04-25T15:28:23.629649+0000","last_became_peered":"2024-04-25T15:28:23.629649+0000","last_unstale":"2024-04-25T15:28:24.648678+0000","last_undegraded":"2024-04-25T15:28:24.648678+0000","last_fullsized":"2024-04-25T15:28:24.648678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648438+0000","last_change":"2024-04-25T15:28:22.632102+0000","last_active":"2024-04-25T15:28:24.648438+0000","last_peered":"2024-04-25T15:28:24.648438+0000","last_clean":"2024-04-25T15:28:24.648438+0000","last_became_active":"2024-04-25T15:28:22.631723+0000","last_became_peered":"2024-04-25T15:28:22.631723+0000","last_unstale":"2024-04-25T15:28:24.648438+0000","last_undegraded":"2024-04-25T15:28:24.648438+0000","last_fullsized":"2024-04-25T15:28:24.648438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:06.295529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282145+0000","last_change":"2024-04-25T15:28:23.639139+0000","last_active":"2024-04-25T15:28:25.282145+0000","last_peered":"2024-04-25T15:28:25.282145+0000","last_clean":"2024-04-25T15:28:25.282145+0000","last_became_active":"2024-04-25T15:28:23.638812+0000","last_became_peered":"2024-04-25T15:28:23.638812+0000","last_unstale":"2024-04-25T15:28:25.282145+0000","last_undegraded":"2024-04-25T15:28:25.282145+0000","last_fullsized":"2024-04-25T15:28:25.282145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:56.852295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281527+0000","last_change":"2024-04-25T15:28:22.631778+0000","last_active":"2024-04-25T15:28:25.281527+0000","last_peered":"2024-04-25T15:28:25.281527+0000","last_clean":"2024-04-25T15:28:25.281527+0000","last_became_active":"2024-04-25T15:28:22.631460+0000","last_became_peered":"2024-04-25T15:28:22.631460+0000","last_unstale":"2024-04-25T15:28:25.281527+0000","last_undegraded":"2024-04-25T15:28:25.281527+0000","last_fullsized":"2024-04-25T15:28:25.281527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.777821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647614+0000","last_change":"2024-04-25T15:28:23.652363+0000","last_active":"2024-04-25T15:28:24.647614+0000","last_peered":"2024-04-25T15:28:24.647614+0000","last_clean":"2024-04-25T15:28:24.647614+0000","last_became_active":"2024-04-25T15:28:23.651744+0000","last_became_peered":"2024-04-25T15:28:23.651744+0000","last_unstale":"2024-04-25T15:28:24.647614+0000","last_undegraded":"2024-04-25T15:28:24.647614+0000","last_fullsized":"2024-04-25T15:28:24.647614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:17.116922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281587+0000","last_change":"2024-04-25T15:28:22.636713+0000","last_active":"2024-04-25T15:28:25.281587+0000","last_peered":"2024-04-25T15:28:25.281587+0000","last_clean":"2024-04-25T15:28:25.281587+0000","last_became_active":"2024-04-25T15:28:22.636389+0000","last_became_peered":"2024-04-25T15:28:22.636389+0000","last_unstale":"2024-04-25T15:28:25.281587+0000","last_undegraded":"2024-04-25T15:28:25.281587+0000","last_fullsized":"2024-04-25T15:28:25.281587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647465+0000","last_change":"2024-04-25T15:28:23.647463+0000","last_active":"2024-04-25T15:28:24.647465+0000","last_peered":"2024-04-25T15:28:24.647465+0000","last_clean":"2024-04-25T15:28:24.647465+0000","last_became_active":"2024-04-25T15:28:23.647133+0000","last_became_peered":"2024-04-25T15:28:23.647133+0000","last_unstale":"2024-04-25T15:28:24.647465+0000","last_undegraded":"2024-04-25T15:28:24.647465+0000","last_fullsized":"2024-04-25T15:28:24.647465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:14.765406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.628461+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628160+0000","last_became_peered":"2024-04-25T15:28:22.628160+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282411+0000","last_change":"2024-04-25T15:28:23.649571+0000","last_active":"2024-04-25T15:28:25.282411+0000","last_peered":"2024-04-25T15:28:25.282411+0000","last_clean":"2024-04-25T15:28:25.282411+0000","last_became_active":"2024-04-25T15:28:23.649214+0000","last_became_peered":"2024-04-25T15:28:23.649214+0000","last_unstale":"2024-04-25T15:28:25.282411+0000","last_undegraded":"2024-04-25T15:28:25.282411+0000","last_fullsized":"2024-04-25T15:28:25.282411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:22.598027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283048+0000","last_change":"2024-04-25T15:28:22.631820+0000","last_active":"2024-04-25T15:28:25.283048+0000","last_peered":"2024-04-25T15:28:25.283048+0000","last_clean":"2024-04-25T15:28:25.283048+0000","last_became_active":"2024-04-25T15:28:22.631501+0000","last_became_peered":"2024-04-25T15:28:22.631501+0000","last_unstale":"2024-04-25T15:28:25.283048+0000","last_undegraded":"2024-04-25T15:28:25.283048+0000","last_fullsized":"2024-04-25T15:28:25.283048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282947+0000","last_change":"2024-04-25T15:28:23.646728+0000","last_active":"2024-04-25T15:28:25.282947+0000","last_peered":"2024-04-25T15:28:25.282947+0000","last_clean":"2024-04-25T15:28:25.282947+0000","last_became_active":"2024-04-25T15:28:23.646234+0000","last_became_peered":"2024-04-25T15:28:23.646234+0000","last_unstale":"2024-04-25T15:28:25.282947+0000","last_undegraded":"2024-04-25T15:28:25.282947+0000","last_fullsized":"2024-04-25T15:28:25.282947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648950+0000","last_change":"2024-04-25T15:28:22.632918+0000","last_active":"2024-04-25T15:28:24.648950+0000","last_peered":"2024-04-25T15:28:24.648950+0000","last_clean":"2024-04-25T15:28:24.648950+0000","last_became_active":"2024-04-25T15:28:22.632551+0000","last_became_peered":"2024-04-25T15:28:22.632551+0000","last_unstale":"2024-04-25T15:28:24.648950+0000","last_undegraded":"2024-04-25T15:28:24.648950+0000","last_fullsized":"2024-04-25T15:28:24.648950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:25.998670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649445+0000","last_change":"2024-04-25T15:28:23.641506+0000","last_active":"2024-04-25T15:28:24.649445+0000","last_peered":"2024-04-25T15:28:24.649445+0000","last_clean":"2024-04-25T15:28:24.649445+0000","last_became_active":"2024-04-25T15:28:23.640967+0000","last_became_peered":"2024-04-25T15:28:23.640967+0000","last_unstale":"2024-04-25T15:28:24.649445+0000","last_undegraded":"2024-04-25T15:28:24.649445+0000","last_fullsized":"2024-04-25T15:28:24.649445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649243+0000","last_change":"2024-04-25T15:28:22.631396+0000","last_active":"2024-04-25T15:28:24.649243+0000","last_peered":"2024-04-25T15:28:24.649243+0000","last_clean":"2024-04-25T15:28:24.649243+0000","last_became_active":"2024-04-25T15:28:22.630989+0000","last_became_peered":"2024-04-25T15:28:22.630989+0000","last_unstale":"2024-04-25T15:28:24.649243+0000","last_undegraded":"2024-04-25T15:28:24.649243+0000","last_fullsized":"2024-04-25T15:28:24.649243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:25.648985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282440+0000","last_change":"2024-04-25T15:28:23.650548+0000","last_active":"2024-04-25T15:28:25.282440+0000","last_peered":"2024-04-25T15:28:25.282440+0000","last_clean":"2024-04-25T15:28:25.282440+0000","last_became_active":"2024-04-25T15:28:23.649636+0000","last_became_peered":"2024-04-25T15:28:23.649636+0000","last_unstale":"2024-04-25T15:28:25.282440+0000","last_undegraded":"2024-04-25T15:28:25.282440+0000","last_fullsized":"2024-04-25T15:28:25.282440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:11.428127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281645+0000","last_change":"2024-04-25T15:28:22.631144+0000","last_active":"2024-04-25T15:28:25.281645+0000","last_peered":"2024-04-25T15:28:25.281645+0000","last_clean":"2024-04-25T15:28:25.281645+0000","last_became_active":"2024-04-25T15:28:22.630762+0000","last_became_peered":"2024-04-25T15:28:22.630762+0000","last_unstale":"2024-04-25T15:28:25.281645+0000","last_undegraded":"2024-04-25T15:28:25.281645+0000","last_fullsized":"2024-04-25T15:28:25.281645+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:10.967925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283204+0000","last_change":"2024-04-25T15:28:23.647208+0000","last_active":"2024-04-25T15:28:25.283204+0000","last_peered":"2024-04-25T15:28:25.283204+0000","last_clean":"2024-04-25T15:28:25.283204+0000","last_became_active":"2024-04-25T15:28:23.646871+0000","last_became_peered":"2024-04-25T15:28:23.646871+0000","last_unstale":"2024-04-25T15:28:25.283204+0000","last_undegraded":"2024-04-25T15:28:25.283204+0000","last_fullsized":"2024-04-25T15:28:25.283204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282312+0000","last_change":"2024-04-25T15:28:22.623948+0000","last_active":"2024-04-25T15:28:25.282312+0000","last_peered":"2024-04-25T15:28:25.282312+0000","last_clean":"2024-04-25T15:28:25.282312+0000","last_became_active":"2024-04-25T15:28:22.623604+0000","last_became_peered":"2024-04-25T15:28:22.623604+0000","last_unstale":"2024-04-25T15:28:25.282312+0000","last_undegraded":"2024-04-25T15:28:25.282312+0000","last_fullsized":"2024-04-25T15:28:25.282312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:13.369418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:28:23.648910+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:28:23.648524+0000","last_became_peered":"2024-04-25T15:28:23.648524+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.853221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281681+0000","last_change":"2024-04-25T15:28:22.635528+0000","last_active":"2024-04-25T15:28:25.281681+0000","last_peered":"2024-04-25T15:28:25.281681+0000","last_clean":"2024-04-25T15:28:25.281681+0000","last_became_active":"2024-04-25T15:28:22.635320+0000","last_became_peered":"2024-04-25T15:28:22.635320+0000","last_unstale":"2024-04-25T15:28:25.281681+0000","last_undegraded":"2024-04-25T15:28:25.281681+0000","last_fullsized":"2024-04-25T15:28:25.281681+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:37.967626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281434+0000","last_change":"2024-04-25T15:28:23.658467+0000","last_active":"2024-04-25T15:28:25.281434+0000","last_peered":"2024-04-25T15:28:25.281434+0000","last_clean":"2024-04-25T15:28:25.281434+0000","last_became_active":"2024-04-25T15:28:23.658051+0000","last_became_peered":"2024-04-25T15:28:23.658051+0000","last_unstale":"2024-04-25T15:28:25.281434+0000","last_undegraded":"2024-04-25T15:28:25.281434+0000","last_fullsized":"2024-04-25T15:28:25.281434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:27.006208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281183+0000","last_change":"2024-04-25T15:28:22.627923+0000","last_active":"2024-04-25T15:28:25.281183+0000","last_peered":"2024-04-25T15:28:25.281183+0000","last_clean":"2024-04-25T15:28:25.281183+0000","last_became_active":"2024-04-25T15:28:22.627595+0000","last_became_peered":"2024-04-25T15:28:22.627595+0000","last_unstale":"2024-04-25T15:28:25.281183+0000","last_undegraded":"2024-04-25T15:28:25.281183+0000","last_fullsized":"2024-04-25T15:28:25.281183+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:04.050831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649208+0000","last_change":"2024-04-25T15:28:23.650714+0000","last_active":"2024-04-25T15:28:24.649208+0000","last_peered":"2024-04-25T15:28:24.649208+0000","last_clean":"2024-04-25T15:28:24.649208+0000","last_became_active":"2024-04-25T15:28:23.650410+0000","last_became_peered":"2024-04-25T15:28:23.650410+0000","last_unstale":"2024-04-25T15:28:24.649208+0000","last_undegraded":"2024-04-25T15:28:24.649208+0000","last_fullsized":"2024-04-25T15:28:24.649208+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:25.989787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647686+0000","last_change":"2024-04-25T15:28:22.629989+0000","last_active":"2024-04-25T15:28:24.647686+0000","last_peered":"2024-04-25T15:28:24.647686+0000","last_clean":"2024-04-25T15:28:24.647686+0000","last_became_active":"2024-04-25T15:28:22.629681+0000","last_became_peered":"2024-04-25T15:28:22.629681+0000","last_unstale":"2024-04-25T15:28:24.647686+0000","last_undegraded":"2024-04-25T15:28:24.647686+0000","last_fullsized":"2024-04-25T15:28:24.647686+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:36.386426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281853+0000","last_change":"2024-04-25T15:28:23.643948+0000","last_active":"2024-04-25T15:28:25.281853+0000","last_peered":"2024-04-25T15:28:25.281853+0000","last_clean":"2024-04-25T15:28:25.281853+0000","last_became_active":"2024-04-25T15:28:23.643600+0000","last_became_peered":"2024-04-25T15:28:23.643600+0000","last_unstale":"2024-04-25T15:28:25.281853+0000","last_undegraded":"2024-04-25T15:28:25.281853+0000","last_fullsized":"2024-04-25T15:28:25.281853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648158+0000","last_change":"2024-04-25T15:28:22.630713+0000","last_active":"2024-04-25T15:28:24.648158+0000","last_peered":"2024-04-25T15:28:24.648158+0000","last_clean":"2024-04-25T15:28:24.648158+0000","last_became_active":"2024-04-25T15:28:22.627937+0000","last_became_peered":"2024-04-25T15:28:22.627937+0000","last_unstale":"2024-04-25T15:28:24.648158+0000","last_undegraded":"2024-04-25T15:28:24.648158+0000","last_fullsized":"2024-04-25T15:28:24.648158+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:34.438583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282846+0000","last_change":"2024-04-25T15:28:23.650199+0000","last_active":"2024-04-25T15:28:25.282846+0000","last_peered":"2024-04-25T15:28:25.282846+0000","last_clean":"2024-04-25T15:28:25.282846+0000","last_became_active":"2024-04-25T15:28:23.649599+0000","last_became_peered":"2024-04-25T15:28:23.649599+0000","last_unstale":"2024-04-25T15:28:25.282846+0000","last_undegraded":"2024-04-25T15:28:25.282846+0000","last_fullsized":"2024-04-25T15:28:25.282846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282154+0000","last_change":"2024-04-25T15:28:22.626743+0000","last_active":"2024-04-25T15:28:25.282154+0000","last_peered":"2024-04-25T15:28:25.282154+0000","last_clean":"2024-04-25T15:28:25.282154+0000","last_became_active":"2024-04-25T15:28:22.626434+0000","last_became_peered":"2024-04-25T15:28:22.626434+0000","last_unstale":"2024-04-25T15:28:25.282154+0000","last_undegraded":"2024-04-25T15:28:25.282154+0000","last_fullsized":"2024-04-25T15:28:25.282154+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:13.395697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649497+0000","last_change":"2024-04-25T15:28:23.633566+0000","last_active":"2024-04-25T15:28:24.649497+0000","last_peered":"2024-04-25T15:28:24.649497+0000","last_clean":"2024-04-25T15:28:24.649497+0000","last_became_active":"2024-04-25T15:28:23.633245+0000","last_became_peered":"2024-04-25T15:28:23.633245+0000","last_unstale":"2024-04-25T15:28:24.649497+0000","last_undegraded":"2024-04-25T15:28:24.649497+0000","last_fullsized":"2024-04-25T15:28:24.649497+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282910+0000","last_change":"2024-04-25T15:28:22.630470+0000","last_active":"2024-04-25T15:28:25.282910+0000","last_peered":"2024-04-25T15:28:25.282910+0000","last_clean":"2024-04-25T15:28:25.282910+0000","last_became_active":"2024-04-25T15:28:22.630069+0000","last_became_peered":"2024-04-25T15:28:22.630069+0000","last_unstale":"2024-04-25T15:28:25.282910+0000","last_undegraded":"2024-04-25T15:28:25.282910+0000","last_fullsized":"2024-04-25T15:28:25.282910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:30.155333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282356+0000","last_change":"2024-04-25T15:28:23.648809+0000","last_active":"2024-04-25T15:28:25.282356+0000","last_peered":"2024-04-25T15:28:25.282356+0000","last_clean":"2024-04-25T15:28:25.282356+0000","last_became_active":"2024-04-25T15:28:23.648507+0000","last_became_peered":"2024-04-25T15:28:23.648507+0000","last_unstale":"2024-04-25T15:28:25.282356+0000","last_undegraded":"2024-04-25T15:28:25.282356+0000","last_fullsized":"2024-04-25T15:28:25.282356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648878+0000","last_change":"2024-04-25T15:28:22.628170+0000","last_active":"2024-04-25T15:28:24.648878+0000","last_peered":"2024-04-25T15:28:24.648878+0000","last_clean":"2024-04-25T15:28:24.648878+0000","last_became_active":"2024-04-25T15:28:22.627191+0000","last_became_peered":"2024-04-25T15:28:22.627191+0000","last_unstale":"2024-04-25T15:28:24.648878+0000","last_undegraded":"2024-04-25T15:28:24.648878+0000","last_fullsized":"2024-04-25T15:28:24.648878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:31.302274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648932+0000","last_change":"2024-04-25T15:28:23.630876+0000","last_active":"2024-04-25T15:28:24.648932+0000","last_peered":"2024-04-25T15:28:24.648932+0000","last_clean":"2024-04-25T15:28:24.648932+0000","last_became_active":"2024-04-25T15:28:23.630558+0000","last_became_peered":"2024-04-25T15:28:23.630558+0000","last_unstale":"2024-04-25T15:28:24.648932+0000","last_undegraded":"2024-04-25T15:28:24.648932+0000","last_fullsized":"2024-04-25T15:28:24.648932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649387+0000","last_change":"2024-04-25T15:28:22.630238+0000","last_active":"2024-04-25T15:28:24.649387+0000","last_peered":"2024-04-25T15:28:24.649387+0000","last_clean":"2024-04-25T15:28:24.649387+0000","last_became_active":"2024-04-25T15:28:22.629922+0000","last_became_peered":"2024-04-25T15:28:22.629922+0000","last_unstale":"2024-04-25T15:28:24.649387+0000","last_undegraded":"2024-04-25T15:28:24.649387+0000","last_fullsized":"2024-04-25T15:28:24.649387+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:13.757868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281519+0000","last_change":"2024-04-25T15:28:23.640269+0000","last_active":"2024-04-25T15:28:25.281519+0000","last_peered":"2024-04-25T15:28:25.281519+0000","last_clean":"2024-04-25T15:28:25.281519+0000","last_became_active":"2024-04-25T15:28:23.639944+0000","last_became_peered":"2024-04-25T15:28:23.639944+0000","last_unstale":"2024-04-25T15:28:25.281519+0000","last_undegraded":"2024-04-25T15:28:25.281519+0000","last_fullsized":"2024-04-25T15:28:25.281519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282853+0000","last_change":"2024-04-25T15:28:22.626538+0000","last_active":"2024-04-25T15:28:25.282853+0000","last_peered":"2024-04-25T15:28:25.282853+0000","last_clean":"2024-04-25T15:28:25.282853+0000","last_became_active":"2024-04-25T15:28:22.626207+0000","last_became_peered":"2024-04-25T15:28:22.626207+0000","last_unstale":"2024-04-25T15:28:25.282853+0000","last_undegraded":"2024-04-25T15:28:25.282853+0000","last_fullsized":"2024-04-25T15:28:25.282853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:50.682621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282764+0000","last_change":"2024-04-25T15:28:23.646030+0000","last_active":"2024-04-25T15:28:25.282764+0000","last_peered":"2024-04-25T15:28:25.282764+0000","last_clean":"2024-04-25T15:28:25.282764+0000","last_became_active":"2024-04-25T15:28:23.645733+0000","last_became_peered":"2024-04-25T15:28:23.645733+0000","last_unstale":"2024-04-25T15:28:25.282764+0000","last_undegraded":"2024-04-25T15:28:25.282764+0000","last_fullsized":"2024-04-25T15:28:25.282764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648741+0000","last_change":"2024-04-25T15:28:22.632573+0000","last_active":"2024-04-25T15:28:24.648741+0000","last_peered":"2024-04-25T15:28:24.648741+0000","last_clean":"2024-04-25T15:28:24.648741+0000","last_became_active":"2024-04-25T15:28:22.632220+0000","last_became_peered":"2024-04-25T15:28:22.632220+0000","last_unstale":"2024-04-25T15:28:24.648741+0000","last_undegraded":"2024-04-25T15:28:24.648741+0000","last_fullsized":"2024-04-25T15:28:24.648741+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:16.109383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648545+0000","last_change":"2024-04-25T15:28:23.640057+0000","last_active":"2024-04-25T15:28:24.648545+0000","last_peered":"2024-04-25T15:28:24.648545+0000","last_clean":"2024-04-25T15:28:24.648545+0000","last_became_active":"2024-04-25T15:28:23.639747+0000","last_became_peered":"2024-04-25T15:28:23.639747+0000","last_unstale":"2024-04-25T15:28:24.648545+0000","last_undegraded":"2024-04-25T15:28:24.648545+0000","last_fullsized":"2024-04-25T15:28:24.648545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:06.046345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281905+0000","last_change":"2024-04-25T15:28:22.632108+0000","last_active":"2024-04-25T15:28:25.281905+0000","last_peered":"2024-04-25T15:28:25.281905+0000","last_clean":"2024-04-25T15:28:25.281905+0000","last_became_active":"2024-04-25T15:28:22.631768+0000","last_became_peered":"2024-04-25T15:28:22.631768+0000","last_unstale":"2024-04-25T15:28:25.281905+0000","last_undegraded":"2024-04-25T15:28:25.281905+0000","last_fullsized":"2024-04-25T15:28:25.281905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":218,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1581056,"data_stored":1007311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":418507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":416873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":418485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":417455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1581056,"data_stored":1007275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:28:29.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:28:29.245 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:28:32.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:28:32.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.334+0000 7fdfc56af640 1 Processor -- start 2024-04-25T15:28:32.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.334+0000 7fdfc56af640 1 -- start start 2024-04-25T15:28:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfc56af640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfc01434d0 0x7fdfc0070480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfc56af640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdfc01612f0 0x7fdfc00d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfc56af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc00709c0 con 0x7fdfc00d0ed0 2024-04-25T15:28:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfc56af640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc01438e0 con 0x7fdfc01434d0 2024-04-25T15:28:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfc56af640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc006f3e0 con 0x7fdfc01612f0 2024-04-25T15:28:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdfc01434d0 0x7fdfc0070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbe7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdfc00d0ed0 0x7fdfc0160be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60466/0 (socket says 172.21.15.67:60466) 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbe7fc640 1 -- 172.21.15.67:0/1458063240 learned_addr learned my addr 172.21.15.67:0/1458063240 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbf7fe640 1 --2- 172.21.15.67:0/1458063240 >> v2:172.21.15.176:3300/0 conn(0x7fdfc01612f0 0x7fdfc00d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbeffd640 1 -- 172.21.15.67:0/1458063240 >> v2:172.21.15.176:3300/0 conn(0x7fdfc01612f0 msgr2=0x7fdfc00d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbeffd640 1 --2- 172.21.15.67:0/1458063240 >> v2:172.21.15.176:3300/0 conn(0x7fdfc01612f0 0x7fdfc00d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbeffd640 1 -- 172.21.15.67:0/1458063240 >> v1:172.21.15.67:6789/0 conn(0x7fdfc00d0ed0 legacy=0x7fdfc0160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbdffb640 1 -- 172.21.15.67:0/1458063240 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214367279 0 0) 0x7fdfc00709c0 con 0x7fdfc00d0ed0 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbeffd640 1 -- 172.21.15.67:0/1458063240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc006f6c0 con 0x7fdfc01434d0 2024-04-25T15:28:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.336+0000 7fdfbf7fe640 1 --2- 172.21.15.67:0/1458063240 >> v2:172.21.15.176:3300/0 conn(0x7fdfc01612f0 0x7fdfc00d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.337+0000 7fdfbeffd640 1 --2- 172.21.15.67:0/1458063240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc01434d0 0x7fdfc0070480 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80097a0 tx=0x7fdfb802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb05c90205bd30c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.337+0000 7fdfbdffb640 1 -- 172.21.15.67:0/1458063240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8037070 con 0x7fdfc01434d0 2024-04-25T15:28:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.337+0000 7fdfbdffb640 1 -- 172.21.15.67:0/1458063240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb8032440 con 0x7fdfc01434d0 2024-04-25T15:28:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.337+0000 7fdfbdffb640 1 -- 172.21.15.67:0/1458063240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8032810 con 0x7fdfc01434d0 2024-04-25T15:28:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 -- 172.21.15.67:0/1458063240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc01434d0 msgr2=0x7fdfc0070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 --2- 172.21.15.67:0/1458063240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc01434d0 0x7fdfc0070480 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80097a0 tx=0x7fdfb802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 -- 172.21.15.67:0/1458063240 shutdown_connections 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 --2- 172.21.15.67:0/1458063240 >> v2:172.21.15.176:3300/0 conn(0x7fdfc01612f0 0x7fdfc00d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 --2- 172.21.15.67:0/1458063240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc01434d0 0x7fdfc0070480 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 -- 172.21.15.67:0/1458063240 >> 172.21.15.67:0/1458063240 conn(0x7fdfc00d1ae0 msgr2=0x7fdfc00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 -- 172.21.15.67:0/1458063240 shutdown_connections 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.338+0000 7fdfc56af640 1 -- 172.21.15.67:0/1458063240 wait complete. 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.340+0000 7fdfc56af640 1 Processor -- start 2024-04-25T15:28:32.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.340+0000 7fdfc56af640 1 -- start start 2024-04-25T15:28:32.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfc56af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfc00d0ed0 0x7fdfc0143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfc56af640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfc01434d0 0x7fdfc015a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfc56af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc006ce50 0x7fdfc006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfc56af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc006f090 con 0x7fdfc006ce50 2024-04-25T15:28:32.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfc56af640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc006fc80 con 0x7fdfc01434d0 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfc56af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdfc00b51a0 con 0x7fdfc00d0ed0 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfbf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc006ce50 0x7fdfc006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfbf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc006ce50 0x7fdfc006d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57012/0 (socket says 172.21.15.67:57012) 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.341+0000 7fdfbf7fe640 1 -- 172.21.15.67:0/3714552763 learned_addr learned my addr 172.21.15.67:0/3714552763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbe7fc640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfc01434d0 0x7fdfc015a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbeffd640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfc00d0ed0 0x7fdfc0143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbf7fe640 1 -- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfc00d0ed0 msgr2=0x7fdfc0143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbf7fe640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfc00d0ed0 0x7fdfc0143a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbf7fe640 1 -- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfc01434d0 msgr2=0x7fdfc015a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbf7fe640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfc01434d0 0x7fdfc015a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbf7fe640 1 -- 172.21.15.67:0/3714552763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc0144ce0 con 0x7fdfc006ce50 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfbf7fe640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc006ce50 0x7fdfc006d260 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000cd50 tx=0x7fdfb000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfb77fe640 1 -- 172.21.15.67:0/3714552763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000f680 con 0x7fdfc006ce50 2024-04-25T15:28:32.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfb77fe640 1 -- 172.21.15.67:0/3714552763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb000f840 con 0x7fdfc006ce50 2024-04-25T15:28:32.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc006d7a0 con 0x7fdfc006ce50 2024-04-25T15:28:32.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfb77fe640 1 -- 172.21.15.67:0/3714552763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000fc30 con 0x7fdfc006ce50 2024-04-25T15:28:32.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.342+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc0144720 con 0x7fdfc006ce50 2024-04-25T15:28:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.344+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc0050740 con 0x7fdfc006ce50 2024-04-25T15:28:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.344+0000 7fdfb77fe640 1 -- 172.21.15.67:0/3714552763 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdfb00213f0 con 0x7fdfc006ce50 2024-04-25T15:28:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.344+0000 7fdfb77fe640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfa00744b0 0x7fdfa00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.345+0000 7fdfb77fe640 1 -- 172.21.15.67:0/3714552763 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fdfb0095e00 con 0x7fdfc006ce50 2024-04-25T15:28:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.345+0000 7fdfbeffd640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfa00744b0 0x7fdfa00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.345+0000 7fdfbeffd640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfa00744b0 0x7fdfa00769c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8034000 tx=0x7fdfb802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.349+0000 7fdfb77fe640 1 -- 172.21.15.67:0/3714552763 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdfc006d7a0 con 0x7fdfc006ce50 2024-04-25T15:28:32.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.530+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdfc00b4600 con 0x7fdfa00744b0 2024-04-25T15:28:32.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.537+0000 7fdfb77fe640 1 -- 172.21.15.67:0/3714552763 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252062 (secure 0 0 0) 0x7fdfc00b4600 con 0x7fdfa00744b0 2024-04-25T15:28:32.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfa00744b0 msgr2=0x7fdfa00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfa00744b0 0x7fdfa00769c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8034000 tx=0x7fdfb802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc006ce50 msgr2=0x7fdfc006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc006ce50 0x7fdfc006d260 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000cd50 tx=0x7fdfb000a650 comp rx=0 tx=0).stop 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 shutdown_connections 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdfa00744b0 0x7fdfa00769c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdfc006ce50 0x7fdfc006d260 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfc01434d0 0x7fdfc015a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 --2- 172.21.15.67:0/3714552763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfc00d0ed0 0x7fdfc0143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 >> 172.21.15.67:0/3714552763 conn(0x7fdfc00d1ae0 msgr2=0x7fdfc015d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 shutdown_connections 2024-04-25T15:28:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.540+0000 7fdfc56af640 1 -- 172.21.15.67:0/3714552763 wait complete. 2024-04-25T15:28:32.552 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-04-25T15:28:30.840164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233416,"kb_used_data":8904,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754741304,"statfs":{"total":773094113280,"available":772855095296,"internally_reserved":0,"allocated":9117696,"data_stored":4520540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":229756465},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002188"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282682+0000","last_change":"2024-04-25T15:28:23.652205+0000","last_active":"2024-04-25T15:28:25.282682+0000","last_peered":"2024-04-25T15:28:25.282682+0000","last_clean":"2024-04-25T15:28:25.282682+0000","last_became_active":"2024-04-25T15:28:23.651726+0000","last_became_peered":"2024-04-25T15:28:23.651726+0000","last_unstale":"2024-04-25T15:28:25.282682+0000","last_undegraded":"2024-04-25T15:28:25.282682+0000","last_fullsized":"2024-04-25T15:28:25.282682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:46.911832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648103+0000","last_change":"2024-04-25T15:28:23.649010+0000","last_active":"2024-04-25T15:28:24.648103+0000","last_peered":"2024-04-25T15:28:24.648103+0000","last_clean":"2024-04-25T15:28:24.648103+0000","last_became_active":"2024-04-25T15:28:23.648637+0000","last_became_peered":"2024-04-25T15:28:23.648637+0000","last_unstale":"2024-04-25T15:28:24.648103+0000","last_undegraded":"2024-04-25T15:28:24.648103+0000","last_fullsized":"2024-04-25T15:28:24.648103+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:28.422017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281638+0000","last_change":"2024-04-25T15:28:23.643910+0000","last_active":"2024-04-25T15:28:25.281638+0000","last_peered":"2024-04-25T15:28:25.281638+0000","last_clean":"2024-04-25T15:28:25.281638+0000","last_became_active":"2024-04-25T15:28:23.643580+0000","last_became_peered":"2024-04-25T15:28:23.643580+0000","last_unstale":"2024-04-25T15:28:25.281638+0000","last_undegraded":"2024-04-25T15:28:25.281638+0000","last_fullsized":"2024-04-25T15:28:25.281638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281992+0000","last_change":"2024-04-25T15:28:23.649272+0000","last_active":"2024-04-25T15:28:25.281992+0000","last_peered":"2024-04-25T15:28:25.281992+0000","last_clean":"2024-04-25T15:28:25.281992+0000","last_became_active":"2024-04-25T15:28:23.647269+0000","last_became_peered":"2024-04-25T15:28:23.647269+0000","last_unstale":"2024-04-25T15:28:25.281992+0000","last_undegraded":"2024-04-25T15:28:25.281992+0000","last_fullsized":"2024-04-25T15:28:25.281992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:27.187638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649087+0000","last_change":"2024-04-25T15:28:23.641251+0000","last_active":"2024-04-25T15:28:24.649087+0000","last_peered":"2024-04-25T15:28:24.649087+0000","last_clean":"2024-04-25T15:28:24.649087+0000","last_became_active":"2024-04-25T15:28:23.640943+0000","last_became_peered":"2024-04-25T15:28:23.640943+0000","last_unstale":"2024-04-25T15:28:24.649087+0000","last_undegraded":"2024-04-25T15:28:24.649087+0000","last_fullsized":"2024-04-25T15:28:24.649087+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:34.342354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281710+0000","last_change":"2024-04-25T15:28:23.652480+0000","last_active":"2024-04-25T15:28:25.281710+0000","last_peered":"2024-04-25T15:28:25.281710+0000","last_clean":"2024-04-25T15:28:25.281710+0000","last_became_active":"2024-04-25T15:28:23.652042+0000","last_became_peered":"2024-04-25T15:28:23.652042+0000","last_unstale":"2024-04-25T15:28:25.281710+0000","last_undegraded":"2024-04-25T15:28:25.281710+0000","last_fullsized":"2024-04-25T15:28:25.281710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:26.656523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281499+0000","last_change":"2024-04-25T15:28:23.645012+0000","last_active":"2024-04-25T15:28:25.281499+0000","last_peered":"2024-04-25T15:28:25.281499+0000","last_clean":"2024-04-25T15:28:25.281499+0000","last_became_active":"2024-04-25T15:28:23.644620+0000","last_became_peered":"2024-04-25T15:28:23.644620+0000","last_unstale":"2024-04-25T15:28:25.281499+0000","last_undegraded":"2024-04-25T15:28:25.281499+0000","last_fullsized":"2024-04-25T15:28:25.281499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:25.372903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281525+0000","last_change":"2024-04-25T15:28:23.645167+0000","last_active":"2024-04-25T15:28:25.281525+0000","last_peered":"2024-04-25T15:28:25.281525+0000","last_clean":"2024-04-25T15:28:25.281525+0000","last_became_active":"2024-04-25T15:28:23.644781+0000","last_became_peered":"2024-04-25T15:28:23.644781+0000","last_unstale":"2024-04-25T15:28:25.281525+0000","last_undegraded":"2024-04-25T15:28:25.281525+0000","last_fullsized":"2024-04-25T15:28:25.281525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:25.989787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282012+0000","last_change":"2024-04-25T15:28:23.658433+0000","last_active":"2024-04-25T15:28:25.282012+0000","last_peered":"2024-04-25T15:28:25.282012+0000","last_clean":"2024-04-25T15:28:25.282012+0000","last_became_active":"2024-04-25T15:28:23.658026+0000","last_became_peered":"2024-04-25T15:28:23.658026+0000","last_unstale":"2024-04-25T15:28:25.282012+0000","last_undegraded":"2024-04-25T15:28:25.282012+0000","last_fullsized":"2024-04-25T15:28:25.282012+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282414+0000","last_change":"2024-04-25T15:28:23.643202+0000","last_active":"2024-04-25T15:28:25.282414+0000","last_peered":"2024-04-25T15:28:25.282414+0000","last_clean":"2024-04-25T15:28:25.282414+0000","last_became_active":"2024-04-25T15:28:23.642274+0000","last_became_peered":"2024-04-25T15:28:23.642274+0000","last_unstale":"2024-04-25T15:28:25.282414+0000","last_undegraded":"2024-04-25T15:28:25.282414+0000","last_fullsized":"2024-04-25T15:28:25.282414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.639963+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.639463+0000","last_became_peered":"2024-04-25T15:28:23.639463+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:18.521086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282830+0000","last_change":"2024-04-25T15:28:23.649937+0000","last_active":"2024-04-25T15:28:25.282830+0000","last_peered":"2024-04-25T15:28:25.282830+0000","last_clean":"2024-04-25T15:28:25.282830+0000","last_became_active":"2024-04-25T15:28:23.649189+0000","last_became_peered":"2024-04-25T15:28:23.649189+0000","last_unstale":"2024-04-25T15:28:25.282830+0000","last_undegraded":"2024-04-25T15:28:25.282830+0000","last_fullsized":"2024-04-25T15:28:25.282830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281903+0000","last_change":"2024-04-25T15:28:23.644231+0000","last_active":"2024-04-25T15:28:25.281903+0000","last_peered":"2024-04-25T15:28:25.281903+0000","last_clean":"2024-04-25T15:28:25.281903+0000","last_became_active":"2024-04-25T15:28:23.643912+0000","last_became_peered":"2024-04-25T15:28:23.643912+0000","last_unstale":"2024-04-25T15:28:25.281903+0000","last_undegraded":"2024-04-25T15:28:25.281903+0000","last_fullsized":"2024-04-25T15:28:25.281903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:11.975463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282212+0000","last_change":"2024-04-25T15:28:23.650840+0000","last_active":"2024-04-25T15:28:25.282212+0000","last_peered":"2024-04-25T15:28:25.282212+0000","last_clean":"2024-04-25T15:28:25.282212+0000","last_became_active":"2024-04-25T15:28:23.650019+0000","last_became_peered":"2024-04-25T15:28:23.650019+0000","last_unstale":"2024-04-25T15:28:25.282212+0000","last_undegraded":"2024-04-25T15:28:25.282212+0000","last_fullsized":"2024-04-25T15:28:25.282212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:31.696277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.639188+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.638896+0000","last_became_peered":"2024-04-25T15:28:23.638896+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:12.900679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648853+0000","last_change":"2024-04-25T15:28:23.637226+0000","last_active":"2024-04-25T15:28:24.648853+0000","last_peered":"2024-04-25T15:28:24.648853+0000","last_clean":"2024-04-25T15:28:24.648853+0000","last_became_active":"2024-04-25T15:28:23.636807+0000","last_became_peered":"2024-04-25T15:28:23.636807+0000","last_unstale":"2024-04-25T15:28:24.648853+0000","last_undegraded":"2024-04-25T15:28:24.648853+0000","last_fullsized":"2024-04-25T15:28:24.648853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:21.505582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649294+0000","last_change":"2024-04-25T15:28:23.649387+0000","last_active":"2024-04-25T15:28:24.649294+0000","last_peered":"2024-04-25T15:28:24.649294+0000","last_clean":"2024-04-25T15:28:24.649294+0000","last_became_active":"2024-04-25T15:28:23.648633+0000","last_became_peered":"2024-04-25T15:28:23.648633+0000","last_unstale":"2024-04-25T15:28:24.649294+0000","last_undegraded":"2024-04-25T15:28:24.649294+0000","last_fullsized":"2024-04-25T15:28:24.649294+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:26.656523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282156+0000","last_change":"2024-04-25T15:28:23.648256+0000","last_active":"2024-04-25T15:28:25.282156+0000","last_peered":"2024-04-25T15:28:25.282156+0000","last_clean":"2024-04-25T15:28:25.282156+0000","last_became_active":"2024-04-25T15:28:23.647895+0000","last_became_peered":"2024-04-25T15:28:23.647895+0000","last_unstale":"2024-04-25T15:28:25.282156+0000","last_undegraded":"2024-04-25T15:28:25.282156+0000","last_fullsized":"2024-04-25T15:28:25.282156+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:31.405763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648935+0000","last_change":"2024-04-25T15:28:23.635903+0000","last_active":"2024-04-25T15:28:24.648935+0000","last_peered":"2024-04-25T15:28:24.648935+0000","last_clean":"2024-04-25T15:28:24.648935+0000","last_became_active":"2024-04-25T15:28:23.635506+0000","last_became_peered":"2024-04-25T15:28:23.635506+0000","last_unstale":"2024-04-25T15:28:24.648935+0000","last_undegraded":"2024-04-25T15:28:24.648935+0000","last_fullsized":"2024-04-25T15:28:24.648935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282313+0000","last_change":"2024-04-25T15:28:23.655054+0000","last_active":"2024-04-25T15:28:25.282313+0000","last_peered":"2024-04-25T15:28:25.282313+0000","last_clean":"2024-04-25T15:28:25.282313+0000","last_became_active":"2024-04-25T15:28:23.654675+0000","last_became_peered":"2024-04-25T15:28:23.654675+0000","last_unstale":"2024-04-25T15:28:25.282313+0000","last_undegraded":"2024-04-25T15:28:25.282313+0000","last_fullsized":"2024-04-25T15:28:25.282313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:29.272131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282209+0000","last_change":"2024-04-25T15:28:23.654074+0000","last_active":"2024-04-25T15:28:25.282209+0000","last_peered":"2024-04-25T15:28:25.282209+0000","last_clean":"2024-04-25T15:28:25.282209+0000","last_became_active":"2024-04-25T15:28:23.653761+0000","last_became_peered":"2024-04-25T15:28:23.653761+0000","last_unstale":"2024-04-25T15:28:25.282209+0000","last_undegraded":"2024-04-25T15:28:25.282209+0000","last_fullsized":"2024-04-25T15:28:25.282209+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:28.122216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647768+0000","last_change":"2024-04-25T15:28:23.637208+0000","last_active":"2024-04-25T15:28:24.647768+0000","last_peered":"2024-04-25T15:28:24.647768+0000","last_clean":"2024-04-25T15:28:24.647768+0000","last_became_active":"2024-04-25T15:28:23.636801+0000","last_became_peered":"2024-04-25T15:28:23.636801+0000","last_unstale":"2024-04-25T15:28:24.647768+0000","last_undegraded":"2024-04-25T15:28:24.647768+0000","last_fullsized":"2024-04-25T15:28:24.647768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:14.403235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282697+0000","last_change":"2024-04-25T15:28:23.646595+0000","last_active":"2024-04-25T15:28:25.282697+0000","last_peered":"2024-04-25T15:28:25.282697+0000","last_clean":"2024-04-25T15:28:25.282697+0000","last_became_active":"2024-04-25T15:28:23.646279+0000","last_became_peered":"2024-04-25T15:28:23.646279+0000","last_unstale":"2024-04-25T15:28:25.282697+0000","last_undegraded":"2024-04-25T15:28:25.282697+0000","last_fullsized":"2024-04-25T15:28:25.282697+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:28.717546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281566+0000","last_change":"2024-04-25T15:28:23.655097+0000","last_active":"2024-04-25T15:28:25.281566+0000","last_peered":"2024-04-25T15:28:25.281566+0000","last_clean":"2024-04-25T15:28:25.281566+0000","last_became_active":"2024-04-25T15:28:23.654517+0000","last_became_peered":"2024-04-25T15:28:23.654517+0000","last_unstale":"2024-04-25T15:28:25.281566+0000","last_undegraded":"2024-04-25T15:28:25.281566+0000","last_fullsized":"2024-04-25T15:28:25.281566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:43:00.233658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647616+0000","last_change":"2024-04-25T15:28:23.648614+0000","last_active":"2024-04-25T15:28:24.647616+0000","last_peered":"2024-04-25T15:28:24.647616+0000","last_clean":"2024-04-25T15:28:24.647616+0000","last_became_active":"2024-04-25T15:28:23.648286+0000","last_became_peered":"2024-04-25T15:28:23.648286+0000","last_unstale":"2024-04-25T15:28:24.647616+0000","last_undegraded":"2024-04-25T15:28:24.647616+0000","last_fullsized":"2024-04-25T15:28:24.647616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:09.027121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282242+0000","last_change":"2024-04-25T15:28:23.639343+0000","last_active":"2024-04-25T15:28:25.282242+0000","last_peered":"2024-04-25T15:28:25.282242+0000","last_clean":"2024-04-25T15:28:25.282242+0000","last_became_active":"2024-04-25T15:28:23.639037+0000","last_became_peered":"2024-04-25T15:28:23.639037+0000","last_unstale":"2024-04-25T15:28:25.282242+0000","last_undegraded":"2024-04-25T15:28:25.282242+0000","last_fullsized":"2024-04-25T15:28:25.282242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:41.056274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649139+0000","last_change":"2024-04-25T15:28:23.642254+0000","last_active":"2024-04-25T15:28:24.649139+0000","last_peered":"2024-04-25T15:28:24.649139+0000","last_clean":"2024-04-25T15:28:24.649139+0000","last_became_active":"2024-04-25T15:28:23.641906+0000","last_became_peered":"2024-04-25T15:28:23.641906+0000","last_unstale":"2024-04-25T15:28:24.649139+0000","last_undegraded":"2024-04-25T15:28:24.649139+0000","last_fullsized":"2024-04-25T15:28:24.649139+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:27.006208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282367+0000","last_change":"2024-04-25T15:28:23.647159+0000","last_active":"2024-04-25T15:28:25.282367+0000","last_peered":"2024-04-25T15:28:25.282367+0000","last_clean":"2024-04-25T15:28:25.282367+0000","last_became_active":"2024-04-25T15:28:23.646103+0000","last_became_peered":"2024-04-25T15:28:23.646103+0000","last_unstale":"2024-04-25T15:28:25.282367+0000","last_undegraded":"2024-04-25T15:28:25.282367+0000","last_fullsized":"2024-04-25T15:28:25.282367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648380+0000","last_change":"2024-04-25T15:28:23.644393+0000","last_active":"2024-04-25T15:28:24.648380+0000","last_peered":"2024-04-25T15:28:24.648380+0000","last_clean":"2024-04-25T15:28:24.648380+0000","last_became_active":"2024-04-25T15:28:23.644073+0000","last_became_peered":"2024-04-25T15:28:23.644073+0000","last_unstale":"2024-04-25T15:28:24.648380+0000","last_undegraded":"2024-04-25T15:28:24.648380+0000","last_fullsized":"2024-04-25T15:28:24.648380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:54.453734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647936+0000","last_change":"2024-04-25T15:28:23.654484+0000","last_active":"2024-04-25T15:28:24.647936+0000","last_peered":"2024-04-25T15:28:24.647936+0000","last_clean":"2024-04-25T15:28:24.647936+0000","last_became_active":"2024-04-25T15:28:23.654191+0000","last_became_peered":"2024-04-25T15:28:23.654191+0000","last_unstale":"2024-04-25T15:28:24.647936+0000","last_undegraded":"2024-04-25T15:28:24.647936+0000","last_fullsized":"2024-04-25T15:28:24.647936+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:51.690159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648920+0000","last_change":"2024-04-25T15:28:23.633612+0000","last_active":"2024-04-25T15:28:24.648920+0000","last_peered":"2024-04-25T15:28:24.648920+0000","last_clean":"2024-04-25T15:28:24.648920+0000","last_became_active":"2024-04-25T15:28:23.633288+0000","last_became_peered":"2024-04-25T15:28:23.633288+0000","last_unstale":"2024-04-25T15:28:24.648920+0000","last_undegraded":"2024-04-25T15:28:24.648920+0000","last_fullsized":"2024-04-25T15:28:24.648920+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:33:42.312490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281367+0000","last_change":"2024-04-25T15:28:23.643192+0000","last_active":"2024-04-25T15:28:25.281367+0000","last_peered":"2024-04-25T15:28:25.281367+0000","last_clean":"2024-04-25T15:28:25.281367+0000","last_became_active":"2024-04-25T15:28:23.642727+0000","last_became_peered":"2024-04-25T15:28:23.642727+0000","last_unstale":"2024-04-25T15:28:25.281367+0000","last_undegraded":"2024-04-25T15:28:25.281367+0000","last_fullsized":"2024-04-25T15:28:25.281367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649633+0000","last_change":"2024-04-25T15:28:23.643113+0000","last_active":"2024-04-25T15:28:24.649633+0000","last_peered":"2024-04-25T15:28:24.649633+0000","last_clean":"2024-04-25T15:28:24.649633+0000","last_became_active":"2024-04-25T15:28:23.637718+0000","last_became_peered":"2024-04-25T15:28:23.637718+0000","last_unstale":"2024-04-25T15:28:24.649633+0000","last_undegraded":"2024-04-25T15:28:24.649633+0000","last_fullsized":"2024-04-25T15:28:24.649633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282773+0000","last_change":"2024-04-25T15:28:22.628837+0000","last_active":"2024-04-25T15:28:25.282773+0000","last_peered":"2024-04-25T15:28:25.282773+0000","last_clean":"2024-04-25T15:28:25.282773+0000","last_became_active":"2024-04-25T15:28:22.628231+0000","last_became_peered":"2024-04-25T15:28:22.628231+0000","last_unstale":"2024-04-25T15:28:25.282773+0000","last_undegraded":"2024-04-25T15:28:25.282773+0000","last_fullsized":"2024-04-25T15:28:25.282773+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:53.446196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282653+0000","last_change":"2024-04-25T15:28:23.653258+0000","last_active":"2024-04-25T15:28:25.282653+0000","last_peered":"2024-04-25T15:28:25.282653+0000","last_clean":"2024-04-25T15:28:25.282653+0000","last_became_active":"2024-04-25T15:28:23.652929+0000","last_became_peered":"2024-04-25T15:28:23.652929+0000","last_unstale":"2024-04-25T15:28:25.282653+0000","last_undegraded":"2024-04-25T15:28:25.282653+0000","last_fullsized":"2024-04-25T15:28:25.282653+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:55.065573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649356+0000","last_change":"2024-04-25T15:28:22.639382+0000","last_active":"2024-04-25T15:28:24.649356+0000","last_peered":"2024-04-25T15:28:24.649356+0000","last_clean":"2024-04-25T15:28:24.649356+0000","last_became_active":"2024-04-25T15:28:22.639130+0000","last_became_peered":"2024-04-25T15:28:22.639130+0000","last_unstale":"2024-04-25T15:28:24.649356+0000","last_undegraded":"2024-04-25T15:28:24.649356+0000","last_fullsized":"2024-04-25T15:28:24.649356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:17.513548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282579+0000","last_change":"2024-04-25T15:28:23.647366+0000","last_active":"2024-04-25T15:28:25.282579+0000","last_peered":"2024-04-25T15:28:25.282579+0000","last_clean":"2024-04-25T15:28:25.282579+0000","last_became_active":"2024-04-25T15:28:23.647027+0000","last_became_peered":"2024-04-25T15:28:23.647027+0000","last_unstale":"2024-04-25T15:28:25.282579+0000","last_undegraded":"2024-04-25T15:28:25.282579+0000","last_fullsized":"2024-04-25T15:28:25.282579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648998+0000","last_change":"2024-04-25T15:28:22.627683+0000","last_active":"2024-04-25T15:28:24.648998+0000","last_peered":"2024-04-25T15:28:24.648998+0000","last_clean":"2024-04-25T15:28:24.648998+0000","last_became_active":"2024-04-25T15:28:22.627379+0000","last_became_peered":"2024-04-25T15:28:22.627379+0000","last_unstale":"2024-04-25T15:28:24.648998+0000","last_undegraded":"2024-04-25T15:28:24.648998+0000","last_fullsized":"2024-04-25T15:28:24.648998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:30.340419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281914+0000","last_change":"2024-04-25T15:28:23.655579+0000","last_active":"2024-04-25T15:28:25.281914+0000","last_peered":"2024-04-25T15:28:25.281914+0000","last_clean":"2024-04-25T15:28:25.281914+0000","last_became_active":"2024-04-25T15:28:23.655206+0000","last_became_peered":"2024-04-25T15:28:23.655206+0000","last_unstale":"2024-04-25T15:28:25.281914+0000","last_undegraded":"2024-04-25T15:28:25.281914+0000","last_fullsized":"2024-04-25T15:28:25.281914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:16.785360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.629301+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628962+0000","last_became_peered":"2024-04-25T15:28:22.628962+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:24.982249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648121+0000","last_change":"2024-04-25T15:28:23.654406+0000","last_active":"2024-04-25T15:28:24.648121+0000","last_peered":"2024-04-25T15:28:24.648121+0000","last_clean":"2024-04-25T15:28:24.648121+0000","last_became_active":"2024-04-25T15:28:23.652920+0000","last_became_peered":"2024-04-25T15:28:23.652920+0000","last_unstale":"2024-04-25T15:28:24.648121+0000","last_undegraded":"2024-04-25T15:28:24.648121+0000","last_fullsized":"2024-04-25T15:28:24.648121+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:31.162871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649152+0000","last_change":"2024-04-25T15:28:22.633397+0000","last_active":"2024-04-25T15:28:24.649152+0000","last_peered":"2024-04-25T15:28:24.649152+0000","last_clean":"2024-04-25T15:28:24.649152+0000","last_became_active":"2024-04-25T15:28:22.633041+0000","last_became_peered":"2024-04-25T15:28:22.633041+0000","last_unstale":"2024-04-25T15:28:24.649152+0000","last_undegraded":"2024-04-25T15:28:24.649152+0000","last_fullsized":"2024-04-25T15:28:24.649152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:08.019583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648965+0000","last_change":"2024-04-25T15:28:23.645615+0000","last_active":"2024-04-25T15:28:24.648965+0000","last_peered":"2024-04-25T15:28:24.648965+0000","last_clean":"2024-04-25T15:28:24.648965+0000","last_became_active":"2024-04-25T15:28:23.645277+0000","last_became_peered":"2024-04-25T15:28:23.645277+0000","last_unstale":"2024-04-25T15:28:24.648965+0000","last_undegraded":"2024-04-25T15:28:24.648965+0000","last_fullsized":"2024-04-25T15:28:24.648965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:19.329565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648771+0000","last_change":"2024-04-25T15:28:22.639411+0000","last_active":"2024-04-25T15:28:24.648771+0000","last_peered":"2024-04-25T15:28:24.648771+0000","last_clean":"2024-04-25T15:28:24.648771+0000","last_became_active":"2024-04-25T15:28:22.639122+0000","last_became_peered":"2024-04-25T15:28:22.639122+0000","last_unstale":"2024-04-25T15:28:24.648771+0000","last_undegraded":"2024-04-25T15:28:24.648771+0000","last_fullsized":"2024-04-25T15:28:24.648771+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:00.247649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281419+0000","last_change":"2024-04-25T15:28:23.639119+0000","last_active":"2024-04-25T15:28:25.281419+0000","last_peered":"2024-04-25T15:28:25.281419+0000","last_clean":"2024-04-25T15:28:25.281419+0000","last_became_active":"2024-04-25T15:28:23.638754+0000","last_became_peered":"2024-04-25T15:28:23.638754+0000","last_unstale":"2024-04-25T15:28:25.281419+0000","last_undegraded":"2024-04-25T15:28:25.281419+0000","last_fullsized":"2024-04-25T15:28:25.281419+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648777+0000","last_change":"2024-04-25T15:28:22.631835+0000","last_active":"2024-04-25T15:28:24.648777+0000","last_peered":"2024-04-25T15:28:24.648777+0000","last_clean":"2024-04-25T15:28:24.648777+0000","last_became_active":"2024-04-25T15:28:22.631539+0000","last_became_peered":"2024-04-25T15:28:22.631539+0000","last_unstale":"2024-04-25T15:28:24.648777+0000","last_undegraded":"2024-04-25T15:28:24.648777+0000","last_fullsized":"2024-04-25T15:28:24.648777+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:18.322027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281369+0000","last_change":"2024-04-25T15:28:23.638974+0000","last_active":"2024-04-25T15:28:25.281369+0000","last_peered":"2024-04-25T15:28:25.281369+0000","last_clean":"2024-04-25T15:28:25.281369+0000","last_became_active":"2024-04-25T15:28:23.638642+0000","last_became_peered":"2024-04-25T15:28:23.638642+0000","last_unstale":"2024-04-25T15:28:25.281369+0000","last_undegraded":"2024-04-25T15:28:25.281369+0000","last_fullsized":"2024-04-25T15:28:25.281369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:06.035044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647693+0000","last_change":"2024-04-25T15:28:22.634256+0000","last_active":"2024-04-25T15:28:24.647693+0000","last_peered":"2024-04-25T15:28:24.647693+0000","last_clean":"2024-04-25T15:28:24.647693+0000","last_became_active":"2024-04-25T15:28:22.633942+0000","last_became_peered":"2024-04-25T15:28:22.633942+0000","last_unstale":"2024-04-25T15:28:24.647693+0000","last_undegraded":"2024-04-25T15:28:24.647693+0000","last_fullsized":"2024-04-25T15:28:24.647693+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:10.420589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282150+0000","last_change":"2024-04-25T15:28:23.639749+0000","last_active":"2024-04-25T15:28:25.282150+0000","last_peered":"2024-04-25T15:28:25.282150+0000","last_clean":"2024-04-25T15:28:25.282150+0000","last_became_active":"2024-04-25T15:28:23.639415+0000","last_became_peered":"2024-04-25T15:28:23.639415+0000","last_unstale":"2024-04-25T15:28:25.282150+0000","last_undegraded":"2024-04-25T15:28:25.282150+0000","last_fullsized":"2024-04-25T15:28:25.282150+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:38:27.203252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282928+0000","last_change":"2024-04-25T15:28:22.628552+0000","last_active":"2024-04-25T15:28:25.282928+0000","last_peered":"2024-04-25T15:28:25.282928+0000","last_clean":"2024-04-25T15:28:25.282928+0000","last_became_active":"2024-04-25T15:28:22.627193+0000","last_became_peered":"2024-04-25T15:28:22.627193+0000","last_unstale":"2024-04-25T15:28:25.282928+0000","last_undegraded":"2024-04-25T15:28:25.282928+0000","last_fullsized":"2024-04-25T15:28:25.282928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282633+0000","last_change":"2024-04-25T15:28:23.651525+0000","last_active":"2024-04-25T15:28:25.282633+0000","last_peered":"2024-04-25T15:28:25.282633+0000","last_clean":"2024-04-25T15:28:25.282633+0000","last_became_active":"2024-04-25T15:28:23.651179+0000","last_became_peered":"2024-04-25T15:28:23.651179+0000","last_unstale":"2024-04-25T15:28:25.282633+0000","last_undegraded":"2024-04-25T15:28:25.282633+0000","last_fullsized":"2024-04-25T15:28:25.282633+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282316+0000","last_change":"2024-04-25T15:28:22.631506+0000","last_active":"2024-04-25T15:28:25.282316+0000","last_peered":"2024-04-25T15:28:25.282316+0000","last_clean":"2024-04-25T15:28:25.282316+0000","last_became_active":"2024-04-25T15:28:22.631161+0000","last_became_peered":"2024-04-25T15:28:22.631161+0000","last_unstale":"2024-04-25T15:28:25.282316+0000","last_undegraded":"2024-04-25T15:28:25.282316+0000","last_fullsized":"2024-04-25T15:28:25.282316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.414479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649203+0000","last_change":"2024-04-25T15:28:23.636915+0000","last_active":"2024-04-25T15:28:24.649203+0000","last_peered":"2024-04-25T15:28:24.649203+0000","last_clean":"2024-04-25T15:28:24.649203+0000","last_became_active":"2024-04-25T15:28:23.636599+0000","last_became_peered":"2024-04-25T15:28:23.636599+0000","last_unstale":"2024-04-25T15:28:24.649203+0000","last_undegraded":"2024-04-25T15:28:24.649203+0000","last_fullsized":"2024-04-25T15:28:24.649203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281907+0000","last_change":"2024-04-25T15:28:22.628373+0000","last_active":"2024-04-25T15:28:25.281907+0000","last_peered":"2024-04-25T15:28:25.281907+0000","last_clean":"2024-04-25T15:28:25.281907+0000","last_became_active":"2024-04-25T15:28:22.628063+0000","last_became_peered":"2024-04-25T15:28:22.628063+0000","last_unstale":"2024-04-25T15:28:25.281907+0000","last_undegraded":"2024-04-25T15:28:25.281907+0000","last_fullsized":"2024-04-25T15:28:25.281907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:17.513548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649303+0000","last_change":"2024-04-25T15:28:23.640559+0000","last_active":"2024-04-25T15:28:24.649303+0000","last_peered":"2024-04-25T15:28:24.649303+0000","last_clean":"2024-04-25T15:28:24.649303+0000","last_became_active":"2024-04-25T15:28:23.638096+0000","last_became_peered":"2024-04-25T15:28:23.638096+0000","last_unstale":"2024-04-25T15:28:24.649303+0000","last_undegraded":"2024-04-25T15:28:24.649303+0000","last_fullsized":"2024-04-25T15:28:24.649303+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:32.309812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282803+0000","last_change":"2024-04-25T15:28:22.627717+0000","last_active":"2024-04-25T15:28:25.282803+0000","last_peered":"2024-04-25T15:28:25.282803+0000","last_clean":"2024-04-25T15:28:25.282803+0000","last_became_active":"2024-04-25T15:28:22.627402+0000","last_became_peered":"2024-04-25T15:28:22.627402+0000","last_unstale":"2024-04-25T15:28:25.282803+0000","last_undegraded":"2024-04-25T15:28:25.282803+0000","last_fullsized":"2024-04-25T15:28:25.282803+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:05.038807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648678+0000","last_change":"2024-04-25T15:28:23.630065+0000","last_active":"2024-04-25T15:28:24.648678+0000","last_peered":"2024-04-25T15:28:24.648678+0000","last_clean":"2024-04-25T15:28:24.648678+0000","last_became_active":"2024-04-25T15:28:23.629649+0000","last_became_peered":"2024-04-25T15:28:23.629649+0000","last_unstale":"2024-04-25T15:28:24.648678+0000","last_undegraded":"2024-04-25T15:28:24.648678+0000","last_fullsized":"2024-04-25T15:28:24.648678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648438+0000","last_change":"2024-04-25T15:28:22.632102+0000","last_active":"2024-04-25T15:28:24.648438+0000","last_peered":"2024-04-25T15:28:24.648438+0000","last_clean":"2024-04-25T15:28:24.648438+0000","last_became_active":"2024-04-25T15:28:22.631723+0000","last_became_peered":"2024-04-25T15:28:22.631723+0000","last_unstale":"2024-04-25T15:28:24.648438+0000","last_undegraded":"2024-04-25T15:28:24.648438+0000","last_fullsized":"2024-04-25T15:28:24.648438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:02:06.295529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282145+0000","last_change":"2024-04-25T15:28:23.639139+0000","last_active":"2024-04-25T15:28:25.282145+0000","last_peered":"2024-04-25T15:28:25.282145+0000","last_clean":"2024-04-25T15:28:25.282145+0000","last_became_active":"2024-04-25T15:28:23.638812+0000","last_became_peered":"2024-04-25T15:28:23.638812+0000","last_unstale":"2024-04-25T15:28:25.282145+0000","last_undegraded":"2024-04-25T15:28:25.282145+0000","last_fullsized":"2024-04-25T15:28:25.282145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:56.852295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281527+0000","last_change":"2024-04-25T15:28:22.631778+0000","last_active":"2024-04-25T15:28:25.281527+0000","last_peered":"2024-04-25T15:28:25.281527+0000","last_clean":"2024-04-25T15:28:25.281527+0000","last_became_active":"2024-04-25T15:28:22.631460+0000","last_became_peered":"2024-04-25T15:28:22.631460+0000","last_unstale":"2024-04-25T15:28:25.281527+0000","last_undegraded":"2024-04-25T15:28:25.281527+0000","last_fullsized":"2024-04-25T15:28:25.281527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:15.777821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647614+0000","last_change":"2024-04-25T15:28:23.652363+0000","last_active":"2024-04-25T15:28:24.647614+0000","last_peered":"2024-04-25T15:28:24.647614+0000","last_clean":"2024-04-25T15:28:24.647614+0000","last_became_active":"2024-04-25T15:28:23.651744+0000","last_became_peered":"2024-04-25T15:28:23.651744+0000","last_unstale":"2024-04-25T15:28:24.647614+0000","last_undegraded":"2024-04-25T15:28:24.647614+0000","last_fullsized":"2024-04-25T15:28:24.647614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:17.116922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281587+0000","last_change":"2024-04-25T15:28:22.636713+0000","last_active":"2024-04-25T15:28:25.281587+0000","last_peered":"2024-04-25T15:28:25.281587+0000","last_clean":"2024-04-25T15:28:25.281587+0000","last_became_active":"2024-04-25T15:28:22.636389+0000","last_became_peered":"2024-04-25T15:28:22.636389+0000","last_unstale":"2024-04-25T15:28:25.281587+0000","last_undegraded":"2024-04-25T15:28:25.281587+0000","last_fullsized":"2024-04-25T15:28:25.281587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647465+0000","last_change":"2024-04-25T15:28:23.647463+0000","last_active":"2024-04-25T15:28:24.647465+0000","last_peered":"2024-04-25T15:28:24.647465+0000","last_clean":"2024-04-25T15:28:24.647465+0000","last_became_active":"2024-04-25T15:28:23.647133+0000","last_became_peered":"2024-04-25T15:28:23.647133+0000","last_unstale":"2024-04-25T15:28:24.647465+0000","last_undegraded":"2024-04-25T15:28:24.647465+0000","last_fullsized":"2024-04-25T15:28:24.647465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:14.765406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649115+0000","last_change":"2024-04-25T15:28:22.628461+0000","last_active":"2024-04-25T15:28:24.649115+0000","last_peered":"2024-04-25T15:28:24.649115+0000","last_clean":"2024-04-25T15:28:24.649115+0000","last_became_active":"2024-04-25T15:28:22.628160+0000","last_became_peered":"2024-04-25T15:28:22.628160+0000","last_unstale":"2024-04-25T15:28:24.649115+0000","last_undegraded":"2024-04-25T15:28:24.649115+0000","last_fullsized":"2024-04-25T15:28:24.649115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282411+0000","last_change":"2024-04-25T15:28:23.649571+0000","last_active":"2024-04-25T15:28:25.282411+0000","last_peered":"2024-04-25T15:28:25.282411+0000","last_clean":"2024-04-25T15:28:25.282411+0000","last_became_active":"2024-04-25T15:28:23.649214+0000","last_became_peered":"2024-04-25T15:28:23.649214+0000","last_unstale":"2024-04-25T15:28:25.282411+0000","last_undegraded":"2024-04-25T15:28:25.282411+0000","last_fullsized":"2024-04-25T15:28:25.282411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:22.598027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283048+0000","last_change":"2024-04-25T15:28:22.631820+0000","last_active":"2024-04-25T15:28:25.283048+0000","last_peered":"2024-04-25T15:28:25.283048+0000","last_clean":"2024-04-25T15:28:25.283048+0000","last_became_active":"2024-04-25T15:28:22.631501+0000","last_became_peered":"2024-04-25T15:28:22.631501+0000","last_unstale":"2024-04-25T15:28:25.283048+0000","last_undegraded":"2024-04-25T15:28:25.283048+0000","last_fullsized":"2024-04-25T15:28:25.283048+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:43.399245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282947+0000","last_change":"2024-04-25T15:28:23.646728+0000","last_active":"2024-04-25T15:28:25.282947+0000","last_peered":"2024-04-25T15:28:25.282947+0000","last_clean":"2024-04-25T15:28:25.282947+0000","last_became_active":"2024-04-25T15:28:23.646234+0000","last_became_peered":"2024-04-25T15:28:23.646234+0000","last_unstale":"2024-04-25T15:28:25.282947+0000","last_undegraded":"2024-04-25T15:28:25.282947+0000","last_fullsized":"2024-04-25T15:28:25.282947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648950+0000","last_change":"2024-04-25T15:28:22.632918+0000","last_active":"2024-04-25T15:28:24.648950+0000","last_peered":"2024-04-25T15:28:24.648950+0000","last_clean":"2024-04-25T15:28:24.648950+0000","last_became_active":"2024-04-25T15:28:22.632551+0000","last_became_peered":"2024-04-25T15:28:22.632551+0000","last_unstale":"2024-04-25T15:28:24.648950+0000","last_undegraded":"2024-04-25T15:28:24.648950+0000","last_fullsized":"2024-04-25T15:28:24.648950+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:25.998670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649445+0000","last_change":"2024-04-25T15:28:23.641506+0000","last_active":"2024-04-25T15:28:24.649445+0000","last_peered":"2024-04-25T15:28:24.649445+0000","last_clean":"2024-04-25T15:28:24.649445+0000","last_became_active":"2024-04-25T15:28:23.640967+0000","last_became_peered":"2024-04-25T15:28:23.640967+0000","last_unstale":"2024-04-25T15:28:24.649445+0000","last_undegraded":"2024-04-25T15:28:24.649445+0000","last_fullsized":"2024-04-25T15:28:24.649445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649243+0000","last_change":"2024-04-25T15:28:22.631396+0000","last_active":"2024-04-25T15:28:24.649243+0000","last_peered":"2024-04-25T15:28:24.649243+0000","last_clean":"2024-04-25T15:28:24.649243+0000","last_became_active":"2024-04-25T15:28:22.630989+0000","last_became_peered":"2024-04-25T15:28:22.630989+0000","last_unstale":"2024-04-25T15:28:24.649243+0000","last_undegraded":"2024-04-25T15:28:24.649243+0000","last_fullsized":"2024-04-25T15:28:24.649243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:25.648985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282440+0000","last_change":"2024-04-25T15:28:23.650548+0000","last_active":"2024-04-25T15:28:25.282440+0000","last_peered":"2024-04-25T15:28:25.282440+0000","last_clean":"2024-04-25T15:28:25.282440+0000","last_became_active":"2024-04-25T15:28:23.649636+0000","last_became_peered":"2024-04-25T15:28:23.649636+0000","last_unstale":"2024-04-25T15:28:25.282440+0000","last_undegraded":"2024-04-25T15:28:25.282440+0000","last_fullsized":"2024-04-25T15:28:25.282440+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:11.428127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281645+0000","last_change":"2024-04-25T15:28:22.631144+0000","last_active":"2024-04-25T15:28:25.281645+0000","last_peered":"2024-04-25T15:28:25.281645+0000","last_clean":"2024-04-25T15:28:25.281645+0000","last_became_active":"2024-04-25T15:28:22.630762+0000","last_became_peered":"2024-04-25T15:28:22.630762+0000","last_unstale":"2024-04-25T15:28:25.281645+0000","last_undegraded":"2024-04-25T15:28:25.281645+0000","last_fullsized":"2024-04-25T15:28:25.281645+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:10.967925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.283204+0000","last_change":"2024-04-25T15:28:23.647208+0000","last_active":"2024-04-25T15:28:25.283204+0000","last_peered":"2024-04-25T15:28:25.283204+0000","last_clean":"2024-04-25T15:28:25.283204+0000","last_became_active":"2024-04-25T15:28:23.646871+0000","last_became_peered":"2024-04-25T15:28:23.646871+0000","last_unstale":"2024-04-25T15:28:25.283204+0000","last_undegraded":"2024-04-25T15:28:25.283204+0000","last_fullsized":"2024-04-25T15:28:25.283204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:22.330771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282312+0000","last_change":"2024-04-25T15:28:22.623948+0000","last_active":"2024-04-25T15:28:25.282312+0000","last_peered":"2024-04-25T15:28:25.282312+0000","last_clean":"2024-04-25T15:28:25.282312+0000","last_became_active":"2024-04-25T15:28:22.623604+0000","last_became_peered":"2024-04-25T15:28:22.623604+0000","last_unstale":"2024-04-25T15:28:25.282312+0000","last_undegraded":"2024-04-25T15:28:25.282312+0000","last_fullsized":"2024-04-25T15:28:25.282312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:13.369418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:28:23.648910+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:28:23.648524+0000","last_became_peered":"2024-04-25T15:28:23.648524+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:05:35.853221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281681+0000","last_change":"2024-04-25T15:28:22.635528+0000","last_active":"2024-04-25T15:28:25.281681+0000","last_peered":"2024-04-25T15:28:25.281681+0000","last_clean":"2024-04-25T15:28:25.281681+0000","last_became_active":"2024-04-25T15:28:22.635320+0000","last_became_peered":"2024-04-25T15:28:22.635320+0000","last_unstale":"2024-04-25T15:28:25.281681+0000","last_undegraded":"2024-04-25T15:28:25.281681+0000","last_fullsized":"2024-04-25T15:28:25.281681+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:37.967626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281434+0000","last_change":"2024-04-25T15:28:23.658467+0000","last_active":"2024-04-25T15:28:25.281434+0000","last_peered":"2024-04-25T15:28:25.281434+0000","last_clean":"2024-04-25T15:28:25.281434+0000","last_became_active":"2024-04-25T15:28:23.658051+0000","last_became_peered":"2024-04-25T15:28:23.658051+0000","last_unstale":"2024-04-25T15:28:25.281434+0000","last_undegraded":"2024-04-25T15:28:25.281434+0000","last_fullsized":"2024-04-25T15:28:25.281434+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:27.006208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281183+0000","last_change":"2024-04-25T15:28:22.627923+0000","last_active":"2024-04-25T15:28:25.281183+0000","last_peered":"2024-04-25T15:28:25.281183+0000","last_clean":"2024-04-25T15:28:25.281183+0000","last_became_active":"2024-04-25T15:28:22.627595+0000","last_became_peered":"2024-04-25T15:28:22.627595+0000","last_unstale":"2024-04-25T15:28:25.281183+0000","last_undegraded":"2024-04-25T15:28:25.281183+0000","last_fullsized":"2024-04-25T15:28:25.281183+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:04.050831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649208+0000","last_change":"2024-04-25T15:28:23.650714+0000","last_active":"2024-04-25T15:28:24.649208+0000","last_peered":"2024-04-25T15:28:24.649208+0000","last_clean":"2024-04-25T15:28:24.649208+0000","last_became_active":"2024-04-25T15:28:23.650410+0000","last_became_peered":"2024-04-25T15:28:23.650410+0000","last_unstale":"2024-04-25T15:28:24.649208+0000","last_undegraded":"2024-04-25T15:28:24.649208+0000","last_fullsized":"2024-04-25T15:28:24.649208+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:25.989787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.647686+0000","last_change":"2024-04-25T15:28:22.629989+0000","last_active":"2024-04-25T15:28:24.647686+0000","last_peered":"2024-04-25T15:28:24.647686+0000","last_clean":"2024-04-25T15:28:24.647686+0000","last_became_active":"2024-04-25T15:28:22.629681+0000","last_became_peered":"2024-04-25T15:28:22.629681+0000","last_unstale":"2024-04-25T15:28:24.647686+0000","last_undegraded":"2024-04-25T15:28:24.647686+0000","last_fullsized":"2024-04-25T15:28:24.647686+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:36.386426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281853+0000","last_change":"2024-04-25T15:28:23.643948+0000","last_active":"2024-04-25T15:28:25.281853+0000","last_peered":"2024-04-25T15:28:25.281853+0000","last_clean":"2024-04-25T15:28:25.281853+0000","last_became_active":"2024-04-25T15:28:23.643600+0000","last_became_peered":"2024-04-25T15:28:23.643600+0000","last_unstale":"2024-04-25T15:28:25.281853+0000","last_undegraded":"2024-04-25T15:28:25.281853+0000","last_fullsized":"2024-04-25T15:28:25.281853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:05.058369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648158+0000","last_change":"2024-04-25T15:28:22.630713+0000","last_active":"2024-04-25T15:28:24.648158+0000","last_peered":"2024-04-25T15:28:24.648158+0000","last_clean":"2024-04-25T15:28:24.648158+0000","last_became_active":"2024-04-25T15:28:22.627937+0000","last_became_peered":"2024-04-25T15:28:22.627937+0000","last_unstale":"2024-04-25T15:28:24.648158+0000","last_undegraded":"2024-04-25T15:28:24.648158+0000","last_fullsized":"2024-04-25T15:28:24.648158+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:34.438583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282846+0000","last_change":"2024-04-25T15:28:23.650199+0000","last_active":"2024-04-25T15:28:25.282846+0000","last_peered":"2024-04-25T15:28:25.282846+0000","last_clean":"2024-04-25T15:28:25.282846+0000","last_became_active":"2024-04-25T15:28:23.649599+0000","last_became_peered":"2024-04-25T15:28:23.649599+0000","last_unstale":"2024-04-25T15:28:25.282846+0000","last_undegraded":"2024-04-25T15:28:25.282846+0000","last_fullsized":"2024-04-25T15:28:25.282846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:20.873603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282154+0000","last_change":"2024-04-25T15:28:22.626743+0000","last_active":"2024-04-25T15:28:25.282154+0000","last_peered":"2024-04-25T15:28:25.282154+0000","last_clean":"2024-04-25T15:28:25.282154+0000","last_became_active":"2024-04-25T15:28:22.626434+0000","last_became_peered":"2024-04-25T15:28:22.626434+0000","last_unstale":"2024-04-25T15:28:25.282154+0000","last_undegraded":"2024-04-25T15:28:25.282154+0000","last_fullsized":"2024-04-25T15:28:25.282154+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:13.395697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649497+0000","last_change":"2024-04-25T15:28:23.633566+0000","last_active":"2024-04-25T15:28:24.649497+0000","last_peered":"2024-04-25T15:28:24.649497+0000","last_clean":"2024-04-25T15:28:24.649497+0000","last_became_active":"2024-04-25T15:28:23.633245+0000","last_became_peered":"2024-04-25T15:28:23.633245+0000","last_unstale":"2024-04-25T15:28:24.649497+0000","last_undegraded":"2024-04-25T15:28:24.649497+0000","last_fullsized":"2024-04-25T15:28:24.649497+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:56.365928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282910+0000","last_change":"2024-04-25T15:28:22.630470+0000","last_active":"2024-04-25T15:28:25.282910+0000","last_peered":"2024-04-25T15:28:25.282910+0000","last_clean":"2024-04-25T15:28:25.282910+0000","last_became_active":"2024-04-25T15:28:22.630069+0000","last_became_peered":"2024-04-25T15:28:22.630069+0000","last_unstale":"2024-04-25T15:28:25.282910+0000","last_undegraded":"2024-04-25T15:28:25.282910+0000","last_fullsized":"2024-04-25T15:28:25.282910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:30.155333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282356+0000","last_change":"2024-04-25T15:28:23.648809+0000","last_active":"2024-04-25T15:28:25.282356+0000","last_peered":"2024-04-25T15:28:25.282356+0000","last_clean":"2024-04-25T15:28:25.282356+0000","last_became_active":"2024-04-25T15:28:23.648507+0000","last_became_peered":"2024-04-25T15:28:23.648507+0000","last_unstale":"2024-04-25T15:28:25.282356+0000","last_undegraded":"2024-04-25T15:28:25.282356+0000","last_fullsized":"2024-04-25T15:28:25.282356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:01:31.347958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648878+0000","last_change":"2024-04-25T15:28:22.628170+0000","last_active":"2024-04-25T15:28:24.648878+0000","last_peered":"2024-04-25T15:28:24.648878+0000","last_clean":"2024-04-25T15:28:24.648878+0000","last_became_active":"2024-04-25T15:28:22.627191+0000","last_became_peered":"2024-04-25T15:28:22.627191+0000","last_unstale":"2024-04-25T15:28:24.648878+0000","last_undegraded":"2024-04-25T15:28:24.648878+0000","last_fullsized":"2024-04-25T15:28:24.648878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:15:31.302274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648932+0000","last_change":"2024-04-25T15:28:23.630876+0000","last_active":"2024-04-25T15:28:24.648932+0000","last_peered":"2024-04-25T15:28:24.648932+0000","last_clean":"2024-04-25T15:28:24.648932+0000","last_became_active":"2024-04-25T15:28:23.630558+0000","last_became_peered":"2024-04-25T15:28:23.630558+0000","last_unstale":"2024-04-25T15:28:24.648932+0000","last_undegraded":"2024-04-25T15:28:24.648932+0000","last_fullsized":"2024-04-25T15:28:24.648932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.649387+0000","last_change":"2024-04-25T15:28:22.630238+0000","last_active":"2024-04-25T15:28:24.649387+0000","last_peered":"2024-04-25T15:28:24.649387+0000","last_clean":"2024-04-25T15:28:24.649387+0000","last_became_active":"2024-04-25T15:28:22.629922+0000","last_became_peered":"2024-04-25T15:28:22.629922+0000","last_unstale":"2024-04-25T15:28:24.649387+0000","last_undegraded":"2024-04-25T15:28:24.649387+0000","last_fullsized":"2024-04-25T15:28:24.649387+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:13.757868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281519+0000","last_change":"2024-04-25T15:28:23.640269+0000","last_active":"2024-04-25T15:28:25.281519+0000","last_peered":"2024-04-25T15:28:25.281519+0000","last_clean":"2024-04-25T15:28:25.281519+0000","last_became_active":"2024-04-25T15:28:23.639944+0000","last_became_peered":"2024-04-25T15:28:23.639944+0000","last_unstale":"2024-04-25T15:28:25.281519+0000","last_undegraded":"2024-04-25T15:28:25.281519+0000","last_fullsized":"2024-04-25T15:28:25.281519+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:25.557691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282853+0000","last_change":"2024-04-25T15:28:22.626538+0000","last_active":"2024-04-25T15:28:25.282853+0000","last_peered":"2024-04-25T15:28:25.282853+0000","last_clean":"2024-04-25T15:28:25.282853+0000","last_became_active":"2024-04-25T15:28:22.626207+0000","last_became_peered":"2024-04-25T15:28:22.626207+0000","last_unstale":"2024-04-25T15:28:25.282853+0000","last_undegraded":"2024-04-25T15:28:25.282853+0000","last_fullsized":"2024-04-25T15:28:25.282853+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:01:50.682621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.282764+0000","last_change":"2024-04-25T15:28:23.646030+0000","last_active":"2024-04-25T15:28:25.282764+0000","last_peered":"2024-04-25T15:28:25.282764+0000","last_clean":"2024-04-25T15:28:25.282764+0000","last_became_active":"2024-04-25T15:28:23.645733+0000","last_became_peered":"2024-04-25T15:28:23.645733+0000","last_unstale":"2024-04-25T15:28:25.282764+0000","last_undegraded":"2024-04-25T15:28:25.282764+0000","last_fullsized":"2024-04-25T15:28:25.282764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:35:10.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648741+0000","last_change":"2024-04-25T15:28:22.632573+0000","last_active":"2024-04-25T15:28:24.648741+0000","last_peered":"2024-04-25T15:28:24.648741+0000","last_clean":"2024-04-25T15:28:24.648741+0000","last_became_active":"2024-04-25T15:28:22.632220+0000","last_became_peered":"2024-04-25T15:28:22.632220+0000","last_unstale":"2024-04-25T15:28:24.648741+0000","last_undegraded":"2024-04-25T15:28:24.648741+0000","last_fullsized":"2024-04-25T15:28:24.648741+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:16.109383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:24.648545+0000","last_change":"2024-04-25T15:28:23.640057+0000","last_active":"2024-04-25T15:28:24.648545+0000","last_peered":"2024-04-25T15:28:24.648545+0000","last_clean":"2024-04-25T15:28:24.648545+0000","last_became_active":"2024-04-25T15:28:23.639747+0000","last_became_peered":"2024-04-25T15:28:23.639747+0000","last_unstale":"2024-04-25T15:28:24.648545+0000","last_undegraded":"2024-04-25T15:28:24.648545+0000","last_fullsized":"2024-04-25T15:28:24.648545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:22.609329+0000","last_clean_scrub_stamp":"2024-04-25T15:28:22.609329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:06.046345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281905+0000","last_change":"2024-04-25T15:28:22.632108+0000","last_active":"2024-04-25T15:28:25.281905+0000","last_peered":"2024-04-25T15:28:25.281905+0000","last_clean":"2024-04-25T15:28:25.281905+0000","last_became_active":"2024-04-25T15:28:22.631768+0000","last_became_peered":"2024-04-25T15:28:22.631768+0000","last_unstale":"2024-04-25T15:28:25.281905+0000","last_undegraded":"2024-04-25T15:28:25.281905+0000","last_fullsized":"2024-04-25T15:28:25.281905+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:21.601791+0000","last_clean_scrub_stamp":"2024-04-25T15:28:21.601791+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:33.334816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":218,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-25T15:28:25.281666+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:25.281666+0000","last_peered":"2024-04-25T15:28:25.281666+0000","last_clean":"2024-04-25T15:28:25.281666+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:25.281666+0000","last_undegraded":"2024-04-25T15:28:25.281666+0000","last_fullsized":"2024-04-25T15:28:25.281666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1581056,"data_stored":1007311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":418507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":416873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":418485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":417455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":417067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":1581056,"data_stored":1007275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:28:32.556 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:28:32.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:28:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.710+0000 7f54f6db7640 1 Processor -- start 2024-04-25T15:28:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.710+0000 7f54f6db7640 1 -- start start 2024-04-25T15:28:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f6db7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f0164640 0x7f54f00b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f6db7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54f00b0a70 0x7f54f00b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f6db7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54f00d3170 con 0x7f54f0164ad0 2024-04-25T15:28:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f6db7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f54f00d2f40 con 0x7f54f00b0a70 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f6db7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f54f0164ee0 con 0x7f54f0164640 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f5354640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54f00b0a70 0x7f54f00b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f5354640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f54f00b0a70 0x7f54f00b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57028/0 (socket says 172.21.15.67:57028) 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f5354640 1 -- 172.21.15.67:0/3041341810 learned_addr learned my addr 172.21.15.67:0/3041341810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54effff640 1 --2- 172.21.15.67:0/3041341810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f0164640 0x7f54f00b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f5354640 1 -- 172.21.15.67:0/3041341810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f0164640 msgr2=0x7f54f00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f5354640 1 --2- 172.21.15.67:0/3041341810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f0164640 0x7f54f00b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f5354640 1 -- 172.21.15.67:0/3041341810 >> v1:172.21.15.176:6789/0 conn(0x7f54f0164ad0 legacy=0x7f54f0163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.712+0000 7f54f5354640 1 -- 172.21.15.67:0/3041341810 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f00d3450 con 0x7f54f00b0a70 2024-04-25T15:28:32.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.713+0000 7f54f5354640 1 --2- 172.21.15.67:0/3041341810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f00b0a70 0x7f54f00b0e80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009870 tx=0x7f54d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a73a634537c4c0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.713+0000 7f54ef7fe640 1 -- 172.21.15.67:0/3041341810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8037070 con 0x7f54f00b0a70 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.713+0000 7f54ef7fe640 1 -- 172.21.15.67:0/3041341810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54d8032440 con 0x7f54f00b0a70 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.713+0000 7f54ef7fe640 1 -- 172.21.15.67:0/3041341810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8032810 con 0x7f54f00b0a70 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 -- 172.21.15.67:0/3041341810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f00b0a70 msgr2=0x7f54f00b0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 --2- 172.21.15.67:0/3041341810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f00b0a70 0x7f54f00b0e80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009870 tx=0x7f54d802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 -- 172.21.15.67:0/3041341810 shutdown_connections 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 --2- 172.21.15.67:0/3041341810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f00b0a70 0x7f54f00b0e80 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 --2- 172.21.15.67:0/3041341810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f0164640 0x7f54f00b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 -- 172.21.15.67:0/3041341810 >> 172.21.15.67:0/3041341810 conn(0x7f54f00d3aa0 msgr2=0x7f54f00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 -- 172.21.15.67:0/3041341810 shutdown_connections 2024-04-25T15:28:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.714+0000 7f54f6db7640 1 -- 172.21.15.67:0/3041341810 wait complete. 2024-04-25T15:28:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.716+0000 7f54f6db7640 1 Processor -- start 2024-04-25T15:28:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.716+0000 7f54f6db7640 1 -- start start 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54f6db7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54f00b0a70 0x7f54f015c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54f6db7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f0164ad0 0x7f54f014ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54f6db7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f01483f0 0x7f54f014b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54f6db7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54f006d9b0 con 0x7f54f0164ad0 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54f6db7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f54f00b31e0 con 0x7f54f00b0a70 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54f6db7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f54f00b9430 con 0x7f54f01483f0 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f0164ad0 0x7f54f014ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f0164ad0 0x7f54f014ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57040/0 (socket says 172.21.15.67:57040) 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54effff640 1 -- 172.21.15.67:0/2015664639 learned_addr learned my addr 172.21.15.67:0/2015664639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.717+0000 7f54f5354640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f01483f0 0x7f54f014b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54f4b53640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54f00b0a70 0x7f54f015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54effff640 1 -- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f01483f0 msgr2=0x7f54f014b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54effff640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f01483f0 0x7f54f014b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54effff640 1 -- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54f00b0a70 msgr2=0x7f54f015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54effff640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54f00b0a70 0x7f54f015c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54effff640 1 -- 172.21.15.67:0/2015664639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f015ded0 con 0x7f54f0164ad0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54effff640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f0164ad0 0x7f54f014ab80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f54e000e870 tx=0x7f54e000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54ed7fa640 1 -- 172.21.15.67:0/2015664639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0017070 con 0x7f54f0164ad0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54ed7fa640 1 -- 172.21.15.67:0/2015664639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e000c8e0 con 0x7f54f0164ad0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54ed7fa640 1 -- 172.21.15.67:0/2015664639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e000ccd0 con 0x7f54f0164ad0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f0161260 con 0x7f54f0164ad0 2024-04-25T15:28:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.718+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f015d110 con 0x7f54f0164ad0 2024-04-25T15:28:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.719+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f00b2830 con 0x7f54f0164ad0 2024-04-25T15:28:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.720+0000 7f54ed7fa640 1 -- 172.21.15.67:0/2015664639 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f54e001a070 con 0x7f54f0164ad0 2024-04-25T15:28:32.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.720+0000 7f54ed7fa640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54d00744e0 0x7f54d00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:32.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.721+0000 7f54ed7fa640 1 -- 172.21.15.67:0/2015664639 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f54e00953e0 con 0x7f54f0164ad0 2024-04-25T15:28:32.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.721+0000 7f54f4b53640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54d00744e0 0x7f54d00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:32.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.721+0000 7f54f4b53640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54d00744e0 0x7f54d00769f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f54f00746e0 tx=0x7f54e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:32.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.725+0000 7f54ed7fa640 1 -- 172.21.15.67:0/2015664639 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54e005f480 con 0x7f54f0164ad0 2024-04-25T15:28:32.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.984+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f54f0160130 con 0x7f54f0164ad0 2024-04-25T15:28:32.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.986+0000 7f54ed7fa640 1 -- 172.21.15.67:0/2015664639 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f54e0063780 con 0x7f54f0164ad0 2024-04-25T15:28:32.987 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:28:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:28:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-25T15:28:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-25T15:28:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:28:32.988 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-25T15:28:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.988+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54d00744e0 msgr2=0x7f54d00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.988+0000 7f54f6db7640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54d00744e0 0x7f54d00769f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f54f00746e0 tx=0x7f54e8009040 comp rx=0 tx=0).stop 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.988+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f0164ad0 msgr2=0x7f54f014ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.988+0000 7f54f6db7640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f0164ad0 0x7f54f014ab80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f54e000e870 tx=0x7f54e000ed30 comp rx=0 tx=0).stop 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 shutdown_connections 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f54d00744e0 0x7f54d00769f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54f01483f0 0x7f54f014b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54f0164ad0 0x7f54f014ab80 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 --2- 172.21.15.67:0/2015664639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54f00b0a70 0x7f54f015c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 >> 172.21.15.67:0/2015664639 conn(0x7f54f00d3aa0 msgr2=0x7f54f0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 shutdown_connections 2024-04-25T15:28:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:32.989+0000 7f54f6db7640 1 -- 172.21.15.67:0/2015664639 wait complete. 2024-04-25T15:28:33.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.112+0000 7f071f69a640 1 Processor -- start 2024-04-25T15:28:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.112+0000 7f071f69a640 1 -- start start 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.113+0000 7f071f69a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0718142730 0x7f0718142b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.113+0000 7f071f69a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07181431d0 con 0x7f07181400a0 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.113+0000 7f071f69a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07181434b0 con 0x7f0718070270 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.113+0000 7f071f69a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f07180b6090 con 0x7f0718142730 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f071d436640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07181400a0 0x7f07180b6890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60476/0 (socket says 172.21.15.67:60476) 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f071d436640 1 -- 172.21.15.67:0/4052416716 learned_addr learned my addr 172.21.15.67:0/4052416716 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f071cc35640 1 --2- 172.21.15.67:0/4052416716 >> v2:172.21.15.67:3300/0 conn(0x7f0718142730 0x7f0718142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f070ffff640 1 -- 172.21.15.67:0/4052416716 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2853490670 0 0) 0x7f07181431d0 con 0x7f07181400a0 2024-04-25T15:28:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f070ffff640 1 -- 172.21.15.67:0/4052416716 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0704003520 con 0x7f07181400a0 2024-04-25T15:28:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f071cc35640 1 -- 172.21.15.67:0/4052416716 >> v1:172.21.15.176:6789/0 conn(0x7f0718070270 legacy=0x7f0718070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f071cc35640 1 -- 172.21.15.67:0/4052416716 >> v1:172.21.15.67:6789/0 conn(0x7f07181400a0 legacy=0x7f07180b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.114+0000 7f071cc35640 1 -- 172.21.15.67:0/4052416716 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07180b6370 con 0x7f0718142730 2024-04-25T15:28:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f071cc35640 1 --2- 172.21.15.67:0/4052416716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718142730 0x7f0718142b40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f07080097a0 tx=0x7f070802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6482c4ad787351ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f070ffff640 1 -- 172.21.15.67:0/4052416716 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708037070 con 0x7f0718142730 2024-04-25T15:28:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f070ffff640 1 -- 172.21.15.67:0/4052416716 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0708032440 con 0x7f0718142730 2024-04-25T15:28:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f070ffff640 1 -- 172.21.15.67:0/4052416716 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708032810 con 0x7f0718142730 2024-04-25T15:28:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f071f69a640 1 -- 172.21.15.67:0/4052416716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718142730 msgr2=0x7f0718142b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f071f69a640 1 --2- 172.21.15.67:0/4052416716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718142730 0x7f0718142b40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f07080097a0 tx=0x7f070802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f071f69a640 1 -- 172.21.15.67:0/4052416716 shutdown_connections 2024-04-25T15:28:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f071f69a640 1 --2- 172.21.15.67:0/4052416716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718142730 0x7f0718142b40 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.115+0000 7f071f69a640 1 -- 172.21.15.67:0/4052416716 >> 172.21.15.67:0/4052416716 conn(0x7f0718073050 msgr2=0x7f0718073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.116+0000 7f071f69a640 1 -- 172.21.15.67:0/4052416716 shutdown_connections 2024-04-25T15:28:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.116+0000 7f071f69a640 1 -- 172.21.15.67:0/4052416716 wait complete. 2024-04-25T15:28:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.117+0000 7f071f69a640 1 Processor -- start 2024-04-25T15:28:33.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.117+0000 7f071f69a640 1 -- start start 2024-04-25T15:28:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071f69a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718070270 0x7f071815eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071f69a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07181400a0 0x7f071815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071f69a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0718142730 0x7f07181614a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071f69a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07180b4b20 con 0x7f0718070270 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071f69a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f07180b49a0 con 0x7f0718142730 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071f69a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07180b4ca0 con 0x7f07181400a0 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718070270 0x7f071815eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718070270 0x7f071815eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57056/0 (socket says 172.21.15.67:57056) 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 -- 172.21.15.67:0/563209060 learned_addr learned my addr 172.21.15.67:0/563209060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071dc37640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0718142730 0x7f07181614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071cc35640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07181400a0 0x7f071815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 -- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07181400a0 msgr2=0x7f071815c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07181400a0 0x7f071815c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 -- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0718142730 msgr2=0x7f07181614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0718142730 0x7f07181614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 -- 172.21.15.67:0/563209060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071815d6c0 con 0x7f0718070270 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071dc37640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0718142730 0x7f07181614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f071d436640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718070270 0x7f071815eed0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f071400c390 tx=0x7f071400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.119+0000 7f070dffb640 1 -- 172.21.15.67:0/563209060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0714017070 con 0x7f0718070270 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.120+0000 7f070dffb640 1 -- 172.21.15.67:0/563209060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0714005d00 con 0x7f0718070270 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.120+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071815c940 con 0x7f0718070270 2024-04-25T15:28:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.120+0000 7f070dffb640 1 -- 172.21.15.67:0/563209060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0714007310 con 0x7f0718070270 2024-04-25T15:28:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.120+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071815cf00 con 0x7f0718070270 2024-04-25T15:28:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.121+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0718050740 con 0x7f0718070270 2024-04-25T15:28:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.121+0000 7f070dffb640 1 -- 172.21.15.67:0/563209060 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f071401a070 con 0x7f0718070270 2024-04-25T15:28:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.122+0000 7f070dffb640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06fc0744b0 0x7f06fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.122+0000 7f070dffb640 1 -- 172.21.15.67:0/563209060 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f0714094620 con 0x7f0718070270 2024-04-25T15:28:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.122+0000 7f071cc35640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06fc0744b0 0x7f06fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.122+0000 7f071cc35640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06fc0744b0 0x7f06fc0769c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0708034000 tx=0x7f070802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.126+0000 7f070dffb640 1 -- 172.21.15.67:0/563209060 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f071405edb0 con 0x7f0718070270 2024-04-25T15:28:33.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.349+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07180b5c60 con 0x7f0718070270 2024-04-25T15:28:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.349+0000 7f070dffb640 1 -- 172.21.15.67:0/563209060 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5186 (secure 0 0 0) 0x7f071405e6b0 con 0x7f0718070270 2024-04-25T15:28:33.350 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:33.350 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/445614904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":445614904},{"type":"v1","addr":"172.21.15.67:6833","nonce":445614904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":6322,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3018066741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3018066741},{"type":"v1","addr":"172.21.15.67:6835","nonce":3018066741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2045345118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2045345118},{"type":"v1","addr":"172.21.15.176:6836","nonce":2045345118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:23.008043+0000","modified":"2024-04-25T15:28:26.033114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6328":{"gid":6328,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6837/2484404027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2484404027},{"type":"v1","addr":"172.21.15.176:6837","nonce":2484404027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6328,"qdb_cluster":[6328]},"id":6}]} 2024-04-25T15:28:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 49 2024-04-25T15:28:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06fc0744b0 msgr2=0x7f06fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06fc0744b0 0x7f06fc0769c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0708034000 tx=0x7f070802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718070270 msgr2=0x7f071815eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718070270 0x7f071815eed0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f071400c390 tx=0x7f071400c850 comp rx=0 tx=0).stop 2024-04-25T15:28:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 shutdown_connections 2024-04-25T15:28:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f06fc0744b0 0x7f06fc0769c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0718142730 0x7f07181614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07181400a0 0x7f071815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 --2- 172.21.15.67:0/563209060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0718070270 0x7f071815eed0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 >> 172.21.15.67:0/563209060 conn(0x7f0718073050 msgr2=0x7f07180b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 shutdown_connections 2024-04-25T15:28:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.351+0000 7f071f69a640 1 -- 172.21.15.67:0/563209060 wait complete. 2024-04-25T15:28:33.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:28:23.008043+0000', 'modified': '2024-04-25T15:28:26.033114+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6328}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6328': {'gid': 6328, 'name': 'd', 'rank': 0, 'incarnation': 46, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.176:6837/2484404027', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6835', 'nonce': 2484404027}, {'type': 'v1', 'addr': '172.21.15.176:6837', 'nonce': 2484404027}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6328, 'qdb_cluster': [6328]} 2024-04-25T15:28:33.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:28:33.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T15:28:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.884+0000 7f52d1c49640 1 Processor -- start 2024-04-25T15:28:33.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.884+0000 7f52d1c49640 1 -- start start 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.885+0000 7f52d1c49640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f52cc0d0e00 0x7f52cc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.885+0000 7f52d1c49640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc06c5c0 0x7f52cc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.885+0000 7f52d1c49640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52cc06cf10 con 0x7f52cc0b7810 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.885+0000 7f52d1c49640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f52cc0d5a60 con 0x7f52cc0d0e00 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.885+0000 7f52d1c49640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc0d1850 con 0x7f52cc06c5c0 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.885+0000 7f52cbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f52cc0b7810 0x7f52cc0b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60482/0 (socket says 172.21.15.67:60482) 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.885+0000 7f52cbfff640 1 -- 172.21.15.67:0/3241031865 learned_addr learned my addr 172.21.15.67:0/3241031865 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52c3fff640 1 --2- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc06c5c0 0x7f52cc06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/3241031865 >> v2:172.21.15.176:3300/0 conn(0x7f52cc0d0e00 0x7f52cc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52caffd640 1 -- 172.21.15.67:0/3241031865 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1646787131 0 0) 0x7f52c4001490 con 0x7f52cc0b7810 2024-04-25T15:28:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52caffd640 1 -- 172.21.15.67:0/3241031865 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52b8003520 con 0x7f52cc0b7810 2024-04-25T15:28:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52cb7fe640 1 -- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc06c5c0 msgr2=0x7f52cc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc06c5c0 0x7f52cc06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52cb7fe640 1 -- 172.21.15.67:0/3241031865 >> v1:172.21.15.67:6789/0 conn(0x7f52cc0b7810 legacy=0x7f52cc0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52caffd640 1 -- 172.21.15.67:0/3241031865 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4116474561 0 0) 0x7f52b8003520 con 0x7f52cc0b7810 2024-04-25T15:28:33.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.886+0000 7f52cb7fe640 1 -- 172.21.15.67:0/3241031865 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc160de0 con 0x7f52cc0d0e00 2024-04-25T15:28:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.887+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc0d0e00 0x7f52cc0d1210 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0097a0 tx=0x7f52bc02a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f03060a82f75b112 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.887+0000 7f52caffd640 1 -- 172.21.15.67:0/3241031865 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc037070 con 0x7f52cc0d0e00 2024-04-25T15:28:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.887+0000 7f52caffd640 1 -- 172.21.15.67:0/3241031865 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52bc02aba0 con 0x7f52cc0d0e00 2024-04-25T15:28:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.887+0000 7f52caffd640 1 -- 172.21.15.67:0/3241031865 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4001bc0 con 0x7f52cc0d0e00 2024-04-25T15:28:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 -- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc0d0e00 msgr2=0x7f52cc0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 --2- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc0d0e00 0x7f52cc0d1210 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f52bc0097a0 tx=0x7f52bc02a110 comp rx=0 tx=0).stop 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 -- 172.21.15.67:0/3241031865 shutdown_connections 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 --2- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc06c5c0 0x7f52cc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 --2- 172.21.15.67:0/3241031865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc0d0e00 0x7f52cc0d1210 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 -- 172.21.15.67:0/3241031865 >> 172.21.15.67:0/3241031865 conn(0x7f52cc0d5d90 msgr2=0x7f52cc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 -- 172.21.15.67:0/3241031865 shutdown_connections 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.888+0000 7f52d1c49640 1 -- 172.21.15.67:0/3241031865 wait complete. 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.890+0000 7f52d1c49640 1 Processor -- start 2024-04-25T15:28:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.890+0000 7f52d1c49640 1 -- start start 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52d1c49640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc06c5c0 0x7f52cc154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52d1c49640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52cc0b7810 0x7f52cc0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52d1c49640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc0d0e00 0x7f52cc1fa400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52d1c49640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc160ad0 con 0x7f52cc0b7810 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52d1c49640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc1613a0 con 0x7f52cc06c5c0 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52d1c49640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc0b5320 con 0x7f52cc0d0e00 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52cc0b7810 0x7f52cc0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52cc0b7810 0x7f52cc0cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57070/0 (socket says 172.21.15.67:57070) 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52c3fff640 1 -- 172.21.15.67:0/754818867 learned_addr learned my addr 172.21.15.67:0/754818867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52cbfff640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc0d0e00 0x7f52cc1fa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.891+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc06c5c0 0x7f52cc154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c3fff640 1 -- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc0d0e00 msgr2=0x7f52cc1fa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c3fff640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc0d0e00 0x7f52cc1fa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c3fff640 1 -- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc06c5c0 msgr2=0x7f52cc154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c3fff640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc06c5c0 0x7f52cc154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c3fff640 1 -- 172.21.15.67:0/754818867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc0c98d0 con 0x7f52cc0b7810 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c3fff640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52cc0b7810 0x7f52cc0cc430 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f52b400e8a0 tx=0x7f52b400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c8ff9640 1 -- 172.21.15.67:0/754818867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4017070 con 0x7f52cc0b7810 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c8ff9640 1 -- 172.21.15.67:0/754818867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b400c940 con 0x7f52cc0b7810 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52cc0cc970 con 0x7f52cc0b7810 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52c8ff9640 1 -- 172.21.15.67:0/754818867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b400cd30 con 0x7f52cc0b7810 2024-04-25T15:28:33.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.892+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52cc0c9e50 con 0x7f52cc0b7810 2024-04-25T15:28:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.894+0000 7f52c8ff9640 1 -- 172.21.15.67:0/754818867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f52b401a070 con 0x7f52cc0b7810 2024-04-25T15:28:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.895+0000 7f52c8ff9640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52ac0744b0 0x7f52ac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.895+0000 7f52c8ff9640 1 -- 172.21.15.67:0/754818867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f52b4095030 con 0x7f52cc0b7810 2024-04-25T15:28:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.895+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52ac0744b0 0x7f52ac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.895+0000 7f52d8bd4740 1 Processor -- start 2024-04-25T15:28:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.895+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52ac0744b0 0x7f52ac0769c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f52bc034000 tx=0x7f52bc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.896+0000 7f52d8bd4740 1 -- start start 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3588270 0x55fce3954600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3950fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3567590 0x55fce392a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fce388e740 con 0x55fce3567590 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55fce382a3a0 con 0x55fce3588270 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55fce388a700 con 0x55fce355f3b0 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3567590 0x55fce392a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3567590 0x55fce392a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57074/0 (socket says 172.21.15.67:57074) 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 -- 172.21.15.67:0/690619835 learned_addr learned my addr 172.21.15.67:0/690619835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3588270 0x55fce3954600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52c3fff640 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3950fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 -- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 msgr2=0x55fce3950fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3950fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 -- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3588270 msgr2=0x55fce3954600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3588270 0x55fce3954600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.897+0000 7f52cbfff640 1 -- 172.21.15.67:0/690619835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fce3846750 con 0x55fce3567590 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.898+0000 7f52cbfff640 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3567590 0x55fce392a260 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f52c400f330 tx=0x7f52c4001530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e380ca06ae5a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.898+0000 7f52c17fa640 1 -- 172.21.15.67:0/690619835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c40072e0 con 0x55fce3567590 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.898+0000 7f52c17fa640 1 -- 172.21.15.67:0/690619835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c4018070 con 0x55fce3567590 2024-04-25T15:28:33.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.898+0000 7f52c17fa640 1 -- 172.21.15.67:0/690619835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c40076c0 con 0x55fce3567590 2024-04-25T15:28:33.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 -- 172.21.15.67:0/690619835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3567590 msgr2=0x55fce392a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3567590 0x55fce392a260 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f52c400f330 tx=0x7f52c4001530 comp rx=0 tx=0).stop 2024-04-25T15:28:33.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 -- 172.21.15.67:0/690619835 shutdown_connections 2024-04-25T15:28:33.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3567590 0x55fce392a260 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3950fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/690619835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3588270 0x55fce3954600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 -- 172.21.15.67:0/690619835 >> 172.21.15.67:0/690619835 conn(0x55fce357c3c0 msgr2=0x55fce39299f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 -- 172.21.15.67:0/690619835 shutdown_connections 2024-04-25T15:28:33.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.899+0000 7f52d8bd4740 1 -- 172.21.15.67:0/690619835 wait complete. 2024-04-25T15:28:33.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.900+0000 7f52d8bd4740 1 -- start start 2024-04-25T15:28:33.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.900+0000 7f52d8bd4740 1 Processor -- start 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce396d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce3956ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3957530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fce3815fe0 con 0x55fce3588270 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55fce396dbd0 con 0x55fce3567590 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55fce3955330 con 0x55fce355f3b0 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52d8bd4740 10 client.0 _ll_register_callbacks cb 0x55fce382c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:28:33.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3957530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3957530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57088/0 (socket says 172.21.15.67:57088) 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52cbfff640 1 -- 172.21.15.67:0/2922684827 learned_addr learned my addr 172.21.15.67:0/2922684827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce396d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.902+0000 7f52c3fff640 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce3956ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52cbfff640 1 -- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 msgr2=0x55fce396d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52cbfff640 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce396d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52cbfff640 1 -- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 msgr2=0x55fce3956ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52cbfff640 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce3956ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52cbfff640 1 -- 172.21.15.67:0/2922684827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fce35a02d0 con 0x55fce3588270 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52cbfff640 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3957530 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f52c4005c00 tx=0x7f52c4005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7c31813f3d5ad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52aa7fc640 1 -- 172.21.15.67:0/2922684827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4027440 con 0x55fce3588270 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52aa7fc640 1 -- 172.21.15.67:0/2922684827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f52c4028c90 con 0x55fce3588270 2024-04-25T15:28:33.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52c4018070 con 0x55fce3588270 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52aa7fc640 1 -- 172.21.15.67:0/2922684827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f52c401d400 con 0x55fce3588270 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52d8bd4740 10 client.6879 fetch_fsmap learned FSMap version 49 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) -- 0x55fce366a6b0 con 0x55fce3588270 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.903+0000 7f52aa7fc640 1 -- 172.21.15.67:0/2922684827 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c4018070 con 0x55fce3588270 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52aa7fc640 1 -- 172.21.15.67:0/2922684827 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 49) ==== 3494+0+0 (secure 0 0 0) 0x7f52c402f050 con 0x55fce3588270 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52d8bd4740 10 client.6879 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52d8bd4740 10 client.6879 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52d8bd4740 20 client.6879 populate_metadata read hostname 'smithi067' 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce3546e40 0x55fce394f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52d8bd4740 4 client.6879 mds_command: new command op to 6328 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 --> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fce389c350 con 0x55fce3546e40 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce3546e40 0x55fce394f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52aa7fc640 10 client.6879 ms_handle_connect on v2:172.21.15.176:6835/2484404027 2024-04-25T15:28:33.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.904+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce3546e40 0x55fce394f640 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54d9fe6c94366e7f server_cookie=4baa7e041dc5cd5e in_seq=0 out_seq=0 2024-04-25T15:28:33.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52aa7fc640 1 -- 172.21.15.67:0/2922684827 <== mds.0 v2:172.21.15.176:6835/2484404027 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fce389c350 con 0x55fce3546e40 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52aa7fc640 10 client.6879 handle_command_reply: tid=0 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 client.6879 shutdown 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 msgr2=0x55fce3957530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3957530 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f52c4005c00 tx=0x7f52c4005b70 comp rx=0 tx=0).stop 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 shutdown_connections 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce3546e40 0x55fce394f640 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3957530 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce3956ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/2922684827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce396d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 >> 172.21.15.67:0/2922684827 conn(0x55fce357c3c0 msgr2=0x55fce3923d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52aa7fc640 0 client.6879 ms_handle_reset on v2:172.21.15.176:6835/2484404027 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 shutdown_connections 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 1 -- 172.21.15.67:0/2922684827 wait complete. 2024-04-25T15:28:33.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.906+0000 7f52d8bd4740 20 client.6879 trim_cache size 0 max 16384 2024-04-25T15:28:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.925+0000 7f52d8bd4740 1 Processor -- start 2024-04-25T15:28:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.925+0000 7f52d8bd4740 1 -- start start 2024-04-25T15:28:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.926+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3970910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.926+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce394fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.926+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3950020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.926+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc201f80 con 0x55fce3588270 2024-04-25T15:28:33.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.926+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc0cb100 con 0x55fce3567590 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.926+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc201930 con 0x55fce355f3b0 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3950020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3950020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57104/0 (socket says 172.21.15.67:57104) 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 -- 172.21.15.67:0/146482880 learned_addr learned my addr 172.21.15.67:0/146482880 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52c3fff640 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce394fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3970910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 -- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 msgr2=0x55fce3970910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3970910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 -- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 msgr2=0x55fce394fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce394fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 -- 172.21.15.67:0/146482880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fce396d9c0 con 0x55fce3588270 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52c3fff640 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce394fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52cbfff640 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3950020 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f52c4001d30 tx=0x7f52c4020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f52ded695fce158 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52c0ff9640 1 -- 172.21.15.67:0/146482880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4027440 con 0x55fce3588270 2024-04-25T15:28:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52c0ff9640 1 -- 172.21.15.67:0/146482880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c4018040 con 0x55fce3588270 2024-04-25T15:28:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.927+0000 7f52c0ff9640 1 -- 172.21.15.67:0/146482880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4027980 con 0x55fce3588270 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 -- 172.21.15.67:0/146482880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 msgr2=0x55fce3950020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3950020 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f52c4001d30 tx=0x7f52c4020040 comp rx=0 tx=0).stop 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 -- 172.21.15.67:0/146482880 shutdown_connections 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce3588270 0x55fce3950020 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce394fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/146482880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce355f3b0 0x55fce3970910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 -- 172.21.15.67:0/146482880 >> 172.21.15.67:0/146482880 conn(0x55fce357c3c0 msgr2=0x55fce391fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 -- 172.21.15.67:0/146482880 shutdown_connections 2024-04-25T15:28:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.928+0000 7f52d8bd4740 1 -- 172.21.15.67:0/146482880 wait complete. 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.929+0000 7f52d8bd4740 1 -- start start 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.929+0000 7f52d8bd4740 1 Processor -- start 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce355f3b0 0x55fce39ebca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce392a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52d8bd4740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce3588270 0x55fce392ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fce39491e0 con 0x55fce355f3b0 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55fce3957ad0 con 0x55fce3567590 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52d8bd4740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55fce3948650 con 0x55fce3588270 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52d8bd4740 10 client.0 _ll_register_callbacks cb 0x7f52b80036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52cb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce355f3b0 0x55fce39ebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52cb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce355f3b0 0x55fce39ebca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57110/0 (socket says 172.21.15.67:57110) 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.930+0000 7f52cb7fe640 1 -- 172.21.15.67:0/439857179 learned_addr learned my addr 172.21.15.67:0/439857179 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cbfff640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce3588270 0x55fce392ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52c3fff640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce392a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cb7fe640 1 -- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce3588270 msgr2=0x55fce392ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce3588270 0x55fce392ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cb7fe640 1 -- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 msgr2=0x55fce392a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce392a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cb7fe640 1 -- 172.21.15.67:0/439857179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fce396bd20 con 0x55fce355f3b0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cbfff640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce3588270 0x55fce392ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52cb7fe640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce355f3b0 0x55fce39ebca0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f52bc02a700 tx=0x7f52bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c241a377bdc55d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52aa7fc640 1 -- 172.21.15.67:0/439857179 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc053060 con 0x55fce355f3b0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52aa7fc640 1 -- 172.21.15.67:0/439857179 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f52bc03e720 con 0x55fce355f3b0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52c4018040 con 0x55fce355f3b0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52aa7fc640 1 -- 172.21.15.67:0/439857179 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f52bc059050 con 0x55fce355f3b0 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52d8bd4740 10 client.6885 fetch_fsmap learned FSMap version 49 2024-04-25T15:28:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) -- 0x55fce3970460 con 0x55fce355f3b0 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.931+0000 7f52aa7fc640 1 -- 172.21.15.67:0/439857179 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c4018040 con 0x55fce355f3b0 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52aa7fc640 1 -- 172.21.15.67:0/439857179 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 49) ==== 3494+0+0 (secure 0 0 0) 0x7f52bc05c020 con 0x55fce355f3b0 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52d8bd4740 10 client.6885 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52d8bd4740 10 client.6885 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52d8bd4740 20 client.6885 populate_metadata read hostname 'smithi067' 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce39f0070 0x55fce394e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52d8bd4740 4 client.6885 mds_command: new command op to 6328 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 --> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] -- command(tid 0: {"prefix": "status"}) -- 0x55fce38961c0 con 0x55fce39f0070 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52c3fff640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce39f0070 0x55fce394e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52aa7fc640 10 client.6885 ms_handle_connect on v2:172.21.15.176:6835/2484404027 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52c3fff640 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce39f0070 0x55fce394e020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b285885bef7ef4f server_cookie=bfa74d82830942a6 in_seq=0 out_seq=0 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52aa7fc640 1 -- 172.21.15.67:0/439857179 <== mds.0 v2:172.21.15.176:6835/2484404027 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55fce38961c0 con 0x55fce39f0070 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52aa7fc640 10 client.6885 handle_command_reply: tid=0 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.932+0000 7f52d8bd4740 1 client.6885 shutdown 2024-04-25T15:28:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce355f3b0 msgr2=0x55fce39ebca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce355f3b0 0x55fce39ebca0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f52bc02a700 tx=0x7f52bc004730 comp rx=0 tx=0).stop 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 shutdown_connections 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:6835/2484404027,v1:172.21.15.176:6837/2484404027] conn(0x55fce39f0070 0x55fce394e020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55fce3588270 0x55fce392ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52aa7fc640 0 client.6885 ms_handle_reset on v2:172.21.15.176:6835/2484404027 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55fce3567590 0x55fce392a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 --2- 172.21.15.67:0/439857179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fce355f3b0 0x55fce39ebca0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 >> 172.21.15.67:0/439857179 conn(0x55fce357c3c0 msgr2=0x55fce3923d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 shutdown_connections 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 1 -- 172.21.15.67:0/439857179 wait complete. 2024-04-25T15:28:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d8bd4740 20 client.6885 trim_cache size 0 max 16384 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b43d0acb-252a-4f7a-bf14-34d789b94b95", 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6328, 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.299223827, 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 49, 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 80, 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 80, 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 15.279636399999999 2024-04-25T15:28:33.939 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:28:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52ac0744b0 msgr2=0x7f52ac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52ac0744b0 0x7f52ac0769c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f52bc034000 tx=0x7f52bc02e000 comp rx=0 tx=0).stop 2024-04-25T15:28:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52cc0b7810 msgr2=0x7f52cc0cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52cc0b7810 0x7f52cc0cc430 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f52b400e8a0 tx=0x7f52b400ed60 comp rx=0 tx=0).stop 2024-04-25T15:28:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 shutdown_connections 2024-04-25T15:28:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f52ac0744b0 0x7f52ac0769c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52cc0d0e00 0x7f52cc1fa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52cc0b7810 0x7f52cc0cc430 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 --2- 172.21.15.67:0/754818867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52cc06c5c0 0x7f52cc154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 >> 172.21.15.67:0/754818867 conn(0x7f52cc0d5d90 msgr2=0x7f52cc1601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 shutdown_connections 2024-04-25T15:28:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:33.933+0000 7f52d1c49640 1 -- 172.21.15.67:0/754818867 wait complete. 2024-04-25T15:28:33.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:28:34.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.056+0000 7f3d9571c640 1 Processor -- start 2024-04-25T15:28:34.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.056+0000 7f3d9571c640 1 -- start start 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d9571c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d900d3790 0x7f3d900d3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d9571c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900b1500 0x7f3d900b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d9571c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d900b0740 con 0x7f3d900d1000 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d9571c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d900d2b30 con 0x7f3d900d3790 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d9571c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d900b0a20 con 0x7f3d900b1500 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8e7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d900d3790 0x7f3d900d3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8e7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d900d3790 0x7f3d900d3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57116/0 (socket says 172.21.15.67:57116) 2024-04-25T15:28:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8e7fc640 1 -- 172.21.15.67:0/2845272550 learned_addr learned my addr 172.21.15.67:0/2845272550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8f7fe640 1 --2- 172.21.15.67:0/2845272550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900b1500 0x7f3d900b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8e7fc640 1 -- 172.21.15.67:0/2845272550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900b1500 msgr2=0x7f3d900b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8e7fc640 1 --2- 172.21.15.67:0/2845272550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900b1500 0x7f3d900b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8e7fc640 1 -- 172.21.15.67:0/2845272550 >> v1:172.21.15.176:6789/0 conn(0x7f3d900d1000 legacy=0x7f3d900d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.058+0000 7f3d8e7fc640 1 -- 172.21.15.67:0/2845272550 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d900b0d00 con 0x7f3d900d3790 2024-04-25T15:28:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.059+0000 7f3d8e7fc640 1 --2- 172.21.15.67:0/2845272550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900d3790 0x7f3d900d3ba0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009870 tx=0x7f3d7802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d6367a95cccb10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.059+0000 7f3d8dffb640 1 -- 172.21.15.67:0/2845272550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78037070 con 0x7f3d900d3790 2024-04-25T15:28:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.059+0000 7f3d8dffb640 1 -- 172.21.15.67:0/2845272550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d78032440 con 0x7f3d900d3790 2024-04-25T15:28:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.059+0000 7f3d8dffb640 1 -- 172.21.15.67:0/2845272550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78032810 con 0x7f3d900d3790 2024-04-25T15:28:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 -- 172.21.15.67:0/2845272550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900d3790 msgr2=0x7f3d900d3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 --2- 172.21.15.67:0/2845272550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900d3790 0x7f3d900d3ba0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009870 tx=0x7f3d7802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 -- 172.21.15.67:0/2845272550 shutdown_connections 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 --2- 172.21.15.67:0/2845272550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900b1500 0x7f3d900b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 --2- 172.21.15.67:0/2845272550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900d3790 0x7f3d900d3ba0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 -- 172.21.15.67:0/2845272550 >> 172.21.15.67:0/2845272550 conn(0x7f3d9008d110 msgr2=0x7f3d9008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 -- 172.21.15.67:0/2845272550 shutdown_connections 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.060+0000 7f3d9571c640 1 -- 172.21.15.67:0/2845272550 wait complete. 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.062+0000 7f3d9571c640 1 Processor -- start 2024-04-25T15:28:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.062+0000 7f3d9571c640 1 -- start start 2024-04-25T15:28:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d9571c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900b1500 0x7f3d900ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d9571c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d900d1000 0x7f3d90152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d9571c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900d3790 0x7f3d901533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d9571c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d900b2ad0 con 0x7f3d900b1500 2024-04-25T15:28:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d9571c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d900d40e0 con 0x7f3d900d1000 2024-04-25T15:28:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d9571c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d900d2780 con 0x7f3d900d3790 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d8effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900b1500 0x7f3d900ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d8effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900b1500 0x7f3d900ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57130/0 (socket says 172.21.15.67:57130) 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.063+0000 7f3d8effd640 1 -- 172.21.15.67:0/3406331126 learned_addr learned my addr 172.21.15.67:0/3406331126 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8e7fc640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d900d1000 0x7f3d90152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8f7fe640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900d3790 0x7f3d901533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8effd640 1 -- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900d3790 msgr2=0x7f3d901533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8effd640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900d3790 0x7f3d901533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8effd640 1 -- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d900d1000 msgr2=0x7f3d90152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8effd640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d900d1000 0x7f3d90152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8effd640 1 -- 172.21.15.67:0/3406331126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d900c9a00 con 0x7f3d900b1500 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8f7fe640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900d3790 0x7f3d901533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8e7fc640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d900d1000 0x7f3d90152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d8effd640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900b1500 0x7f3d900ca9d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800c3c0 tx=0x7f3d8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d877fe640 1 -- 172.21.15.67:0/3406331126 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d880052d0 con 0x7f3d900b1500 2024-04-25T15:28:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d877fe640 1 -- 172.21.15.67:0/3406331126 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d88005490 con 0x7f3d900b1500 2024-04-25T15:28:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d900c9ce0 con 0x7f3d900b1500 2024-04-25T15:28:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d877fe640 1 -- 172.21.15.67:0/3406331126 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88007260 con 0x7f3d900b1500 2024-04-25T15:28:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.064+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d90152820 con 0x7f3d900b1500 2024-04-25T15:28:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.065+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d90050740 con 0x7f3d900b1500 2024-04-25T15:28:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.066+0000 7f3d877fe640 1 -- 172.21.15.67:0/3406331126 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d88007400 con 0x7f3d900b1500 2024-04-25T15:28:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.066+0000 7f3d877fe640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d700744b0 0x7f3d700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.066+0000 7f3d877fe640 1 -- 172.21.15.67:0/3406331126 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f3d88094660 con 0x7f3d900b1500 2024-04-25T15:28:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.066+0000 7f3d8e7fc640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d700744b0 0x7f3d700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:34.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.067+0000 7f3d8e7fc640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d700744b0 0x7f3d700769c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3d78034000 tx=0x7f3d7802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.071+0000 7f3d877fe640 1 -- 172.21.15.67:0/3406331126 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d8805ee80 con 0x7f3d900b1500 2024-04-25T15:28:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.297+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3d90152a50 con 0x7f3d900b1500 2024-04-25T15:28:34.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.297+0000 7f3d877fe640 1 -- 172.21.15.67:0/3406331126 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) ==== 53+0+73 (secure 0 0 0) 0x7f3d88065e20 con 0x7f3d900b1500 2024-04-25T15:28:34.298 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:28:34.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d700744b0 msgr2=0x7f3d700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:34.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d700744b0 0x7f3d700769c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3d78034000 tx=0x7f3d7802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:34.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900b1500 msgr2=0x7f3d900ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:34.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900b1500 0x7f3d900ca9d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800c3c0 tx=0x7f3d8800c880 comp rx=0 tx=0).stop 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 shutdown_connections 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3d700744b0 0x7f3d700769c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d900d3790 0x7f3d901533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d900d1000 0x7f3d90152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 --2- 172.21.15.67:0/3406331126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d900b1500 0x7f3d900ca9d0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 >> 172.21.15.67:0/3406331126 conn(0x7f3d9008d110 msgr2=0x7f3d90071810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.299+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 shutdown_connections 2024-04-25T15:28:34.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34.300+0000 7f3d9571c640 1 -- 172.21.15.67:0/3406331126 wait complete. 2024-04-25T15:28:34.311 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:28:34.312 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:28:34.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:34.312 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:28:34.328 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42585sec preferred_lft 42585sec 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7482:42ff:fe30:5ffc/64 scope link 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:28:34.329 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:28:34.330 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::885:bff:fe04:f6cb/64 scope link 2024-04-25T15:28:34.330 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:34.330 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:28:34.330 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ca:af:51:1f:d5:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:28:34.330 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8af:51ff:fe1f:d5b0/64 scope link 2024-04-25T15:28:34.330 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:34.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:34.330 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:28:34.391 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:28:34.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:34.393 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:28:34.449 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:28:34.449 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:28:34.449 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:28:34.449 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:28:34.450 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:28:34.450 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:28:34.450 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:28:34.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:28:34.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:28:34.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:28:34.450 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:28:34.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:28:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:28:34.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:34.575 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:28:34.591 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:28:34.591 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:28:34.591 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:28:34.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:34.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:28:34.656 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:28:34.656 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:28:34.711 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:28:34.712 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:28:34.712 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:28:34.766 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T15:28:34.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:34.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:34.860 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:34.860 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:28:34.871 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:28:34.872 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:28:34.872 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:28:34.873 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:28:34.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:34.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:28:34.901 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:28:34.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:28:34.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:28:35.005 INFO:teuthology.orchestra.run.smithi067.stdout:client6894 2024-04-25T15:28:35.005 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:35.005 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:35.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:28:35.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.172+0000 7effb659d640 1 Processor -- start 2024-04-25T15:28:35.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.172+0000 7effb659d640 1 -- start start 2024-04-25T15:28:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb659d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00b4e30 0x7effb00d28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb659d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effb00d2e20 0x7effb00b7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb659d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb00d3230 con 0x7effb00b7ab0 2024-04-25T15:28:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb659d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7effb0163dc0 con 0x7effb00d2e20 2024-04-25T15:28:35.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb659d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7effb00b5240 con 0x7effb00b4e30 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb4d9a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effb00d2e20 0x7effb00b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb4d9a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effb00d2e20 0x7effb00b7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57146/0 (socket says 172.21.15.67:57146) 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb4d9a640 1 -- 172.21.15.67:0/3034203905 learned_addr learned my addr 172.21.15.67:0/3034203905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb5d9c640 1 --1- 172.21.15.67:0/3034203905 >> v1:172.21.15.67:6789/0 conn(0x7effb00b7ab0 0x7effb00d4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60496/0 (socket says 172.21.15.67:60496) 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb559b640 1 --2- 172.21.15.67:0/3034203905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00b4e30 0x7effb00d28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb4d9a640 1 -- 172.21.15.67:0/3034203905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00b4e30 msgr2=0x7effb00d28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb4d9a640 1 --2- 172.21.15.67:0/3034203905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00b4e30 0x7effb00d28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb4d9a640 1 -- 172.21.15.67:0/3034203905 >> v1:172.21.15.67:6789/0 conn(0x7effb00b7ab0 legacy=0x7effb00d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effb4d9a640 1 -- 172.21.15.67:0/3034203905 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb00ba150 con 0x7effb00d2e20 2024-04-25T15:28:35.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.174+0000 7effaffff640 1 -- 172.21.15.67:0/3034203905 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649831598 0 0) 0x7effa0001490 con 0x7effb00b7ab0 2024-04-25T15:28:35.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.175+0000 7effb4d9a640 1 --2- 172.21.15.67:0/3034203905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00d2e20 0x7effb00b7570 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7effa4009870 tx=0x7effa402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e367e9e9e9d02d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:35.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.175+0000 7effaffff640 1 -- 172.21.15.67:0/3034203905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4037070 con 0x7effb00d2e20 2024-04-25T15:28:35.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.175+0000 7effaffff640 1 -- 172.21.15.67:0/3034203905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa4032440 con 0x7effb00d2e20 2024-04-25T15:28:35.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.175+0000 7effaffff640 1 -- 172.21.15.67:0/3034203905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4032810 con 0x7effb00d2e20 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 -- 172.21.15.67:0/3034203905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00d2e20 msgr2=0x7effb00b7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 --2- 172.21.15.67:0/3034203905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00d2e20 0x7effb00b7570 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7effa4009870 tx=0x7effa402a270 comp rx=0 tx=0).stop 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 -- 172.21.15.67:0/3034203905 shutdown_connections 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 --2- 172.21.15.67:0/3034203905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00d2e20 0x7effb00b7570 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 --2- 172.21.15.67:0/3034203905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00b4e30 0x7effb00d28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 -- 172.21.15.67:0/3034203905 >> 172.21.15.67:0/3034203905 conn(0x7effb006e020 msgr2=0x7effb006e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 -- 172.21.15.67:0/3034203905 shutdown_connections 2024-04-25T15:28:35.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.176+0000 7effb659d640 1 -- 172.21.15.67:0/3034203905 wait complete. 2024-04-25T15:28:35.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.178+0000 7effb659d640 1 Processor -- start 2024-04-25T15:28:35.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.178+0000 7effb659d640 1 -- start start 2024-04-25T15:28:35.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb659d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00b4e30 0x7effb015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:35.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb659d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb00b7ab0 0x7effb01630c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:35.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb659d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00d2e20 0x7effb0156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:35.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb659d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effb01642f0 con 0x7effb00b4e30 2024-04-25T15:28:35.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb659d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7effb0164010 con 0x7effb00b7ab0 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb659d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7effb01645d0 con 0x7effb00d2e20 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb559b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00b4e30 0x7effb015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb559b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00b4e30 0x7effb015bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57156/0 (socket says 172.21.15.67:57156) 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb559b640 1 -- 172.21.15.67:0/1475106024 learned_addr learned my addr 172.21.15.67:0/1475106024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb559b640 1 -- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00d2e20 msgr2=0x7effb0156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb5d9c640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00d2e20 0x7effb0156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.179+0000 7effb4d9a640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb00b7ab0 0x7effb01630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb559b640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00d2e20 0x7effb0156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb559b640 1 -- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb00b7ab0 msgr2=0x7effb01630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb559b640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb00b7ab0 0x7effb01630c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb559b640 1 -- 172.21.15.67:0/1475106024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb00cf960 con 0x7effb00b4e30 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb5d9c640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00d2e20 0x7effb0156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb4d9a640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb00b7ab0 0x7effb01630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:35.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb559b640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00b4e30 0x7effb015bb20 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7effa800c820 tx=0x7effa800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:35.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effadffb640 1 -- 172.21.15.67:0/1475106024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8015070 con 0x7effb00b4e30 2024-04-25T15:28:35.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effadffb640 1 -- 172.21.15.67:0/1475106024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa8004020 con 0x7effb00b4e30 2024-04-25T15:28:35.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb00cf680 con 0x7effb00b4e30 2024-04-25T15:28:35.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effadffb640 1 -- 172.21.15.67:0/1475106024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa80043f0 con 0x7effb00b4e30 2024-04-25T15:28:35.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.180+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb015b320 con 0x7effb00b4e30 2024-04-25T15:28:35.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.181+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb0050740 con 0x7effb00b4e30 2024-04-25T15:28:35.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.182+0000 7effadffb640 1 -- 172.21.15.67:0/1475106024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7effa8007830 con 0x7effb00b4e30 2024-04-25T15:28:35.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.182+0000 7effadffb640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7eff9c0744b0 0x7eff9c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:35.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.182+0000 7effadffb640 1 -- 172.21.15.67:0/1475106024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7effa8095030 con 0x7effb00b4e30 2024-04-25T15:28:35.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.182+0000 7effb4d9a640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7eff9c0744b0 0x7eff9c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:35.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.183+0000 7effb4d9a640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7eff9c0744b0 0x7eff9c0769c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7effa4034000 tx=0x7effa402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:35.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.187+0000 7effadffb640 1 -- 172.21.15.67:0/1475106024 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effa805f740 con 0x7effb00b4e30 2024-04-25T15:28:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.412+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7effb00ba6d0 con 0x7effb00b4e30 2024-04-25T15:28:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.412+0000 7effadffb640 1 -- 172.21.15.67:0/1475106024 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) ==== 53+0+73 (secure 0 0 0) 0x7effa805f040 con 0x7effb00b4e30 2024-04-25T15:28:35.413 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7eff9c0744b0 msgr2=0x7eff9c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7eff9c0744b0 0x7eff9c0769c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7effa4034000 tx=0x7effa402e000 comp rx=0 tx=0).stop 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00b4e30 msgr2=0x7effb015bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00b4e30 0x7effb015bb20 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7effa800c820 tx=0x7effa800cce0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 shutdown_connections 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7eff9c0744b0 0x7eff9c0769c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb00d2e20 0x7effb0156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb00b7ab0 0x7effb01630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 --2- 172.21.15.67:0/1475106024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb00b4e30 0x7effb015bb20 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:35.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 >> 172.21.15.67:0/1475106024 conn(0x7effb006e020 msgr2=0x7effb00bb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:35.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 shutdown_connections 2024-04-25T15:28:35.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:35.414+0000 7effb659d640 1 -- 172.21.15.67:0/1475106024 wait complete. 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:28:35.426 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:28:35.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:35.427 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:28:35.445 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:28:35.445 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:28:35.446 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42573sec preferred_lft 42573sec 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:35.447 INFO:teuthology.orchestra.run.smithi176.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:28:35.448 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:28:35.448 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::248e:2bff:fe69:7f63/64 scope link 2024-04-25T15:28:35.448 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:28:35.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:35.448 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:28:35.512 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:28:35.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:35.514 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:28:35.572 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:28:35.572 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:28:35.573 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:28:35.573 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:28:35.573 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:28:35.573 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:28:35.573 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:28:35.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:28:35.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:28:35.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:28:35.573 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:28:35.667 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:28:35.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:28:35.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:35.706 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:28:35.720 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:28:35.721 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:28:35.721 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:28:35.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:35.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:28:35.785 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:28:35.785 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:35.840 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:28:35.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:28:35.841 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:35.895 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T15:28:35.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:35.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:35.987 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:35.988 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:35.996 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:28:35.996 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:28:35.996 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:28:35.996 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:28:35.996 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:28:35.997 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:28:35.997 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:28:35.997 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:28:35.997 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:28:35.997 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:28:35.998 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:28:35.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:35.999 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:36.021 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:28:36.109 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:28:36.111 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:28:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:28:36.116 INFO:teuthology.orchestra.run.smithi176.stdout:client6478 2024-04-25T15:28:36.117 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:36.117 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:36.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:36.132 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-25T15:28:36.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:36.156 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-25T15:28:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:28:36.216 INFO:teuthology.orchestra.run.smithi067.stderr:./subdir: ceph.quota.max_files: No such attribute 2024-04-25T15:28:36.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:36.217 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-25T15:28:36.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:28:36.233 INFO:teuthology.orchestra.run.smithi176.stderr:./subdir: ceph.quota.max_files: No such attribute 2024-04-25T15:28:36.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:36.233 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 10 ./subdir) 2024-04-25T15:28:36.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:36.278 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-25T15:28:36.338 INFO:teuthology.orchestra.run.smithi067.stdout:10 2024-04-25T15:28:36.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:36.338 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-04-25T15:28:36.353 INFO:teuthology.orchestra.run.smithi176.stdout:10 2024-04-25T15:28:36.353 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:28:36.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:36.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:36.467 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:28:36.467 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:36.467 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:36.468 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:36.485 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:36.485 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:36.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:36.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.703+0000 7fc803a67640 1 Processor -- start 2024-04-25T15:28:36.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.703+0000 7fc803a67640 1 -- start start 2024-04-25T15:28:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc803a67640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7fc06ccd0 0x7fc7fc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc803a67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d4a50 0x7fc7fc0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc803a67640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7fc0d1850 con 0x7fc7fc0d0e00 2024-04-25T15:28:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc803a67640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc7fc160de0 con 0x7fc7fc06ccd0 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc803a67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc7fc1610c0 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc801002640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7fc06ccd0 0x7fc7fc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc801803640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc7fc0d0e00 0x7fc7fc06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60504/0 (socket says 172.21.15.67:60504) 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc801803640 1 -- 172.21.15.67:0/2389208243 learned_addr learned my addr 172.21.15.67:0/2389208243 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc802004640 1 --2- 172.21.15.67:0/2389208243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d4a50 0x7fc7fc0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc801002640 1 -- 172.21.15.67:0/2389208243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d4a50 msgr2=0x7fc7fc0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc801002640 1 --2- 172.21.15.67:0/2389208243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d4a50 0x7fc7fc0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc801002640 1 -- 172.21.15.67:0/2389208243 >> v1:172.21.15.67:6789/0 conn(0x7fc7fc0d0e00 legacy=0x7fc7fc06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc801002640 1 -- 172.21.15.67:0/2389208243 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7fc1613a0 con 0x7fc7fc06ccd0 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc802004640 1 --2- 172.21.15.67:0/2389208243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d4a50 0x7fc7fc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.705+0000 7fc800801640 1 -- 172.21.15.67:0/2389208243 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 18949587 0 0) 0x7fc7fc0d1850 con 0x7fc7fc0d0e00 2024-04-25T15:28:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.706+0000 7fc801002640 1 --2- 172.21.15.67:0/2389208243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc06ccd0 0x7fc7fc0d1210 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00097a0 tx=0x7fc7f002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96a917da1455a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.706+0000 7fc800801640 1 -- 172.21.15.67:0/2389208243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0037070 con 0x7fc7fc06ccd0 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.706+0000 7fc800801640 1 -- 172.21.15.67:0/2389208243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f0032440 con 0x7fc7fc06ccd0 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.706+0000 7fc800801640 1 -- 172.21.15.67:0/2389208243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0032810 con 0x7fc7fc06ccd0 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 -- 172.21.15.67:0/2389208243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc06ccd0 msgr2=0x7fc7fc0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 --2- 172.21.15.67:0/2389208243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc06ccd0 0x7fc7fc0d1210 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00097a0 tx=0x7fc7f002a180 comp rx=0 tx=0).stop 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 -- 172.21.15.67:0/2389208243 shutdown_connections 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 --2- 172.21.15.67:0/2389208243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d4a50 0x7fc7fc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 --2- 172.21.15.67:0/2389208243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc06ccd0 0x7fc7fc0d1210 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 -- 172.21.15.67:0/2389208243 >> 172.21.15.67:0/2389208243 conn(0x7fc7fc0d5d90 msgr2=0x7fc7fc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 -- 172.21.15.67:0/2389208243 shutdown_connections 2024-04-25T15:28:36.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.707+0000 7fc803a67640 1 -- 172.21.15.67:0/2389208243 wait complete. 2024-04-25T15:28:36.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.709+0000 7fc803a67640 1 Processor -- start 2024-04-25T15:28:36.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.709+0000 7fc803a67640 1 -- start start 2024-04-25T15:28:36.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc803a67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d0e00 0x7fc7fc15ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc803a67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc0d4a50 0x7fc7fc0b7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc803a67640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7fc15b4c0 0x7fc7fc0b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc803a67640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc164e00 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc803a67640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc7fc0b5320 con 0x7fc7fc15b4c0 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc803a67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc7fc164f80 con 0x7fc7fc0d0e00 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc0d4a50 0x7fc7fc0b7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc0d4a50 0x7fc7fc0b7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57174/0 (socket says 172.21.15.67:57174) 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 -- 172.21.15.67:0/4147619206 learned_addr learned my addr 172.21.15.67:0/4147619206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801803640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d0e00 0x7fc7fc15ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc802004640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7fc15b4c0 0x7fc7fc0b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 -- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d0e00 msgr2=0x7fc7fc15ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d0e00 0x7fc7fc15ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 -- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7fc15b4c0 msgr2=0x7fc7fc0b7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:36.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7fc15b4c0 0x7fc7fc0b7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 -- 172.21.15.67:0/4147619206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7fc0c9670 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc801002640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc0d4a50 0x7fc7fc0b7860 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00098c0 tx=0x7fc7f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.711+0000 7fc7fa7fc640 1 -- 172.21.15.67:0/4147619206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0037040 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.712+0000 7fc7fa7fc640 1 -- 172.21.15.67:0/4147619206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f0030770 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.712+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7fc0ceb00 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.712+0000 7fc7fa7fc640 1 -- 172.21.15.67:0/4147619206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0030b40 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.712+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7fc0cc3f0 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.713+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7fc0cfe60 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.713+0000 7fc7fa7fc640 1 -- 172.21.15.67:0/4147619206 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc7f0044020 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.714+0000 7fc7fa7fc640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7e80744e0 0x7fc7e80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:36.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.714+0000 7fc7fa7fc640 1 -- 172.21.15.67:0/4147619206 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fc7f00b5520 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.714+0000 7fc801803640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7e80744e0 0x7fc7e80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:36.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.714+0000 7fc801803640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7e80744e0 0x7fc7e80769f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4007fd0 tx=0x7fc7f4007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:36.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.719+0000 7fc7fa7fc640 1 -- 172.21.15.67:0/4147619206 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7f0032e20 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.905+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7fc050740 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.906+0000 7fc7fa7fc640 1 -- 172.21.15.67:0/4147619206 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16624 (secure 0 0 0) 0x7fc7f007f6f0 con 0x7fc7fc0d4a50 2024-04-25T15:28:36.907 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:36.908 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:24.632893+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:36.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7e80744e0 msgr2=0x7fc7e80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:36.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7e80744e0 0x7fc7e80769f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4007fd0 tx=0x7fc7f4007de0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc0d4a50 msgr2=0x7fc7fc0b7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc0d4a50 0x7fc7fc0b7860 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00098c0 tx=0x7fc7f00046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 shutdown_connections 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc7e80744e0 0x7fc7e80769f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7fc15b4c0 0x7fc7fc0b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7fc0d4a50 0x7fc7fc0b7860 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 --2- 172.21.15.67:0/4147619206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7fc0d0e00 0x7fc7fc15ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 >> 172.21.15.67:0/4147619206 conn(0x7fc7fc0d5d90 msgr2=0x7fc7fc0b9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 shutdown_connections 2024-04-25T15:28:36.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:36.908+0000 7fc803a67640 1 -- 172.21.15.67:0/4147619206 wait complete. 2024-04-25T15:28:36.921 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:28:21.329066+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:28:21.948629+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:36.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:37.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.036+0000 7f0966a1a640 1 Processor -- start 2024-04-25T15:28:37.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.036+0000 7f0966a1a640 1 -- start start 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.037+0000 7f0966a1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09600cd9a0 0x7f096015f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.037+0000 7f0966a1a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09600cdea0 con 0x7f09600b0af0 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.037+0000 7f0966a1a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09600ce180 con 0x7f096006dce0 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.037+0000 7f0966a1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f09600ceab0 con 0x7f09600cd9a0 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.037+0000 7f095f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f09600b0af0 0x7f096006d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60506/0 (socket says 172.21.15.67:60506) 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.037+0000 7f095f7fe640 1 -- 172.21.15.67:0/3449363331 learned_addr learned my addr 172.21.15.67:0/3449363331 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.037+0000 7f095ffff640 1 --2- 172.21.15.67:0/3449363331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09600cd9a0 0x7f096015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.038+0000 7f095effd640 1 -- 172.21.15.67:0/3449363331 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348952697 0 0) 0x7f09600cdea0 con 0x7f09600b0af0 2024-04-25T15:28:37.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.038+0000 7f095effd640 1 -- 172.21.15.67:0/3449363331 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0948003520 con 0x7f09600b0af0 2024-04-25T15:28:37.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.038+0000 7f095effd640 1 -- 172.21.15.67:0/3449363331 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1808116656 0 0) 0x7f09600ce180 con 0x7f096006dce0 2024-04-25T15:28:37.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.038+0000 7f095effd640 1 -- 172.21.15.67:0/3449363331 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09600cdea0 con 0x7f096006dce0 2024-04-25T15:28:37.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.038+0000 7f095ffff640 1 -- 172.21.15.67:0/3449363331 >> v1:172.21.15.176:6789/0 conn(0x7f096006dce0 legacy=0x7f096015faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.038+0000 7f095ffff640 1 -- 172.21.15.67:0/3449363331 >> v1:172.21.15.67:6789/0 conn(0x7f09600b0af0 legacy=0x7f096006d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.038+0000 7f095ffff640 1 -- 172.21.15.67:0/3449363331 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09600ce460 con 0x7f09600cd9a0 2024-04-25T15:28:37.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f095ffff640 1 --2- 172.21.15.67:0/3449363331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09600cd9a0 0x7f096015f360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09500097d0 tx=0x7f095002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d1fff23eacfddfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f095effd640 1 -- 172.21.15.67:0/3449363331 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950037070 con 0x7f09600cd9a0 2024-04-25T15:28:37.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f095effd640 1 -- 172.21.15.67:0/3449363331 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0950032440 con 0x7f09600cd9a0 2024-04-25T15:28:37.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f095effd640 1 -- 172.21.15.67:0/3449363331 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950032810 con 0x7f09600cd9a0 2024-04-25T15:28:37.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f0966a1a640 1 -- 172.21.15.67:0/3449363331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09600cd9a0 msgr2=0x7f096015f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3449363331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09600cd9a0 0x7f096015f360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09500097d0 tx=0x7f095002a210 comp rx=0 tx=0).stop 2024-04-25T15:28:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f0966a1a640 1 -- 172.21.15.67:0/3449363331 shutdown_connections 2024-04-25T15:28:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3449363331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09600cd9a0 0x7f096015f360 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.039+0000 7f0966a1a640 1 -- 172.21.15.67:0/3449363331 >> 172.21.15.67:0/3449363331 conn(0x7f0960073090 msgr2=0x7f09600734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.040+0000 7f0966a1a640 1 -- 172.21.15.67:0/3449363331 shutdown_connections 2024-04-25T15:28:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.040+0000 7f0966a1a640 1 -- 172.21.15.67:0/3449363331 wait complete. 2024-04-25T15:28:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.041+0000 7f0966a1a640 1 Processor -- start 2024-04-25T15:28:37.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.041+0000 7f0966a1a640 1 -- start start 2024-04-25T15:28:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f0966a1a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09600b0af0 0x7f096013c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f0966a1a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09600cd9a0 0x7f09601587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f0966a1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f096013ccb0 0x7f09600b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f0966a1a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f09600cef40 con 0x7f09600cd9a0 2024-04-25T15:28:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f0966a1a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f09600b0f00 con 0x7f09600b0af0 2024-04-25T15:28:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f0966a1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f09600cf0c0 con 0x7f096013ccb0 2024-04-25T15:28:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09600cd9a0 0x7f09601587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09600cd9a0 0x7f09601587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57182/0 (socket says 172.21.15.67:57182) 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 -- 172.21.15.67:0/3862787915 learned_addr learned my addr 172.21.15.67:0/3862787915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095ffff640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09600b0af0 0x7f096013c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f0964fb7640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f096013ccb0 0x7f09600b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 -- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f096013ccb0 msgr2=0x7f09600b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f096013ccb0 0x7f09600b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 -- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09600b0af0 msgr2=0x7f096013c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09600b0af0 0x7f096013c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 -- 172.21.15.67:0/3862787915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096013d250 con 0x7f09600cd9a0 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095ffff640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09600b0af0 0x7f096013c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.043+0000 7f095f7fe640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09600cd9a0 0x7f09601587b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f095400c3c0 tx=0x7f095400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.044+0000 7f095cff9640 1 -- 172.21.15.67:0/3862787915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954017070 con 0x7f09600cd9a0 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.044+0000 7f095cff9640 1 -- 172.21.15.67:0/3862787915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0954005da0 con 0x7f09600cd9a0 2024-04-25T15:28:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.044+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096013d530 con 0x7f09600cd9a0 2024-04-25T15:28:37.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.044+0000 7f095cff9640 1 -- 172.21.15.67:0/3862787915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954007310 con 0x7f09600cd9a0 2024-04-25T15:28:37.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.044+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096013c700 con 0x7f09600cd9a0 2024-04-25T15:28:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.045+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0960050740 con 0x7f09600cd9a0 2024-04-25T15:28:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.045+0000 7f095cff9640 1 -- 172.21.15.67:0/3862787915 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f095401a070 con 0x7f09600cd9a0 2024-04-25T15:28:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.046+0000 7f095cff9640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f094c0744b0 0x7f094c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.046+0000 7f095cff9640 1 -- 172.21.15.67:0/3862787915 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f0954094590 con 0x7f09600cd9a0 2024-04-25T15:28:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.046+0000 7f095ffff640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f094c0744b0 0x7f094c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.047+0000 7f095ffff640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f094c0744b0 0x7f094c0769c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0950034000 tx=0x7f095002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.050+0000 7f095cff9640 1 -- 172.21.15.67:0/3862787915 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f095405edb0 con 0x7f09600cd9a0 2024-04-25T15:28:37.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.233+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f09600008d0 con 0x7f09600cd9a0 2024-04-25T15:28:37.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.233+0000 7f095cff9640 1 -- 172.21.15.67:0/3862787915 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7f095405e6b0 con 0x7f09600cd9a0 2024-04-25T15:28:37.234 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f094c0744b0 msgr2=0x7f094c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f094c0744b0 0x7f094c0769c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0950034000 tx=0x7f095002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09600cd9a0 msgr2=0x7f09601587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09600cd9a0 0x7f09601587b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f095400c3c0 tx=0x7f095400c880 comp rx=0 tx=0).stop 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 shutdown_connections 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f094c0744b0 0x7f094c0769c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f096013ccb0 0x7f09600b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09600cd9a0 0x7f09601587b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 --2- 172.21.15.67:0/3862787915 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09600b0af0 0x7f096013c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 >> 172.21.15.67:0/3862787915 conn(0x7f0960073090 msgr2=0x7f096006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 shutdown_connections 2024-04-25T15:28:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.235+0000 7f0966a1a640 1 -- 172.21.15.67:0/3862787915 wait complete. 2024-04-25T15:28:37.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:28:37.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.361+0000 7ff630c92640 1 Processor -- start 2024-04-25T15:28:37.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.361+0000 7ff630c92640 1 -- start start 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.362+0000 7ff630c92640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff62c164ad0 0x7ff62c0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.362+0000 7ff630c92640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff62c163f30 0x7ff62c164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff630c92640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff62c164880 con 0x7ff62c0d1f30 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff630c92640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff62c164ee0 con 0x7ff62c163f30 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff630c92640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff62c0b31e0 con 0x7ff62c164ad0 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff629d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff62c163f30 0x7ff62c164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff629d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff62c163f30 0x7ff62c164340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57190/0 (socket says 172.21.15.67:57190) 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff629d9b640 1 -- 172.21.15.67:0/4204057740 learned_addr learned my addr 172.21.15.67:0/4204057740 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff62a59c640 1 --2- 172.21.15.67:0/4204057740 >> v2:172.21.15.176:3300/0 conn(0x7ff62c164ad0 0x7ff62c0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff629d9b640 1 -- 172.21.15.67:0/4204057740 >> v2:172.21.15.176:3300/0 conn(0x7ff62c164ad0 msgr2=0x7ff62c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff629d9b640 1 --2- 172.21.15.67:0/4204057740 >> v2:172.21.15.176:3300/0 conn(0x7ff62c164ad0 0x7ff62c0b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff629d9b640 1 -- 172.21.15.67:0/4204057740 >> v1:172.21.15.176:6789/0 conn(0x7ff62c0d1f30 legacy=0x7ff62c0d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.363+0000 7ff629d9b640 1 -- 172.21.15.67:0/4204057740 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c0d3170 con 0x7ff62c163f30 2024-04-25T15:28:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.364+0000 7ff629d9b640 1 --2- 172.21.15.67:0/4204057740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c163f30 0x7ff62c164340 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff62c074530 tx=0x7ff62002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffd5f3fc6feae56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.364+0000 7ff62959a640 1 -- 172.21.15.67:0/4204057740 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620037070 con 0x7ff62c163f30 2024-04-25T15:28:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.364+0000 7ff62959a640 1 -- 172.21.15.67:0/4204057740 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff620032440 con 0x7ff62c163f30 2024-04-25T15:28:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.364+0000 7ff62959a640 1 -- 172.21.15.67:0/4204057740 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620032810 con 0x7ff62c163f30 2024-04-25T15:28:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 -- 172.21.15.67:0/4204057740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c163f30 msgr2=0x7ff62c164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 --2- 172.21.15.67:0/4204057740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c163f30 0x7ff62c164340 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff62c074530 tx=0x7ff62002a270 comp rx=0 tx=0).stop 2024-04-25T15:28:37.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 -- 172.21.15.67:0/4204057740 shutdown_connections 2024-04-25T15:28:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 --2- 172.21.15.67:0/4204057740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c163f30 0x7ff62c164340 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 --2- 172.21.15.67:0/4204057740 >> v2:172.21.15.176:3300/0 conn(0x7ff62c164ad0 0x7ff62c0b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 -- 172.21.15.67:0/4204057740 >> 172.21.15.67:0/4204057740 conn(0x7ff62c0d3aa0 msgr2=0x7ff62c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 -- 172.21.15.67:0/4204057740 shutdown_connections 2024-04-25T15:28:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.365+0000 7ff630c92640 1 -- 172.21.15.67:0/4204057740 wait complete. 2024-04-25T15:28:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.367+0000 7ff630c92640 1 Processor -- start 2024-04-25T15:28:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.367+0000 7ff630c92640 1 -- start start 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff630c92640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c0d1f30 0x7ff62c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff630c92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff62c163f30 0x7ff62c15bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff630c92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff62c164ad0 0x7ff62c1631c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff630c92640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c06d9b0 con 0x7ff62c0d1f30 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff630c92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c06d6d0 con 0x7ff62c163f30 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff630c92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff62c0b9430 con 0x7ff62c164ad0 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff62a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c0d1f30 0x7ff62c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff62a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c0d1f30 0x7ff62c15c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57204/0 (socket says 172.21.15.67:57204) 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff62a59c640 1 -- 172.21.15.67:0/245630591 learned_addr learned my addr 172.21.15.67:0/245630591 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff62ad9d640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff62c164ad0 0x7ff62c1631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.368+0000 7ff629d9b640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff62c163f30 0x7ff62c15bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff62a59c640 1 -- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff62c164ad0 msgr2=0x7ff62c1631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff62a59c640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff62c164ad0 0x7ff62c1631c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff62a59c640 1 -- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff62c163f30 msgr2=0x7ff62c15bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff62a59c640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff62c163f30 0x7ff62c15bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff62a59c640 1 -- 172.21.15.67:0/245630591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c162120 con 0x7ff62c0d1f30 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff62a59c640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c0d1f30 0x7ff62c15c800 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00d5a0 tx=0x7ff61c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff61affd640 1 -- 172.21.15.67:0/245630591 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c019070 con 0x7ff62c0d1f30 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff61affd640 1 -- 172.21.15.67:0/245630591 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff61c004240 con 0x7ff62c0d1f30 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62c162400 con 0x7ff62c0d1f30 2024-04-25T15:28:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff61affd640 1 -- 172.21.15.67:0/245630591 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61c003610 con 0x7ff62c0d1f30 2024-04-25T15:28:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.369+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62c1629c0 con 0x7ff62c0d1f30 2024-04-25T15:28:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.370+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff62c0c72f0 con 0x7ff62c0d1f30 2024-04-25T15:28:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.371+0000 7ff61affd640 1 -- 172.21.15.67:0/245630591 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff61c01c070 con 0x7ff62c0d1f30 2024-04-25T15:28:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.371+0000 7ff61affd640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff60c0744b0 0x7ff60c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.371+0000 7ff61affd640 1 -- 172.21.15.67:0/245630591 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7ff61c094ad0 con 0x7ff62c0d1f30 2024-04-25T15:28:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.372+0000 7ff629d9b640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff60c0744b0 0x7ff60c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.372+0000 7ff629d9b640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff60c0744b0 0x7ff60c0769c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff620034000 tx=0x7ff62002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.376+0000 7ff61affd640 1 -- 172.21.15.67:0/245630591 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff61c05ec10 con 0x7ff62c0d1f30 2024-04-25T15:28:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.560+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff62c0d4770 con 0x7ff62c0d1f30 2024-04-25T15:28:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.560+0000 7ff61affd640 1 -- 172.21.15.67:0/245630591 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7ff61c062f10 con 0x7ff62c0d1f30 2024-04-25T15:28:37.561 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff60c0744b0 msgr2=0x7ff60c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff60c0744b0 0x7ff60c0769c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff620034000 tx=0x7ff62002e000 comp rx=0 tx=0).stop 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c0d1f30 msgr2=0x7ff62c15c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c0d1f30 0x7ff62c15c800 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00d5a0 tx=0x7ff61c00da60 comp rx=0 tx=0).stop 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 shutdown_connections 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff60c0744b0 0x7ff60c0769c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff62c164ad0 0x7ff62c1631c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff62c163f30 0x7ff62c15bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 --2- 172.21.15.67:0/245630591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff62c0d1f30 0x7ff62c15c800 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 >> 172.21.15.67:0/245630591 conn(0x7ff62c0d3aa0 msgr2=0x7ff62c0d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 shutdown_connections 2024-04-25T15:28:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.562+0000 7ff630c92640 1 -- 172.21.15.67:0/245630591 wait complete. 2024-04-25T15:28:37.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:28:37.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.688+0000 7ff6ef896640 1 Processor -- start 2024-04-25T15:28:37.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.689+0000 7ff6ef896640 1 -- start start 2024-04-25T15:28:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ef896640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff6e8161710 0x7ff6e815e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ef896640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e806c5c0 0x7ff6e806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ef896640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6e8163fa0 0x7ff6e81643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ef896640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff6e81609b0 con 0x7ff6e8163fa0 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ef896640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff6e806cf10 con 0x7ff6e8161710 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ef896640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e8164cd0 con 0x7ff6e806c5c0 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ede33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6e8163fa0 0x7ff6e81643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ede33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6e8163fa0 0x7ff6e81643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57744/0 (socket says 172.21.15.67:57744) 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ede33640 1 -- 172.21.15.67:0/2372250189 learned_addr learned my addr 172.21.15.67:0/2372250189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ed632640 1 --2- 172.21.15.67:0/2372250189 >> v2:172.21.15.176:3300/0 conn(0x7ff6e8161710 0x7ff6e815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.690+0000 7ff6ece31640 1 --2- 172.21.15.67:0/2372250189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e806c5c0 0x7ff6e806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6ede33640 1 -- 172.21.15.67:0/2372250189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e806c5c0 msgr2=0x7ff6e806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6ede33640 1 --2- 172.21.15.67:0/2372250189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e806c5c0 0x7ff6e806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6ede33640 1 -- 172.21.15.67:0/2372250189 >> v2:172.21.15.176:3300/0 conn(0x7ff6e8161710 msgr2=0x7ff6e815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6ede33640 1 --2- 172.21.15.67:0/2372250189 >> v2:172.21.15.176:3300/0 conn(0x7ff6e8161710 0x7ff6e815e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6ede33640 1 -- 172.21.15.67:0/2372250189 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e8160c30 con 0x7ff6e8163fa0 2024-04-25T15:28:37.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6ed632640 1 --2- 172.21.15.67:0/2372250189 >> v2:172.21.15.176:3300/0 conn(0x7ff6e8161710 0x7ff6e815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6ede33640 1 --2- 172.21.15.67:0/2372250189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8163fa0 0x7ff6e81643b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80d5cf0 tx=0x7ff6e402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181254c4d6adaa4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6dffff640 1 -- 172.21.15.67:0/2372250189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4036070 con 0x7ff6e8163fa0 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6dffff640 1 -- 172.21.15.67:0/2372250189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e4031600 con 0x7ff6e8163fa0 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.691+0000 7ff6dffff640 1 -- 172.21.15.67:0/2372250189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e40319d0 con 0x7ff6e8163fa0 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 -- 172.21.15.67:0/2372250189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8163fa0 msgr2=0x7ff6e81643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 --2- 172.21.15.67:0/2372250189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8163fa0 0x7ff6e81643b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80d5cf0 tx=0x7ff6e402a610 comp rx=0 tx=0).stop 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 -- 172.21.15.67:0/2372250189 shutdown_connections 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 --2- 172.21.15.67:0/2372250189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8163fa0 0x7ff6e81643b0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 --2- 172.21.15.67:0/2372250189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e806c5c0 0x7ff6e806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 --2- 172.21.15.67:0/2372250189 >> v2:172.21.15.176:3300/0 conn(0x7ff6e8161710 0x7ff6e815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 -- 172.21.15.67:0/2372250189 >> 172.21.15.67:0/2372250189 conn(0x7ff6e80d5d90 msgr2=0x7ff6e80d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 -- 172.21.15.67:0/2372250189 shutdown_connections 2024-04-25T15:28:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.692+0000 7ff6ef896640 1 -- 172.21.15.67:0/2372250189 wait complete. 2024-04-25T15:28:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.694+0000 7ff6ef896640 1 Processor -- start 2024-04-25T15:28:37.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.694+0000 7ff6ef896640 1 -- start start 2024-04-25T15:28:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ef896640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e806c5c0 0x7ff6e80c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ef896640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8161710 0x7ff6e80c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ef896640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8163fa0 0x7ff6e80d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ef896640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81648f0 con 0x7ff6e806c5c0 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ef896640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e81611f0 con 0x7ff6e8161710 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ef896640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e8164a70 con 0x7ff6e8163fa0 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ed632640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e806c5c0 0x7ff6e80c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ed632640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e806c5c0 0x7ff6e80c88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57760/0 (socket says 172.21.15.67:57760) 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ed632640 1 -- 172.21.15.67:0/3882862958 learned_addr learned my addr 172.21.15.67:0/3882862958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.695+0000 7ff6ede33640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8163fa0 0x7ff6e80d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ece31640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8161710 0x7ff6e80c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ed632640 1 -- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8163fa0 msgr2=0x7ff6e80d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ed632640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8163fa0 0x7ff6e80d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ed632640 1 -- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8161710 msgr2=0x7ff6e80c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ed632640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8161710 0x7ff6e80c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ed632640 1 -- 172.21.15.67:0/3882862958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e80c9f10 con 0x7ff6e806c5c0 2024-04-25T15:28:37.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ed632640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e806c5c0 0x7ff6e80c88f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000c820 tx=0x7ff6e000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ddffb640 1 -- 172.21.15.67:0/3882862958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0015070 con 0x7ff6e806c5c0 2024-04-25T15:28:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ddffb640 1 -- 172.21.15.67:0/3882862958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e0007a30 con 0x7ff6e806c5c0 2024-04-25T15:28:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e80cb590 con 0x7ff6e806c5c0 2024-04-25T15:28:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ddffb640 1 -- 172.21.15.67:0/3882862958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0007e20 con 0x7ff6e806c5c0 2024-04-25T15:28:37.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.696+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e80cbb50 con 0x7ff6e806c5c0 2024-04-25T15:28:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.697+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e8050740 con 0x7ff6e806c5c0 2024-04-25T15:28:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.698+0000 7ff6ddffb640 1 -- 172.21.15.67:0/3882862958 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff6e0018070 con 0x7ff6e806c5c0 2024-04-25T15:28:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.698+0000 7ff6ddffb640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6cc0744b0 0x7ff6cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.698+0000 7ff6ddffb640 1 -- 172.21.15.67:0/3882862958 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7ff6e0094d10 con 0x7ff6e806c5c0 2024-04-25T15:28:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.699+0000 7ff6ece31640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6cc0744b0 0x7ff6cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:37.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.699+0000 7ff6ece31640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6cc0744b0 0x7ff6cc0769c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80bba90 tx=0x7ff6d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:37.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.703+0000 7ff6ddffb640 1 -- 172.21.15.67:0/3882862958 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff6e005eee0 con 0x7ff6e806c5c0 2024-04-25T15:28:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.889+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff6e81643b0 con 0x7ff6e806c5c0 2024-04-25T15:28:37.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.889+0000 7ff6ddffb640 1 -- 172.21.15.67:0/3882862958 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7ff6e00631e0 con 0x7ff6e806c5c0 2024-04-25T15:28:37.890 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:28:37.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6cc0744b0 msgr2=0x7ff6cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6cc0744b0 0x7ff6cc0769c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80bba90 tx=0x7ff6d8008040 comp rx=0 tx=0).stop 2024-04-25T15:28:37.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e806c5c0 msgr2=0x7ff6e80c88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e806c5c0 0x7ff6e80c88f0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000c820 tx=0x7ff6e000cce0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 shutdown_connections 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff6cc0744b0 0x7ff6cc0769c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8163fa0 0x7ff6e80d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8161710 0x7ff6e80c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 --2- 172.21.15.67:0/3882862958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e806c5c0 0x7ff6e80c88f0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 >> 172.21.15.67:0/3882862958 conn(0x7ff6e80d5d90 msgr2=0x7ff6e80d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 shutdown_connections 2024-04-25T15:28:37.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:37.891+0000 7ff6ef896640 1 -- 172.21.15.67:0/3882862958 wait complete. 2024-04-25T15:28:37.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:28:38.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.017+0000 7f2e18d72640 1 Processor -- start 2024-04-25T15:28:38.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.017+0000 7f2e18d72640 1 -- start start 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.018+0000 7f2e18d72640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e1406d8e0 0x7f2e140d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.018+0000 7f2e18d72640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2e1406dcf0 0x7f2e140d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.018+0000 7f2e18d72640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e140b3c10 con 0x7f2e140d04a0 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.018+0000 7f2e18d72640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2e140b9520 con 0x7f2e1406d8e0 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.018+0000 7f2e18d72640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2e140b4820 con 0x7f2e1406dcf0 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e1259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e1406d8e0 0x7f2e140d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e1259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e1406d8e0 0x7f2e140d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57764/0 (socket says 172.21.15.67:57764) 2024-04-25T15:28:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e1259c640 1 -- 172.21.15.67:0/806354335 learned_addr learned my addr 172.21.15.67:0/806354335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e0bfff640 1 --2- 172.21.15.67:0/806354335 >> v2:172.21.15.176:3300/0 conn(0x7f2e1406dcf0 0x7f2e140d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e1259c640 1 -- 172.21.15.67:0/806354335 >> v2:172.21.15.176:3300/0 conn(0x7f2e1406dcf0 msgr2=0x7f2e140d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e1259c640 1 --2- 172.21.15.67:0/806354335 >> v2:172.21.15.176:3300/0 conn(0x7f2e1406dcf0 0x7f2e140d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e1259c640 1 -- 172.21.15.67:0/806354335 >> v1:172.21.15.176:6789/0 conn(0x7f2e140d04a0 legacy=0x7f2e140d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e1259c640 1 -- 172.21.15.67:0/806354335 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e140b3ef0 con 0x7f2e1406d8e0 2024-04-25T15:28:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.019+0000 7f2e0bfff640 1 --2- 172.21.15.67:0/806354335 >> v2:172.21.15.176:3300/0 conn(0x7f2e1406dcf0 0x7f2e140d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.020+0000 7f2e1259c640 1 --2- 172.21.15.67:0/806354335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e140d3a20 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f2e1406f790 tx=0x7f2e0402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c28df0f6e0182f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.020+0000 7f2e11d9b640 1 -- 172.21.15.67:0/806354335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04037070 con 0x7f2e1406d8e0 2024-04-25T15:28:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.020+0000 7f2e11d9b640 1 -- 172.21.15.67:0/806354335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e04032440 con 0x7f2e1406d8e0 2024-04-25T15:28:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.020+0000 7f2e11d9b640 1 -- 172.21.15.67:0/806354335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04032810 con 0x7f2e1406d8e0 2024-04-25T15:28:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.020+0000 7f2e18d72640 1 -- 172.21.15.67:0/806354335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 msgr2=0x7f2e140d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.020+0000 7f2e18d72640 1 --2- 172.21.15.67:0/806354335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e140d3a20 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f2e1406f790 tx=0x7f2e0402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.021+0000 7f2e18d72640 1 -- 172.21.15.67:0/806354335 shutdown_connections 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.021+0000 7f2e18d72640 1 --2- 172.21.15.67:0/806354335 >> v2:172.21.15.176:3300/0 conn(0x7f2e1406dcf0 0x7f2e140d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.021+0000 7f2e18d72640 1 --2- 172.21.15.67:0/806354335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e140d3a20 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.021+0000 7f2e18d72640 1 -- 172.21.15.67:0/806354335 >> 172.21.15.67:0/806354335 conn(0x7f2e1408d110 msgr2=0x7f2e1408d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.021+0000 7f2e18d72640 1 -- 172.21.15.67:0/806354335 shutdown_connections 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.021+0000 7f2e18d72640 1 -- 172.21.15.67:0/806354335 wait complete. 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.022+0000 7f2e18d72640 1 Processor -- start 2024-04-25T15:28:38.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.022+0000 7f2e18d72640 1 -- start start 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e18d72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e141e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e18d72640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e1406dcf0 0x7f2e141f5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e18d72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e140d04a0 0x7f2e140a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e18d72640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e140b38e0 con 0x7f2e1406d8e0 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e18d72640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2e140b44b0 con 0x7f2e1406dcf0 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e18d72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2e140d0fc0 con 0x7f2e140d04a0 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e141e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e141e6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57772/0 (socket says 172.21.15.67:57772) 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 -- 172.21.15.67:0/3317826882 learned_addr learned my addr 172.21.15.67:0/3317826882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e0bfff640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e1406dcf0 0x7f2e141f5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e12d9d640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e140d04a0 0x7f2e140a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 -- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e140d04a0 msgr2=0x7f2e140a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e140d04a0 0x7f2e140a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 -- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e1406dcf0 msgr2=0x7f2e141f5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e1406dcf0 0x7f2e141f5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 -- 172.21.15.67:0/3317826882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e14154910 con 0x7f2e1406d8e0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e12d9d640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e140d04a0 0x7f2e140a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.024+0000 7f2e1259c640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e141e6a30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f2e0402a1e0 tx=0x7f2e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.025+0000 7f2e0affd640 1 -- 172.21.15.67:0/3317826882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04037040 con 0x7f2e1406d8e0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.025+0000 7f2e0affd640 1 -- 172.21.15.67:0/3317826882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e04042070 con 0x7f2e1406d8e0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.025+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e141e75a0 con 0x7f2e1406d8e0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.025+0000 7f2e0affd640 1 -- 172.21.15.67:0/3317826882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04030bb0 con 0x7f2e1406d8e0 2024-04-25T15:28:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.025+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e141e7b60 con 0x7f2e1406d8e0 2024-04-25T15:28:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.026+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e14050740 con 0x7f2e1406d8e0 2024-04-25T15:28:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.026+0000 7f2e0affd640 1 -- 172.21.15.67:0/3317826882 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2e04047460 con 0x7f2e1406d8e0 2024-04-25T15:28:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.027+0000 7f2e0affd640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2df40744b0 0x7f2df40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.027+0000 7f2e0affd640 1 -- 172.21.15.67:0/3317826882 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f2e040b5de0 con 0x7f2e1406d8e0 2024-04-25T15:28:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.027+0000 7f2e0bfff640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2df40744b0 0x7f2df40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.028+0000 7f2e0bfff640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2df40744b0 0x7f2df40769c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2e141f96d0 tx=0x7f2dfc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.031+0000 7f2e0affd640 1 -- 172.21.15.67:0/3317826882 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e04080600 con 0x7f2e1406d8e0 2024-04-25T15:28:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.229+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2e140d4010 con 0x7f2e1406d8e0 2024-04-25T15:28:38.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.230+0000 7f2e0affd640 1 -- 172.21.15.67:0/3317826882 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v80) ==== 80+0+0 (secure 0 0 0) 0x7f2e0407ff00 con 0x7f2e1406d8e0 2024-04-25T15:28:38.231 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.231+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2df40744b0 msgr2=0x7f2df40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.231+0000 7f2e18d72640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2df40744b0 0x7f2df40769c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2e141f96d0 tx=0x7f2dfc008040 comp rx=0 tx=0).stop 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.231+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 msgr2=0x7f2e141e6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.231+0000 7f2e18d72640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e141e6a30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f2e0402a1e0 tx=0x7f2e040046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 shutdown_connections 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2df40744b0 0x7f2df40769c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e140d04a0 0x7f2e140a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e1406dcf0 0x7f2e141f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 --2- 172.21.15.67:0/3317826882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406d8e0 0x7f2e141e6a30 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 >> 172.21.15.67:0/3317826882 conn(0x7f2e1408d110 msgr2=0x7f2e14071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 shutdown_connections 2024-04-25T15:28:38.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.232+0000 7f2e18d72640 1 -- 172.21.15.67:0/3317826882 wait complete. 2024-04-25T15:28:38.244 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T15:28:38.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:38.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T15:28:38.271 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:28:38.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:38.272 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:28:38.327 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:38.327 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:38.327 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:38.346 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:38.346 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:38.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:38.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.514+0000 7effbb4f2640 1 Processor -- start 2024-04-25T15:28:38.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.514+0000 7effbb4f2640 1 -- start start 2024-04-25T15:28:38.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effbb4f2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7effb406d8e0 0x7effb40d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effbb4f2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb40d12c0 con 0x7effb406dcf0 2024-04-25T15:28:38.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effbb4f2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb40b3c10 con 0x7effb40d2d70 2024-04-25T15:28:38.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effbb4f2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7effb40b9520 con 0x7effb406d8e0 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb8a8d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7effb406dcf0 0x7effb40d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37100/0 (socket says 172.21.15.67:37100) 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb8a8d640 1 -- 172.21.15.67:0/3266498363 learned_addr learned my addr 172.21.15.67:0/3266498363 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb928e640 1 --2- 172.21.15.67:0/3266498363 >> v2:172.21.15.176:3300/0 conn(0x7effb406d8e0 0x7effb40d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002716906 0 0) 0x7effb40d12c0 con 0x7effb406dcf0 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff9c003520 con 0x7effb406dcf0 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 413808038 0 0) 0x7eff9c003520 con 0x7effb406dcf0 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 >> v2:172.21.15.176:3300/0 conn(0x7effb406d8e0 msgr2=0x7effb40d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 --2- 172.21.15.67:0/3266498363 >> v2:172.21.15.176:3300/0 conn(0x7effb406d8e0 0x7effb40d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 >> v1:172.21.15.176:6789/0 conn(0x7effb40d2d70 legacy=0x7effb40d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb40b3ef0 con 0x7effb406dcf0 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7effa8001670 con 0x7effb406dcf0 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 >> v1:172.21.15.67:6789/0 conn(0x7effb406dcf0 legacy=0x7effb40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff9c0039b0 0x7eff9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9c004800 0x7eff9c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.516+0000 7effb3fff640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c005250 0x7eff9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effb40b9520 con 0x7eff9c004800 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c00c1e0 con 0x7eff9c0039b0 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c00c3e0 con 0x7eff9c005250 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb8a8d640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9c004800 0x7eff9c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb928e640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff9c0039b0 0x7eff9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb9a8f640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c005250 0x7eff9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb8a8d640 1 -- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c005250 msgr2=0x7eff9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb8a8d640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c005250 0x7eff9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb8a8d640 1 -- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff9c0039b0 msgr2=0x7eff9c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb8a8d640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff9c0039b0 0x7eff9c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb8a8d640 1 -- 172.21.15.67:0/3266498363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c003520 con 0x7eff9c004800 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.517+0000 7effb9a8f640 1 -- 172.21.15.67:0/3266498363 reap_dead start 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effb8a8d640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9c004800 0x7eff9c004c10 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7effa800c240 tx=0x7eff9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdb7518425c2762 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8007cf0 con 0x7eff9c004800 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa80058d0 con 0x7eff9c004800 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effb3fff640 1 -- 172.21.15.67:0/3266498363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8005d20 con 0x7eff9c004800 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effbb4f2640 1 -- 172.21.15.67:0/3266498363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9c004800 msgr2=0x7eff9c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effbb4f2640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9c004800 0x7eff9c004c10 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7effa800c240 tx=0x7eff9c005830 comp rx=0 tx=0).stop 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effbb4f2640 1 -- 172.21.15.67:0/3266498363 shutdown_connections 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effbb4f2640 1 --2- 172.21.15.67:0/3266498363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9c004800 0x7eff9c004c10 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.518+0000 7effbb4f2640 1 -- 172.21.15.67:0/3266498363 >> 172.21.15.67:0/3266498363 conn(0x7effb408d110 msgr2=0x7effb408d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.519+0000 7effbb4f2640 1 -- 172.21.15.67:0/3266498363 shutdown_connections 2024-04-25T15:28:38.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.519+0000 7effbb4f2640 1 -- 172.21.15.67:0/3266498363 wait complete. 2024-04-25T15:28:38.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.520+0000 7effbb4f2640 1 Processor -- start 2024-04-25T15:28:38.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.520+0000 7effbb4f2640 1 -- start start 2024-04-25T15:28:38.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effbb4f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c004800 0x7effb4151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effbb4f2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb415a0b0 0x7effb4152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effbb4f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb415e640 0x7effb415ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effbb4f2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effb40b44b0 con 0x7effb415a0b0 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effbb4f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7effb40b4820 con 0x7effb415e640 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effbb4f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7effb40b38e0 con 0x7eff9c004800 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb415a0b0 0x7effb4152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb415a0b0 0x7effb4152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57798/0 (socket says 172.21.15.67:57798) 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 -- 172.21.15.67:0/4154260841 learned_addr learned my addr 172.21.15.67:0/4154260841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb9a8f640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb415e640 0x7effb415ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb928e640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c004800 0x7effb4151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 -- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c004800 msgr2=0x7effb4151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c004800 0x7effb4151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 -- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb415e640 msgr2=0x7effb415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb415e640 0x7effb415ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 -- 172.21.15.67:0/4154260841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb415dba0 con 0x7effb415a0b0 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb8a8d640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb415a0b0 0x7effb4152450 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7effa800c360 tx=0x7effb40b40c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb1ffb640 1 -- 172.21.15.67:0/4154260841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8038040 con 0x7effb415a0b0 2024-04-25T15:28:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb1ffb640 1 -- 172.21.15.67:0/4154260841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa8000ed0 con 0x7effb415a0b0 2024-04-25T15:28:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb415fb90 con 0x7effb415a0b0 2024-04-25T15:28:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.522+0000 7effb1ffb640 1 -- 172.21.15.67:0/4154260841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8001c30 con 0x7effb415a0b0 2024-04-25T15:28:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.523+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb415e120 con 0x7effb415a0b0 2024-04-25T15:28:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.524+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb4050740 con 0x7effb415a0b0 2024-04-25T15:28:38.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.524+0000 7effb1ffb640 1 -- 172.21.15.67:0/4154260841 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7effa8043020 con 0x7effb415a0b0 2024-04-25T15:28:38.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.525+0000 7effb1ffb640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7effa00744e0 0x7effa00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.525+0000 7effb1ffb640 1 -- 172.21.15.67:0/4154260841 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7effa80b49c0 con 0x7effb415a0b0 2024-04-25T15:28:38.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.525+0000 7effb928e640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7effa00744e0 0x7effa00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.526+0000 7effb928e640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7effa00744e0 0x7effa00769f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7effb40744c0 tx=0x7effac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.530+0000 7effb1ffb640 1 -- 172.21.15.67:0/4154260841 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effa80b6e20 con 0x7effb415a0b0 2024-04-25T15:28:38.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.718+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7effb40b6860 con 0x7effb415a0b0 2024-04-25T15:28:38.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.719+0000 7effb1ffb640 1 -- 172.21.15.67:0/4154260841 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16624 (secure 0 0 0) 0x7effa807f150 con 0x7effb415a0b0 2024-04-25T15:28:38.721 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:38.721 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:24.632893+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:38.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7effa00744e0 msgr2=0x7effa00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7effa00744e0 0x7effa00769f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7effb40744c0 tx=0x7effac008040 comp rx=0 tx=0).stop 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb415a0b0 msgr2=0x7effb4152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb415a0b0 0x7effb4152450 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7effa800c360 tx=0x7effb40b40c0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 shutdown_connections 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7effa00744e0 0x7effa00769f0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb415e640 0x7effb415ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb415a0b0 0x7effb4152450 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 --2- 172.21.15.67:0/4154260841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff9c004800 0x7effb4151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.721+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 >> 172.21.15.67:0/4154260841 conn(0x7effb408d110 msgr2=0x7effb40b59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.722+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 shutdown_connections 2024-04-25T15:28:38.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.722+0000 7effbb4f2640 1 -- 172.21.15.67:0/4154260841 wait complete. 2024-04-25T15:28:38.734 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:28:21.329066+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:28:21.948629+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:38.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.848+0000 7fc3c8f96640 1 Processor -- start 2024-04-25T15:28:38.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.848+0000 7fc3c8f96640 1 -- start start 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c8f96640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc3c40b2e60 0x7fc3c40d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c8f96640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 0x7fc3c4164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c8f96640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3c4164970 con 0x7fc3c40d1e20 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c8f96640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc3c40d23f0 con 0x7fc3c40b2e60 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c8f96640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc3c40d26d0 con 0x7fc3c4164020 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3c40d1e20 0x7fc3c408d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37116/0 (socket says 172.21.15.67:37116) 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c259c640 1 -- 172.21.15.67:0/2019546669 learned_addr learned my addr 172.21.15.67:0/2019546669 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c2d9d640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 0x7fc3c4164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c1d9b640 1 --2- 172.21.15.67:0/2019546669 >> v2:172.21.15.176:3300/0 conn(0x7fc3c40b2e60 0x7fc3c40d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869120591 0 0) 0x7fc3ac001110 con 0x7fc3c40d1e20 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.850+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b0003520 con 0x7fc3c40d1e20 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 360518935 0 0) 0x7fc3b0003520 con 0x7fc3c40d1e20 2024-04-25T15:28:38.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 msgr2=0x7fc3c4164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 0x7fc3c4164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 >> v2:172.21.15.176:3300/0 conn(0x7fc3c40b2e60 msgr2=0x7fc3c40d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 --2- 172.21.15.67:0/2019546669 >> v2:172.21.15.176:3300/0 conn(0x7fc3c40b2e60 0x7fc3c40d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c40d29b0 con 0x7fc3c40d1e20 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fc3ac001840 con 0x7fc3c40d1e20 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 >> v1:172.21.15.67:6789/0 conn(0x7fc3c40d1e20 legacy=0x7fc3c408d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b00039b0 0x7fc3b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 0x7fc3b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3b0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c40d23f0 con 0x7fc3b0004d40 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c40d26d0 con 0x7fc3b00039b0 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b000ba40 con 0x7fc3c4164020 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c2d9d640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3b0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c259c640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b00039b0 0x7fc3b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c1d9b640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 0x7fc3b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c2d9d640 1 -- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 msgr2=0x7fc3b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c2d9d640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c4164020 0x7fc3b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c2d9d640 1 -- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b00039b0 msgr2=0x7fc3b0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c2d9d640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3b00039b0 0x7fc3b0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.851+0000 7fc3c2d9d640 1 -- 172.21.15.67:0/2019546669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b0003520 con 0x7fc3b0004d40 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.852+0000 7fc3c2d9d640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3b0005150 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc3c408e190 tx=0x7fc3b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33326de79ad1f33b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.852+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc037070 con 0x7fc3b0004d40 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.852+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3bc032620 con 0x7fc3b0004d40 2024-04-25T15:28:38.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.852+0000 7fc3c159a640 1 -- 172.21.15.67:0/2019546669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc009c10 con 0x7fc3b0004d40 2024-04-25T15:28:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.853+0000 7fc3c8f96640 1 -- 172.21.15.67:0/2019546669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 msgr2=0x7fc3b0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.853+0000 7fc3c8f96640 1 --2- 172.21.15.67:0/2019546669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3b0005150 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fc3c408e190 tx=0x7fc3b0005830 comp rx=0 tx=0).stop 2024-04-25T15:28:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.853+0000 7fc3c2d9d640 1 -- 172.21.15.67:0/2019546669 reap_dead start 2024-04-25T15:28:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.853+0000 7fc3c8f96640 1 -- 172.21.15.67:0/2019546669 shutdown_connections 2024-04-25T15:28:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.853+0000 7fc3c8f96640 1 -- 172.21.15.67:0/2019546669 >> 172.21.15.67:0/2019546669 conn(0x7fc3c40d0530 msgr2=0x7fc3c40d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.853+0000 7fc3c8f96640 1 -- 172.21.15.67:0/2019546669 shutdown_connections 2024-04-25T15:28:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.853+0000 7fc3c8f96640 1 -- 172.21.15.67:0/2019546669 wait complete. 2024-04-25T15:28:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.855+0000 7fc3c8f96640 1 Processor -- start 2024-04-25T15:28:38.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.855+0000 7fc3c8f96640 1 -- start start 2024-04-25T15:28:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c8f96640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3c40c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c8f96640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3c41f5ea0 0x7fc3c40c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c8f96640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c41f62b0 0x7fc3c40c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c8f96640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c4164430 con 0x7fc3b0004d40 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c8f96640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c40d2f70 con 0x7fc3c41f5ea0 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c8f96640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc3c41645b0 con 0x7fc3c41f62b0 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3c40c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3c40c9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57806/0 (socket says 172.21.15.67:57806) 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c259c640 1 -- 172.21.15.67:0/538664408 learned_addr learned my addr 172.21.15.67:0/538664408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c2d9d640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c41f62b0 0x7fc3c40c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.856+0000 7fc3c1d9b640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3c41f5ea0 0x7fc3c40c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c259c640 1 -- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c41f62b0 msgr2=0x7fc3c40c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c259c640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c41f62b0 0x7fc3c40c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c259c640 1 -- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3c41f5ea0 msgr2=0x7fc3c40c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c259c640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3c41f5ea0 0x7fc3c40c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c259c640 1 -- 172.21.15.67:0/538664408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c4163140 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c2d9d640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c41f62b0 0x7fc3c40c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c259c640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3c40c9d80 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0019b0 tx=0x7fc3ac0077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3baffd640 1 -- 172.21.15.67:0/538664408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac00f8e0 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3baffd640 1 -- 172.21.15.67:0/538664408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac018070 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c4163420 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3baffd640 1 -- 172.21.15.67:0/538664408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac00fca0 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.857+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c4153050 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.858+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c4050740 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.858+0000 7fc3baffd640 1 -- 172.21.15.67:0/538664408 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc3ac01d460 con 0x7fc3b0004d40 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.859+0000 7fc3baffd640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc3a40744b0 0x7fc3a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:38.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.859+0000 7fc3baffd640 1 -- 172.21.15.67:0/538664408 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fc3ac0953d0 con 0x7fc3b0004d40 2024-04-25T15:28:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.859+0000 7fc3c1d9b640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc3a40744b0 0x7fc3a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:38.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.860+0000 7fc3c1d9b640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc3a40744b0 0x7fc3a40769c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc3c41639a0 tx=0x7fc3b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:38.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:38.864+0000 7fc3baffd640 1 -- 172.21.15.67:0/538664408 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3c4164970 con 0x7fc3b0004d40 2024-04-25T15:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.050+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc3c40d1200 con 0x7fc3b0004d40 2024-04-25T15:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.050+0000 7fc3baffd640 1 -- 172.21.15.67:0/538664408 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7fc3ac05fbf0 con 0x7fc3b0004d40 2024-04-25T15:28:39.051 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.052+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc3a40744b0 msgr2=0x7fc3a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc3a40744b0 0x7fc3a40769c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc3c41639a0 tx=0x7fc3b4008040 comp rx=0 tx=0).stop 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 msgr2=0x7fc3c40c9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3c40c9d80 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0019b0 tx=0x7fc3ac0077a0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 shutdown_connections 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc3a40744b0 0x7fc3a40769c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc3c41f62b0 0x7fc3c40c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3c41f5ea0 0x7fc3c40c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 --2- 172.21.15.67:0/538664408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b0004d40 0x7fc3c40c9d80 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 >> 172.21.15.67:0/538664408 conn(0x7fc3c40d0530 msgr2=0x7fc3c40b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 shutdown_connections 2024-04-25T15:28:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.053+0000 7fc3c8f96640 1 -- 172.21.15.67:0/538664408 wait complete. 2024-04-25T15:28:39.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:28:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.178+0000 7fd38a54d640 1 Processor -- start 2024-04-25T15:28:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.178+0000 7fd38a54d640 1 -- start start 2024-04-25T15:28:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd38a54d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd384164b00 0x7fd3840b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd38a54d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3841615f0 con 0x7fd384160fa0 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd38a54d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd384164020 con 0x7fd38415df80 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd38a54d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd3841613b0 con 0x7fd384164b00 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd388aea640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd384160fa0 0x7fd3840b4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37132/0 (socket says 172.21.15.67:37132) 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd388aea640 1 -- 172.21.15.67:0/81127424 learned_addr learned my addr 172.21.15.67:0/81127424 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd383fff640 1 --2- 172.21.15.67:0/81127424 >> v2:172.21.15.176:3300/0 conn(0x7fd384164b00 0x7fd3840b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812620086 0 0) 0x7fd3841615f0 con 0x7fd384160fa0 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd370003520 con 0x7fd384160fa0 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4098002093 0 0) 0x7fd384164020 con 0x7fd38415df80 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3841615f0 con 0x7fd38415df80 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 178203291 0 0) 0x7fd370003520 con 0x7fd384160fa0 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 >> v2:172.21.15.176:3300/0 conn(0x7fd384164b00 msgr2=0x7fd3840b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 --2- 172.21.15.67:0/81127424 >> v2:172.21.15.176:3300/0 conn(0x7fd384164b00 0x7fd3840b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 >> v1:172.21.15.176:6789/0 conn(0x7fd38415df80 legacy=0x7fd384160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd384164300 con 0x7fd384160fa0 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fd37c0019f0 con 0x7fd384160fa0 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 >> v1:172.21.15.67:6789/0 conn(0x7fd384160fa0 legacy=0x7fd3840b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd370003b40 0x7fd370003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.180+0000 7fd382ffd640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370004800 0x7fd370004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd382ffd640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3700051a0 0x7fd37000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841613b0 con 0x7fd370003b40 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd37000c140 con 0x7fd370004800 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd37000c340 con 0x7fd3700051a0 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd383fff640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd370003b40 0x7fd370003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd3837fe640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370004800 0x7fd370004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd388aea640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3700051a0 0x7fd37000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd383fff640 1 -- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3700051a0 msgr2=0x7fd37000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd383fff640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3700051a0 0x7fd37000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd383fff640 1 -- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370004800 msgr2=0x7fd370004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd383fff640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370004800 0x7fd370004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd383fff640 1 -- 172.21.15.67:0/81127424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd370003520 con 0x7fd370003b40 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd388aea640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3700051a0 0x7fd37000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.181+0000 7fd388aea640 1 -- 172.21.15.67:0/81127424 reap_dead start 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd383fff640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd370003b40 0x7fd370003f90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd3700057e0 tx=0x7fd370005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcedc2dac773334a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c036070 con 0x7fd370003b40 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd36c031400 con 0x7fd370003b40 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd382ffd640 1 -- 172.21.15.67:0/81127424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c009690 con 0x7fd370003b40 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd38a54d640 1 -- 172.21.15.67:0/81127424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd370003b40 msgr2=0x7fd370003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd38a54d640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd370003b40 0x7fd370003f90 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd3700057e0 tx=0x7fd370005830 comp rx=0 tx=0).stop 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd38a54d640 1 -- 172.21.15.67:0/81127424 shutdown_connections 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd38a54d640 1 --2- 172.21.15.67:0/81127424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd370003b40 0x7fd370003f90 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd38a54d640 1 -- 172.21.15.67:0/81127424 >> 172.21.15.67:0/81127424 conn(0x7fd3840d2bc0 msgr2=0x7fd3840d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.182+0000 7fd38a54d640 1 -- 172.21.15.67:0/81127424 shutdown_connections 2024-04-25T15:28:39.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.183+0000 7fd38a54d640 1 -- 172.21.15.67:0/81127424 wait complete. 2024-04-25T15:28:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.184+0000 7fd38a54d640 1 Processor -- start 2024-04-25T15:28:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.184+0000 7fd38a54d640 1 -- start start 2024-04-25T15:28:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd38a54d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370003b40 0x7fd3840cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd38a54d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3840c9830 0x7fd3840cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd38a54d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38415a4a0 0x7fd38415a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd38a54d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd384160c20 con 0x7fd38415a4a0 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd38a54d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd384160aa0 con 0x7fd370003b40 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd38a54d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd384160da0 con 0x7fd3840c9830 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38415a4a0 0x7fd38415a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38415a4a0 0x7fd38415a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57822/0 (socket says 172.21.15.67:57822) 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 -- 172.21.15.67:0/3376588322 learned_addr learned my addr 172.21.15.67:0/3376588322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd383fff640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370003b40 0x7fd3840cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd3837fe640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3840c9830 0x7fd3840cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 -- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3840c9830 msgr2=0x7fd3840cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3840c9830 0x7fd3840cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 -- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370003b40 msgr2=0x7fd3840cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370003b40 0x7fd3840cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 -- 172.21.15.67:0/3376588322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3841eb960 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd388aea640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38415a4a0 0x7fd38415a8b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd37c001fd0 tx=0x7fd37c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.186+0000 7fd380ff9640 1 -- 172.21.15.67:0/3376588322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c00eb90 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.187+0000 7fd380ff9640 1 -- 172.21.15.67:0/3376588322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd37c018070 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.187+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3841ebc40 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.187+0000 7fd380ff9640 1 -- 172.21.15.67:0/3376588322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c013420 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.187+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3841ec200 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.188+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd384050740 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.188+0000 7fd380ff9640 1 -- 172.21.15.67:0/3376588322 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd37c0135c0 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.189+0000 7fd380ff9640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd3640744b0 0x7fd3640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.189+0000 7fd380ff9640 1 -- 172.21.15.67:0/3376588322 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fd37c095440 con 0x7fd38415a4a0 2024-04-25T15:28:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.189+0000 7fd383fff640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd3640744b0 0x7fd3640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.189+0000 7fd383fff640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd3640744b0 0x7fd3640769c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd36c033000 tx=0x7fd36c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.193+0000 7fd380ff9640 1 -- 172.21.15.67:0/3376588322 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd37c05fbd0 con 0x7fd38415a4a0 2024-04-25T15:28:39.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.380+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd3840c9ca0 con 0x7fd38415a4a0 2024-04-25T15:28:39.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.380+0000 7fd380ff9640 1 -- 172.21.15.67:0/3376588322 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7fd37c05f4d0 con 0x7fd38415a4a0 2024-04-25T15:28:39.381 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:28:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd3640744b0 msgr2=0x7fd3640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd3640744b0 0x7fd3640769c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd36c033000 tx=0x7fd36c008030 comp rx=0 tx=0).stop 2024-04-25T15:28:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38415a4a0 msgr2=0x7fd38415a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38415a4a0 0x7fd38415a8b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd37c001fd0 tx=0x7fd37c00e530 comp rx=0 tx=0).stop 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 shutdown_connections 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd3640744b0 0x7fd3640769c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd38415a4a0 0x7fd38415a8b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3840c9830 0x7fd3840cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 --2- 172.21.15.67:0/3376588322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd370003b40 0x7fd3840cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 >> 172.21.15.67:0/3376588322 conn(0x7fd3840d2bc0 msgr2=0x7fd38415e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 shutdown_connections 2024-04-25T15:28:39.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.382+0000 7fd38a54d640 1 -- 172.21.15.67:0/3376588322 wait complete. 2024-04-25T15:28:39.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:28:39.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.508+0000 7fe8f1030640 1 Processor -- start 2024-04-25T15:28:39.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.508+0000 7fe8f1030640 1 -- start start 2024-04-25T15:28:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.509+0000 7fe8f1030640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe8ec0b2e60 0x7fe8ec0d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.509+0000 7fe8f1030640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8ec0d23f0 con 0x7fe8ec0d1e20 2024-04-25T15:28:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.509+0000 7fe8f1030640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8ec0d26d0 con 0x7fe8ec164020 2024-04-25T15:28:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.509+0000 7fe8f1030640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe8ec0d29b0 con 0x7fe8ec0b2e60 2024-04-25T15:28:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.509+0000 7fe8ead9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8ec0d1e20 0x7fe8ec08d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37144/0 (socket says 172.21.15.67:37144) 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.509+0000 7fe8ead9d640 1 -- 172.21.15.67:0/654419664 learned_addr learned my addr 172.21.15.67:0/654419664 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8e3fff640 1 --2- 172.21.15.67:0/654419664 >> v2:172.21.15.176:3300/0 conn(0x7fe8ec0b2e60 0x7fe8ec0d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139083254 0 0) 0x7fe8ec0d23f0 con 0x7fe8ec0d1e20 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8d8003520 con 0x7fe8ec0d1e20 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499590249 0 0) 0x7fe8ec0d26d0 con 0x7fe8ec164020 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8ec0d23f0 con 0x7fe8ec164020 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1845416900 0 0) 0x7fe8d8003520 con 0x7fe8ec0d1e20 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 >> v2:172.21.15.176:3300/0 conn(0x7fe8ec0b2e60 msgr2=0x7fe8ec0d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 --2- 172.21.15.67:0/654419664 >> v2:172.21.15.176:3300/0 conn(0x7fe8ec0b2e60 0x7fe8ec0d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 >> v1:172.21.15.176:6789/0 conn(0x7fe8ec164020 legacy=0x7fe8ec164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ec0d2c90 con 0x7fe8ec0d1e20 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fe8dc001670 con 0x7fe8ec0d1e20 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8e3fff640 1 --2- 172.21.15.67:0/654419664 >> v2:172.21.15.176:3300/0 conn(0x7fe8ec0b2e60 0x7fe8ec0d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 >> v1:172.21.15.67:6789/0 conn(0x7fe8ec0d1e20 legacy=0x7fe8ec08d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8d8003b40 0x7fe8d8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d8004800 0x7fe8d8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80051a0 0x7fe8d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec0d29b0 con 0x7fe8d80051a0 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8d800c090 con 0x7fe8d8004800 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8d800c290 con 0x7fe8d8003b40 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8eb59e640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80051a0 0x7fe8d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8ead9d640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8d8003b40 0x7fe8d8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.510+0000 7fe8e3fff640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d8004800 0x7fe8d8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8eb59e640 1 -- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8d8003b40 msgr2=0x7fe8d8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8eb59e640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8d8003b40 0x7fe8d8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8eb59e640 1 -- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d8004800 msgr2=0x7fe8d8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8eb59e640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d8004800 0x7fe8d8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8eb59e640 1 -- 172.21.15.67:0/654419664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8d8003520 con 0x7fe8d80051a0 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8eb59e640 1 -- 172.21.15.67:0/654419664 reap_dead start 2024-04-25T15:28:39.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8eb59e640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80051a0 0x7fe8d800b9c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0b5570 tx=0x7fe8d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a7d6038709655d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4001bf0 con 0x7fe8d80051a0 2024-04-25T15:28:39.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e4003b70 con 0x7fe8d80051a0 2024-04-25T15:28:39.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.511+0000 7fe8ea59c640 1 -- 172.21.15.67:0/654419664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e40050d0 con 0x7fe8d80051a0 2024-04-25T15:28:39.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.512+0000 7fe8f1030640 1 -- 172.21.15.67:0/654419664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80051a0 msgr2=0x7fe8d800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.512+0000 7fe8f1030640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80051a0 0x7fe8d800b9c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec0b5570 tx=0x7fe8d8005830 comp rx=0 tx=0).stop 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.512+0000 7fe8f1030640 1 -- 172.21.15.67:0/654419664 shutdown_connections 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.512+0000 7fe8f1030640 1 --2- 172.21.15.67:0/654419664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8d80051a0 0x7fe8d800b9c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.512+0000 7fe8f1030640 1 -- 172.21.15.67:0/654419664 >> 172.21.15.67:0/654419664 conn(0x7fe8ec0d0530 msgr2=0x7fe8ec0d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.512+0000 7fe8f1030640 1 -- 172.21.15.67:0/654419664 shutdown_connections 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.512+0000 7fe8f1030640 1 -- 172.21.15.67:0/654419664 wait complete. 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.514+0000 7fe8f1030640 1 Processor -- start 2024-04-25T15:28:39.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.514+0000 7fe8f1030640 1 -- start start 2024-04-25T15:28:39.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.515+0000 7fe8f1030640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d80051a0 0x7fe8ec0b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.515+0000 7fe8f1030640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ec0b85e0 0x7fe8ec1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.515+0000 7fe8f1030640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8ec15d7b0 0x7fe8ec15dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.515+0000 7fe8f1030640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec0d3610 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.515+0000 7fe8f1030640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec0d3490 con 0x7fe8d80051a0 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.515+0000 7fe8f1030640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8ec0d3790 con 0x7fe8ec0b85e0 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8ec15d7b0 0x7fe8ec15dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8ec15d7b0 0x7fe8ec15dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57848/0 (socket says 172.21.15.67:57848) 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 -- 172.21.15.67:0/669757348 learned_addr learned my addr 172.21.15.67:0/669757348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8ead9d640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d80051a0 0x7fe8ec0b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8e3fff640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ec0b85e0 0x7fe8ec1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 -- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ec0b85e0 msgr2=0x7fe8ec1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ec0b85e0 0x7fe8ec1551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 -- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d80051a0 msgr2=0x7fe8ec0b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d80051a0 0x7fe8ec0b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 -- 172.21.15.67:0/669757348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ec15e100 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8ead9d640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d80051a0 0x7fe8ec0b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8e3fff640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ec0b85e0 0x7fe8ec1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8eb59e640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8ec15d7b0 0x7fe8ec15dbc0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40012e0 tx=0x7fe8e4001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8e37fe640 1 -- 172.21.15.67:0/669757348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4003970 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8e37fe640 1 -- 172.21.15.67:0/669757348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e40052e0 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ec155730 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8e37fe640 1 -- 172.21.15.67:0/669757348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e40054b0 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.516+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ec0cb020 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.517+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ec050740 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.518+0000 7fe8e37fe640 1 -- 172.21.15.67:0/669757348 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe8e4042020 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.518+0000 7fe8e37fe640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.519+0000 7fe8e37fe640 1 -- 172.21.15.67:0/669757348 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fe8e40b4d70 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.519+0000 7fe8ead9d640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.519+0000 7fe8ead9d640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc001eb0 tx=0x7fe8dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.523+0000 7fe8e37fe640 1 -- 172.21.15.67:0/669757348 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8ec15e100 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.710+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe8ec0b18e0 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.711+0000 7fe8e37fe640 1 -- 172.21.15.67:0/669757348 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7fe8e407f510 con 0x7fe8ec15d7b0 2024-04-25T15:28:39.712 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:28:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8cc0744b0 msgr2=0x7fe8cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc001eb0 tx=0x7fe8dc009040 comp rx=0 tx=0).stop 2024-04-25T15:28:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8ec15d7b0 msgr2=0x7fe8ec15dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8ec15d7b0 0x7fe8ec15dbc0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40012e0 tx=0x7fe8e4001650 comp rx=0 tx=0).stop 2024-04-25T15:28:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 shutdown_connections 2024-04-25T15:28:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8ec15d7b0 0x7fe8ec15dbc0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ec0b85e0 0x7fe8ec1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 --2- 172.21.15.67:0/669757348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8d80051a0 0x7fe8ec0b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 >> 172.21.15.67:0/669757348 conn(0x7fe8ec0d0530 msgr2=0x7fe8ec0b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:39.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 shutdown_connections 2024-04-25T15:28:39.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.713+0000 7fe8f1030640 1 -- 172.21.15.67:0/669757348 wait complete. 2024-04-25T15:28:39.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:28:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.837+0000 7f2da30b1640 1 Processor -- start 2024-04-25T15:28:39.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.837+0000 7f2da30b1640 1 -- start start 2024-04-25T15:28:39.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da30b1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2d9c0cf770 0x7f2d9c162590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da30b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 0x7f2d9c0b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da30b1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9c0cfc30 con 0x7f2d9c0b91d0 2024-04-25T15:28:39.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da30b1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2d9c0d1ff0 con 0x7f2d9c0cf770 2024-04-25T15:28:39.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da30b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9c0d2890 con 0x7f2d9c162ad0 2024-04-25T15:28:39.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da0e4d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2d9c0b91d0 0x7f2d9c161e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37146/0 (socket says 172.21.15.67:37146) 2024-04-25T15:28:39.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da0e4d640 1 -- 172.21.15.67:0/3069024167 learned_addr learned my addr 172.21.15.67:0/3069024167 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2d9bfff640 1 --2- 172.21.15.67:0/3069024167 >> v2:172.21.15.176:3300/0 conn(0x7f2d9c0cf770 0x7f2d9c162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.838+0000 7f2da164e640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 0x7f2d9c0b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927173257 0 0) 0x7f2d9c0cfc30 con 0x7f2d9c0b91d0 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d88003520 con 0x7f2d9c0b91d0 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2529743984 0 0) 0x7f2d88003520 con 0x7f2d9c0b91d0 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 msgr2=0x7f2d9c0b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 0x7f2d9c0b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 >> v2:172.21.15.176:3300/0 conn(0x7f2d9c0cf770 msgr2=0x7f2d9c162590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 --2- 172.21.15.67:0/3069024167 >> v2:172.21.15.176:3300/0 conn(0x7f2d9c0cf770 0x7f2d9c162590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c0d2240 con 0x7f2d9c0b91d0 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f2d940019f0 con 0x7f2d9c0b91d0 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2da164e640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 0x7f2d9c0b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 >> v1:172.21.15.67:6789/0 conn(0x7f2d9c0b91d0 legacy=0x7f2d9c161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d880039b0 0x7f2d88003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 0x7f2d88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d88005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c0d1ff0 con 0x7f2d88004d40 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c0d2890 con 0x7f2d880039b0 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d8800ba10 con 0x7f2d9c162ad0 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2da164e640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d88005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2da0e4d640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d880039b0 0x7f2d88003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2d9bfff640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 0x7f2d88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2da164e640 1 -- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 msgr2=0x7f2d88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2da164e640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c162ad0 0x7f2d88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2da164e640 1 -- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d880039b0 msgr2=0x7f2d88003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.839+0000 7f2da164e640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d880039b0 0x7f2d88003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.840+0000 7f2da164e640 1 -- 172.21.15.67:0/3069024167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d88003520 con 0x7f2d88004d40 2024-04-25T15:28:39.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.840+0000 7f2da164e640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d88005150 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009cc0 tx=0x7f2d88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a08d9b70e1da2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.840+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c036070 con 0x7f2d88004d40 2024-04-25T15:28:39.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.840+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d8c031400 con 0x7f2d88004d40 2024-04-25T15:28:39.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.840+0000 7f2d9b7fe640 1 -- 172.21.15.67:0/3069024167 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c009860 con 0x7f2d88004d40 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.841+0000 7f2da30b1640 1 -- 172.21.15.67:0/3069024167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 msgr2=0x7f2d88005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.841+0000 7f2da30b1640 1 --2- 172.21.15.67:0/3069024167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d88005150 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009cc0 tx=0x7f2d88005830 comp rx=0 tx=0).stop 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.841+0000 7f2da164e640 1 -- 172.21.15.67:0/3069024167 reap_dead start 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.841+0000 7f2da30b1640 1 -- 172.21.15.67:0/3069024167 shutdown_connections 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.841+0000 7f2da30b1640 1 -- 172.21.15.67:0/3069024167 >> 172.21.15.67:0/3069024167 conn(0x7f2d9c0d3aa0 msgr2=0x7f2d9c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.841+0000 7f2da30b1640 1 -- 172.21.15.67:0/3069024167 shutdown_connections 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.841+0000 7f2da30b1640 1 -- 172.21.15.67:0/3069024167 wait complete. 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.843+0000 7f2da30b1640 1 Processor -- start 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.843+0000 7f2da30b1640 1 -- start start 2024-04-25T15:28:39.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.844+0000 7f2da30b1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d9c159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.844+0000 7f2da30b1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c0b5750 0x7f2d9c15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.844+0000 7f2da30b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c06dac0 0x7f2d9c1f9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.844+0000 7f2da30b1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c0b24a0 con 0x7f2d88004d40 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.844+0000 7f2da30b1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c0b2320 con 0x7f2d9c0b5750 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.844+0000 7f2da30b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9c0b0690 con 0x7f2d9c06dac0 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d9c159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d9c159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57880/0 (socket says 172.21.15.67:57880) 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 -- 172.21.15.67:0/1874119569 learned_addr learned my addr 172.21.15.67:0/1874119569 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2d9bfff640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c0b5750 0x7f2d9c15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da164e640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c06dac0 0x7f2d9c1f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 -- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c06dac0 msgr2=0x7f2d9c1f9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c06dac0 0x7f2d9c1f9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 -- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c0b5750 msgr2=0x7f2d9c15a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c0b5750 0x7f2d9c15a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 -- 172.21.15.67:0/1874119569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c0cd7d0 con 0x7f2d88004d40 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da164e640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c06dac0 0x7f2d9c1f9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2d9bfff640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c0b5750 0x7f2d9c15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da0e4d640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d9c159be0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f2d940079f0 tx=0x7f2d9400a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2d997fa640 1 -- 172.21.15.67:0/1874119569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94017020 con 0x7f2d88004d40 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2d997fa640 1 -- 172.21.15.67:0/1874119569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9400ae00 con 0x7f2d88004d40 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9c0cdab0 con 0x7f2d88004d40 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2d997fa640 1 -- 172.21.15.67:0/1874119569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94013840 con 0x7f2d88004d40 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.845+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9c0cbb00 con 0x7f2d88004d40 2024-04-25T15:28:39.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.846+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d9c0cb610 con 0x7f2d88004d40 2024-04-25T15:28:39.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.847+0000 7f2d997fa640 1 -- 172.21.15.67:0/1874119569 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2d9401a020 con 0x7f2d88004d40 2024-04-25T15:28:39.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.848+0000 7f2d997fa640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d7c0744b0 0x7f2d7c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:39.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.848+0000 7f2d997fa640 1 -- 172.21.15.67:0/1874119569 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f2d94095e00 con 0x7f2d88004d40 2024-04-25T15:28:39.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.848+0000 7f2d9bfff640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d7c0744b0 0x7f2d7c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:39.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.848+0000 7f2d9bfff640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d7c0744b0 0x7f2d7c0769c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0ca230 tx=0x7f2d84008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:39.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:39.852+0000 7f2d997fa640 1 -- 172.21.15.67:0/1874119569 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d94021020 con 0x7f2d88004d40 2024-04-25T15:28:40.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.049+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2d9c050740 con 0x7f2d88004d40 2024-04-25T15:28:40.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.050+0000 7f2d997fa640 1 -- 172.21.15.67:0/1874119569 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v80) ==== 80+0+0 (secure 0 0 0) 0x7f2d94060620 con 0x7f2d88004d40 2024-04-25T15:28:40.050 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.051+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d7c0744b0 msgr2=0x7f2d7c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.051+0000 7f2da30b1640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d7c0744b0 0x7f2d7c0769c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0ca230 tx=0x7f2d84008040 comp rx=0 tx=0).stop 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.051+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 msgr2=0x7f2d9c159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.051+0000 7f2da30b1640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d9c159be0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f2d940079f0 tx=0x7f2d9400a750 comp rx=0 tx=0).stop 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 shutdown_connections 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2d7c0744b0 0x7f2d7c0769c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d9c06dac0 0x7f2d9c1f9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c0b5750 0x7f2d9c15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 --2- 172.21.15.67:0/1874119569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d88004d40 0x7f2d9c159be0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 >> 172.21.15.67:0/1874119569 conn(0x7f2d9c0d3aa0 msgr2=0x7f2d9c072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 shutdown_connections 2024-04-25T15:28:40.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.052+0000 7f2da30b1640 1 -- 172.21.15.67:0/1874119569 wait complete. 2024-04-25T15:28:40.064 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:28:40.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:40.064 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-25T15:28:40.096 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:28:40.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:28:40.097 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:28:40.154 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:40.154 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:40.154 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:40.170 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:28:40.170 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:28:40.170 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:40.170 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:40.211 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:28:40.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.322+0000 7fb9af196640 1 Processor -- start 2024-04-25T15:28:40.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.322+0000 7fb9af196640 1 -- start start 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.323+0000 7fb9af196640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9a8164b00 0x7fb9a80b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.323+0000 7fb9af196640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb9a815df80 0x7fb9a80b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.323+0000 7fb9af196640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a81615f0 con 0x7fb9a8160890 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.323+0000 7fb9af196640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a81613b0 con 0x7fb9a8164b00 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.323+0000 7fb9af196640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a8164670 con 0x7fb9a815df80 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9acf32640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9a8164b00 0x7fb9a80b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9ad733640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb9a8160890 0x7fb9a8160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37152/0 (socket says 172.21.15.67:37152) 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9ad733640 1 -- 172.21.15.67:0/4110268606 learned_addr learned my addr 172.21.15.67:0/4110268606 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9a7fff640 1 --2- 172.21.15.67:0/4110268606 >> v2:172.21.15.176:3300/0 conn(0x7fb9a815df80 0x7fb9a80b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9acf32640 1 -- 172.21.15.67:0/4110268606 >> v2:172.21.15.176:3300/0 conn(0x7fb9a815df80 msgr2=0x7fb9a80b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9acf32640 1 --2- 172.21.15.67:0/4110268606 >> v2:172.21.15.176:3300/0 conn(0x7fb9a815df80 0x7fb9a80b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9acf32640 1 -- 172.21.15.67:0/4110268606 >> v1:172.21.15.67:6789/0 conn(0x7fb9a8160890 legacy=0x7fb9a8160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9a77fe640 1 -- 172.21.15.67:0/4110268606 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3126034261 0 0) 0x7fb9a81615f0 con 0x7fb9a8160890 2024-04-25T15:28:40.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.324+0000 7fb9acf32640 1 -- 172.21.15.67:0/4110268606 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a8164020 con 0x7fb9a8164b00 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9acf32640 1 --2- 172.21.15.67:0/4110268606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a80b7b80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80d2540 tx=0x7fb99802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2c5bc6f7cd20ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9a77fe640 1 -- 172.21.15.67:0/4110268606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998037070 con 0x7fb9a8164b00 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9a77fe640 1 -- 172.21.15.67:0/4110268606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb998032440 con 0x7fb9a8164b00 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9a77fe640 1 -- 172.21.15.67:0/4110268606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998032810 con 0x7fb9a8164b00 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9af196640 1 -- 172.21.15.67:0/4110268606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 msgr2=0x7fb9a80b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9af196640 1 --2- 172.21.15.67:0/4110268606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a80b7b80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80d2540 tx=0x7fb99802a1a0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9af196640 1 -- 172.21.15.67:0/4110268606 shutdown_connections 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9af196640 1 --2- 172.21.15.67:0/4110268606 >> v2:172.21.15.176:3300/0 conn(0x7fb9a815df80 0x7fb9a80b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.325+0000 7fb9af196640 1 --2- 172.21.15.67:0/4110268606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a80b7b80 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.326+0000 7fb9af196640 1 -- 172.21.15.67:0/4110268606 >> 172.21.15.67:0/4110268606 conn(0x7fb9a80d2bc0 msgr2=0x7fb9a80d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.326+0000 7fb9af196640 1 -- 172.21.15.67:0/4110268606 shutdown_connections 2024-04-25T15:28:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.326+0000 7fb9af196640 1 -- 172.21.15.67:0/4110268606 wait complete. 2024-04-25T15:28:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.327+0000 7fb9af196640 1 Processor -- start 2024-04-25T15:28:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.327+0000 7fb9af196640 1 -- start start 2024-04-25T15:28:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9af196640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9a815df80 0x7fb9a81f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9af196640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a8160890 0x7fb9a8158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9af196640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a81552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9af196640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a80b4a00 con 0x7fb9a8164b00 2024-04-25T15:28:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9af196640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a80b4880 con 0x7fb9a8160890 2024-04-25T15:28:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9af196640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a80b4b80 con 0x7fb9a815df80 2024-04-25T15:28:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a81552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a81552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57896/0 (socket says 172.21.15.67:57896) 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 -- 172.21.15.67:0/2266718018 learned_addr learned my addr 172.21.15.67:0/2266718018 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9a7fff640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a8160890 0x7fb9a8158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9acf32640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9a815df80 0x7fb9a81f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 -- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9a815df80 msgr2=0x7fb9a81f68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9a815df80 0x7fb9a81f68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 -- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a8160890 msgr2=0x7fb9a8158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a8160890 0x7fb9a8158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 -- 172.21.15.67:0/2266718018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a8155800 con 0x7fb9a8164b00 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.329+0000 7fb9ad733640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a81552c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000c6d0 tx=0x7fb9a000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.330+0000 7fb9a57fa640 1 -- 172.21.15.67:0/2266718018 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0018070 con 0x7fb9a8164b00 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.330+0000 7fb9a57fa640 1 -- 172.21.15.67:0/2266718018 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a0004d60 con 0x7fb9a8164b00 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.330+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a81f9ed0 con 0x7fb9a8164b00 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.330+0000 7fb9a57fa640 1 -- 172.21.15.67:0/2266718018 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a000a880 con 0x7fb9a8164b00 2024-04-25T15:28:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.330+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a81f7410 con 0x7fb9a8164b00 2024-04-25T15:28:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.331+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a8050740 con 0x7fb9a8164b00 2024-04-25T15:28:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.331+0000 7fb9a57fa640 1 -- 172.21.15.67:0/2266718018 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb9a001b070 con 0x7fb9a8164b00 2024-04-25T15:28:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.332+0000 7fb9a57fa640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9880744e0 0x7fb9880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.332+0000 7fb9a57fa640 1 -- 172.21.15.67:0/2266718018 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fb9a00948f0 con 0x7fb9a8164b00 2024-04-25T15:28:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.332+0000 7fb9acf32640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9880744e0 0x7fb9880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.333+0000 7fb9acf32640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9880744e0 0x7fb9880769f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb998034000 tx=0x7fb99802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.336+0000 7fb9a57fa640 1 -- 172.21.15.67:0/2266718018 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9a005ea30 con 0x7fb9a8164b00 2024-04-25T15:28:40.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.559+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9a81f6ec0 con 0x7fb9a8164b00 2024-04-25T15:28:40.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.560+0000 7fb9a57fa640 1 -- 172.21.15.67:0/2266718018 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5186 (secure 0 0 0) 0x7fb9a0062d30 con 0x7fb9a8164b00 2024-04-25T15:28:40.561 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:40.561 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/445614904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":445614904},{"type":"v1","addr":"172.21.15.67:6833","nonce":445614904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":6322,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3018066741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3018066741},{"type":"v1","addr":"172.21.15.67:6835","nonce":3018066741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2045345118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2045345118},{"type":"v1","addr":"172.21.15.176:6836","nonce":2045345118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:23.008043+0000","modified":"2024-04-25T15:28:26.033114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6328":{"gid":6328,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6837/2484404027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2484404027},{"type":"v1","addr":"172.21.15.176:6837","nonce":2484404027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6328,"qdb_cluster":[6328]},"id":6}]} 2024-04-25T15:28:40.561 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 49 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9880744e0 msgr2=0x7fb9880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9880744e0 0x7fb9880769f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb998034000 tx=0x7fb99802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 msgr2=0x7fb9a81552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a81552c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000c6d0 tx=0x7fb9a000cb90 comp rx=0 tx=0).stop 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 shutdown_connections 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb9880744e0 0x7fb9880769f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9a8164b00 0x7fb9a81552c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a8160890 0x7fb9a8158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 --2- 172.21.15.67:0/2266718018 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9a815df80 0x7fb9a81f68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 >> 172.21.15.67:0/2266718018 conn(0x7fb9a80d2bc0 msgr2=0x7fb9a8160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 shutdown_connections 2024-04-25T15:28:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.562+0000 7fb9af196640 1 -- 172.21.15.67:0/2266718018 wait complete. 2024-04-25T15:28:40.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:40.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.685+0000 7fd274242640 1 Processor -- start 2024-04-25T15:28:40.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.685+0000 7fd274242640 1 -- start start 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd274242640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd26c164b00 0x7fd26c0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd274242640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd274242640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd26c1615f0 con 0x7fd26c160890 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd274242640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd26c1613b0 con 0x7fd26c164b00 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd274242640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd26c164670 con 0x7fd26c15df80 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd271fde640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd26c164b00 0x7fd26c0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd271fde640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd26c164b00 0x7fd26c0b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57900/0 (socket says 172.21.15.67:57900) 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd271fde640 1 -- 172.21.15.67:0/2819904812 learned_addr learned my addr 172.21.15.67:0/2819904812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd2717dd640 1 --2- 172.21.15.67:0/2819904812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd271fde640 1 -- 172.21.15.67:0/2819904812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 msgr2=0x7fd26c0b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:40.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd271fde640 1 --2- 172.21.15.67:0/2819904812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd271fde640 1 -- 172.21.15.67:0/2819904812 >> v1:172.21.15.176:6789/0 conn(0x7fd26c160890 legacy=0x7fd26c160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.687+0000 7fd271fde640 1 -- 172.21.15.67:0/2819904812 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26c164020 con 0x7fd26c164b00 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.688+0000 7fd271fde640 1 --2- 172.21.15.67:0/2819904812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c164b00 0x7fd26c0b7b80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd268009870 tx=0x7fd26802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0a7029866a08f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.688+0000 7fd270fdc640 1 -- 172.21.15.67:0/2819904812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268037070 con 0x7fd26c164b00 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.688+0000 7fd270fdc640 1 -- 172.21.15.67:0/2819904812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd268032440 con 0x7fd26c164b00 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.688+0000 7fd270fdc640 1 -- 172.21.15.67:0/2819904812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268032810 con 0x7fd26c164b00 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 -- 172.21.15.67:0/2819904812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c164b00 msgr2=0x7fd26c0b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 --2- 172.21.15.67:0/2819904812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c164b00 0x7fd26c0b7b80 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd268009870 tx=0x7fd26802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 -- 172.21.15.67:0/2819904812 shutdown_connections 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 --2- 172.21.15.67:0/2819904812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 --2- 172.21.15.67:0/2819904812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c164b00 0x7fd26c0b7b80 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 -- 172.21.15.67:0/2819904812 >> 172.21.15.67:0/2819904812 conn(0x7fd26c0d2bc0 msgr2=0x7fd26c0d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:40.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 -- 172.21.15.67:0/2819904812 shutdown_connections 2024-04-25T15:28:40.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.689+0000 7fd274242640 1 -- 172.21.15.67:0/2819904812 wait complete. 2024-04-25T15:28:40.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.691+0000 7fd274242640 1 Processor -- start 2024-04-25T15:28:40.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.691+0000 7fd274242640 1 -- start start 2024-04-25T15:28:40.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd274242640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd274242640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c160890 0x7fd26c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd274242640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd26c164b00 0x7fd26c159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd274242640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd26c0b4a00 con 0x7fd26c160890 2024-04-25T15:28:40.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd274242640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd26c0b4880 con 0x7fd26c164b00 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd274242640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd26c0b4b80 con 0x7fd26c15df80 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd2717dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c160890 0x7fd26c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd2717dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c160890 0x7fd26c158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57914/0 (socket says 172.21.15.67:57914) 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd2717dd640 1 -- 172.21.15.67:0/3704735176 learned_addr learned my addr 172.21.15.67:0/3704735176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd271fde640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.692+0000 7fd2727df640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd26c164b00 0x7fd26c159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2717dd640 1 -- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 msgr2=0x7fd26c0c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2717dd640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2717dd640 1 -- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd26c164b00 msgr2=0x7fd26c159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2717dd640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd26c164b00 0x7fd26c159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2717dd640 1 -- 172.21.15.67:0/3704735176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26c154ad0 con 0x7fd26c160890 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2727df640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd26c164b00 0x7fd26c159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2717dd640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c160890 0x7fd26c158ae0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00d5f0 tx=0x7fd25c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2627fc640 1 -- 172.21.15.67:0/3704735176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c0125f0 con 0x7fd26c160890 2024-04-25T15:28:40.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2627fc640 1 -- 172.21.15.67:0/3704735176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd25c019070 con 0x7fd26c160890 2024-04-25T15:28:40.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26c154db0 con 0x7fd26c160890 2024-04-25T15:28:40.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd2627fc640 1 -- 172.21.15.67:0/3704735176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c0129d0 con 0x7fd26c160890 2024-04-25T15:28:40.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.693+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd26c155370 con 0x7fd26c160890 2024-04-25T15:28:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.694+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd26c155b70 con 0x7fd26c160890 2024-04-25T15:28:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.695+0000 7fd2627fc640 1 -- 172.21.15.67:0/3704735176 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd25c012b70 con 0x7fd26c160890 2024-04-25T15:28:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.695+0000 7fd2627fc640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2500744b0 0x7fd2500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.695+0000 7fd2627fc640 1 -- 172.21.15.67:0/3704735176 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fd25c0956a0 con 0x7fd26c160890 2024-04-25T15:28:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.695+0000 7fd271fde640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2500744b0 0x7fd2500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.696+0000 7fd271fde640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2500744b0 0x7fd2500769c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd268034000 tx=0x7fd26802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:40.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.700+0000 7fd2627fc640 1 -- 172.21.15.67:0/3704735176 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd26c155b70 con 0x7fd26c160890 2024-04-25T15:28:40.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.886+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd26c050740 con 0x7fd26c160890 2024-04-25T15:28:40.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.887+0000 7fd2627fc640 1 -- 172.21.15.67:0/3704735176 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16624 (secure 0 0 0) 0x7fd25c05fe30 con 0x7fd26c160890 2024-04-25T15:28:40.888 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:40.888 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:24.632893+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2500744b0 msgr2=0x7fd2500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2500744b0 0x7fd2500769c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd268034000 tx=0x7fd26802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c160890 msgr2=0x7fd26c158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c160890 0x7fd26c158ae0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00d5f0 tx=0x7fd25c00dab0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 shutdown_connections 2024-04-25T15:28:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd2500744b0 0x7fd2500769c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd26c164b00 0x7fd26c159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd26c160890 0x7fd26c158ae0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 --2- 172.21.15.67:0/3704735176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd26c15df80 0x7fd26c0c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 >> 172.21.15.67:0/3704735176 conn(0x7fd26c0d2bc0 msgr2=0x7fd26c15f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 shutdown_connections 2024-04-25T15:28:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:40.889+0000 7fd274242640 1 -- 172.21.15.67:0/3704735176 wait complete. 2024-04-25T15:28:40.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:28:21.329066+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:28:21.948629+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:40.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:41.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.016+0000 7fb0f077d640 1 Processor -- start 2024-04-25T15:28:41.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.016+0000 7fb0f077d640 1 -- start start 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0f077d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0e8161a50 0x7fb0e80b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0f077d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e806dd70 0x7fb0e80af570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0f077d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e80b1bd0 con 0x7fb0e80b17c0 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0f077d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb0e80b3250 con 0x7fb0e8161a50 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0f077d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e80b3530 con 0x7fb0e806dd70 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0ee519640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0e8161a50 0x7fb0e80b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0ee519640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0e8161a50 0x7fb0e80b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57924/0 (socket says 172.21.15.67:57924) 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.017+0000 7fb0ee519640 1 -- 172.21.15.67:0/888574184 learned_addr learned my addr 172.21.15.67:0/888574184 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.018+0000 7fb0ee519640 1 -- 172.21.15.67:0/888574184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e806dd70 msgr2=0x7fb0e80af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.018+0000 7fb0eed1a640 1 --2- 172.21.15.67:0/888574184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e806dd70 0x7fb0e80af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.018+0000 7fb0ee519640 1 --2- 172.21.15.67:0/888574184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e806dd70 0x7fb0e80af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.018+0000 7fb0ee519640 1 -- 172.21.15.67:0/888574184 >> v1:172.21.15.176:6789/0 conn(0x7fb0e80b17c0 legacy=0x7fb0e806d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:41.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.018+0000 7fb0ee519640 1 -- 172.21.15.67:0/888574184 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e80b3810 con 0x7fb0e8161a50 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.018+0000 7fb0ee519640 1 --2- 172.21.15.67:0/888574184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e8161a50 0x7fb0e80b1280 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009870 tx=0x7fb0e402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897f3267d591e0ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0ed517640 1 -- 172.21.15.67:0/888574184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4037070 con 0x7fb0e8161a50 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0ed517640 1 -- 172.21.15.67:0/888574184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0e4032440 con 0x7fb0e8161a50 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0ed517640 1 -- 172.21.15.67:0/888574184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4032810 con 0x7fb0e8161a50 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 -- 172.21.15.67:0/888574184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e8161a50 msgr2=0x7fb0e80b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 --2- 172.21.15.67:0/888574184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e8161a50 0x7fb0e80b1280 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009870 tx=0x7fb0e402a180 comp rx=0 tx=0).stop 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 -- 172.21.15.67:0/888574184 shutdown_connections 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 --2- 172.21.15.67:0/888574184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e806dd70 0x7fb0e80af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 --2- 172.21.15.67:0/888574184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e8161a50 0x7fb0e80b1280 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 -- 172.21.15.67:0/888574184 >> 172.21.15.67:0/888574184 conn(0x7fb0e809b7d0 msgr2=0x7fb0e80b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 -- 172.21.15.67:0/888574184 shutdown_connections 2024-04-25T15:28:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.019+0000 7fb0f077d640 1 -- 172.21.15.67:0/888574184 wait complete. 2024-04-25T15:28:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.021+0000 7fb0f077d640 1 Processor -- start 2024-04-25T15:28:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.021+0000 7fb0f077d640 1 -- start start 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.022+0000 7fb0f077d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e806dd70 0x7fb0e809fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.022+0000 7fb0f077d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e80b17c0 0x7fb0e8151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.022+0000 7fb0f077d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0e8161a50 0x7fb0e80cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.022+0000 7fb0f077d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e8161e60 con 0x7fb0e806dd70 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.022+0000 7fb0f077d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e80af050 con 0x7fb0e8161a50 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.022+0000 7fb0f077d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e80b2e70 con 0x7fb0e80b17c0 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e806dd70 0x7fb0e809fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e806dd70 0x7fb0e809fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57934/0 (socket says 172.21.15.67:57934) 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 -- 172.21.15.67:0/2328512441 learned_addr learned my addr 172.21.15.67:0/2328512441 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0edd18640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e80b17c0 0x7fb0e8151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0eed1a640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0e8161a50 0x7fb0e80cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 -- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e80b17c0 msgr2=0x7fb0e8151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e80b17c0 0x7fb0e8151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 -- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0e8161a50 msgr2=0x7fb0e80cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0e8161a50 0x7fb0e80cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 -- 172.21.15.67:0/2328512441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e8151c70 con 0x7fb0e806dd70 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0edd18640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e80b17c0 0x7fb0e8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0eed1a640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0e8161a50 0x7fb0e80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0ee519640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e806dd70 0x7fb0e809fc20 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009870 tx=0x7fb0e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0deffd640 1 -- 172.21.15.67:0/2328512441 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4041020 con 0x7fb0e806dd70 2024-04-25T15:28:41.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0deffd640 1 -- 172.21.15.67:0/2328512441 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0e4030970 con 0x7fb0e806dd70 2024-04-25T15:28:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e809ee60 con 0x7fb0e806dd70 2024-04-25T15:28:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0deffd640 1 -- 172.21.15.67:0/2328512441 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4037040 con 0x7fb0e806dd70 2024-04-25T15:28:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.023+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e809f420 con 0x7fb0e806dd70 2024-04-25T15:28:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.024+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e8050740 con 0x7fb0e806dd70 2024-04-25T15:28:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.025+0000 7fb0deffd640 1 -- 172.21.15.67:0/2328512441 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb0e4044020 con 0x7fb0e806dd70 2024-04-25T15:28:41.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.025+0000 7fb0deffd640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb0cc0744b0 0x7fb0cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.026+0000 7fb0deffd640 1 -- 172.21.15.67:0/2328512441 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fb0e40b5c40 con 0x7fb0e806dd70 2024-04-25T15:28:41.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.026+0000 7fb0edd18640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb0cc0744b0 0x7fb0cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.027+0000 7fb0edd18640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb0cc0744b0 0x7fb0cc0769c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8070700 tx=0x7fb0d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.030+0000 7fb0deffd640 1 -- 172.21.15.67:0/2328512441 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0e404b030 con 0x7fb0e806dd70 2024-04-25T15:28:41.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.214+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb0e80b7240 con 0x7fb0e806dd70 2024-04-25T15:28:41.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.214+0000 7fb0deffd640 1 -- 172.21.15.67:0/2328512441 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) ==== 93+0+10 (secure 0 0 0) 0x7fb0e80b1bd0 con 0x7fb0e806dd70 2024-04-25T15:28:41.215 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:41.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb0cc0744b0 msgr2=0x7fb0cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb0cc0744b0 0x7fb0cc0769c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8070700 tx=0x7fb0d8008040 comp rx=0 tx=0).stop 2024-04-25T15:28:41.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e806dd70 msgr2=0x7fb0e809fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e806dd70 0x7fb0e809fc20 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009870 tx=0x7fb0e40046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 shutdown_connections 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb0cc0744b0 0x7fb0cc0769c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0e8161a50 0x7fb0e80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0e80b17c0 0x7fb0e8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 --2- 172.21.15.67:0/2328512441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0e806dd70 0x7fb0e809fc20 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 >> 172.21.15.67:0/2328512441 conn(0x7fb0e809b7d0 msgr2=0x7fb0e806c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 shutdown_connections 2024-04-25T15:28:41.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.216+0000 7fb0f077d640 1 -- 172.21.15.67:0/2328512441 wait complete. 2024-04-25T15:28:41.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:28:41.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.342+0000 7fa52d89f640 1 Processor -- start 2024-04-25T15:28:41.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.343+0000 7fa52d89f640 1 -- start start 2024-04-25T15:28:41.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa52d89f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa5280d1e20 0x7fa5280d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa52d89f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280b2e60 0x7fa52808d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa52d89f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5280d23f0 con 0x7fa528164020 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa52d89f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa5280d26d0 con 0x7fa5280d1e20 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa52d89f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa5280d29b0 con 0x7fa5280b2e60 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa5277fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa528164020 0x7fa528164430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37162/0 (socket says 172.21.15.67:37162) 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa5277fe640 1 -- 172.21.15.67:0/1829152772 learned_addr learned my addr 172.21.15.67:0/1829152772 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa526ffd640 1 --2- 172.21.15.67:0/1829152772 >> v2:172.21.15.176:3300/0 conn(0x7fa5280d1e20 0x7fa5280d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa5267fc640 1 --2- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280b2e60 0x7fa52808d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa525ffb640 1 -- 172.21.15.67:0/1829152772 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150693765 0 0) 0x7fa5280d23f0 con 0x7fa528164020 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.344+0000 7fa525ffb640 1 -- 172.21.15.67:0/1829152772 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa514003520 con 0x7fa528164020 2024-04-25T15:28:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.345+0000 7fa525ffb640 1 -- 172.21.15.67:0/1829152772 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 20970801 0 0) 0x7fa514003520 con 0x7fa528164020 2024-04-25T15:28:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.345+0000 7fa526ffd640 1 -- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280b2e60 msgr2=0x7fa52808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.345+0000 7fa526ffd640 1 --2- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280b2e60 0x7fa52808d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.345+0000 7fa526ffd640 1 -- 172.21.15.67:0/1829152772 >> v1:172.21.15.67:6789/0 conn(0x7fa528164020 legacy=0x7fa528164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.345+0000 7fa526ffd640 1 -- 172.21.15.67:0/1829152772 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5280d2c90 con 0x7fa5280d1e20 2024-04-25T15:28:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.345+0000 7fa5267fc640 1 --2- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280b2e60 0x7fa52808d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.345+0000 7fa526ffd640 1 --2- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5280d3490 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa5200097a0 tx=0x7fa52002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=969f8508ab3275a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa525ffb640 1 -- 172.21.15.67:0/1829152772 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520037070 con 0x7fa5280d1e20 2024-04-25T15:28:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa525ffb640 1 -- 172.21.15.67:0/1829152772 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52002ac30 con 0x7fa5280d1e20 2024-04-25T15:28:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa525ffb640 1 -- 172.21.15.67:0/1829152772 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5180019f0 con 0x7fa5280d1e20 2024-04-25T15:28:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 -- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 msgr2=0x7fa5280d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 --2- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5280d3490 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa5200097a0 tx=0x7fa52002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 -- 172.21.15.67:0/1829152772 shutdown_connections 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 --2- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280b2e60 0x7fa52808d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 --2- 172.21.15.67:0/1829152772 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5280d3490 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 -- 172.21.15.67:0/1829152772 >> 172.21.15.67:0/1829152772 conn(0x7fa5280d0530 msgr2=0x7fa5280d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 -- 172.21.15.67:0/1829152772 shutdown_connections 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.346+0000 7fa52d89f640 1 -- 172.21.15.67:0/1829152772 wait complete. 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.348+0000 7fa52d89f640 1 Processor -- start 2024-04-25T15:28:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.348+0000 7fa52d89f640 1 -- start start 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa52d89f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5280b2e60 0x7fa5280cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa52d89f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5281f77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa52d89f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa528164020 0x7fa5281596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa52d89f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa5280ce070 con 0x7fa5280b2e60 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa52d89f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa52808dcb0 con 0x7fa5280d1e20 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa52d89f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa5280ce1f0 con 0x7fa528164020 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5280b2e60 0x7fa5280cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5280b2e60 0x7fa5280cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57944/0 (socket says 172.21.15.67:57944) 2024-04-25T15:28:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 -- 172.21.15.67:0/2929558943 learned_addr learned my addr 172.21.15.67:0/2929558943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa5267fc640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5281f77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa5277fe640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa528164020 0x7fa5281596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 -- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa528164020 msgr2=0x7fa5281596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa528164020 0x7fa5281596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 -- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 msgr2=0x7fa5281f77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5281f77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 -- 172.21.15.67:0/2929558943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa528162a30 con 0x7fa5280b2e60 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa5267fc640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5281f77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa5277fe640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa528164020 0x7fa5281596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa526ffd640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5280b2e60 0x7fa5280cd4b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa520035040 tx=0x7fa5200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa51ffff640 1 -- 172.21.15.67:0/2929558943 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520041020 con 0x7fa5280b2e60 2024-04-25T15:28:41.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa51ffff640 1 -- 172.21.15.67:0/2929558943 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52003d620 con 0x7fa5280b2e60 2024-04-25T15:28:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5281f6ce0 con 0x7fa5280b2e60 2024-04-25T15:28:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.350+0000 7fa51ffff640 1 -- 172.21.15.67:0/2929558943 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa520037040 con 0x7fa5280b2e60 2024-04-25T15:28:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.351+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa528161510 con 0x7fa5280b2e60 2024-04-25T15:28:41.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.352+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa528050740 con 0x7fa5280b2e60 2024-04-25T15:28:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.352+0000 7fa51ffff640 1 -- 172.21.15.67:0/2929558943 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa520044020 con 0x7fa5280b2e60 2024-04-25T15:28:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.352+0000 7fa51ffff640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa5080744b0 0x7fa5080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.353+0000 7fa51ffff640 1 -- 172.21.15.67:0/2929558943 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fa5200b56f0 con 0x7fa5280b2e60 2024-04-25T15:28:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.353+0000 7fa5267fc640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa5080744b0 0x7fa5080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.353+0000 7fa5267fc640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa5080744b0 0x7fa5080769c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa528071310 tx=0x7fa510008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.357+0000 7fa51ffff640 1 -- 172.21.15.67:0/2929558943 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa52004b030 con 0x7fa5280b2e60 2024-04-25T15:28:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.543+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa5280b8920 con 0x7fa5280b2e60 2024-04-25T15:28:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.543+0000 7fa51ffff640 1 -- 172.21.15.67:0/2929558943 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) ==== 104+0+11 (secure 0 0 0) 0x7fa52007ffa0 con 0x7fa5280b2e60 2024-04-25T15:28:41.544 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.545+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa5080744b0 msgr2=0x7fa5080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.545+0000 7fa52d89f640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa5080744b0 0x7fa5080769c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa528071310 tx=0x7fa510008040 comp rx=0 tx=0).stop 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.545+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5280b2e60 msgr2=0x7fa5280cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.545+0000 7fa52d89f640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5280b2e60 0x7fa5280cd4b0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa520035040 tx=0x7fa5200046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.545+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 shutdown_connections 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.545+0000 7fa52d89f640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa5080744b0 0x7fa5080769c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.546+0000 7fa52d89f640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa528164020 0x7fa5281596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.546+0000 7fa52d89f640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280d1e20 0x7fa5281f77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.546+0000 7fa52d89f640 1 --2- 172.21.15.67:0/2929558943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5280b2e60 0x7fa5280cd4b0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.546+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 >> 172.21.15.67:0/2929558943 conn(0x7fa5280d0530 msgr2=0x7fa52815fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:41.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.546+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 shutdown_connections 2024-04-25T15:28:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.546+0000 7fa52d89f640 1 -- 172.21.15.67:0/2929558943 wait complete. 2024-04-25T15:28:41.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:28:41.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.671+0000 7feebfa0c640 1 Processor -- start 2024-04-25T15:28:41.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.672+0000 7feebfa0c640 1 -- start start 2024-04-25T15:28:41.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebfa0c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7feeb80ce040 0x7feeb80b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebfa0c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feeb81649f0 0x7feeb80b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebfa0c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb806e350 0x7feeb806e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebfa0c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7feeb806eca0 con 0x7feeb81649f0 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebfa0c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7feeb806d770 con 0x7feeb80ce040 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebfa0c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feeb806da50 con 0x7feeb806e350 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feeb81649f0 0x7feeb80b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feeb81649f0 0x7feeb80b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57954/0 (socket says 172.21.15.67:57954) 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 -- 172.21.15.67:0/3312812487 learned_addr learned my addr 172.21.15.67:0/3312812487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebdfa9640 1 --2- 172.21.15.67:0/3312812487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb806e350 0x7feeb806e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebd7a8640 1 --2- 172.21.15.67:0/3312812487 >> v2:172.21.15.176:3300/0 conn(0x7feeb80ce040 0x7feeb80b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 -- 172.21.15.67:0/3312812487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb806e350 msgr2=0x7feeb806e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 --2- 172.21.15.67:0/3312812487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb806e350 0x7feeb806e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 -- 172.21.15.67:0/3312812487 >> v2:172.21.15.176:3300/0 conn(0x7feeb80ce040 msgr2=0x7feeb80b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 --2- 172.21.15.67:0/3312812487 >> v2:172.21.15.176:3300/0 conn(0x7feeb80ce040 0x7feeb80b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.673+0000 7feebcfa7640 1 -- 172.21.15.67:0/3312812487 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb80d2200 con 0x7feeb81649f0 2024-04-25T15:28:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.674+0000 7feebcfa7640 1 --2- 172.21.15.67:0/3312812487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb81649f0 0x7feeb80b90a0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7feeb80d39e0 tx=0x7feeac02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f4565bd4d41ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.674+0000 7feeb7fff640 1 -- 172.21.15.67:0/3312812487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac037070 con 0x7feeb81649f0 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.674+0000 7feeb7fff640 1 -- 172.21.15.67:0/3312812487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeac032440 con 0x7feeb81649f0 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.674+0000 7feeb7fff640 1 -- 172.21.15.67:0/3312812487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac032810 con 0x7feeb81649f0 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 -- 172.21.15.67:0/3312812487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb81649f0 msgr2=0x7feeb80b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 --2- 172.21.15.67:0/3312812487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb81649f0 0x7feeb80b90a0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7feeb80d39e0 tx=0x7feeac02a140 comp rx=0 tx=0).stop 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 -- 172.21.15.67:0/3312812487 shutdown_connections 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 --2- 172.21.15.67:0/3312812487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb806e350 0x7feeb806e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 --2- 172.21.15.67:0/3312812487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb81649f0 0x7feeb80b90a0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 --2- 172.21.15.67:0/3312812487 >> v2:172.21.15.176:3300/0 conn(0x7feeb80ce040 0x7feeb80b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 -- 172.21.15.67:0/3312812487 >> 172.21.15.67:0/3312812487 conn(0x7feeb80d3a80 msgr2=0x7feeb80d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 -- 172.21.15.67:0/3312812487 shutdown_connections 2024-04-25T15:28:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.675+0000 7feebfa0c640 1 -- 172.21.15.67:0/3312812487 wait complete. 2024-04-25T15:28:41.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.677+0000 7feebfa0c640 1 Processor -- start 2024-04-25T15:28:41.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.677+0000 7feebfa0c640 1 -- start start 2024-04-25T15:28:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebfa0c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeb806e350 0x7feeb80b8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebfa0c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb80ce040 0x7feeb815c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebfa0c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb81649f0 0x7feeb815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebfa0c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feeb8164e00 con 0x7feeb80ce040 2024-04-25T15:28:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebfa0c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feeb80d1f10 con 0x7feeb806e350 2024-04-25T15:28:41.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebfa0c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feeb80b5400 con 0x7feeb81649f0 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebcfa7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb80ce040 0x7feeb815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebcfa7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb80ce040 0x7feeb815c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57968/0 (socket says 172.21.15.67:57968) 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.678+0000 7feebcfa7640 1 -- 172.21.15.67:0/595463925 learned_addr learned my addr 172.21.15.67:0/595463925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebd7a8640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeb806e350 0x7feeb80b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebdfa9640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb81649f0 0x7feeb815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebcfa7640 1 -- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb81649f0 msgr2=0x7feeb815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebcfa7640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb81649f0 0x7feeb815cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebcfa7640 1 -- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeb806e350 msgr2=0x7feeb80b8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebcfa7640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeb806e350 0x7feeb80b8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebcfa7640 1 -- 172.21.15.67:0/595463925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb80b4d90 con 0x7feeb80ce040 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebd7a8640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeb806e350 0x7feeb80b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebcfa7640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb80ce040 0x7feeb815c720 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7feeac02a180 tx=0x7feeac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feeb5ffb640 1 -- 172.21.15.67:0/595463925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac041020 con 0x7feeb80ce040 2024-04-25T15:28:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feeb5ffb640 1 -- 172.21.15.67:0/595463925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeac03d620 con 0x7feeb80ce040 2024-04-25T15:28:41.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb80b1f60 con 0x7feeb80ce040 2024-04-25T15:28:41.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feeb5ffb640 1 -- 172.21.15.67:0/595463925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac037040 con 0x7feeb80ce040 2024-04-25T15:28:41.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.679+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb80b3510 con 0x7feeb80ce040 2024-04-25T15:28:41.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.680+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb8050740 con 0x7feeb80ce040 2024-04-25T15:28:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.681+0000 7feeb5ffb640 1 -- 172.21.15.67:0/595463925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7feeac044020 con 0x7feeb80ce040 2024-04-25T15:28:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.681+0000 7feeb5ffb640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feea40744e0 0x7feea40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.682+0000 7feeb5ffb640 1 -- 172.21.15.67:0/595463925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7feeac0b5830 con 0x7feeb80ce040 2024-04-25T15:28:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.682+0000 7feebd7a8640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feea40744e0 0x7feea40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:41.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.682+0000 7feebd7a8640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feea40744e0 0x7feea40769f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7feeb808ce10 tx=0x7feeb0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:41.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.686+0000 7feeb5ffb640 1 -- 172.21.15.67:0/595463925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feeac04b030 con 0x7feeb80ce040 2024-04-25T15:28:41.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.871+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feeb80d4780 con 0x7feeb80ce040 2024-04-25T15:28:41.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.871+0000 7feeb5ffb640 1 -- 172.21.15.67:0/595463925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) ==== 100+0+11 (secure 0 0 0) 0x7feeac0800e0 con 0x7feeb80ce040 2024-04-25T15:28:41.873 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feea40744e0 msgr2=0x7feea40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feea40744e0 0x7feea40769f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7feeb808ce10 tx=0x7feeb0008040 comp rx=0 tx=0).stop 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb80ce040 msgr2=0x7feeb815c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb80ce040 0x7feeb815c720 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7feeac02a180 tx=0x7feeac0046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 shutdown_connections 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7feea40744e0 0x7feea40769f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feeb81649f0 0x7feeb815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb80ce040 0x7feeb815c720 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 --2- 172.21.15.67:0/595463925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feeb806e350 0x7feeb80b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.873+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 >> 172.21.15.67:0/595463925 conn(0x7feeb80d3a80 msgr2=0x7feeb80d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.874+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 shutdown_connections 2024-04-25T15:28:41.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.874+0000 7feebfa0c640 1 -- 172.21.15.67:0/595463925 wait complete. 2024-04-25T15:28:41.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:41.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.997+0000 7f792c37a640 1 Processor -- start 2024-04-25T15:28:41.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.997+0000 7f792c37a640 1 -- start start 2024-04-25T15:28:42.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792c37a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79241443c0 0x7f792406fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792c37a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7924070370 0x7f79240cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792c37a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f79240cfb20 0x7f79240ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792c37a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f79240708d0 con 0x7f7924070370 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792c37a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f792415f160 con 0x7f79240cfb20 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792c37a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79240cff30 con 0x7f79241443c0 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7924070370 0x7f79240cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7924070370 0x7f79240cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57972/0 (socket says 172.21.15.67:57972) 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 -- 172.21.15.67:0/3349924810 learned_addr learned my addr 172.21.15.67:0/3349924810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792a917640 1 --2- 172.21.15.67:0/3349924810 >> v2:172.21.15.176:3300/0 conn(0x7f79240cfb20 0x7f79240ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792a116640 1 --2- 172.21.15.67:0/3349924810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79241443c0 0x7f792406fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 -- 172.21.15.67:0/3349924810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79241443c0 msgr2=0x7f792406fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 --2- 172.21.15.67:0/3349924810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79241443c0 0x7f792406fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 -- 172.21.15.67:0/3349924810 >> v2:172.21.15.176:3300/0 conn(0x7f79240cfb20 msgr2=0x7f79240ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 --2- 172.21.15.67:0/3349924810 >> v2:172.21.15.176:3300/0 conn(0x7f79240cfb20 0x7f79240ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f7929915640 1 -- 172.21.15.67:0/3349924810 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792415f5f0 con 0x7f7924070370 2024-04-25T15:28:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792a116640 1 --2- 172.21.15.67:0/3349924810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79241443c0 0x7f792406fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:41.999+0000 7f792a917640 1 --2- 172.21.15.67:0/3349924810 >> v2:172.21.15.176:3300/0 conn(0x7f79240cfb20 0x7f79240ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:42.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.000+0000 7f7929915640 1 --2- 172.21.15.67:0/3349924810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924070370 0x7f79240cf5e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f792406f8f0 tx=0x7f791402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0d83e8ba6ada6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.000+0000 7f7929114640 1 -- 172.21.15.67:0/3349924810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914037070 con 0x7f7924070370 2024-04-25T15:28:42.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.000+0000 7f7929114640 1 -- 172.21.15.67:0/3349924810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7914032440 con 0x7f7924070370 2024-04-25T15:28:42.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.000+0000 7f7929114640 1 -- 172.21.15.67:0/3349924810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914032810 con 0x7f7924070370 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 -- 172.21.15.67:0/3349924810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924070370 msgr2=0x7f79240cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 --2- 172.21.15.67:0/3349924810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924070370 0x7f79240cf5e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f792406f8f0 tx=0x7f791402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 -- 172.21.15.67:0/3349924810 shutdown_connections 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 --2- 172.21.15.67:0/3349924810 >> v2:172.21.15.176:3300/0 conn(0x7f79240cfb20 0x7f79240ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 --2- 172.21.15.67:0/3349924810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924070370 0x7f79240cf5e0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 --2- 172.21.15.67:0/3349924810 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79241443c0 0x7f792406fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 -- 172.21.15.67:0/3349924810 >> 172.21.15.67:0/3349924810 conn(0x7f79240d1ae0 msgr2=0x7f79240d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 -- 172.21.15.67:0/3349924810 shutdown_connections 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.001+0000 7f792c37a640 1 -- 172.21.15.67:0/3349924810 wait complete. 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.003+0000 7f792c37a640 1 Processor -- start 2024-04-25T15:28:42.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.003+0000 7f792c37a640 1 -- start start 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792c37a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7924070370 0x7f79241399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792c37a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79240cfb20 0x7f79240aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792c37a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79241443c0 0x7f79241f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792c37a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f792415edf0 con 0x7f79240cfb20 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792c37a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f792415eb10 con 0x7f79241443c0 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792c37a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79241447d0 con 0x7f7924070370 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f7929915640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79240cfb20 0x7f79240aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f7929915640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79240cfb20 0x7f79240aa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57976/0 (socket says 172.21.15.67:57976) 2024-04-25T15:28:42.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f7929915640 1 -- 172.21.15.67:0/2410270162 learned_addr learned my addr 172.21.15.67:0/2410270162 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792a116640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7924070370 0x7f79241399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.004+0000 7f792a917640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79241443c0 0x7f79241f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f7929915640 1 -- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7924070370 msgr2=0x7f79241399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f7929915640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7924070370 0x7f79241399c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f7929915640 1 -- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79241443c0 msgr2=0x7f79241f78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f7929915640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79241443c0 0x7f79241f78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f7929915640 1 -- 172.21.15.67:0/2410270162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7924139f00 con 0x7f79240cfb20 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f7929915640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79240cfb20 0x7f79240aa230 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f7914000c00 tx=0x7f79140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f791affd640 1 -- 172.21.15.67:0/2410270162 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914037040 con 0x7f79240cfb20 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f791affd640 1 -- 172.21.15.67:0/2410270162 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7914042070 con 0x7f79240cfb20 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792413ae10 con 0x7f79240cfb20 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f791affd640 1 -- 172.21.15.67:0/2410270162 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914030bb0 con 0x7f79240cfb20 2024-04-25T15:28:42.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.005+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f792413b3d0 con 0x7f79240cfb20 2024-04-25T15:28:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.006+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f792406f090 con 0x7f79240cfb20 2024-04-25T15:28:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.007+0000 7f791affd640 1 -- 172.21.15.67:0/2410270162 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7914047460 con 0x7f79240cfb20 2024-04-25T15:28:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.007+0000 7f791affd640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79080744e0 0x7f79080769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.007+0000 7f791affd640 1 -- 172.21.15.67:0/2410270162 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f79140b5d30 con 0x7f79240cfb20 2024-04-25T15:28:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.007+0000 7f792a116640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79080744e0 0x7f79080769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.008+0000 7f792a116640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79080744e0 0x7f79080769f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f79240d1860 tx=0x7f791c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.012+0000 7f791affd640 1 -- 172.21.15.67:0/2410270162 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f792406f090 con 0x7f79240cfb20 2024-04-25T15:28:42.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.235+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7924050740 con 0x7f79240cfb20 2024-04-25T15:28:42.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.236+0000 7f791affd640 1 -- 172.21.15.67:0/2410270162 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5186 (secure 0 0 0) 0x7f791407ff00 con 0x7f79240cfb20 2024-04-25T15:28:42.237 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:42.237 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/445614904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":445614904},{"type":"v1","addr":"172.21.15.67:6833","nonce":445614904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":6322,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3018066741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3018066741},{"type":"v1","addr":"172.21.15.67:6835","nonce":3018066741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2045345118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2045345118},{"type":"v1","addr":"172.21.15.176:6836","nonce":2045345118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:23.008043+0000","modified":"2024-04-25T15:28:26.033114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6328":{"gid":6328,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6837/2484404027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2484404027},{"type":"v1","addr":"172.21.15.176:6837","nonce":2484404027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6328,"qdb_cluster":[6328]},"id":6}]} 2024-04-25T15:28:42.237 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 49 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79080744e0 msgr2=0x7f79080769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79080744e0 0x7f79080769f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f79240d1860 tx=0x7f791c008040 comp rx=0 tx=0).stop 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79240cfb20 msgr2=0x7f79240aa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79240cfb20 0x7f79240aa230 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f7914000c00 tx=0x7f79140046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 shutdown_connections 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f79080744e0 0x7f79080769f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79241443c0 0x7f79241f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79240cfb20 0x7f79240aa230 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 --2- 172.21.15.67:0/2410270162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7924070370 0x7f79241399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 >> 172.21.15.67:0/2410270162 conn(0x7f79240d1ae0 msgr2=0x7f7924160900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:42.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 shutdown_connections 2024-04-25T15:28:42.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.238+0000 7f792c37a640 1 -- 172.21.15.67:0/2410270162 wait complete. 2024-04-25T15:28:42.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:42.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.361+0000 7f4c86de0640 1 Processor -- start 2024-04-25T15:28:42.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.361+0000 7f4c86de0640 1 -- start start 2024-04-25T15:28:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c86de0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c800d4130 0x7f4c800d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c86de0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c8006dcf0 0x7f4c800d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c86de0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c800b3c10 con 0x7f4c8006d8e0 2024-04-25T15:28:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c86de0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4c800b9520 con 0x7f4c8006dcf0 2024-04-25T15:28:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c86de0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c800b4820 con 0x7f4c800d4130 2024-04-25T15:28:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c8537d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c8006dcf0 0x7f4c800d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c84b7c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c8006d8e0 0x7f4c800d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37174/0 (socket says 172.21.15.67:37174) 2024-04-25T15:28:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c84b7c640 1 -- 172.21.15.67:0/3995407499 learned_addr learned my addr 172.21.15.67:0/3995407499 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c7ffff640 1 --2- 172.21.15.67:0/3995407499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c800d4130 0x7f4c800d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c8537d640 1 -- 172.21.15.67:0/3995407499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c800d4130 msgr2=0x7f4c800d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c8537d640 1 --2- 172.21.15.67:0/3995407499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c800d4130 0x7f4c800d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c8537d640 1 -- 172.21.15.67:0/3995407499 >> v1:172.21.15.67:6789/0 conn(0x7f4c8006d8e0 legacy=0x7f4c800d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c8537d640 1 -- 172.21.15.67:0/3995407499 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c800b3ef0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.363+0000 7f4c7ffff640 1 --2- 172.21.15.67:0/3995407499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c800d4130 0x7f4c800d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:42.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.364+0000 7f4c8537d640 1 --2- 172.21.15.67:0/3995407499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c800d22a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009d90 tx=0x7f4c7802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d340e2d545409b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.364+0000 7f4c7f7fe640 1 -- 172.21.15.67:0/3995407499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78031600 con 0x7f4c8006dcf0 2024-04-25T15:28:42.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.364+0000 7f4c7f7fe640 1 -- 172.21.15.67:0/3995407499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c780317a0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.364+0000 7f4c7f7fe640 1 -- 172.21.15.67:0/3995407499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78031b70 con 0x7f4c8006dcf0 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 -- 172.21.15.67:0/3995407499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 msgr2=0x7f4c800d22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 --2- 172.21.15.67:0/3995407499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c800d22a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009d90 tx=0x7f4c7802a5d0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 -- 172.21.15.67:0/3995407499 shutdown_connections 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 --2- 172.21.15.67:0/3995407499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c800d22a0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 --2- 172.21.15.67:0/3995407499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c800d4130 0x7f4c800d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 -- 172.21.15.67:0/3995407499 >> 172.21.15.67:0/3995407499 conn(0x7f4c8008d110 msgr2=0x7f4c8008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 -- 172.21.15.67:0/3995407499 shutdown_connections 2024-04-25T15:28:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.365+0000 7f4c86de0640 1 -- 172.21.15.67:0/3995407499 wait complete. 2024-04-25T15:28:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.367+0000 7f4c86de0640 1 Processor -- start 2024-04-25T15:28:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.367+0000 7f4c86de0640 1 -- start start 2024-04-25T15:28:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c86de0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c8006d8e0 0x7f4c8009e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c86de0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c801611f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c86de0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c800a0470 0x7f4c800a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c86de0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c800b38e0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c86de0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4c800b44b0 con 0x7f4c800a0470 2024-04-25T15:28:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c86de0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c800d27e0 con 0x7f4c8006d8e0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c801611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c801611f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57984/0 (socket says 172.21.15.67:57984) 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c7ffff640 1 -- 172.21.15.67:0/2728782473 learned_addr learned my addr 172.21.15.67:0/2728782473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c84b7c640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c8006d8e0 0x7f4c8009e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.368+0000 7f4c8537d640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c800a0470 0x7f4c800a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7ffff640 1 -- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c8006d8e0 msgr2=0x7f4c8009e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7ffff640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c8006d8e0 0x7f4c8009e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7ffff640 1 -- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c800a0470 msgr2=0x7f4c800a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7ffff640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c800a0470 0x7f4c800a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7ffff640 1 -- 172.21.15.67:0/2728782473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c800a0dc0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7ffff640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c801611f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d800 tx=0x7f4c7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7d7fa640 1 -- 172.21.15.67:0/2728782473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74004030 con 0x7f4c8006dcf0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7d7fa640 1 -- 172.21.15.67:0/2728782473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c740041d0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c8009e9e0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c7d7fa640 1 -- 172.21.15.67:0/2728782473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c740096a0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.369+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c8009efa0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.370+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c80050740 con 0x7f4c8006dcf0 2024-04-25T15:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.371+0000 7f4c7d7fa640 1 -- 172.21.15.67:0/2728782473 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4c74009840 con 0x7f4c8006dcf0 2024-04-25T15:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.371+0000 7f4c7d7fa640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c6c0744e0 0x7f4c6c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.371+0000 7f4c7d7fa640 1 -- 172.21.15.67:0/2728782473 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f4c74095500 con 0x7f4c8006dcf0 2024-04-25T15:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.371+0000 7f4c84b7c640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c6c0744e0 0x7f4c6c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.372+0000 7f4c84b7c640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c6c0744e0 0x7f4c6c0769f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4c70000f60 tx=0x7f4c70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.376+0000 7f4c7d7fa640 1 -- 172.21.15.67:0/2728782473 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c800a0dc0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.562+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c800d0070 con 0x7f4c8006dcf0 2024-04-25T15:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.563+0000 7f4c7d7fa640 1 -- 172.21.15.67:0/2728782473 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16624 (secure 0 0 0) 0x7f4c7405fca0 con 0x7f4c8006dcf0 2024-04-25T15:28:42.564 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:42.564 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:24.632893+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c6c0744e0 msgr2=0x7f4c6c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c6c0744e0 0x7f4c6c0769f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4c70000f60 tx=0x7f4c70009040 comp rx=0 tx=0).stop 2024-04-25T15:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 msgr2=0x7f4c801611f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c801611f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d800 tx=0x7f4c7400dcc0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 shutdown_connections 2024-04-25T15:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4c6c0744e0 0x7f4c6c0769f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c800a0470 0x7f4c800a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8006dcf0 0x7f4c801611f0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 --2- 172.21.15.67:0/2728782473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c8006d8e0 0x7f4c8009e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 >> 172.21.15.67:0/2728782473 conn(0x7f4c8008d110 msgr2=0x7f4c801e9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 shutdown_connections 2024-04-25T15:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.565+0000 7f4c86de0640 1 -- 172.21.15.67:0/2728782473 wait complete. 2024-04-25T15:28:42.578 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:28:42.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.689+0000 7fb7b071f640 1 Processor -- start 2024-04-25T15:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.689+0000 7fb7b071f640 1 -- start start 2024-04-25T15:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7b071f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a8164b00 0x7fb7a80b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7b071f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb7a815df80 0x7fb7a80b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7b071f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7a8160890 0x7fb7a8160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7b071f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb7a8164670 con 0x7fb7a8160890 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7b071f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb7a81615f0 con 0x7fb7a815df80 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7b071f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a8164020 con 0x7fb7a8164b00 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7aecbc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7a8160890 0x7fb7a8160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7aecbc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7a8160890 0x7fb7a8160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57988/0 (socket says 172.21.15.67:57988) 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.690+0000 7fb7aecbc640 1 -- 172.21.15.67:0/2197447487 learned_addr learned my addr 172.21.15.67:0/2197447487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7ae4bb640 1 --2- 172.21.15.67:0/2197447487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a8164b00 0x7fb7a80b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7adcba640 1 --2- 172.21.15.67:0/2197447487 >> v2:172.21.15.176:3300/0 conn(0x7fb7a815df80 0x7fb7a80b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7aecbc640 1 -- 172.21.15.67:0/2197447487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a8164b00 msgr2=0x7fb7a80b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7aecbc640 1 --2- 172.21.15.67:0/2197447487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a8164b00 0x7fb7a80b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7aecbc640 1 -- 172.21.15.67:0/2197447487 >> v2:172.21.15.176:3300/0 conn(0x7fb7a815df80 msgr2=0x7fb7a80b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7aecbc640 1 --2- 172.21.15.67:0/2197447487 >> v2:172.21.15.176:3300/0 conn(0x7fb7a815df80 0x7fb7a80b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7aecbc640 1 -- 172.21.15.67:0/2197447487 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a8164300 con 0x7fb7a8160890 2024-04-25T15:28:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.691+0000 7fb7aecbc640 1 --2- 172.21.15.67:0/2197447487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8160890 0x7fb7a8160ca0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb7a808e440 tx=0x7fb7a002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fdd396a6a94137 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7ad4b9640 1 -- 172.21.15.67:0/2197447487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0036070 con 0x7fb7a8160890 2024-04-25T15:28:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7ad4b9640 1 -- 172.21.15.67:0/2197447487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a0031600 con 0x7fb7a8160890 2024-04-25T15:28:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7ad4b9640 1 -- 172.21.15.67:0/2197447487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a00319d0 con 0x7fb7a8160890 2024-04-25T15:28:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 -- 172.21.15.67:0/2197447487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8160890 msgr2=0x7fb7a8160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 --2- 172.21.15.67:0/2197447487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8160890 0x7fb7a8160ca0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb7a808e440 tx=0x7fb7a002a6a0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 -- 172.21.15.67:0/2197447487 shutdown_connections 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 --2- 172.21.15.67:0/2197447487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8160890 0x7fb7a8160ca0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 --2- 172.21.15.67:0/2197447487 >> v2:172.21.15.176:3300/0 conn(0x7fb7a815df80 0x7fb7a80b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 --2- 172.21.15.67:0/2197447487 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a8164b00 0x7fb7a80b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 -- 172.21.15.67:0/2197447487 >> 172.21.15.67:0/2197447487 conn(0x7fb7a80d2bc0 msgr2=0x7fb7a80d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.692+0000 7fb7b071f640 1 -- 172.21.15.67:0/2197447487 shutdown_connections 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.693+0000 7fb7b071f640 1 -- 172.21.15.67:0/2197447487 wait complete. 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.694+0000 7fb7b071f640 1 Processor -- start 2024-04-25T15:28:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.694+0000 7fb7b071f640 1 -- start start 2024-04-25T15:28:42.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.695+0000 7fb7b071f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a815df80 0x7fb7a814f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.695+0000 7fb7b071f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a8160890 0x7fb7a81ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.695+0000 7fb7b071f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8164b00 0x7fb7a81e9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.695+0000 7fb7b071f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a80b4a00 con 0x7fb7a8164b00 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.695+0000 7fb7b071f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a80b4880 con 0x7fb7a8160890 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.695+0000 7fb7b071f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a80b4b80 con 0x7fb7a815df80 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8164b00 0x7fb7a81e9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8164b00 0x7fb7a81e9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58000/0 (socket says 172.21.15.67:58000) 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 -- 172.21.15.67:0/4053738654 learned_addr learned my addr 172.21.15.67:0/4053738654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7adcba640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a8160890 0x7fb7a81ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7ae4bb640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a815df80 0x7fb7a814f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 -- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a815df80 msgr2=0x7fb7a814f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a815df80 0x7fb7a814f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 -- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a8160890 msgr2=0x7fb7a81ebb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a8160890 0x7fb7a81ebb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 -- 172.21.15.67:0/4053738654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a81eabd0 con 0x7fb7a8164b00 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7aecbc640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8164b00 0x7fb7a81e9ce0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0009f40 tx=0x7fb7a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb79effd640 1 -- 172.21.15.67:0/4053738654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0036050 con 0x7fb7a8164b00 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb79effd640 1 -- 172.21.15.67:0/4053738654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a0042070 con 0x7fb7a8164b00 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a81eaeb0 con 0x7fb7a8164b00 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb79effd640 1 -- 172.21.15.67:0/4053738654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0009510 con 0x7fb7a8164b00 2024-04-25T15:28:42.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.696+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a8162f60 con 0x7fb7a8164b00 2024-04-25T15:28:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.698+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a81631c0 con 0x7fb7a8164b00 2024-04-25T15:28:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.698+0000 7fb79effd640 1 -- 172.21.15.67:0/4053738654 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb7a0007bd0 con 0x7fb7a8164b00 2024-04-25T15:28:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.698+0000 7fb79effd640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb78c0744b0 0x7fb78c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.699+0000 7fb79effd640 1 -- 172.21.15.67:0/4053738654 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fb7a00b6050 con 0x7fb7a8164b00 2024-04-25T15:28:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.699+0000 7fb7ae4bb640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb78c0744b0 0x7fb78c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:42.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.699+0000 7fb7ae4bb640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb78c0744b0 0x7fb78c0769c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4006fd0 tx=0x7fb7a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:42.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.703+0000 7fb79effd640 1 -- 172.21.15.67:0/4053738654 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7a0080870 con 0x7fb7a8164b00 2024-04-25T15:28:42.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.924+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7a8161b20 con 0x7fb7a8164b00 2024-04-25T15:28:42.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.925+0000 7fb79effd640 1 -- 172.21.15.67:0/4053738654 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5186 (secure 0 0 0) 0x7fb7a0080170 con 0x7fb7a8164b00 2024-04-25T15:28:42.926 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:42.926 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/445614904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":445614904},{"type":"v1","addr":"172.21.15.67:6833","nonce":445614904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":6322,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3018066741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3018066741},{"type":"v1","addr":"172.21.15.67:6835","nonce":3018066741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2045345118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2045345118},{"type":"v1","addr":"172.21.15.176:6836","nonce":2045345118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:23.008043+0000","modified":"2024-04-25T15:28:26.033114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6328":{"gid":6328,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6837/2484404027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2484404027},{"type":"v1","addr":"172.21.15.176:6837","nonce":2484404027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6328,"qdb_cluster":[6328]},"id":6}]} 2024-04-25T15:28:42.926 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 49 2024-04-25T15:28:42.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb78c0744b0 msgr2=0x7fb78c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb78c0744b0 0x7fb78c0769c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4006fd0 tx=0x7fb7a4008040 comp rx=0 tx=0).stop 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8164b00 msgr2=0x7fb7a81e9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8164b00 0x7fb7a81e9ce0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0009f40 tx=0x7fb7a0004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 shutdown_connections 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb78c0744b0 0x7fb78c0769c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a8164b00 0x7fb7a81e9ce0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a8160890 0x7fb7a81ebb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 --2- 172.21.15.67:0/4053738654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a815df80 0x7fb7a814f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 >> 172.21.15.67:0/4053738654 conn(0x7fb7a80d2bc0 msgr2=0x7fb7a80d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 shutdown_connections 2024-04-25T15:28:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:42.927+0000 7fb7b071f640 1 -- 172.21.15.67:0/4053738654 wait complete. 2024-04-25T15:28:42.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:43.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.051+0000 7fed46af2640 1 Processor -- start 2024-04-25T15:28:43.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.051+0000 7fed46af2640 1 -- start start 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.052+0000 7fed46af2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b21a0 0x7fed400ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.052+0000 7fed46af2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed400b7840 0x7fed400b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed46af2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed400b8190 con 0x7fed400cfd40 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed46af2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fed400d27e0 con 0x7fed400b7840 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed46af2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fed400d1c10 con 0x7fed400b21a0 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed4508f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed400b7840 0x7fed400b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed4508f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed400b7840 0x7fed400b7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58010/0 (socket says 172.21.15.67:58010) 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed4508f640 1 -- 172.21.15.67:0/3237247641 learned_addr learned my addr 172.21.15.67:0/3237247641 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:43.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed3ffff640 1 --2- 172.21.15.67:0/3237247641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b21a0 0x7fed400ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed4508f640 1 -- 172.21.15.67:0/3237247641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b21a0 msgr2=0x7fed400ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed4508f640 1 --2- 172.21.15.67:0/3237247641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b21a0 0x7fed400ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed4508f640 1 -- 172.21.15.67:0/3237247641 >> v1:172.21.15.176:6789/0 conn(0x7fed400cfd40 legacy=0x7fed400b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.053+0000 7fed4508f640 1 -- 172.21.15.67:0/3237247641 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed400b25b0 con 0x7fed400b7840 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.054+0000 7fed4508f640 1 --2- 172.21.15.67:0/3237247641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b7840 0x7fed400b7c50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fed38009e50 tx=0x7fed3802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662ff7671578e9d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.054+0000 7fed3f7fe640 1 -- 172.21.15.67:0/3237247641 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38036070 con 0x7fed400b7840 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.054+0000 7fed3f7fe640 1 -- 172.21.15.67:0/3237247641 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed38031600 con 0x7fed400b7840 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.054+0000 7fed3f7fe640 1 -- 172.21.15.67:0/3237247641 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380319d0 con 0x7fed400b7840 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 -- 172.21.15.67:0/3237247641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b7840 msgr2=0x7fed400b7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 --2- 172.21.15.67:0/3237247641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b7840 0x7fed400b7c50 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fed38009e50 tx=0x7fed3802a680 comp rx=0 tx=0).stop 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 -- 172.21.15.67:0/3237247641 shutdown_connections 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 --2- 172.21.15.67:0/3237247641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b7840 0x7fed400b7c50 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 --2- 172.21.15.67:0/3237247641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b21a0 0x7fed400ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 -- 172.21.15.67:0/3237247641 >> 172.21.15.67:0/3237247641 conn(0x7fed4009b7d0 msgr2=0x7fed4009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 -- 172.21.15.67:0/3237247641 shutdown_connections 2024-04-25T15:28:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.055+0000 7fed46af2640 1 -- 172.21.15.67:0/3237247641 wait complete. 2024-04-25T15:28:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.057+0000 7fed46af2640 1 Processor -- start 2024-04-25T15:28:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.057+0000 7fed46af2640 1 -- start start 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed46af2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b21a0 0x7fed40162a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed46af2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b7840 0x7fed401f9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed46af2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed400cfd40 0x7fed400b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed46af2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fed400d2470 con 0x7fed400b21a0 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed46af2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fed400d2190 con 0x7fed400cfd40 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed46af2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fed400ce650 con 0x7fed400b7840 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b21a0 0x7fed40162a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b21a0 0x7fed40162a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58016/0 (socket says 172.21.15.67:58016) 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 -- 172.21.15.67:0/3154302211 learned_addr learned my addr 172.21.15.67:0/3154302211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4508f640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed400cfd40 0x7fed400b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed3ffff640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b7840 0x7fed401f9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 -- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b7840 msgr2=0x7fed401f9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b7840 0x7fed401f9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 -- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed400cfd40 msgr2=0x7fed400b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed400cfd40 0x7fed400b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.058+0000 7fed4488e640 1 -- 172.21.15.67:0/3154302211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed401637f0 con 0x7fed400b21a0 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed3ffff640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b7840 0x7fed401f9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed4508f640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed400cfd40 0x7fed400b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed4488e640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b21a0 0x7fed40162a70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c3c0 tx=0x7fed3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed3d7fa640 1 -- 172.21.15.67:0/3154302211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed300052d0 con 0x7fed400b21a0 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed3d7fa640 1 -- 172.21.15.67:0/3154302211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed30005490 con 0x7fed400b21a0 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed3d7fa640 1 -- 172.21.15.67:0/3154302211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed30007260 con 0x7fed400b21a0 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed401f64a0 con 0x7fed400b21a0 2024-04-25T15:28:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.059+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed40163510 con 0x7fed400b21a0 2024-04-25T15:28:43.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.060+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed40050740 con 0x7fed400b21a0 2024-04-25T15:28:43.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.061+0000 7fed3d7fa640 1 -- 172.21.15.67:0/3154302211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fed30007400 con 0x7fed400b21a0 2024-04-25T15:28:43.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.061+0000 7fed3d7fa640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fed2c0744b0 0x7fed2c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.061+0000 7fed3d7fa640 1 -- 172.21.15.67:0/3154302211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7fed300946f0 con 0x7fed400b21a0 2024-04-25T15:28:43.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.061+0000 7fed3ffff640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fed2c0744b0 0x7fed2c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.062+0000 7fed3ffff640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fed2c0744b0 0x7fed2c0769c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fed4015a7b0 tx=0x7fed34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.066+0000 7fed3d7fa640 1 -- 172.21.15.67:0/3154302211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed400b8190 con 0x7fed400b21a0 2024-04-25T15:28:43.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.291+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed400b40f0 con 0x7fed400b21a0 2024-04-25T15:28:43.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.291+0000 7fed3d7fa640 1 -- 172.21.15.67:0/3154302211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5186 (secure 0 0 0) 0x7fed3005ee80 con 0x7fed400b21a0 2024-04-25T15:28:43.292 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:43.293 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/445614904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":445614904},{"type":"v1","addr":"172.21.15.67:6833","nonce":445614904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":6322,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3018066741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3018066741},{"type":"v1","addr":"172.21.15.67:6835","nonce":3018066741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2045345118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2045345118},{"type":"v1","addr":"172.21.15.176:6836","nonce":2045345118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:23.008043+0000","modified":"2024-04-25T15:28:26.033114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6328},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6328":{"gid":6328,"name":"d","rank":0,"incarnation":46,"state":"up:active","state_seq":3,"addr":"172.21.15.176:6837/2484404027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2484404027},{"type":"v1","addr":"172.21.15.176:6837","nonce":2484404027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6328,"qdb_cluster":[6328]},"id":6}]} 2024-04-25T15:28:43.293 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 49 2024-04-25T15:28:43.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.293+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fed2c0744b0 msgr2=0x7fed2c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.293+0000 7fed46af2640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fed2c0744b0 0x7fed2c0769c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fed4015a7b0 tx=0x7fed34008040 comp rx=0 tx=0).stop 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.293+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b21a0 msgr2=0x7fed40162a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.293+0000 7fed46af2640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b21a0 0x7fed40162a70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fed3000c3c0 tx=0x7fed3000c880 comp rx=0 tx=0).stop 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 shutdown_connections 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fed2c0744b0 0x7fed2c0769c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed400cfd40 0x7fed400b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed400b7840 0x7fed401f9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 --2- 172.21.15.67:0/3154302211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed400b21a0 0x7fed40162a70 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 >> 172.21.15.67:0/3154302211 conn(0x7fed4009b7d0 msgr2=0x7fed400b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 shutdown_connections 2024-04-25T15:28:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.294+0000 7fed46af2640 1 -- 172.21.15.67:0/3154302211 wait complete. 2024-04-25T15:28:43.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:43.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.418+0000 7f769f372640 1 Processor -- start 2024-04-25T15:28:43.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.418+0000 7f769f372640 1 -- start start 2024-04-25T15:28:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769f372640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f76980d0de0 0x7f76980d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769f372640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f769806c5c0 0x7f769806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769f372640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f769806cf10 con 0x7f76980b77f0 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769f372640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f76980d5a40 con 0x7f769806c5c0 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769f372640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f76980d1830 con 0x7f76980d0de0 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769c90d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f769806c5c0 0x7f769806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769d90f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76980b77f0 0x7f76980b7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37176/0 (socket says 172.21.15.67:37176) 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.419+0000 7f769d90f640 1 -- 172.21.15.67:0/1180549745 learned_addr learned my addr 172.21.15.67:0/1180549745 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f769d10e640 1 --2- 172.21.15.67:0/1180549745 >> v2:172.21.15.176:3300/0 conn(0x7f76980d0de0 0x7f76980d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f769c90d640 1 -- 172.21.15.67:0/1180549745 >> v2:172.21.15.176:3300/0 conn(0x7f76980d0de0 msgr2=0x7f76980d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f769c90d640 1 --2- 172.21.15.67:0/1180549745 >> v2:172.21.15.176:3300/0 conn(0x7f76980d0de0 0x7f76980d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f769c90d640 1 -- 172.21.15.67:0/1180549745 >> v1:172.21.15.67:6789/0 conn(0x7f76980b77f0 legacy=0x7f76980b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f769c90d640 1 -- 172.21.15.67:0/1180549745 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7698160dc0 con 0x7f769806c5c0 2024-04-25T15:28:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f7697fff640 1 -- 172.21.15.67:0/1180549745 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306768278 0 0) 0x7f7688001490 con 0x7f76980b77f0 2024-04-25T15:28:43.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f769c90d640 1 --2- 172.21.15.67:0/1180549745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769806c5c0 0x7f769806c9d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f76980d5bc0 tx=0x7f768c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1933b5d0bf1ef9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.420+0000 7f7697fff640 1 -- 172.21.15.67:0/1180549745 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c037070 con 0x7f769806c5c0 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f7697fff640 1 -- 172.21.15.67:0/1180549745 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768c032440 con 0x7f769806c5c0 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f7697fff640 1 -- 172.21.15.67:0/1180549745 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c02ae60 con 0x7f769806c5c0 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 -- 172.21.15.67:0/1180549745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769806c5c0 msgr2=0x7f769806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 --2- 172.21.15.67:0/1180549745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769806c5c0 0x7f769806c9d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f76980d5bc0 tx=0x7f768c02a110 comp rx=0 tx=0).stop 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 -- 172.21.15.67:0/1180549745 shutdown_connections 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 --2- 172.21.15.67:0/1180549745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f769806c5c0 0x7f769806c9d0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 --2- 172.21.15.67:0/1180549745 >> v2:172.21.15.176:3300/0 conn(0x7f76980d0de0 0x7f76980d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 -- 172.21.15.67:0/1180549745 >> 172.21.15.67:0/1180549745 conn(0x7f76980d5d70 msgr2=0x7f76980d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 -- 172.21.15.67:0/1180549745 shutdown_connections 2024-04-25T15:28:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.421+0000 7f769f372640 1 -- 172.21.15.67:0/1180549745 wait complete. 2024-04-25T15:28:43.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.423+0000 7f769f372640 1 Processor -- start 2024-04-25T15:28:43.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.423+0000 7f769f372640 1 -- start start 2024-04-25T15:28:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.424+0000 7f769f372640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f769806c5c0 0x7f76981f9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.424+0000 7f769f372640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76980b77f0 0x7f7698152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.424+0000 7f769f372640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76980d0de0 0x7f76980ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.424+0000 7f769f372640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7698160ab0 con 0x7f76980d0de0 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.424+0000 7f769f372640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7698161380 con 0x7f76980b77f0 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.424+0000 7f769f372640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f76980b5300 con 0x7f769806c5c0 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.424+0000 7f769d90f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76980d0de0 0x7f76980ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76980d0de0 0x7f76980ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58038/0 (socket says 172.21.15.67:58038) 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 -- 172.21.15.67:0/3612892124 learned_addr learned my addr 172.21.15.67:0/3612892124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769c90d640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76980b77f0 0x7f7698152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d10e640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f769806c5c0 0x7f76981f9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 -- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f769806c5c0 msgr2=0x7f76981f9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f769806c5c0 0x7f76981f9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 -- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76980b77f0 msgr2=0x7f7698152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76980b77f0 0x7f7698152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 -- 172.21.15.67:0/3612892124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76981fa850 con 0x7f76980d0de0 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d10e640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f769806c5c0 0x7f76981f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769d90f640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76980d0de0 0x7f76980ce7d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f768800c700 tx=0x7f768800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f7695ffb640 1 -- 172.21.15.67:0/3612892124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7688018070 con 0x7f76980d0de0 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f7695ffb640 1 -- 172.21.15.67:0/3612892124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768800ce10 con 0x7f76980d0de0 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76981fab30 con 0x7f76980d0de0 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f7695ffb640 1 -- 172.21.15.67:0/3612892124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768800a860 con 0x7f76980d0de0 2024-04-25T15:28:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.425+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76980cc910 con 0x7f76980d0de0 2024-04-25T15:28:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.426+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76981fa5c0 con 0x7f76980d0de0 2024-04-25T15:28:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.427+0000 7f7695ffb640 1 -- 172.21.15.67:0/3612892124 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f768801b070 con 0x7f76980d0de0 2024-04-25T15:28:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.427+0000 7f7695ffb640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76840744e0 0x7f76840769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.428+0000 7f7695ffb640 1 -- 172.21.15.67:0/3612892124 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f76880946e0 con 0x7f76980d0de0 2024-04-25T15:28:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.428+0000 7f769d10e640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76840744e0 0x7f76840769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.428+0000 7f769d10e640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76840744e0 0x7f76840769f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7690006fd0 tx=0x7f7690008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.432+0000 7f7695ffb640 1 -- 172.21.15.67:0/3612892124 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f769806cf10 con 0x7f76980d0de0 2024-04-25T15:28:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.621+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7698050740 con 0x7f76980d0de0 2024-04-25T15:28:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.622+0000 7f7695ffb640 1 -- 172.21.15.67:0/3612892124 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16624 (secure 0 0 0) 0x7f76880650d0 con 0x7f76980d0de0 2024-04-25T15:28:43.623 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:43.623 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":80,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:24.632893+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:21.329066+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:21.948629+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76840744e0 msgr2=0x7f76840769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76840744e0 0x7f76840769f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7690006fd0 tx=0x7f7690008040 comp rx=0 tx=0).stop 2024-04-25T15:28:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76980d0de0 msgr2=0x7f76980ce7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76980d0de0 0x7f76980ce7d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f768800c700 tx=0x7f768800cbc0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 shutdown_connections 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76840744e0 0x7f76840769f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76980d0de0 0x7f76980ce7d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76980b77f0 0x7f7698152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 --2- 172.21.15.67:0/3612892124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f769806c5c0 0x7f76981f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 >> 172.21.15.67:0/3612892124 conn(0x7f76980d5d70 msgr2=0x7f76980bb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 shutdown_connections 2024-04-25T15:28:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.624+0000 7f769f372640 1 -- 172.21.15.67:0/3612892124 wait complete. 2024-04-25T15:28:43.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:28:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.749+0000 7f3e902a8640 1 Processor -- start 2024-04-25T15:28:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.749+0000 7f3e902a8640 1 -- start start 2024-04-25T15:28:43.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.750+0000 7f3e902a8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3e880b4830 0x7f3e8806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.750+0000 7f3e902a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b2dc0 0x7f3e880b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e902a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8806d870 con 0x7f3e880d4570 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e902a8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3e880b4c40 con 0x7f3e880b4830 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e902a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e8806e460 con 0x7f3e880b2dc0 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8e845640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b2dc0 0x7f3e880b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8d843640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3e880b4830 0x7f3e8806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8e845640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b2dc0 0x7f3e880b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:32816/0 (socket says 172.21.15.67:32816) 2024-04-25T15:28:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8e845640 1 -- 172.21.15.67:0/3881154310 learned_addr learned my addr 172.21.15.67:0/3881154310 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8e845640 1 -- 172.21.15.67:0/3881154310 >> v2:172.21.15.176:3300/0 conn(0x7f3e880b4830 msgr2=0x7f3e8806e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8e845640 1 --2- 172.21.15.67:0/3881154310 >> v2:172.21.15.176:3300/0 conn(0x7f3e880b4830 0x7f3e8806e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8e845640 1 -- 172.21.15.67:0/3881154310 >> v1:172.21.15.176:6789/0 conn(0x7f3e880d4570 legacy=0x7f3e880d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8e845640 1 -- 172.21.15.67:0/3881154310 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8806db10 con 0x7f3e880b2dc0 2024-04-25T15:28:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.751+0000 7f3e8d843640 1 --2- 172.21.15.67:0/3881154310 >> v2:172.21.15.176:3300/0 conn(0x7f3e880b4830 0x7f3e8806e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.752+0000 7f3e8e845640 1 --2- 172.21.15.67:0/3881154310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b2dc0 0x7f3e880b31d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e8806e5e0 tx=0x7f3e8002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e4861518ac3db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.752+0000 7f3e8d042640 1 -- 172.21.15.67:0/3881154310 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80036070 con 0x7f3e880b2dc0 2024-04-25T15:28:43.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.752+0000 7f3e8d042640 1 -- 172.21.15.67:0/3881154310 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e80031600 con 0x7f3e880b2dc0 2024-04-25T15:28:43.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.752+0000 7f3e8d042640 1 -- 172.21.15.67:0/3881154310 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e800319d0 con 0x7f3e880b2dc0 2024-04-25T15:28:43.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 -- 172.21.15.67:0/3881154310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b2dc0 msgr2=0x7f3e880b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3881154310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b2dc0 0x7f3e880b31d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e8806e5e0 tx=0x7f3e8002a680 comp rx=0 tx=0).stop 2024-04-25T15:28:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 -- 172.21.15.67:0/3881154310 shutdown_connections 2024-04-25T15:28:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3881154310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b2dc0 0x7f3e880b31d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3881154310 >> v2:172.21.15.176:3300/0 conn(0x7f3e880b4830 0x7f3e8806e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 -- 172.21.15.67:0/3881154310 >> 172.21.15.67:0/3881154310 conn(0x7f3e880d3a50 msgr2=0x7f3e880d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 -- 172.21.15.67:0/3881154310 shutdown_connections 2024-04-25T15:28:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.753+0000 7f3e902a8640 1 -- 172.21.15.67:0/3881154310 wait complete. 2024-04-25T15:28:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.755+0000 7f3e902a8640 1 Processor -- start 2024-04-25T15:28:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.755+0000 7f3e902a8640 1 -- start start 2024-04-25T15:28:43.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e902a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e880b2dc0 0x7f3e880c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e902a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b4830 0x7f3e8809fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e902a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e880d4570 0x7f3e88159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e902a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e880d0fb0 con 0x7f3e880d4570 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e902a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3e880d0bc0 con 0x7f3e880b2dc0 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e902a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e880b3710 con 0x7f3e880b4830 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e8e845640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e880d4570 0x7f3e88159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e8e845640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e880d4570 0x7f3e88159d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58048/0 (socket says 172.21.15.67:58048) 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.756+0000 7f3e8e845640 1 -- 172.21.15.67:0/3879443632 learned_addr learned my addr 172.21.15.67:0/3879443632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8e044640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e880b2dc0 0x7f3e880c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8d843640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b4830 0x7f3e8809fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8e845640 1 -- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b4830 msgr2=0x7f3e8809fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8e845640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b4830 0x7f3e8809fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8e845640 1 -- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e880b2dc0 msgr2=0x7f3e880c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8e845640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e880b2dc0 0x7f3e880c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8e845640 1 -- 172.21.15.67:0/3879443632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8815a2d0 con 0x7f3e880d4570 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e8e845640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e880d4570 0x7f3e88159d90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009f40 tx=0x7f3e80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e7effd640 1 -- 172.21.15.67:0/3879443632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80036050 con 0x7f3e880d4570 2024-04-25T15:28:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e7effd640 1 -- 172.21.15.67:0/3879443632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e80042070 con 0x7f3e880d4570 2024-04-25T15:28:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e880a05c0 con 0x7f3e880d4570 2024-04-25T15:28:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e7effd640 1 -- 172.21.15.67:0/3879443632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80009510 con 0x7f3e880d4570 2024-04-25T15:28:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.757+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e880a0b80 con 0x7f3e880d4570 2024-04-25T15:28:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.758+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e88050740 con 0x7f3e880d4570 2024-04-25T15:28:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.759+0000 7f3e7effd640 1 -- 172.21.15.67:0/3879443632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3e80007bf0 con 0x7f3e880d4570 2024-04-25T15:28:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.760+0000 7f3e7effd640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3e6c0744b0 0x7f3e6c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.760+0000 7f3e7effd640 1 -- 172.21.15.67:0/3879443632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f3e800b6960 con 0x7f3e880d4570 2024-04-25T15:28:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.760+0000 7f3e8e044640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3e6c0744b0 0x7f3e6c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.760+0000 7f3e8e044640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3e6c0744b0 0x7f3e6c0769c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3e84001630 tx=0x7f3e84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:43.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.764+0000 7f3e7effd640 1 -- 172.21.15.67:0/3879443632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e80081260 con 0x7f3e880d4570 2024-04-25T15:28:43.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.986+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3e880d1220 con 0x7f3e880d4570 2024-04-25T15:28:43.988 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:28:43.987+0000 7f3e02b50640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:28:43.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.988+0000 7f3e7effd640 1 -- 172.21.15.67:0/3879443632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) ==== 161+0+0 (secure 0 0 0) 0x7f3e80080b60 con 0x7f3e880d4570 2024-04-25T15:28:43.989 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3e6c0744b0 msgr2=0x7f3e6c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3e6c0744b0 0x7f3e6c0769c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3e84001630 tx=0x7f3e84009040 comp rx=0 tx=0).stop 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e880d4570 msgr2=0x7f3e88159d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e880d4570 0x7f3e88159d90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009f40 tx=0x7f3e80004a40 comp rx=0 tx=0).stop 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 shutdown_connections 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3e6c0744b0 0x7f3e6c0769c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e880d4570 0x7f3e88159d90 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e880b4830 0x7f3e8809fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 --2- 172.21.15.67:0/3879443632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e880b2dc0 0x7f3e880c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 >> 172.21.15.67:0/3879443632 conn(0x7f3e880d3a50 msgr2=0x7f3e880b88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 shutdown_connections 2024-04-25T15:28:43.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:43.991+0000 7f3e902a8640 1 -- 172.21.15.67:0/3879443632 wait complete. 2024-04-25T15:28:43.994 INFO:tasks.ceph.mds.d.smithi176.stderr: -3499> 2024-04-25T15:28:25.611+0000 7f4ef1a9f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:28:44.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:28:44.047 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:28:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.117+0000 7efc82996640 1 Processor -- start 2024-04-25T15:28:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.117+0000 7efc82996640 1 -- start start 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc82996640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc7c06d5d0 0x7efc7c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc82996640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc7c06df20 con 0x7efc7c0b56e0 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc82996640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc7c0709a0 con 0x7efc7c143660 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc82996640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efc7c070c70 con 0x7efc7c06d5d0 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc7b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc7c06d5d0 0x7efc7c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc80f33640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efc7c0b56e0 0x7efc7c0b5af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37182/0 (socket says 172.21.15.67:37182) 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc80f33640 1 -- 172.21.15.67:0/3901360541 learned_addr learned my addr 172.21.15.67:0/3901360541 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc7b7fe640 1 -- 172.21.15.67:0/3901360541 >> v1:172.21.15.176:6789/0 conn(0x7efc7c143660 legacy=0x7efc7c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc7b7fe640 1 -- 172.21.15.67:0/3901360541 >> v1:172.21.15.67:6789/0 conn(0x7efc7c0b56e0 legacy=0x7efc7c0b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc7b7fe640 1 -- 172.21.15.67:0/3901360541 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7c0b4920 con 0x7efc7c06d5d0 2024-04-25T15:28:44.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.119+0000 7efc7affd640 1 -- 172.21.15.67:0/3901360541 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280023991 0 0) 0x7efc74001490 con 0x7efc7c0b56e0 2024-04-25T15:28:44.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.120+0000 7efc7b7fe640 1 --2- 172.21.15.67:0/3901360541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c06d5d0 0x7efc7c06d9e0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0097a0 tx=0x7efc6c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee04772796476e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:44.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.120+0000 7efc7affd640 1 -- 172.21.15.67:0/3901360541 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c037070 con 0x7efc7c06d5d0 2024-04-25T15:28:44.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.120+0000 7efc7affd640 1 -- 172.21.15.67:0/3901360541 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc6c032440 con 0x7efc7c06d5d0 2024-04-25T15:28:44.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.120+0000 7efc7affd640 1 -- 172.21.15.67:0/3901360541 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c032810 con 0x7efc7c06d5d0 2024-04-25T15:28:44.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.121+0000 7efc82996640 1 -- 172.21.15.67:0/3901360541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c06d5d0 msgr2=0x7efc7c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:44.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.121+0000 7efc82996640 1 --2- 172.21.15.67:0/3901360541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c06d5d0 0x7efc7c06d9e0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efc6c0097a0 tx=0x7efc6c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.121+0000 7efc82996640 1 -- 172.21.15.67:0/3901360541 shutdown_connections 2024-04-25T15:28:44.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.121+0000 7efc82996640 1 --2- 172.21.15.67:0/3901360541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c06d5d0 0x7efc7c06d9e0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.121+0000 7efc82996640 1 -- 172.21.15.67:0/3901360541 >> 172.21.15.67:0/3901360541 conn(0x7efc7c073120 msgr2=0x7efc7c073530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:44.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.121+0000 7efc82996640 1 -- 172.21.15.67:0/3901360541 shutdown_connections 2024-04-25T15:28:44.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.121+0000 7efc82996640 1 -- 172.21.15.67:0/3901360541 wait complete. 2024-04-25T15:28:44.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.122+0000 7efc82996640 1 Processor -- start 2024-04-25T15:28:44.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.123+0000 7efc82996640 1 -- start start 2024-04-25T15:28:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc82996640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc7c06d5d0 0x7efc7c1530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc82996640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc7c0b56e0 0x7efc7c1535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc82996640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c143660 0x7efc7c1556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc82996640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c0b51c0 con 0x7efc7c143660 2024-04-25T15:28:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc82996640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efc7c0b4ee0 con 0x7efc7c0b56e0 2024-04-25T15:28:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc82996640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efc7c0ce0e0 con 0x7efc7c06d5d0 2024-04-25T15:28:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c143660 0x7efc7c1556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c143660 0x7efc7c1556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58060/0 (socket says 172.21.15.67:58060) 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 -- 172.21.15.67:0/1055722267 learned_addr learned my addr 172.21.15.67:0/1055722267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc7b7fe640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc7c0b56e0 0x7efc7c1535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc7bfff640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc7c06d5d0 0x7efc7c1530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 -- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc7c06d5d0 msgr2=0x7efc7c1530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc7c06d5d0 0x7efc7c1530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 -- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc7c0b56e0 msgr2=0x7efc7c1535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc7c0b56e0 0x7efc7c1535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.124+0000 7efc80f33640 1 -- 172.21.15.67:0/1055722267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7c1f4c20 con 0x7efc7c143660 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.125+0000 7efc80f33640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c143660 0x7efc7c1556a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7efc7400c730 tx=0x7efc7400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.125+0000 7efc78ff9640 1 -- 172.21.15.67:0/1055722267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc74018070 con 0x7efc7c143660 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.125+0000 7efc78ff9640 1 -- 172.21.15.67:0/1055722267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc7400ce40 con 0x7efc7c143660 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.125+0000 7efc78ff9640 1 -- 172.21.15.67:0/1055722267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7400a800 con 0x7efc7c143660 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.125+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7c15a740 con 0x7efc7c143660 2024-04-25T15:28:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.125+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc7c159c60 con 0x7efc7c143660 2024-04-25T15:28:44.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.126+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc7c0707c0 con 0x7efc7c143660 2024-04-25T15:28:44.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.127+0000 7efc78ff9640 1 -- 172.21.15.67:0/1055722267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efc7401b070 con 0x7efc7c143660 2024-04-25T15:28:44.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.127+0000 7efc78ff9640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc680744b0 0x7efc680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:44.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.127+0000 7efc78ff9640 1 -- 172.21.15.67:0/1055722267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5795+0+0 (secure 0 0 0) 0x7efc74094780 con 0x7efc7c143660 2024-04-25T15:28:44.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.127+0000 7efc7bfff640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc680744b0 0x7efc680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:44.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.128+0000 7efc7bfff640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc680744b0 0x7efc680769c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efc70000f60 tx=0x7efc70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:44.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.131+0000 7efc78ff9640 1 -- 172.21.15.67:0/1055722267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc7c0707c0 con 0x7efc7c143660 2024-04-25T15:28:44.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.357+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7efc7c050740 con 0x7efc7c143660 2024-04-25T15:28:44.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.992+0000 7efc78ff9640 1 -- 172.21.15.67:0/1055722267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) ==== 104+0+0 (secure 0 0 0) 0x7efc7c06df20 con 0x7efc7c143660 2024-04-25T15:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc680744b0 msgr2=0x7efc680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc680744b0 0x7efc680769c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efc70000f60 tx=0x7efc70009040 comp rx=0 tx=0).stop 2024-04-25T15:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c143660 msgr2=0x7efc7c1556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c143660 0x7efc7c1556a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7efc7400c730 tx=0x7efc7400cbf0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 shutdown_connections 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7efc680744b0 0x7efc680769c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc7c143660 0x7efc7c1556a0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc7c0b56e0 0x7efc7c1535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 --2- 172.21.15.67:0/1055722267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc7c06d5d0 0x7efc7c1530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.995+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 >> 172.21.15.67:0/1055722267 conn(0x7efc7c073120 msgr2=0x7efc7c0b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.996+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 shutdown_connections 2024-04-25T15:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:44.996+0000 7efc82996640 1 -- 172.21.15.67:0/1055722267 wait complete. 2024-04-25T15:28:45.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:28:45.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.132+0000 7f5256251640 1 Processor -- start 2024-04-25T15:28:45.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.132+0000 7f5256251640 1 -- start start 2024-04-25T15:28:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f5256251640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f52500b1990 0x7f525006d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f5256251640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f525006dba0 0x7f52500af570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f5256251640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525006dfb0 con 0x7f5250161a50 2024-04-25T15:28:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f5256251640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f52500b3250 con 0x7f525006dba0 2024-04-25T15:28:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f5256251640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f52500b3530 con 0x7f52500b1990 2024-04-25T15:28:45.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f525006dba0 0x7f52500af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f525006dba0 0x7f52500af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58076/0 (socket says 172.21.15.67:58076) 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524ffff640 1 -- 172.21.15.67:0/409127228 learned_addr learned my addr 172.21.15.67:0/409127228 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524effd640 1 --2- 172.21.15.67:0/409127228 >> v2:172.21.15.176:3300/0 conn(0x7f52500b1990 0x7f525006d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524ffff640 1 -- 172.21.15.67:0/409127228 >> v2:172.21.15.176:3300/0 conn(0x7f52500b1990 msgr2=0x7f525006d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524ffff640 1 --2- 172.21.15.67:0/409127228 >> v2:172.21.15.176:3300/0 conn(0x7f52500b1990 0x7f525006d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524ffff640 1 -- 172.21.15.67:0/409127228 >> v1:172.21.15.176:6789/0 conn(0x7f5250161a50 legacy=0x7f52500b1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524ffff640 1 -- 172.21.15.67:0/409127228 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52500b3810 con 0x7f525006dba0 2024-04-25T15:28:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.134+0000 7f524effd640 1 --2- 172.21.15.67:0/409127228 >> v2:172.21.15.176:3300/0 conn(0x7f52500b1990 0x7f525006d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:45.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.135+0000 7f524ffff640 1 --2- 172.21.15.67:0/409127228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52500af570 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5250073150 tx=0x7f524802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4496b584f2a6944 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:45.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.135+0000 7f524e7fc640 1 -- 172.21.15.67:0/409127228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248036070 con 0x7f525006dba0 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.135+0000 7f524e7fc640 1 -- 172.21.15.67:0/409127228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5248031600 con 0x7f525006dba0 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.135+0000 7f524e7fc640 1 -- 172.21.15.67:0/409127228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52480319d0 con 0x7f525006dba0 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 -- 172.21.15.67:0/409127228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 msgr2=0x7f52500af570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 --2- 172.21.15.67:0/409127228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52500af570 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5250073150 tx=0x7f524802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 -- 172.21.15.67:0/409127228 shutdown_connections 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 --2- 172.21.15.67:0/409127228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52500af570 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 --2- 172.21.15.67:0/409127228 >> v2:172.21.15.176:3300/0 conn(0x7f52500b1990 0x7f525006d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 -- 172.21.15.67:0/409127228 >> 172.21.15.67:0/409127228 conn(0x7f525009b7d0 msgr2=0x7f52500b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 -- 172.21.15.67:0/409127228 shutdown_connections 2024-04-25T15:28:45.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.136+0000 7f5256251640 1 -- 172.21.15.67:0/409127228 wait complete. 2024-04-25T15:28:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.138+0000 7f5256251640 1 Processor -- start 2024-04-25T15:28:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.138+0000 7f5256251640 1 -- start start 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f5256251640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52501483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f5256251640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5250161a50 0x7f52501588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f5256251640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5250146390 0x7f525015ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f5256251640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5250161e60 con 0x7f525006dba0 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f5256251640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f52500af050 con 0x7f5250161a50 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f5256251640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f52500b2e70 con 0x7f5250146390 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f524f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52501483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f524f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52501483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58078/0 (socket says 172.21.15.67:58078) 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f524f7fe640 1 -- 172.21.15.67:0/854846430 learned_addr learned my addr 172.21.15.67:0/854846430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.139+0000 7f524f7fe640 1 -- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5250146390 msgr2=0x7f525015ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f524f7fe640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5250146390 0x7f525015ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f524effd640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5250161a50 0x7f52501588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f524f7fe640 1 -- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5250161a50 msgr2=0x7f52501588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f524f7fe640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5250161a50 0x7f52501588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f524f7fe640 1 -- 172.21.15.67:0/854846430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5250148920 con 0x7f525006dba0 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f524f7fe640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52501483e0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f524000c470 tx=0x7f524000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f5237fff640 1 -- 172.21.15.67:0/854846430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52400052d0 con 0x7f525006dba0 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f5237fff640 1 -- 172.21.15.67:0/854846430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5240005490 con 0x7f525006dba0 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5250158de0 con 0x7f525006dba0 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f5237fff640 1 -- 172.21.15.67:0/854846430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5240007260 con 0x7f525006dba0 2024-04-25T15:28:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.140+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52501490e0 con 0x7f525006dba0 2024-04-25T15:28:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.141+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5250050740 con 0x7f525006dba0 2024-04-25T15:28:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.142+0000 7f5237fff640 1 -- 172.21.15.67:0/854846430 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5240007400 con 0x7f525006dba0 2024-04-25T15:28:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.142+0000 7f5237fff640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f523c0744b0 0x7f523c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.142+0000 7f5237fff640 1 -- 172.21.15.67:0/854846430 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5795+0+0 (secure 0 0 0) 0x7f5240094710 con 0x7f525006dba0 2024-04-25T15:28:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.143+0000 7f524effd640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f523c0744b0 0x7f523c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.143+0000 7f524effd640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f523c0744b0 0x7f523c0769c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5250076190 tx=0x7f5244008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.147+0000 7f5237fff640 1 -- 172.21.15.67:0/854846430 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5250148920 con 0x7f525006dba0 2024-04-25T15:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.333+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f525015e0c0 con 0x7f525006dba0 2024-04-25T15:28:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.994+0000 7f5237fff640 1 -- 172.21.15.67:0/854846430 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v82) ==== 181+0+0 (secure 0 0 0) 0x7f524005ee40 con 0x7f525006dba0 2024-04-25T15:28:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:28:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.997+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f523c0744b0 msgr2=0x7f523c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f523c0744b0 0x7f523c0769c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5250076190 tx=0x7f5244008040 comp rx=0 tx=0).stop 2024-04-25T15:28:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 msgr2=0x7f52501483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52501483e0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f524000c470 tx=0x7f524000c930 comp rx=0 tx=0).stop 2024-04-25T15:28:45.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 shutdown_connections 2024-04-25T15:28:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f523c0744b0 0x7f523c0769c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5250146390 0x7f525015ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5250161a50 0x7f52501588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 --2- 172.21.15.67:0/854846430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525006dba0 0x7f52501483e0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 >> 172.21.15.67:0/854846430 conn(0x7f525009b7d0 msgr2=0x7f52501600e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.998+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 shutdown_connections 2024-04-25T15:28:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:45.999+0000 7f5256251640 1 -- 172.21.15.67:0/854846430 wait complete. 2024-04-25T15:28:46.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:28:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.134+0000 7fbac5b6c640 1 Processor -- start 2024-04-25T15:28:46.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.134+0000 7fbac5b6c640 1 -- start start 2024-04-25T15:28:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbac5b6c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbac0164b00 0x7fbac00b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbac5b6c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbac5b6c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbac0160890 0x7fbac0160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbac5b6c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbac0164670 con 0x7fbac0160890 2024-04-25T15:28:46.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbac5b6c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbac01615f0 con 0x7fbac0164b00 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbac5b6c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbac0164020 con 0x7fbac015df80 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbac0160890 0x7fbac0160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbac0160890 0x7fbac0160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58086/0 (socket says 172.21.15.67:58086) 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 -- 172.21.15.67:0/2220626133 learned_addr learned my addr 172.21.15.67:0/2220626133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabeffd640 1 --2- 172.21.15.67:0/2220626133 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 -- 172.21.15.67:0/2220626133 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 msgr2=0x7fbac00b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 --2- 172.21.15.67:0/2220626133 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac00b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabf7fe640 1 --2- 172.21.15.67:0/2220626133 >> v2:172.21.15.176:3300/0 conn(0x7fbac0164b00 0x7fbac00b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 -- 172.21.15.67:0/2220626133 >> v2:172.21.15.176:3300/0 conn(0x7fbac0164b00 msgr2=0x7fbac00b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 --2- 172.21.15.67:0/2220626133 >> v2:172.21.15.176:3300/0 conn(0x7fbac0164b00 0x7fbac00b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.136+0000 7fbabffff640 1 -- 172.21.15.67:0/2220626133 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac0164300 con 0x7fbac0160890 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.137+0000 7fbabffff640 1 --2- 172.21.15.67:0/2220626133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0160ca0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fbac008e410 tx=0x7fbab802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f28983aa7c6ed6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.137+0000 7fbabe7fc640 1 -- 172.21.15.67:0/2220626133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab8036070 con 0x7fbac0160890 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.137+0000 7fbabe7fc640 1 -- 172.21.15.67:0/2220626133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbab8031600 con 0x7fbac0160890 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.137+0000 7fbabe7fc640 1 -- 172.21.15.67:0/2220626133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab80319d0 con 0x7fbac0160890 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 -- 172.21.15.67:0/2220626133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 msgr2=0x7fbac0160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/2220626133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0160ca0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fbac008e410 tx=0x7fbab802a6a0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 -- 172.21.15.67:0/2220626133 shutdown_connections 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/2220626133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0160ca0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/2220626133 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac00b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/2220626133 >> v2:172.21.15.176:3300/0 conn(0x7fbac0164b00 0x7fbac00b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 -- 172.21.15.67:0/2220626133 >> 172.21.15.67:0/2220626133 conn(0x7fbac00d2bc0 msgr2=0x7fbac00d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 -- 172.21.15.67:0/2220626133 shutdown_connections 2024-04-25T15:28:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.138+0000 7fbac5b6c640 1 -- 172.21.15.67:0/2220626133 wait complete. 2024-04-25T15:28:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.140+0000 7fbac5b6c640 1 Processor -- start 2024-04-25T15:28:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.140+0000 7fbac5b6c640 1 -- start start 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbac5b6c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac0156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbac5b6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbac5b6c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbac0164b00 0x7fbac0153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbac5b6c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbac00b4a00 con 0x7fbac0160890 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbac5b6c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbac00b4880 con 0x7fbac0164b00 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbac5b6c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbac00b4b80 con 0x7fbac015df80 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbabeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbabeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58098/0 (socket says 172.21.15.67:58098) 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.141+0000 7fbabeffd640 1 -- 172.21.15.67:0/882483616 learned_addr learned my addr 172.21.15.67:0/882483616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabf7fe640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac0156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabffff640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbac0164b00 0x7fbac0153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:46.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabeffd640 1 -- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 msgr2=0x7fbac0156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabeffd640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac0156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabeffd640 1 -- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbac0164b00 msgr2=0x7fbac0153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabeffd640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbac0164b00 0x7fbac0153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabeffd640 1 -- 172.21.15.67:0/882483616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac01f66f0 con 0x7fbac0160890 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbabeffd640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0153430 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c5a0 tx=0x7fbab400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbaa7fff640 1 -- 172.21.15.67:0/882483616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab4015070 con 0x7fbac0160890 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbaa7fff640 1 -- 172.21.15.67:0/882483616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbab4004220 con 0x7fbac0160890 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac01f6970 con 0x7fbac0160890 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbaa7fff640 1 -- 172.21.15.67:0/882483616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab4003610 con 0x7fbac0160890 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.142+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac01f6f30 con 0x7fbac0160890 2024-04-25T15:28:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.143+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac0050740 con 0x7fbac0160890 2024-04-25T15:28:46.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.144+0000 7fbaa7fff640 1 -- 172.21.15.67:0/882483616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbab40037b0 con 0x7fbac0160890 2024-04-25T15:28:46.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.144+0000 7fbaa7fff640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbaac0744b0 0x7fbaac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:46.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.144+0000 7fbaa7fff640 1 -- 172.21.15.67:0/882483616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5357+0+0 (secure 0 0 0) 0x7fbab4094e50 con 0x7fbac0160890 2024-04-25T15:28:46.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.145+0000 7fbabf7fe640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbaac0744b0 0x7fbaac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:46.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.145+0000 7fbabf7fe640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbaac0744b0 0x7fbaac0769c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbab0006fd0 tx=0x7fbab0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.149+0000 7fbaa7fff640 1 -- 172.21.15.67:0/882483616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbac01f66f0 con 0x7fbac0160890 2024-04-25T15:28:46.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.335+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbac00d4170 con 0x7fbac0160890 2024-04-25T15:28:47.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:46.999+0000 7fbaa7fff640 1 -- 172.21.15.67:0/882483616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v83) ==== 169+0+0 (secure 0 0 0) 0x7fbab405f740 con 0x7fbac0160890 2024-04-25T15:28:47.000 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:28:47.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbaac0744b0 msgr2=0x7fbaac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbaac0744b0 0x7fbaac0769c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbab0006fd0 tx=0x7fbab0008040 comp rx=0 tx=0).stop 2024-04-25T15:28:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 msgr2=0x7fbac0153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0153430 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c5a0 tx=0x7fbab400ca60 comp rx=0 tx=0).stop 2024-04-25T15:28:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 shutdown_connections 2024-04-25T15:28:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbaac0744b0 0x7fbaac0769c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbac0164b00 0x7fbac0153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbac0160890 0x7fbac0153430 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 --2- 172.21.15.67:0/882483616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbac015df80 0x7fbac0156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 >> 172.21.15.67:0/882483616 conn(0x7fbac00d2bc0 msgr2=0x7fbac015e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 shutdown_connections 2024-04-25T15:28:47.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.007+0000 7fbac5b6c640 1 -- 172.21.15.67:0/882483616 wait complete. 2024-04-25T15:28:47.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:28:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.148+0000 7f9627ae0640 1 Processor -- start 2024-04-25T15:28:47.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.148+0000 7f9627ae0640 1 -- start start 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.149+0000 7f9627ae0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f96200d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.149+0000 7f9627ae0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9620164020 0x7f9620164430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.149+0000 7f9627ae0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9620164970 con 0x7f96200d1e20 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.149+0000 7f9627ae0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f96200d23f0 con 0x7f9620164020 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.149+0000 7f9627ae0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96200d26d0 con 0x7f96200b2e60 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.149+0000 7f962587c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96200d1e20 0x7f962008d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37190/0 (socket says 172.21.15.67:37190) 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.149+0000 7f962607d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9620164020 0x7f9620164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.150+0000 7f962587c640 1 -- 172.21.15.67:0/2175727602 learned_addr learned my addr 172.21.15.67:0/2175727602 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.150+0000 7f962507b640 1 --2- 172.21.15.67:0/2175727602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f96200d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.150+0000 7f962607d640 1 -- 172.21.15.67:0/2175727602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 msgr2=0x7f96200d3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.150+0000 7f962607d640 1 --2- 172.21.15.67:0/2175727602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f96200d3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.150+0000 7f962607d640 1 -- 172.21.15.67:0/2175727602 >> v1:172.21.15.67:6789/0 conn(0x7f96200d1e20 legacy=0x7f962008d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.150+0000 7f962607d640 1 -- 172.21.15.67:0/2175727602 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96200d29b0 con 0x7f9620164020 2024-04-25T15:28:47.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.150+0000 7f962487a640 1 -- 172.21.15.67:0/2175727602 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077174323 0 0) 0x7f961c001110 con 0x7f96200d1e20 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f962607d640 1 --2- 172.21.15.67:0/2175727602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9620164020 0x7f9620164430 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f9618009d90 tx=0x7f961802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacc22a4d4bff407 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f962487a640 1 -- 172.21.15.67:0/2175727602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618007740 con 0x7f9620164020 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f962487a640 1 -- 172.21.15.67:0/2175727602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96180078e0 con 0x7f9620164020 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f962487a640 1 -- 172.21.15.67:0/2175727602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618007cb0 con 0x7f9620164020 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f9627ae0640 1 -- 172.21.15.67:0/2175727602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9620164020 msgr2=0x7f9620164430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f9627ae0640 1 --2- 172.21.15.67:0/2175727602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9620164020 0x7f9620164430 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f9618009d90 tx=0x7f961802a6f0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f9627ae0640 1 -- 172.21.15.67:0/2175727602 shutdown_connections 2024-04-25T15:28:47.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f9627ae0640 1 --2- 172.21.15.67:0/2175727602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9620164020 0x7f9620164430 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f9627ae0640 1 --2- 172.21.15.67:0/2175727602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f96200d3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.151+0000 7f9627ae0640 1 -- 172.21.15.67:0/2175727602 >> 172.21.15.67:0/2175727602 conn(0x7f96200d0530 msgr2=0x7f96200d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.152+0000 7f9627ae0640 1 -- 172.21.15.67:0/2175727602 shutdown_connections 2024-04-25T15:28:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.152+0000 7f9627ae0640 1 -- 172.21.15.67:0/2175727602 wait complete. 2024-04-25T15:28:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.153+0000 7f9627ae0640 1 Processor -- start 2024-04-25T15:28:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.153+0000 7f9627ae0640 1 -- start start 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f9627ae0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f962015e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f9627ae0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96200d1e20 0x7f962014ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f9627ae0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9620164020 0x7f96200b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f9627ae0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96200ce070 con 0x7f96200d1e20 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f9627ae0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f96200d2f70 con 0x7f9620164020 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f9627ae0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96200ce1f0 con 0x7f96200b2e60 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96200d1e20 0x7f962014ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96200d1e20 0x7f962014ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58120/0 (socket says 172.21.15.67:58120) 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 -- 172.21.15.67:0/299278718 learned_addr learned my addr 172.21.15.67:0/299278718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:47.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962587c640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f962015e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962607d640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9620164020 0x7f96200b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 -- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 msgr2=0x7f962015e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f962015e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 -- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9620164020 msgr2=0x7f96200b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9620164020 0x7f96200b7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 -- 172.21.15.67:0/299278718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962014e410 con 0x7f96200d1e20 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.155+0000 7f962507b640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96200d1e20 0x7f962014ded0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f961000e8a0 tx=0x7f961000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.156+0000 7f96167fc640 1 -- 172.21.15.67:0/299278718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610017070 con 0x7f96200d1e20 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.156+0000 7f96167fc640 1 -- 172.21.15.67:0/299278718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961000c940 con 0x7f96200d1e20 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.156+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96200b49f0 con 0x7f96200d1e20 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.156+0000 7f96167fc640 1 -- 172.21.15.67:0/299278718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961000cd30 con 0x7f96200d1e20 2024-04-25T15:28:47.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.156+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96200b4fb0 con 0x7f96200d1e20 2024-04-25T15:28:47.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.157+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9620050740 con 0x7f96200d1e20 2024-04-25T15:28:47.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.157+0000 7f96167fc640 1 -- 172.21.15.67:0/299278718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f961001a070 con 0x7f96200d1e20 2024-04-25T15:28:47.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.158+0000 7f96167fc640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96040744b0 0x7f96040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.158+0000 7f96167fc640 1 -- 172.21.15.67:0/299278718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7f9610094cb0 con 0x7f96200d1e20 2024-04-25T15:28:47.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.158+0000 7f962587c640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96040744b0 0x7f96040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.159+0000 7f962587c640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96040744b0 0x7f96040769c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f961c007fd0 tx=0x7f961c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:47.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.163+0000 7f96167fc640 1 -- 172.21.15.67:0/299278718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f961005f180 con 0x7f96200d1e20 2024-04-25T15:28:47.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.346+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f9620071000 con 0x7f96200d1e20 2024-04-25T15:28:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.349+0000 7f96167fc640 1 -- 172.21.15.67:0/299278718 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f9610063480 con 0x7f96200d1e20 2024-04-25T15:28:47.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96040744b0 msgr2=0x7f96040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96040744b0 0x7f96040769c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f961c007fd0 tx=0x7f961c007de0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96200d1e20 msgr2=0x7f962014ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96200d1e20 0x7f962014ded0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f961000e8a0 tx=0x7f961000ed60 comp rx=0 tx=0).stop 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 shutdown_connections 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f96040744b0 0x7f96040769c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9620164020 0x7f96200b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96200d1e20 0x7f962014ded0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 --2- 172.21.15.67:0/299278718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96200b2e60 0x7f962015e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 >> 172.21.15.67:0/299278718 conn(0x7f96200d0530 msgr2=0x7f9620159050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 shutdown_connections 2024-04-25T15:28:47.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.351+0000 7f9627ae0640 1 -- 172.21.15.67:0/299278718 wait complete. 2024-04-25T15:28:47.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr' 2024-04-25T15:28:47.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.474+0000 7fb30f500640 1 Processor -- start 2024-04-25T15:28:47.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.474+0000 7fb30f500640 1 -- start start 2024-04-25T15:28:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30f500640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb30806d8e0 0x7fb3080d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30f500640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3080d12c0 con 0x7fb30806dcf0 2024-04-25T15:28:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30f500640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3080b3c10 con 0x7fb3080d2d70 2024-04-25T15:28:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30f500640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb3080b9520 con 0x7fb30806d8e0 2024-04-25T15:28:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30d29c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb30806d8e0 0x7fb3080d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30ca9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb30806dcf0 0x7fb3080d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37200/0 (socket says 172.21.15.67:37200) 2024-04-25T15:28:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30ca9b640 1 -- 172.21.15.67:0/1584525790 learned_addr learned my addr 172.21.15.67:0/1584525790 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:47.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30d29c640 1 -- 172.21.15.67:0/1584525790 >> v1:172.21.15.176:6789/0 conn(0x7fb3080d2d70 legacy=0x7fb3080d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:47.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30d29c640 1 -- 172.21.15.67:0/1584525790 >> v1:172.21.15.67:6789/0 conn(0x7fb30806dcf0 legacy=0x7fb3080d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb30d29c640 1 -- 172.21.15.67:0/1584525790 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3080b3ef0 con 0x7fb30806d8e0 2024-04-25T15:28:47.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.476+0000 7fb307fff640 1 -- 172.21.15.67:0/1584525790 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951997810 0 0) 0x7fb3080d12c0 con 0x7fb30806dcf0 2024-04-25T15:28:47.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.477+0000 7fb30d29c640 1 --2- 172.21.15.67:0/1584525790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb3080d3a20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fb3000097a0 tx=0x7fb30002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e30baf629c49c90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:47.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.477+0000 7fb307fff640 1 -- 172.21.15.67:0/1584525790 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300037070 con 0x7fb30806d8e0 2024-04-25T15:28:47.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.477+0000 7fb307fff640 1 -- 172.21.15.67:0/1584525790 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30002ae50 con 0x7fb30806d8e0 2024-04-25T15:28:47.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.477+0000 7fb307fff640 1 -- 172.21.15.67:0/1584525790 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300032660 con 0x7fb30806d8e0 2024-04-25T15:28:47.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.478+0000 7fb30f500640 1 -- 172.21.15.67:0/1584525790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 msgr2=0x7fb3080d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.478+0000 7fb30f500640 1 --2- 172.21.15.67:0/1584525790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb3080d3a20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fb3000097a0 tx=0x7fb30002a140 comp rx=0 tx=0).stop 2024-04-25T15:28:47.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.478+0000 7fb30f500640 1 -- 172.21.15.67:0/1584525790 shutdown_connections 2024-04-25T15:28:47.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.478+0000 7fb30f500640 1 --2- 172.21.15.67:0/1584525790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb3080d3a20 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.478+0000 7fb30f500640 1 -- 172.21.15.67:0/1584525790 >> 172.21.15.67:0/1584525790 conn(0x7fb30808d110 msgr2=0x7fb30808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:47.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.478+0000 7fb30f500640 1 -- 172.21.15.67:0/1584525790 shutdown_connections 2024-04-25T15:28:47.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.478+0000 7fb30f500640 1 -- 172.21.15.67:0/1584525790 wait complete. 2024-04-25T15:28:47.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.480+0000 7fb30f500640 1 Processor -- start 2024-04-25T15:28:47.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.480+0000 7fb30f500640 1 -- start start 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30f500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb30809e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30f500640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb30806dcf0 0x7fb308159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30f500640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3080d2d70 0x7fb30815f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30f500640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb3080b44b0 con 0x7fb30806d8e0 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30f500640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb3080b4820 con 0x7fb3080d2d70 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30f500640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb3080b38e0 con 0x7fb30806dcf0 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb30809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb30809e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58140/0 (socket says 172.21.15.67:58140) 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 -- 172.21.15.67:0/3892257407 learned_addr learned my addr 172.21.15.67:0/3892257407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:47.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30ca9b640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb30806dcf0 0x7fb308159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30da9d640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3080d2d70 0x7fb30815f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 -- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb30806dcf0 msgr2=0x7fb308159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb30806dcf0 0x7fb308159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 -- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3080d2d70 msgr2=0x7fb30815f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3080d2d70 0x7fb30815f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.481+0000 7fb30d29c640 1 -- 172.21.15.67:0/3892257407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30815fc00 con 0x7fb30806d8e0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.482+0000 7fb30d29c640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb30809e440 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb30002a730 tx=0x7fb3000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.482+0000 7fb305ffb640 1 -- 172.21.15.67:0/3892257407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300042070 con 0x7fb30806d8e0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.482+0000 7fb305ffb640 1 -- 172.21.15.67:0/3892257407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30003e670 con 0x7fb30806d8e0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.482+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30815fee0 con 0x7fb30806d8e0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.482+0000 7fb305ffb640 1 -- 172.21.15.67:0/3892257407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb300037040 con 0x7fb30806d8e0 2024-04-25T15:28:47.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.482+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30815d670 con 0x7fb30806d8e0 2024-04-25T15:28:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.483+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb30809ea30 con 0x7fb30806d8e0 2024-04-25T15:28:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.484+0000 7fb305ffb640 1 -- 172.21.15.67:0/3892257407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb300045070 con 0x7fb30806d8e0 2024-04-25T15:28:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.484+0000 7fb305ffb640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2f40744e0 0x7fb2f40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.484+0000 7fb305ffb640 1 -- 172.21.15.67:0/3892257407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7fb3000b5410 con 0x7fb30806d8e0 2024-04-25T15:28:47.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.485+0000 7fb30ca9b640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2f40744e0 0x7fb2f40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:47.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.485+0000 7fb30ca9b640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2f40744e0 0x7fb2f40769f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb308161410 tx=0x7fb2fc007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:47.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.489+0000 7fb305ffb640 1 -- 172.21.15.67:0/3892257407 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb30809ea30 con 0x7fb30806d8e0 2024-04-25T15:28:47.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:47.747+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]} v 0) -- 0x7fb308050740 con 0x7fb30806d8e0 2024-04-25T15:28:48.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.000+0000 7fb305ffb640 1 -- 172.21.15.67:0/3892257407 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]}]=0 v212) ==== 140+0+0 (secure 0 0 0) 0x7fb30007f850 con 0x7fb30806d8e0 2024-04-25T15:28:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2f40744e0 msgr2=0x7fb2f40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2f40744e0 0x7fb2f40769f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb308161410 tx=0x7fb2fc007de0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 msgr2=0x7fb30809e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb30809e440 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb30002a730 tx=0x7fb3000046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 shutdown_connections 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb2f40744e0 0x7fb2f40769f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3080d2d70 0x7fb30815f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb30806dcf0 0x7fb308159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 --2- 172.21.15.67:0/3892257407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb30806d8e0 0x7fb30809e440 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 >> 172.21.15.67:0/3892257407 conn(0x7fb30808d110 msgr2=0x7fb3080b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 shutdown_connections 2024-04-25T15:28:48.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.002+0000 7fb30f500640 1 -- 172.21.15.67:0/3892257407 wait complete. 2024-04-25T15:28:48.015 INFO:tasks.cephfs_test_runner:That quota changes made from one client are visible to another ... ok 2024-04-25T15:28:48.016 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_write (tasks.cephfs.test_quota.TestQuota) That when a client modifies the quota on a directory used 2024-04-25T15:28:48.016 INFO:tasks.cephfs_test_runner:test_remote_update_write (tasks.cephfs.test_quota.TestQuota) 2024-04-25T15:28:48.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:48.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.126+0000 7fa61d605640 1 Processor -- start 2024-04-25T15:28:48.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.126+0000 7fa61d605640 1 -- start start 2024-04-25T15:28:48.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.127+0000 7fa61d605640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa61815abd0 0x7fa61806c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa61d605640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa61d605640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61806cf10 con 0x7fa61806cb00 2024-04-25T15:28:48.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa61d605640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa618161380 con 0x7fa61815abd0 2024-04-25T15:28:48.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa61d605640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa618164b60 con 0x7fa618164750 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa616ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa61815abd0 0x7fa61806c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa6167fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa61806cb00 0x7fa618164040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54526/0 (socket says 172.21.15.67:54526) 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa6167fc640 1 -- 172.21.15.67:0/3126960680 learned_addr learned my addr 172.21.15.67:0/3126960680 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa6177fe640 1 --2- 172.21.15.67:0/3126960680 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa616ffd640 1 -- 172.21.15.67:0/3126960680 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 msgr2=0x7fa6180d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa616ffd640 1 --2- 172.21.15.67:0/3126960680 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180d4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa616ffd640 1 -- 172.21.15.67:0/3126960680 >> v1:172.21.15.67:6789/0 conn(0x7fa61806cb00 legacy=0x7fa618164040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa616ffd640 1 -- 172.21.15.67:0/3126960680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa618164db0 con 0x7fa61815abd0 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa615ffb640 1 -- 172.21.15.67:0/3126960680 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531957476 0 0) 0x7fa60c001110 con 0x7fa61806cb00 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.128+0000 7fa6177fe640 1 --2- 172.21.15.67:0/3126960680 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.129+0000 7fa616ffd640 1 --2- 172.21.15.67:0/3126960680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa61806c5c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097a0 tx=0x7fa60802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01ffb8426f6f830 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.129+0000 7fa615ffb640 1 -- 172.21.15.67:0/3126960680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608037070 con 0x7fa61815abd0 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.129+0000 7fa615ffb640 1 -- 172.21.15.67:0/3126960680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa608032440 con 0x7fa61815abd0 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.129+0000 7fa615ffb640 1 -- 172.21.15.67:0/3126960680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608032810 con 0x7fa61815abd0 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 -- 172.21.15.67:0/3126960680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 msgr2=0x7fa61806c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 --2- 172.21.15.67:0/3126960680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa61806c5c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097a0 tx=0x7fa60802a180 comp rx=0 tx=0).stop 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 -- 172.21.15.67:0/3126960680 shutdown_connections 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 --2- 172.21.15.67:0/3126960680 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 --2- 172.21.15.67:0/3126960680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa61806c5c0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 -- 172.21.15.67:0/3126960680 >> 172.21.15.67:0/3126960680 conn(0x7fa6180d5df0 msgr2=0x7fa6180d6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:48.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 -- 172.21.15.67:0/3126960680 shutdown_connections 2024-04-25T15:28:48.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.130+0000 7fa61d605640 1 -- 172.21.15.67:0/3126960680 wait complete. 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.132+0000 7fa61d605640 1 Processor -- start 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.132+0000 7fa61d605640 1 -- start start 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa61d605640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa61806cb00 0x7fa61814ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa61d605640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa6180b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa61d605640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa61d605640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa6181609e0 con 0x7fa61815abd0 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa61d605640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa618161010 con 0x7fa61806cb00 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa61d605640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa618161810 con 0x7fa618164750 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa6180b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa6180b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41954/0 (socket says 172.21.15.67:41954) 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 -- 172.21.15.67:0/3942316620 learned_addr learned my addr 172.21.15.67:0/3942316620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6177fe640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa616ffd640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa61806cb00 0x7fa61814ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 -- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 msgr2=0x7fa6180b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 -- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa61806cb00 msgr2=0x7fa61814ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa61806cb00 0x7fa61814ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6167fc640 1 -- 172.21.15.67:0/3942316620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6180b8140 con 0x7fa61815abd0 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa6177fe640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.133+0000 7fa616ffd640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa61806cb00 0x7fa61814ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.134+0000 7fa6167fc640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa6180b6a70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c460 tx=0x7fa60c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.134+0000 7fa5ff7fe640 1 -- 172.21.15.67:0/3942316620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c018070 con 0x7fa61815abd0 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.134+0000 7fa5ff7fe640 1 -- 172.21.15.67:0/3942316620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa60c007500 con 0x7fa61815abd0 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.134+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6180b7530 con 0x7fa61815abd0 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.134+0000 7fa5ff7fe640 1 -- 172.21.15.67:0/3942316620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c004a20 con 0x7fa61815abd0 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.134+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6180b7af0 con 0x7fa61815abd0 2024-04-25T15:28:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.135+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6180b72a0 con 0x7fa61815abd0 2024-04-25T15:28:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.136+0000 7fa5ff7fe640 1 -- 172.21.15.67:0/3942316620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa60c01b070 con 0x7fa61815abd0 2024-04-25T15:28:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.136+0000 7fa5ff7fe640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6040744e0 0x7fa6040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.136+0000 7fa5ff7fe640 1 -- 172.21.15.67:0/3942316620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7fa60c0940c0 con 0x7fa61815abd0 2024-04-25T15:28:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.136+0000 7fa616ffd640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6040744e0 0x7fa6040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.137+0000 7fa616ffd640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6040744e0 0x7fa6040769f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa608034000 tx=0x7fa60802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.141+0000 7fa5ff7fe640 1 -- 172.21.15.67:0/3942316620 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa61806cf10 con 0x7fa61815abd0 2024-04-25T15:28:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.327+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa618050740 con 0x7fa61815abd0 2024-04-25T15:28:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.328+0000 7fa5ff7fe640 1 -- 172.21.15.67:0/3942316620 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+12987 (secure 0 0 0) 0x7fa60c05ec70 con 0x7fa61815abd0 2024-04-25T15:28:48.329 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:48.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":83,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:46.995463+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.176:6837/2484404027":"2024-04-26T15:28:43.988097+0000","172.21.15.176:6835/2484404027":"2024-04-26T15:28:43.988097+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6040744e0 msgr2=0x7fa6040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6040744e0 0x7fa6040769f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa608034000 tx=0x7fa60802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 msgr2=0x7fa6180b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa6180b6a70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00c460 tx=0x7fa60c00c920 comp rx=0 tx=0).stop 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 shutdown_connections 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6040744e0 0x7fa6040769f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa618164750 0x7fa6180b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61815abd0 0x7fa6180b6a70 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 --2- 172.21.15.67:0/3942316620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa61806cb00 0x7fa61814ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 >> 172.21.15.67:0/3942316620 conn(0x7fa6180d5df0 msgr2=0x7fa6181635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 shutdown_connections 2024-04-25T15:28:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.330+0000 7fa61d605640 1 -- 172.21.15.67:0/3942316620 wait complete. 2024-04-25T15:28:48.344 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:48.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.457+0000 7f26400af640 1 Processor -- start 2024-04-25T15:28:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.457+0000 7f26400af640 1 -- start start 2024-04-25T15:28:48.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.458+0000 7f26400af640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f263806ea60 0x7f26380b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f26400af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26381649f0 0x7f26380b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f26400af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f263806d770 con 0x7f26380ce040 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f26400af640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f263806da50 con 0x7f263806ea60 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f26400af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f26380d2200 con 0x7f26381649f0 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263e64c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26381649f0 0x7f26380b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263e64c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26381649f0 0x7f26380b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:33552/0 (socket says 172.21.15.67:33552) 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263d64a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f263806ea60 0x7f26380b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263e64c640 1 -- 172.21.15.67:0/1913815713 learned_addr learned my addr 172.21.15.67:0/1913815713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263d64a640 1 -- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26381649f0 msgr2=0x7f26380b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263d64a640 1 --2- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26381649f0 0x7f26380b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263d64a640 1 -- 172.21.15.67:0/1913815713 >> v1:172.21.15.176:6789/0 conn(0x7f26380ce040 legacy=0x7f263806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263d64a640 1 -- 172.21.15.67:0/1913815713 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26380d24e0 con 0x7f263806ea60 2024-04-25T15:28:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.459+0000 7f263e64c640 1 --2- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26381649f0 0x7f26380b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.460+0000 7f263d64a640 1 --2- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f263806ea60 0x7f26380b2d60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f26280097a0 tx=0x7f262802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e332062ac5a142f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.460+0000 7f263ce49640 1 -- 172.21.15.67:0/1913815713 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628037070 con 0x7f263806ea60 2024-04-25T15:28:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.460+0000 7f263ce49640 1 -- 172.21.15.67:0/1913815713 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262802ae50 con 0x7f263806ea60 2024-04-25T15:28:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.460+0000 7f263ce49640 1 -- 172.21.15.67:0/1913815713 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628032660 con 0x7f263806ea60 2024-04-25T15:28:48.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 -- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f263806ea60 msgr2=0x7f26380b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 --2- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f263806ea60 0x7f26380b2d60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f26280097a0 tx=0x7f262802a100 comp rx=0 tx=0).stop 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 -- 172.21.15.67:0/1913815713 shutdown_connections 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 --2- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26381649f0 0x7f26380b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 --2- 172.21.15.67:0/1913815713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f263806ea60 0x7f26380b2d60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 -- 172.21.15.67:0/1913815713 >> 172.21.15.67:0/1913815713 conn(0x7f26380d3a80 msgr2=0x7f26380d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 -- 172.21.15.67:0/1913815713 shutdown_connections 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.461+0000 7f26400af640 1 -- 172.21.15.67:0/1913815713 wait complete. 2024-04-25T15:28:48.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.463+0000 7f26400af640 1 Processor -- start 2024-04-25T15:28:48.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.463+0000 7f26400af640 1 -- start start 2024-04-25T15:28:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f26400af640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26380ce040 0x7f26380b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f26400af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26381649f0 0x7f263815e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f26400af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26380b72d0 0x7f26380d0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f26400af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f26380b5400 con 0x7f26381649f0 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f26400af640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2638164e00 con 0x7f26380ce040 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f26400af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f26380b5580 con 0x7f26380b72d0 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f263d64a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26381649f0 0x7f263815e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f263d64a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26381649f0 0x7f263815e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41956/0 (socket says 172.21.15.67:41956) 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f263d64a640 1 -- 172.21.15.67:0/378958452 learned_addr learned my addr 172.21.15.67:0/378958452 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.464+0000 7f263de4b640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26380ce040 0x7f26380b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f263e64c640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26380b72d0 0x7f26380d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f263d64a640 1 -- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26380b72d0 msgr2=0x7f26380d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f263d64a640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26380b72d0 0x7f26380d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f263d64a640 1 -- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26380ce040 msgr2=0x7f26380b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f263d64a640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26380ce040 0x7f26380b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f263d64a640 1 -- 172.21.15.67:0/378958452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263815e570 con 0x7f26381649f0 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f263d64a640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26381649f0 0x7f263815e030 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f26280098c0 tx=0x7f26280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f262e7fc640 1 -- 172.21.15.67:0/378958452 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628041030 con 0x7f26381649f0 2024-04-25T15:28:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f262e7fc640 1 -- 172.21.15.67:0/378958452 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262802ac50 con 0x7f26381649f0 2024-04-25T15:28:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26380b15c0 con 0x7f26381649f0 2024-04-25T15:28:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f262e7fc640 1 -- 172.21.15.67:0/378958452 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628037040 con 0x7f26381649f0 2024-04-25T15:28:48.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.465+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26380b1b80 con 0x7f26381649f0 2024-04-25T15:28:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.466+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26380b6f40 con 0x7f26381649f0 2024-04-25T15:28:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.467+0000 7f262e7fc640 1 -- 172.21.15.67:0/378958452 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2628044070 con 0x7f26381649f0 2024-04-25T15:28:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.467+0000 7f262e7fc640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f261c0744b0 0x7f261c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.468+0000 7f262e7fc640 1 -- 172.21.15.67:0/378958452 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7f26280b53b0 con 0x7f26381649f0 2024-04-25T15:28:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.468+0000 7f263de4b640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f261c0744b0 0x7f261c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.468+0000 7f263de4b640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f261c0744b0 0x7f261c0769c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f263808ce10 tx=0x7f2634009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.472+0000 7f262e7fc640 1 -- 172.21.15.67:0/378958452 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f263815e570 con 0x7f26381649f0 2024-04-25T15:28:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.655+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2638050740 con 0x7f26381649f0 2024-04-25T15:28:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.656+0000 7f262e7fc640 1 -- 172.21.15.67:0/378958452 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f262807f880 con 0x7f26381649f0 2024-04-25T15:28:48.658 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f261c0744b0 msgr2=0x7f261c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f261c0744b0 0x7f261c0769c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f263808ce10 tx=0x7f2634009040 comp rx=0 tx=0).stop 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26381649f0 msgr2=0x7f263815e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26381649f0 0x7f263815e030 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f26280098c0 tx=0x7f26280046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 shutdown_connections 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f261c0744b0 0x7f261c0769c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f26380b72d0 0x7f26380d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26381649f0 0x7f263815e030 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 --2- 172.21.15.67:0/378958452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26380ce040 0x7f26380b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 >> 172.21.15.67:0/378958452 conn(0x7f26380d3a80 msgr2=0x7f26380d3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 shutdown_connections 2024-04-25T15:28:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.658+0000 7f26400af640 1 -- 172.21.15.67:0/378958452 wait complete. 2024-04-25T15:28:48.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write' 2024-04-25T15:28:48.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.780+0000 7f1a4d3b2640 1 Processor -- start 2024-04-25T15:28:48.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.780+0000 7f1a4d3b2640 1 -- start start 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a4d3b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 0x7f1a48160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a4d3b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a481615f0 con 0x7f1a48164b00 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a4d3b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a48164020 con 0x7f1a480b8290 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a4d3b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a48164670 con 0x7f1a4815df80 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a46ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1a48164b00 0x7f1a480b7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54528/0 (socket says 172.21.15.67:54528) 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a46ffd640 1 -- 172.21.15.67:0/1841277818 learned_addr learned my addr 172.21.15.67:0/1841277818 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a477fe640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 0x7f1a48160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040806676 0 0) 0x7f1a481615f0 con 0x7f1a48164b00 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a34003520 con 0x7f1a48164b00 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.782+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1743787652 0 0) 0x7f1a34003520 con 0x7f1a48164b00 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 msgr2=0x7f1a48160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 0x7f1a48160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 >> v1:172.21.15.176:6789/0 conn(0x7f1a480b8290 legacy=0x7f1a48160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a48164300 con 0x7f1a48164b00 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f1a40001670 con 0x7f1a48164b00 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 >> v1:172.21.15.67:6789/0 conn(0x7f1a48164b00 legacy=0x7f1a480b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a340039e0 0x7f1a34003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a34004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 0x7f1a34005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a48164670 con 0x7f1a34004800 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3400ba70 con 0x7f1a340039e0 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3400bc70 con 0x7f1a4815df80 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a467fc640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a34004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a46ffd640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a340039e0 0x7f1a34003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a477fe640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 0x7f1a34005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a467fc640 1 -- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 msgr2=0x7f1a34005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a467fc640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4815df80 0x7f1a34005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a467fc640 1 -- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a340039e0 msgr2=0x7f1a34003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a467fc640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a340039e0 0x7f1a34003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.783+0000 7f1a467fc640 1 -- 172.21.15.67:0/1841277818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a34003520 con 0x7f1a34004800 2024-04-25T15:28:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.784+0000 7f1a467fc640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a34004c10 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1a30001910 tx=0x7f1a34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2822a19c52b6378f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.784+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30037070 con 0x7f1a34004800 2024-04-25T15:28:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.784+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a3002ac60 con 0x7f1a34004800 2024-04-25T15:28:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.784+0000 7f1a45ffb640 1 -- 172.21.15.67:0/1841277818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a300014e0 con 0x7f1a34004800 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.785+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/1841277818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 msgr2=0x7f1a34004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.785+0000 7f1a4d3b2640 1 --2- 172.21.15.67:0/1841277818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a34004c10 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1a30001910 tx=0x7f1a34005830 comp rx=0 tx=0).stop 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.785+0000 7f1a477fe640 1 -- 172.21.15.67:0/1841277818 reap_dead start 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.785+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/1841277818 shutdown_connections 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.785+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/1841277818 >> 172.21.15.67:0/1841277818 conn(0x7f1a480d2bc0 msgr2=0x7f1a480d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.785+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/1841277818 shutdown_connections 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.785+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/1841277818 wait complete. 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.787+0000 7f1a4d3b2640 1 Processor -- start 2024-04-25T15:28:48.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.787+0000 7f1a4d3b2640 1 -- start start 2024-04-25T15:28:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a4d3b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a481552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a4d3b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a481f5fa0 0x7f1a481f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a4d3b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4814d5a0 0x7f1a4814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a4d3b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a48161370 con 0x7f1a34004800 2024-04-25T15:28:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a4d3b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481611f0 con 0x7f1a4814d5a0 2024-04-25T15:28:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a4d3b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a480b4880 con 0x7f1a481f5fa0 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a481552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a481552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41974/0 (socket says 172.21.15.67:41974) 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 -- 172.21.15.67:0/830702129 learned_addr learned my addr 172.21.15.67:0/830702129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 -- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a481f5fa0 msgr2=0x7f1a481f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a481f5fa0 0x7f1a481f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 -- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4814d5a0 msgr2=0x7f1a4814d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4814d5a0 0x7f1a4814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 -- 172.21.15.67:0/830702129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a48162380 con 0x7f1a34004800 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.788+0000 7f1a46ffd640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a481552f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1a400072c0 tx=0x7f1a40007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.789+0000 7f1a3f7fe640 1 -- 172.21.15.67:0/830702129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40001650 con 0x7f1a34004800 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.789+0000 7f1a3f7fe640 1 -- 172.21.15.67:0/830702129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a40018070 con 0x7f1a34004800 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.789+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a48162660 con 0x7f1a34004800 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.789+0000 7f1a3f7fe640 1 -- 172.21.15.67:0/830702129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4000fb80 con 0x7f1a34004800 2024-04-25T15:28:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.789+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a48162c20 con 0x7f1a34004800 2024-04-25T15:28:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.790+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a48050740 con 0x7f1a34004800 2024-04-25T15:28:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.791+0000 7f1a3f7fe640 1 -- 172.21.15.67:0/830702129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1a4000fd20 con 0x7f1a34004800 2024-04-25T15:28:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.791+0000 7f1a3f7fe640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a280744e0 0x7f1a280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.791+0000 7f1a3f7fe640 1 -- 172.21.15.67:0/830702129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7f1a40094ed0 con 0x7f1a34004800 2024-04-25T15:28:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.791+0000 7f1a467fc640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a280744e0 0x7f1a280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.792+0000 7f1a467fc640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a280744e0 0x7f1a280769f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1a30001910 tx=0x7f1a3002c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:48.796+0000 7f1a3f7fe640 1 -- 172.21.15.67:0/830702129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a4005f960 con 0x7f1a34004800 2024-04-25T15:28:49.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:49.053+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]} v 0) -- 0x7f1a480d4170 con 0x7f1a34004800 2024-04-25T15:28:50.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.011+0000 7f1a3f7fe640 1 -- 172.21.15.67:0/830702129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]}]=0 v214) ==== 140+0+0 (secure 0 0 0) 0x7f1a4005f260 con 0x7f1a34004800 2024-04-25T15:28:50.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a280744e0 msgr2=0x7f1a280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a280744e0 0x7f1a280769f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1a30001910 tx=0x7f1a3002c000 comp rx=0 tx=0).stop 2024-04-25T15:28:50.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 msgr2=0x7f1a481552f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a481552f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1a400072c0 tx=0x7f1a40007780 comp rx=0 tx=0).stop 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 shutdown_connections 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1a280744e0 0x7f1a280769f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4814d5a0 0x7f1a4814d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a481f5fa0 0x7f1a481f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 --2- 172.21.15.67:0/830702129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a34004800 0x7f1a481552f0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 >> 172.21.15.67:0/830702129 conn(0x7f1a480d2bc0 msgr2=0x7f1a480d5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 shutdown_connections 2024-04-25T15:28:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.014+0000 7f1a4d3b2640 1 -- 172.21.15.67:0/830702129 wait complete. 2024-04-25T15:28:50.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T15:28:50.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.150+0000 7f0ea549f640 1 Processor -- start 2024-04-25T15:28:50.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.150+0000 7f0ea549f640 1 -- start start 2024-04-25T15:28:50.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0ea549f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ea00d0e00 0x7f0ea00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0ea549f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0ea006c5c0 0x7f0ea006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0ea549f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ea006cf10 con 0x7f0ea00b7810 2024-04-25T15:28:50.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0ea549f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0ea00d5a60 con 0x7f0ea00d0e00 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0ea549f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0ea00d1850 con 0x7f0ea006c5c0 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ea00d0e00 0x7f0ea00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0ea00d0e00 0x7f0ea00d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41986/0 (socket says 172.21.15.67:41986) 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9effd640 1 -- 172.21.15.67:0/481066896 learned_addr learned my addr 172.21.15.67:0/481066896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9e7fc640 1 --2- 172.21.15.67:0/481066896 >> v2:172.21.15.176:3300/0 conn(0x7f0ea006c5c0 0x7f0ea006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9effd640 1 -- 172.21.15.67:0/481066896 >> v2:172.21.15.176:3300/0 conn(0x7f0ea006c5c0 msgr2=0x7f0ea006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9effd640 1 --2- 172.21.15.67:0/481066896 >> v2:172.21.15.176:3300/0 conn(0x7f0ea006c5c0 0x7f0ea006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9effd640 1 -- 172.21.15.67:0/481066896 >> v1:172.21.15.176:6789/0 conn(0x7f0ea00b7810 legacy=0x7f0ea00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.152+0000 7f0e9effd640 1 -- 172.21.15.67:0/481066896 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea0160de0 con 0x7f0ea00d0e00 2024-04-25T15:28:50.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.153+0000 7f0e9effd640 1 --2- 172.21.15.67:0/481066896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00d1210 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00d5be0 tx=0x7f0e9802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d431b65fc6b541e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:50.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.153+0000 7f0e9dffb640 1 -- 172.21.15.67:0/481066896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98037070 con 0x7f0ea00d0e00 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.153+0000 7f0e9dffb640 1 -- 172.21.15.67:0/481066896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e98032440 con 0x7f0ea00d0e00 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.153+0000 7f0e9dffb640 1 -- 172.21.15.67:0/481066896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e98032810 con 0x7f0ea00d0e00 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 -- 172.21.15.67:0/481066896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00d0e00 msgr2=0x7f0ea00d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 --2- 172.21.15.67:0/481066896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00d1210 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00d5be0 tx=0x7f0e9802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 -- 172.21.15.67:0/481066896 shutdown_connections 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 --2- 172.21.15.67:0/481066896 >> v2:172.21.15.176:3300/0 conn(0x7f0ea006c5c0 0x7f0ea006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 --2- 172.21.15.67:0/481066896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00d1210 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 -- 172.21.15.67:0/481066896 >> 172.21.15.67:0/481066896 conn(0x7f0ea00d5d90 msgr2=0x7f0ea00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 -- 172.21.15.67:0/481066896 shutdown_connections 2024-04-25T15:28:50.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.154+0000 7f0ea549f640 1 -- 172.21.15.67:0/481066896 wait complete. 2024-04-25T15:28:50.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.156+0000 7f0ea549f640 1 Processor -- start 2024-04-25T15:28:50.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.156+0000 7f0ea549f640 1 -- start start 2024-04-25T15:28:50.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0ea549f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0ea006c5c0 0x7f0ea014d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0ea549f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00b7810 0x7f0ea00ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0ea549f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0ea549f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea0160ad0 con 0x7f0ea00b7810 2024-04-25T15:28:50.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0ea549f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01613a0 con 0x7f0ea00d0e00 2024-04-25T15:28:50.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0ea549f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea00b5320 con 0x7f0ea006c5c0 2024-04-25T15:28:50.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0e9e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00b7810 0x7f0ea00ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0e9e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00b7810 0x7f0ea00ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41988/0 (socket says 172.21.15.67:41988) 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.157+0000 7f0e9e7fc640 1 -- 172.21.15.67:0/1416480401 learned_addr learned my addr 172.21.15.67:0/1416480401 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9effd640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0ea006c5c0 0x7f0ea014d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9f7fe640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9e7fc640 1 -- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0ea006c5c0 msgr2=0x7f0ea014d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9e7fc640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0ea006c5c0 0x7f0ea014d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9e7fc640 1 -- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ea00d0e00 msgr2=0x7f0ea00cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9e7fc640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9e7fc640 1 -- 172.21.15.67:0/1416480401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea00cd140 con 0x7f0ea00b7810 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9f7fe640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e9e7fc640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00b7810 0x7f0ea00ca840 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800d620 tx=0x7f0e8800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e977fe640 1 -- 172.21.15.67:0/1416480401 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8800b620 con 0x7f0ea00b7810 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e977fe640 1 -- 172.21.15.67:0/1416480401 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e88019070 con 0x7f0ea00b7810 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0e977fe640 1 -- 172.21.15.67:0/1416480401 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8800ba00 con 0x7f0ea00b7810 2024-04-25T15:28:50.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea014ae10 con 0x7f0ea00b7810 2024-04-25T15:28:50.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.158+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea014b3d0 con 0x7f0ea00b7810 2024-04-25T15:28:50.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.159+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea00cceb0 con 0x7f0ea00b7810 2024-04-25T15:28:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.160+0000 7f0e977fe640 1 -- 172.21.15.67:0/1416480401 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0e8800bba0 con 0x7f0ea00b7810 2024-04-25T15:28:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.160+0000 7f0e977fe640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0e800744e0 0x7f0e800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.160+0000 7f0e977fe640 1 -- 172.21.15.67:0/1416480401 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7f0e880952e0 con 0x7f0ea00b7810 2024-04-25T15:28:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.161+0000 7f0e9effd640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0e800744e0 0x7f0e800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.161+0000 7f0e9effd640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0e800744e0 0x7f0e800769f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0e98034000 tx=0x7f0e9802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:50.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.165+0000 7f0e977fe640 1 -- 172.21.15.67:0/1416480401 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e8805fe00 con 0x7f0ea00b7810 2024-04-25T15:28:50.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.349+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f0ea0050740 con 0x7f0ea00b7810 2024-04-25T15:28:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.353+0000 7f0e977fe640 1 -- 172.21.15.67:0/1416480401 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7f0e8805f700 con 0x7f0ea00b7810 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0e800744e0 msgr2=0x7f0e800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0e800744e0 0x7f0e800769f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0e98034000 tx=0x7f0e9802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00b7810 msgr2=0x7f0ea00ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00b7810 0x7f0ea00ca840 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0e8800d620 tx=0x7f0e8800dae0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 shutdown_connections 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f0e800744e0 0x7f0e800769f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ea00d0e00 0x7f0ea00cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ea00b7810 0x7f0ea00ca840 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 --2- 172.21.15.67:0/1416480401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0ea006c5c0 0x7f0ea014d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 >> 172.21.15.67:0/1416480401 conn(0x7f0ea00d5d90 msgr2=0x7f0ea00b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:50.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 shutdown_connections 2024-04-25T15:28:50.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.355+0000 7f0ea549f640 1 -- 172.21.15.67:0/1416480401 wait complete. 2024-04-25T15:28:50.368 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:28:50.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:50.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:28:50.451 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:28:50.451 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:50.451 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:50.466 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:28:50.466 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:50.466 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:50.508 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:28:50.508 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:28:50.508 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:50.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:28:50.524 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:28:50.524 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:28:50.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:28:50.566 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T15:28:50.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:28:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.677+0000 7f7daf91f640 1 Processor -- start 2024-04-25T15:28:50.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.677+0000 7f7daf91f640 1 -- start start 2024-04-25T15:28:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.678+0000 7f7daf91f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806d8e0 0x7f7da80d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.678+0000 7f7daf91f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7da806dcf0 0x7f7da80d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.678+0000 7f7daf91f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da80b3c10 con 0x7f7da80d04a0 2024-04-25T15:28:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.678+0000 7f7daf91f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7da80b9520 con 0x7f7da806dcf0 2024-04-25T15:28:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.678+0000 7f7daf91f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7da80b4820 con 0x7f7da806d8e0 2024-04-25T15:28:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7dad6bb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806d8e0 0x7f7da80d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7dad6bb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806d8e0 0x7f7da80d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:33604/0 (socket says 172.21.15.67:33604) 2024-04-25T15:28:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7daceba640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7da806dcf0 0x7f7da80d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7dad6bb640 1 -- 172.21.15.67:0/55647689 learned_addr learned my addr 172.21.15.67:0/55647689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7dad6bb640 1 -- 172.21.15.67:0/55647689 >> v2:172.21.15.176:3300/0 conn(0x7f7da806dcf0 msgr2=0x7f7da80d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7dad6bb640 1 --2- 172.21.15.67:0/55647689 >> v2:172.21.15.176:3300/0 conn(0x7f7da806dcf0 0x7f7da80d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7dad6bb640 1 -- 172.21.15.67:0/55647689 >> v1:172.21.15.176:6789/0 conn(0x7f7da80d04a0 legacy=0x7f7da80d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7dad6bb640 1 -- 172.21.15.67:0/55647689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da80b3ef0 con 0x7f7da806d8e0 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.679+0000 7f7daceba640 1 --2- 172.21.15.67:0/55647689 >> v2:172.21.15.176:3300/0 conn(0x7f7da806dcf0 0x7f7da80d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.680+0000 7f7dad6bb640 1 --2- 172.21.15.67:0/55647689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806d8e0 0x7f7da80d3a20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009870 tx=0x7f7da002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99b16559b426c336 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.680+0000 7f7da7fff640 1 -- 172.21.15.67:0/55647689 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da0037070 con 0x7f7da806d8e0 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.680+0000 7f7da7fff640 1 -- 172.21.15.67:0/55647689 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da0032440 con 0x7f7da806d8e0 2024-04-25T15:28:50.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.680+0000 7f7da7fff640 1 -- 172.21.15.67:0/55647689 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da0032810 con 0x7f7da806d8e0 2024-04-25T15:28:50.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 -- 172.21.15.67:0/55647689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806d8e0 msgr2=0x7f7da80d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 --2- 172.21.15.67:0/55647689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806d8e0 0x7f7da80d3a20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009870 tx=0x7f7da002a1c0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 -- 172.21.15.67:0/55647689 shutdown_connections 2024-04-25T15:28:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 --2- 172.21.15.67:0/55647689 >> v2:172.21.15.176:3300/0 conn(0x7f7da806dcf0 0x7f7da80d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 --2- 172.21.15.67:0/55647689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806d8e0 0x7f7da80d3a20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 -- 172.21.15.67:0/55647689 >> 172.21.15.67:0/55647689 conn(0x7f7da808d110 msgr2=0x7f7da808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 -- 172.21.15.67:0/55647689 shutdown_connections 2024-04-25T15:28:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.681+0000 7f7daf91f640 1 -- 172.21.15.67:0/55647689 wait complete. 2024-04-25T15:28:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.683+0000 7f7daf91f640 1 Processor -- start 2024-04-25T15:28:50.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.683+0000 7f7daf91f640 1 -- start start 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7daf91f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da806d8e0 0x7f7da8160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7daf91f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806dcf0 0x7f7da809f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7daf91f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da80d04a0 0x7f7da81e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7daf91f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80b38e0 con 0x7f7da806d8e0 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7daf91f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80b44b0 con 0x7f7da80d04a0 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7daf91f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7da80d0fc0 con 0x7f7da806dcf0 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da806d8e0 0x7f7da8160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da806d8e0 0x7f7da8160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41994/0 (socket says 172.21.15.67:41994) 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 -- 172.21.15.67:0/3098007326 learned_addr learned my addr 172.21.15.67:0/3098007326 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7daceba640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806dcf0 0x7f7da809f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dadebc640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da80d04a0 0x7f7da81e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 -- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806dcf0 msgr2=0x7f7da809f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806dcf0 0x7f7da809f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 -- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da80d04a0 msgr2=0x7f7da81e6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da80d04a0 0x7f7da81e6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 -- 172.21.15.67:0/3098007326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da81e78b0 con 0x7f7da806d8e0 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dadebc640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da80d04a0 0x7f7da81e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.684+0000 7f7dad6bb640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da806d8e0 0x7f7da8160270 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7da002a200 tx=0x7f7da00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.685+0000 7f7da5ffb640 1 -- 172.21.15.67:0/3098007326 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da0037040 con 0x7f7da806d8e0 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.685+0000 7f7da5ffb640 1 -- 172.21.15.67:0/3098007326 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da0042070 con 0x7f7da806d8e0 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.685+0000 7f7da5ffb640 1 -- 172.21.15.67:0/3098007326 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da0030880 con 0x7f7da806d8e0 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.685+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da81e7b90 con 0x7f7da806d8e0 2024-04-25T15:28:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.685+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da80c7760 con 0x7f7da806d8e0 2024-04-25T15:28:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.686+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da8050740 con 0x7f7da806d8e0 2024-04-25T15:28:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.687+0000 7f7da5ffb640 1 -- 172.21.15.67:0/3098007326 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7da002ad10 con 0x7f7da806d8e0 2024-04-25T15:28:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.687+0000 7f7da5ffb640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d940744b0 0x7f7d940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.687+0000 7f7da5ffb640 1 -- 172.21.15.67:0/3098007326 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7f7da00b53d0 con 0x7f7da806d8e0 2024-04-25T15:28:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.687+0000 7f7daceba640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d940744b0 0x7f7d940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.688+0000 7f7daceba640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d940744b0 0x7f7d940769c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7da8071bf0 tx=0x7f7d9c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:50.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.692+0000 7f7da5ffb640 1 -- 172.21.15.67:0/3098007326 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7da007fef0 con 0x7f7da806d8e0 2024-04-25T15:28:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.913+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7da80d33c0 con 0x7f7da806d8e0 2024-04-25T15:28:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.914+0000 7f7da5ffb640 1 -- 172.21.15.67:0/3098007326 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+3872 (secure 0 0 0) 0x7f7da007f7f0 con 0x7f7da806d8e0 2024-04-25T15:28:50.916 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:50.916 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/445614904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":445614904},{"type":"v1","addr":"172.21.15.67:6833","nonce":445614904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":44},{"gid":5993,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1382651923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":1382651923},{"type":"v1","addr":"172.21.15.176:6837","nonce":1382651923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51},{"gid":6322,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3018066741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3018066741},{"type":"v1","addr":"172.21.15.67:6835","nonce":3018066741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45},{"gid":6325,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/2045345118","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2045345118},{"type":"v1","addr":"172.21.15.176:6836","nonce":2045345118}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":45}],"filesystems":[]} 2024-04-25T15:28:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 51 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d940744b0 msgr2=0x7f7d940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d940744b0 0x7f7d940769c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7da8071bf0 tx=0x7f7d9c009190 comp rx=0 tx=0).stop 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da806d8e0 msgr2=0x7f7da8160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da806d8e0 0x7f7da8160270 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7da002a200 tx=0x7f7da00046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 shutdown_connections 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7d940744b0 0x7f7d940769c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da80d04a0 0x7f7da81e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da806dcf0 0x7f7da809f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 --2- 172.21.15.67:0/3098007326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7da806d8e0 0x7f7da8160270 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:50.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 >> 172.21.15.67:0/3098007326 conn(0x7f7da808d110 msgr2=0x7f7da80b8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:50.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 shutdown_connections 2024-04-25T15:28:50.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:50.916+0000 7f7daf91f640 1 -- 172.21.15.67:0/3098007326 wait complete. 2024-04-25T15:28:50.929 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:28:50.930 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T15:28:50.930 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:28:50.930 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:50.930 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:28:50.930 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T15:28:50.930 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:28:50.930 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:50.931 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:28:50.931 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T15:28:50.931 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:28:50.931 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:50.931 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:28:50.931 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T15:28:50.931 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:28:50.931 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:28:50.940 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:28:50.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:28:50.941 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:28:50.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:28:50.945 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:28:50.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:28:50.947 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:28:50.947 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:28:50.983 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:28:50.984 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:50.984 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:28:50.984 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:50.988 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:28:50.989 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:50.990 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:28:50.990 DEBUG:teuthology.parallel:result is None 2024-04-25T15:28:50.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T15:28:51.072 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:28:51.072 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:28:51.082 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:28:51.082 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:28:51.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.108+0000 7f6fe8b7a640 1 Processor -- start 2024-04-25T15:28:51.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.108+0000 7f6fe8b7a640 1 -- start start 2024-04-25T15:28:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe8b7a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6fe406ccd0 0x7f6fe40d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe8b7a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe40d1850 con 0x7f6fe40d0e00 2024-04-25T15:28:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe8b7a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe4160de0 con 0x7f6fe40d4a50 2024-04-25T15:28:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe8b7a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe41610c0 con 0x7f6fe406ccd0 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6fe40d0e00 0x7f6fe406c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54550/0 (socket says 172.21.15.67:54550) 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe259c640 1 -- 172.21.15.67:0/2716954872 learned_addr learned my addr 172.21.15.67:0/2716954872 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fdbfff640 1 --2- 172.21.15.67:0/2716954872 >> v2:172.21.15.176:3300/0 conn(0x7f6fe406ccd0 0x7f6fe40d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1061502292 0 0) 0x7f6fe40d1850 con 0x7f6fe40d0e00 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd0003520 con 0x7f6fe40d0e00 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524002225 0 0) 0x7f6fe4160de0 con 0x7f6fe40d4a50 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe40d1850 con 0x7f6fe40d4a50 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.110+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4174213116 0 0) 0x7f6fd0003520 con 0x7f6fe40d0e00 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.111+0000 7f6fdbfff640 1 -- 172.21.15.67:0/2716954872 >> v1:172.21.15.176:6789/0 conn(0x7f6fe40d4a50 legacy=0x7f6fe40d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.111+0000 7f6fdbfff640 1 -- 172.21.15.67:0/2716954872 >> v1:172.21.15.67:6789/0 conn(0x7f6fe40d0e00 legacy=0x7f6fe406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.111+0000 7f6fdbfff640 1 -- 172.21.15.67:0/2716954872 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe41613a0 con 0x7f6fe406ccd0 2024-04-25T15:28:51.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.111+0000 7f6fdbfff640 1 --2- 172.21.15.67:0/2716954872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe406ccd0 0x7f6fe40d1210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0097a0 tx=0x7f6fcc02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=157feb0fac437a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:51.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.111+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc037070 con 0x7f6fe406ccd0 2024-04-25T15:28:51.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fcc02abc0 con 0x7f6fe406ccd0 2024-04-25T15:28:51.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe1d9b640 1 -- 172.21.15.67:0/2716954872 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4001670 con 0x7f6fe406ccd0 2024-04-25T15:28:51.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2716954872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe406ccd0 msgr2=0x7f6fe40d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:51.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2716954872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe406ccd0 0x7f6fe40d1210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0097a0 tx=0x7f6fcc02a130 comp rx=0 tx=0).stop 2024-04-25T15:28:51.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2716954872 shutdown_connections 2024-04-25T15:28:51.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2716954872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe406ccd0 0x7f6fe40d1210 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:51.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2716954872 >> 172.21.15.67:0/2716954872 conn(0x7f6fe40d5d90 msgr2=0x7f6fe40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:51.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2716954872 shutdown_connections 2024-04-25T15:28:51.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.112+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2716954872 wait complete. 2024-04-25T15:28:51.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.114+0000 7f6fe8b7a640 1 Processor -- start 2024-04-25T15:28:51.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.114+0000 7f6fe8b7a640 1 -- start start 2024-04-25T15:28:51.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.115+0000 7f6fe8b7a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe40d0e00 0x7f6fe41583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:51.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe8b7a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fe40d4a50 0x7f6fe40c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:51.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe8b7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fe40c99f0 0x7f6fe40c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:51.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe8b7a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe4164e00 con 0x7f6fe40c99f0 2024-04-25T15:28:51.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe8b7a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe40b5320 con 0x7f6fe40d0e00 2024-04-25T15:28:51.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe8b7a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe4164f80 con 0x7f6fe40d4a50 2024-04-25T15:28:51.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fe40c99f0 0x7f6fe40c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:51.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fe40c99f0 0x7f6fe40c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42020/0 (socket says 172.21.15.67:42020) 2024-04-25T15:28:51.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 -- 172.21.15.67:0/2437966711 learned_addr learned my addr 172.21.15.67:0/2437966711 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:51.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe259c640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe40d0e00 0x7f6fe41583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:51.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fdbfff640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fe40d4a50 0x7f6fe40c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:51.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 -- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fe40d4a50 msgr2=0x7f6fe40c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fe40d4a50 0x7f6fe40c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 -- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe40d0e00 msgr2=0x7f6fe41583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe40d0e00 0x7f6fe41583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 -- 172.21.15.67:0/2437966711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe40cc230 con 0x7f6fe40c99f0 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe259c640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe40d0e00 0x7f6fe41583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fdbfff640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fe40d4a50 0x7f6fe40c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe2d9d640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fe40c99f0 0x7f6fe40c9e00 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c710 tx=0x7f6fdc00cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fdaffd640 1 -- 172.21.15.67:0/2437966711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc003b70 con 0x7f6fe40c99f0 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fdaffd640 1 -- 172.21.15.67:0/2437966711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fdc003d10 con 0x7f6fe40c99f0 2024-04-25T15:28:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe40b6230 con 0x7f6fe40c99f0 2024-04-25T15:28:51.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fdaffd640 1 -- 172.21.15.67:0/2437966711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc00aa50 con 0x7f6fe40c99f0 2024-04-25T15:28:51.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.116+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe40b67f0 con 0x7f6fe40c99f0 2024-04-25T15:28:51.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.117+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe4050740 con 0x7f6fe40c99f0 2024-04-25T15:28:51.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.118+0000 7f6fdaffd640 1 -- 172.21.15.67:0/2437966711 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6fdc00abf0 con 0x7f6fe40c99f0 2024-04-25T15:28:51.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.119+0000 7f6fdaffd640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6fc40744b0 0x7f6fc40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:51.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.119+0000 7f6fdaffd640 1 -- 172.21.15.67:0/2437966711 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 4943+0+0 (secure 0 0 0) 0x7f6fdc094c50 con 0x7f6fe40c99f0 2024-04-25T15:28:51.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.119+0000 7f6fe259c640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6fc40744b0 0x7f6fc40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:51.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.119+0000 7f6fe259c640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6fc40744b0 0x7f6fc40769c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40017e0 tx=0x7f6fd4001610 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:51.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.123+0000 7f6fdaffd640 1 -- 172.21.15.67:0/2437966711 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fdc09a020 con 0x7f6fe40c99f0 2024-04-25T15:28:51.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:51.322+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6fe40d6610 con 0x7f6fe40c99f0 2024-04-25T15:28:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.373+0000 7f6fdaffd640 1 -- 172.21.15.67:0/2437966711 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v88) ==== 97+0+0 (secure 0 0 0) 0x7f6fdc00ce20 con 0x7f6fe40c99f0 2024-04-25T15:28:52.376 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-25T15:28:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.376+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6fc40744b0 msgr2=0x7f6fc40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.376+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6fc40744b0 0x7f6fc40769c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40017e0 tx=0x7f6fd4001610 comp rx=0 tx=0).stop 2024-04-25T15:28:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.376+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fe40c99f0 msgr2=0x7f6fe40c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fe40c99f0 0x7f6fe40c9e00 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c710 tx=0x7f6fdc00cbd0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 shutdown_connections 2024-04-25T15:28:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6fc40744b0 0x7f6fc40769c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fe40c99f0 0x7f6fe40c9e00 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fe40d4a50 0x7f6fe40c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 --2- 172.21.15.67:0/2437966711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fe40d0e00 0x7f6fe41583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 >> 172.21.15.67:0/2437966711 conn(0x7f6fe40d5d90 msgr2=0x7f6fe40d5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 shutdown_connections 2024-04-25T15:28:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.377+0000 7f6fe8b7a640 1 -- 172.21.15.67:0/2437966711 wait complete. 2024-04-25T15:28:52.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T15:28:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.516+0000 7f441acbd640 1 Processor -- start 2024-04-25T15:28:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.516+0000 7f441acbd640 1 -- start start 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f441acbd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f441406d8e0 0x7f44140d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f441acbd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406dcf0 0x7f44140d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f441acbd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44140b3c10 con 0x7f44140d04a0 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f441acbd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f44140b9520 con 0x7f441406d8e0 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f441acbd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44140b4820 con 0x7f441406dcf0 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f4418a59640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f441406d8e0 0x7f44140d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f4413fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406dcf0 0x7f44140d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f4418a59640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f441406d8e0 0x7f44140d3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39738/0 (socket says 172.21.15.67:39738) 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f4413fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406dcf0 0x7f44140d3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:33666/0 (socket says 172.21.15.67:33666) 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.517+0000 7f4418a59640 1 -- 172.21.15.67:0/4217004629 learned_addr learned my addr 172.21.15.67:0/4217004629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.518+0000 7f4418a59640 1 -- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406dcf0 msgr2=0x7f44140d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.518+0000 7f4418a59640 1 --2- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406dcf0 0x7f44140d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.518+0000 7f4418a59640 1 -- 172.21.15.67:0/4217004629 >> v1:172.21.15.176:6789/0 conn(0x7f44140d04a0 legacy=0x7f44140d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.518+0000 7f44137fe640 1 -- 172.21.15.67:0/4217004629 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660647367 0 0) 0x7f44140b3c10 con 0x7f44140d04a0 2024-04-25T15:28:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.518+0000 7f4418a59640 1 -- 172.21.15.67:0/4217004629 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44140b3ef0 con 0x7f441406d8e0 2024-04-25T15:28:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.518+0000 7f4413fff640 1 --2- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406dcf0 0x7f44140d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f4418a59640 1 --2- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441406d8e0 0x7f44140d3a20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4404009870 tx=0x7f440402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=633e870c2f2fdb83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f44137fe640 1 -- 172.21.15.67:0/4217004629 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404037070 con 0x7f441406d8e0 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f44137fe640 1 -- 172.21.15.67:0/4217004629 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4404032440 con 0x7f441406d8e0 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f44137fe640 1 -- 172.21.15.67:0/4217004629 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404032810 con 0x7f441406d8e0 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f441acbd640 1 -- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441406d8e0 msgr2=0x7f44140d3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f441acbd640 1 --2- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441406d8e0 0x7f44140d3a20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4404009870 tx=0x7f440402a1c0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f441acbd640 1 -- 172.21.15.67:0/4217004629 shutdown_connections 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f441acbd640 1 --2- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406dcf0 0x7f44140d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f441acbd640 1 --2- 172.21.15.67:0/4217004629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f441406d8e0 0x7f44140d3a20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f441acbd640 1 -- 172.21.15.67:0/4217004629 >> 172.21.15.67:0/4217004629 conn(0x7f441408d110 msgr2=0x7f441408d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.519+0000 7f441acbd640 1 -- 172.21.15.67:0/4217004629 shutdown_connections 2024-04-25T15:28:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.520+0000 7f441acbd640 1 -- 172.21.15.67:0/4217004629 wait complete. 2024-04-25T15:28:52.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.521+0000 7f441acbd640 1 Processor -- start 2024-04-25T15:28:52.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.521+0000 7f441acbd640 1 -- start start 2024-04-25T15:28:52.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f441acbd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406d8e0 0x7f441415f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f441acbd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441406dcf0 0x7f44141e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f441acbd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44140d04a0 0x7f44140a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f441acbd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f44140b38e0 con 0x7f441406dcf0 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f441acbd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f44140b44b0 con 0x7f44140d04a0 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f441acbd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44140d0fc0 con 0x7f441406d8e0 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441406dcf0 0x7f44141e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441406dcf0 0x7f44141e6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42034/0 (socket says 172.21.15.67:42034) 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 -- 172.21.15.67:0/2799902944 learned_addr learned my addr 172.21.15.67:0/2799902944 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4418a59640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406d8e0 0x7f441415f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f441925a640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44140d04a0 0x7f44140a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 -- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406d8e0 msgr2=0x7f441415f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406d8e0 0x7f441415f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 -- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44140d04a0 msgr2=0x7f44140a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44140d04a0 0x7f44140a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 -- 172.21.15.67:0/2799902944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44141e7410 con 0x7f441406dcf0 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.523+0000 7f4413fff640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441406dcf0 0x7f44141e6a30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f440800e960 tx=0x7f440800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.524+0000 7f44117fa640 1 -- 172.21.15.67:0/2799902944 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408017070 con 0x7f441406dcf0 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.524+0000 7f44117fa640 1 -- 172.21.15.67:0/2799902944 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440800ca00 con 0x7f441406dcf0 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.524+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44141e76f0 con 0x7f441406dcf0 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.524+0000 7f44117fa640 1 -- 172.21.15.67:0/2799902944 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440800cdd0 con 0x7f441406dcf0 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.524+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f441409e790 con 0x7f441406dcf0 2024-04-25T15:28:52.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.525+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44141e7160 con 0x7f441406dcf0 2024-04-25T15:28:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.525+0000 7f44117fa640 1 -- 172.21.15.67:0/2799902944 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f440801a070 con 0x7f441406dcf0 2024-04-25T15:28:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.526+0000 7f44117fa640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44000744e0 0x7f44000769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.526+0000 7f44117fa640 1 -- 172.21.15.67:0/2799902944 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7f4408095190 con 0x7f441406dcf0 2024-04-25T15:28:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.526+0000 7f4418a59640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44000744e0 0x7f44000769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.527+0000 7f4418a59640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44000744e0 0x7f44000769f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4404034000 tx=0x7f440402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:52.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.531+0000 7f44117fa640 1 -- 172.21.15.67:0/2799902944 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f440805fc70 con 0x7f441406dcf0 2024-04-25T15:28:52.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.785+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4414050740 con 0x7f441406dcf0 2024-04-25T15:28:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.785+0000 7f44117fa640 1 -- 172.21.15.67:0/2799902944 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7f440805f570 con 0x7f441406dcf0 2024-04-25T15:28:52.787 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:52.787 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:28:52.787 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-25T15:28:52.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmv0XONhAA40hM29Um5EemJuHmsIZlCg==", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmkxPoBxAAi6i2KKQEVGPHh+WXVErCAA==", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:52.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmbNDHAhAAFpdpecZdHNHGw2BY4X+N3Q==", 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-25T15:28:52.789 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKblNDxAAcID20Yny8UB6uwZgYJYfBw==", 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCydSpmOniEKhAAvH0KZWIbOeVPXT3XkW9RMg==", 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.790 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC1dSpmlGAmKRAAUV5M2gFc1l7gIVFgnI1rHw==", 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.791 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC4dSpmNbHwJxAAwk0PdFlAnn2/EVUKqFgHOA==", 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQC7dSpmrdWuJxAA4DtHhFMLZOgDDjZW6iuP7g==", 2024-04-25T15:28:52.792 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDBdSpmb6V1KBAAj8AEKFnqEEfHtSLojnvQpA==", 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.793 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDEdSpmFImkJxAASJ/FWODkknQwk6xnkJsczA==", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDHdSpmKfEvJhAAAT7M1iEk4gA0p//4PARCXQ==", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQDKdSpmqgprJBAAf9Uc8B9+SecXfDqYwIa4nA==", 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q==", 2024-04-25T15:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmKkc3GxAA04ROvaGpRWhQQK9YYaLeZQ==", 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpm0a2MFxAAhXqVvUTU9+9WQ7jME4SoCg==", 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-25T15:28:52.797 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCvdSpmm0XVHhAAkNJT6JwMbBDIqRaVKztGuA==", 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmVIPuGRAAswFIakCxyA7OvvoJIBYUiQ==", 2024-04-25T15:28:52.798 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmJq4IMxAAM2VEdOCqr3PK3K+S9zcyPg==", 2024-04-25T15:28:52.799 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQCudSpmOr+2LhAAySfno4B13NT7vqY4ewHgag==", 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-25T15:28:52.800 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-25T15:28:52.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-25T15:28:52.801 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.801 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:52.801 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:28:52.801 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:28:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44000744e0 msgr2=0x7f44000769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44000744e0 0x7f44000769f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4404034000 tx=0x7f440402e000 comp rx=0 tx=0).stop 2024-04-25T15:28:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441406dcf0 msgr2=0x7f44141e6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441406dcf0 0x7f44141e6a30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f440800e960 tx=0x7f440800ee20 comp rx=0 tx=0).stop 2024-04-25T15:28:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 shutdown_connections 2024-04-25T15:28:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44000744e0 0x7f44000769f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44140d04a0 0x7f44140a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f441406dcf0 0x7f44141e6a30 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 --2- 172.21.15.67:0/2799902944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f441406d8e0 0x7f441415f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 >> 172.21.15.67:0/2799902944 conn(0x7f441408d110 msgr2=0x7f4414071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:52.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 shutdown_connections 2024-04-25T15:28:52.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.787+0000 7f441acbd640 1 -- 172.21.15.67:0/2799902944 wait complete. 2024-04-25T15:28:52.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.914+0000 7fcbe7140640 1 Processor -- start 2024-04-25T15:28:52.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.914+0000 7fcbe7140640 1 -- start start 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe7140640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcbe00ce520 0x7fcbe006c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe7140640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe015e320 0x7fcbe00b7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe7140640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbe00b7d80 0x7fcbe00d2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe7140640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe00ce930 con 0x7fcbe00b7d80 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe7140640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe00b8190 con 0x7fcbe00ce520 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe7140640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe00d0450 con 0x7fcbe015e320 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbe00b7d80 0x7fcbe00d2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbe00b7d80 0x7fcbe00d2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42042/0 (socket says 172.21.15.67:42042) 2024-04-25T15:28:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 -- 172.21.15.67:0/2657036631 learned_addr learned my addr 172.21.15.67:0/2657036631 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe4edc640 1 --2- 172.21.15.67:0/2657036631 >> v2:172.21.15.176:3300/0 conn(0x7fcbe00ce520 0x7fcbe006c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbdffff640 1 --2- 172.21.15.67:0/2657036631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe015e320 0x7fcbe00b7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 -- 172.21.15.67:0/2657036631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe015e320 msgr2=0x7fcbe00b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 --2- 172.21.15.67:0/2657036631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe015e320 0x7fcbe00b7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 -- 172.21.15.67:0/2657036631 >> v2:172.21.15.176:3300/0 conn(0x7fcbe00ce520 msgr2=0x7fcbe006c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 --2- 172.21.15.67:0/2657036631 >> v2:172.21.15.176:3300/0 conn(0x7fcbe00ce520 0x7fcbe006c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe56dd640 1 -- 172.21.15.67:0/2657036631 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe006cc50 con 0x7fcbe00b7d80 2024-04-25T15:28:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.916+0000 7fcbe4edc640 1 --2- 172.21.15.67:0/2657036631 >> v2:172.21.15.176:3300/0 conn(0x7fcbe00ce520 0x7fcbe006c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.917+0000 7fcbe56dd640 1 --2- 172.21.15.67:0/2657036631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00b7d80 0x7fcbe00d2a20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00b8620 tx=0x7fcbd802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07ff931a185b2c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.917+0000 7fcbdf7fe640 1 -- 172.21.15.67:0/2657036631 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8036070 con 0x7fcbe00b7d80 2024-04-25T15:28:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.917+0000 7fcbdf7fe640 1 -- 172.21.15.67:0/2657036631 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd8031600 con 0x7fcbe00b7d80 2024-04-25T15:28:52.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.917+0000 7fcbdf7fe640 1 -- 172.21.15.67:0/2657036631 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80319d0 con 0x7fcbe00b7d80 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 -- 172.21.15.67:0/2657036631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00b7d80 msgr2=0x7fcbe00d2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 --2- 172.21.15.67:0/2657036631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00b7d80 0x7fcbe00d2a20 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00b8620 tx=0x7fcbd802a610 comp rx=0 tx=0).stop 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 -- 172.21.15.67:0/2657036631 shutdown_connections 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 --2- 172.21.15.67:0/2657036631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00b7d80 0x7fcbe00d2a20 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 --2- 172.21.15.67:0/2657036631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe015e320 0x7fcbe00b7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 --2- 172.21.15.67:0/2657036631 >> v2:172.21.15.176:3300/0 conn(0x7fcbe00ce520 0x7fcbe006c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 -- 172.21.15.67:0/2657036631 >> 172.21.15.67:0/2657036631 conn(0x7fcbe009b7d0 msgr2=0x7fcbe009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 -- 172.21.15.67:0/2657036631 shutdown_connections 2024-04-25T15:28:52.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.918+0000 7fcbe7140640 1 -- 172.21.15.67:0/2657036631 wait complete. 2024-04-25T15:28:52.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.920+0000 7fcbe7140640 1 Processor -- start 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.920+0000 7fcbe7140640 1 -- start start 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe7140640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe00b7d80 0x7fcbe0150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe7140640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00ce520 0x7fcbe00cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe7140640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe015e320 0x7fcbe0154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe7140640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe006d4f0 con 0x7fcbe00ce520 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe7140640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe006d210 con 0x7fcbe015e320 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe7140640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe006d7d0 con 0x7fcbe00b7d80 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbdffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00ce520 0x7fcbe00cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbdffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00ce520 0x7fcbe00cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42048/0 (socket says 172.21.15.67:42048) 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbdffff640 1 -- 172.21.15.67:0/824514859 learned_addr learned my addr 172.21.15.67:0/824514859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe4edc640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe00b7d80 0x7fcbe0150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.921+0000 7fcbe56dd640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe015e320 0x7fcbe0154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdffff640 1 -- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe00b7d80 msgr2=0x7fcbe0150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdffff640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe00b7d80 0x7fcbe0150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdffff640 1 -- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe015e320 msgr2=0x7fcbe0154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdffff640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe015e320 0x7fcbe0154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdffff640 1 -- 172.21.15.67:0/824514859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe01514e0 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbe4edc640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe00b7d80 0x7fcbe0150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdffff640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00ce520 0x7fcbe00cbce0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400d620 tx=0x7fcbd400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdd7fa640 1 -- 172.21.15.67:0/824514859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400b620 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdd7fa640 1 -- 172.21.15.67:0/824514859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd4019070 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe00cca40 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbdd7fa640 1 -- 172.21.15.67:0/824514859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400ba00 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.922+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe00af880 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.923+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe0050740 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.924+0000 7fcbdd7fa640 1 -- 172.21.15.67:0/824514859 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcbd400bba0 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.924+0000 7fcbdd7fa640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbcc0744e0 0x7fcbcc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.924+0000 7fcbdd7fa640 1 -- 172.21.15.67:0/824514859 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7fcbd4095310 con 0x7fcbe00ce520 2024-04-25T15:28:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.924+0000 7fcbe4edc640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbcc0744e0 0x7fcbcc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.925+0000 7fcbe4edc640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbcc0744e0 0x7fcbcc0769f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0006fd0 tx=0x7fcbd0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:52.929+0000 7fcbdd7fa640 1 -- 172.21.15.67:0/824514859 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcbd405fd70 con 0x7fcbe00ce520 2024-04-25T15:28:53.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.114+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbe006ffb0 con 0x7fcbe00ce520 2024-04-25T15:28:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.115+0000 7fcbdd7fa640 1 -- 172.21.15.67:0/824514859 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+12856 (secure 0 0 0) 0x7fcbd405f670 con 0x7fcbe00ce520 2024-04-25T15:28:53.117 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:53.117 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":88,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:52.373302+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbcc0744e0 msgr2=0x7fcbcc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbcc0744e0 0x7fcbcc0769f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0006fd0 tx=0x7fcbd0008040 comp rx=0 tx=0).stop 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00ce520 msgr2=0x7fcbe00cbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00ce520 0x7fcbe00cbce0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400d620 tx=0x7fcbd400dae0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 shutdown_connections 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcbcc0744e0 0x7fcbcc0769f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbe015e320 0x7fcbe0154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbe00ce520 0x7fcbe00cbce0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 --2- 172.21.15.67:0/824514859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbe00b7d80 0x7fcbe0150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 >> 172.21.15.67:0/824514859 conn(0x7fcbe009b7d0 msgr2=0x7fcbe015d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 shutdown_connections 2024-04-25T15:28:53.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.117+0000 7fcbe7140640 1 -- 172.21.15.67:0/824514859 wait complete. 2024-04-25T15:28:53.131 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:28:53.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:28:53.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.243+0000 7f497454d640 1 Processor -- start 2024-04-25T15:28:53.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.243+0000 7f497454d640 1 -- start start 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f497454d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f496c164ad0 0x7f496c0b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f497454d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f496c0d3170 con 0x7f496c0b0a70 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f497454d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f496c0d3450 con 0x7f496c163f30 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f497454d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f496c0d2f40 con 0x7f496c164ad0 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f4972aea640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f496c0b0a70 0x7f496c0b0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54562/0 (socket says 172.21.15.67:54562) 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f4972aea640 1 -- 172.21.15.67:0/3664938267 learned_addr learned my addr 172.21.15.67:0/3664938267 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49722e9640 1 --2- 172.21.15.67:0/3664938267 >> v2:172.21.15.176:3300/0 conn(0x7f496c164ad0 0x7f496c0b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177015791 0 0) 0x7f496c0d3170 con 0x7f496c0b0a70 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4954003520 con 0x7f496c0b0a70 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4235171824 0 0) 0x7f4954003520 con 0x7f496c0b0a70 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 >> v2:172.21.15.176:3300/0 conn(0x7f496c164ad0 msgr2=0x7f496c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:53.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 --2- 172.21.15.67:0/3664938267 >> v2:172.21.15.176:3300/0 conn(0x7f496c164ad0 0x7f496c0b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 >> v1:172.21.15.176:6789/0 conn(0x7f496c163f30 legacy=0x7f496c164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c06d6d0 con 0x7f496c0b0a70 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f495c0019f0 con 0x7f496c0b0a70 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.245+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 >> v1:172.21.15.67:6789/0 conn(0x7f496c0b0a70 legacy=0x7f496c0b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f49712e7640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49540039b0 0x7f4954003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f49712e7640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4954004800 0x7f4954004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f49712e7640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f495400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f495400c130 con 0x7f49540051c0 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f495400c2e0 con 0x7f4954004800 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f495400c4e0 con 0x7f49540039b0 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f4972aea640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f495400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f4971ae8640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4954004800 0x7f4954004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f49722e9640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49540039b0 0x7f4954003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f4972aea640 1 -- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49540039b0 msgr2=0x7f4954003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f4972aea640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49540039b0 0x7f4954003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f4972aea640 1 -- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4954004800 msgr2=0x7f4954004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f4972aea640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4954004800 0x7f4954004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.246+0000 7f4972aea640 1 -- 172.21.15.67:0/3664938267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4954003520 con 0x7f49540051c0 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.247+0000 7f4972aea640 1 -- 172.21.15.67:0/3664938267 reap_dead start 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.247+0000 7f4972aea640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f495400b9c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f495c00a850 tx=0x7f4954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40561d00a9f2018 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.247+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c00a4a0 con 0x7f49540051c0 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.247+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f495c02f070 con 0x7f49540051c0 2024-04-25T15:28:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.247+0000 7f49712e7640 1 -- 172.21.15.67:0/3664938267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c02bc20 con 0x7f49540051c0 2024-04-25T15:28:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.247+0000 7f497454d640 1 -- 172.21.15.67:0/3664938267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 msgr2=0x7f495400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.247+0000 7f497454d640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f495400b9c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f495c00a850 tx=0x7f4954005830 comp rx=0 tx=0).stop 2024-04-25T15:28:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.248+0000 7f497454d640 1 -- 172.21.15.67:0/3664938267 shutdown_connections 2024-04-25T15:28:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.248+0000 7f497454d640 1 --2- 172.21.15.67:0/3664938267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f495400b9c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.248+0000 7f497454d640 1 -- 172.21.15.67:0/3664938267 >> 172.21.15.67:0/3664938267 conn(0x7f496c0d3aa0 msgr2=0x7f496c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.248+0000 7f497454d640 1 -- 172.21.15.67:0/3664938267 shutdown_connections 2024-04-25T15:28:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.248+0000 7f497454d640 1 -- 172.21.15.67:0/3664938267 wait complete. 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.249+0000 7f497454d640 1 Processor -- start 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.249+0000 7f497454d640 1 -- start start 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f497454d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f496c1596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f497454d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f496c0cd280 0x7f496c0cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f497454d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f496c0cd690 0x7f496c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f497454d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f496c0b9430 con 0x7f49540051c0 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f497454d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f496c164ee0 con 0x7f496c0cd280 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f497454d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f496c0b95b0 con 0x7f496c0cd690 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f496c1596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f496c1596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42062/0 (socket says 172.21.15.67:42062) 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 -- 172.21.15.67:0/1561997750 learned_addr learned my addr 172.21.15.67:0/1561997750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f4972aea640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f496c0cd690 0x7f496c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f4971ae8640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f496c0cd280 0x7f496c0cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 -- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f496c0cd690 msgr2=0x7f496c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f496c0cd690 0x7f496c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 -- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f496c0cd280 msgr2=0x7f496c0cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f496c0cd280 0x7f496c0cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 -- 172.21.15.67:0/1561997750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c0caae0 con 0x7f49540051c0 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.251+0000 7f49722e9640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f496c1596b0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4964011930 tx=0x7f4964011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.252+0000 7f496affd640 1 -- 172.21.15.67:0/1561997750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496400fa30 con 0x7f49540051c0 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.252+0000 7f496affd640 1 -- 172.21.15.67:0/1561997750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496400fbd0 con 0x7f49540051c0 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.252+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c0cafe0 con 0x7f49540051c0 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.252+0000 7f496affd640 1 -- 172.21.15.67:0/1561997750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964014440 con 0x7f49540051c0 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.252+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c0cb5a0 con 0x7f49540051c0 2024-04-25T15:28:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.253+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496c06ecf0 con 0x7f49540051c0 2024-04-25T15:28:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.253+0000 7f496affd640 1 -- 172.21.15.67:0/1561997750 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4964012040 con 0x7f49540051c0 2024-04-25T15:28:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.254+0000 7f496affd640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49580744b0 0x7f49580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.254+0000 7f496affd640 1 -- 172.21.15.67:0/1561997750 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7f49640952f0 con 0x7f49540051c0 2024-04-25T15:28:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.254+0000 7f4971ae8640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49580744b0 0x7f49580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.255+0000 7f4971ae8640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49580744b0 0x7f49580769c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f496c0b25d0 tx=0x7f4960009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.259+0000 7f496affd640 1 -- 172.21.15.67:0/1561997750 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4964019030 con 0x7f49540051c0 2024-04-25T15:28:53.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.445+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f496c0d4770 con 0x7f49540051c0 2024-04-25T15:28:53.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.445+0000 7f496affd640 1 -- 172.21.15.67:0/1561997750 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f496405f810 con 0x7f49540051c0 2024-04-25T15:28:53.447 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49580744b0 msgr2=0x7f49580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49580744b0 0x7f49580769c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f496c0b25d0 tx=0x7f4960009040 comp rx=0 tx=0).stop 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 msgr2=0x7f496c1596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f496c1596b0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4964011930 tx=0x7f4964011df0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 shutdown_connections 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49580744b0 0x7f49580769c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f496c0cd690 0x7f496c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f496c0cd280 0x7f496c0cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 --2- 172.21.15.67:0/1561997750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49540051c0 0x7f496c1596b0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 >> 172.21.15.67:0/1561997750 conn(0x7f496c0d3aa0 msgr2=0x7f496c06e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 shutdown_connections 2024-04-25T15:28:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.447+0000 7f497454d640 1 -- 172.21.15.67:0/1561997750 wait complete. 2024-04-25T15:28:53.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:28:53.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.571+0000 7f61fed8f640 1 Processor -- start 2024-04-25T15:28:53.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.571+0000 7f61fed8f640 1 -- start start 2024-04-25T15:28:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fed8f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f61f806ccd0 0x7f61f8164040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fed8f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61f8164580 0x7f61f80d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fed8f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f8164db0 con 0x7f61f815abd0 2024-04-25T15:28:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fed8f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f61f8161380 con 0x7f61f8164580 2024-04-25T15:28:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fed8f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f61f8160d30 con 0x7f61f806ccd0 2024-04-25T15:28:53.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fd32c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61f8164580 0x7f61f80d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fcb2b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f61f815abd0 0x7f61f806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54578/0 (socket says 172.21.15.67:54578) 2024-04-25T15:28:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fcb2b640 1 -- 172.21.15.67:0/2265002642 learned_addr learned my addr 172.21.15.67:0/2265002642 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61f7fff640 1 --2- 172.21.15.67:0/2265002642 >> v2:172.21.15.176:3300/0 conn(0x7f61f806ccd0 0x7f61f8164040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fd32c640 1 -- 172.21.15.67:0/2265002642 >> v2:172.21.15.176:3300/0 conn(0x7f61f806ccd0 msgr2=0x7f61f8164040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fd32c640 1 --2- 172.21.15.67:0/2265002642 >> v2:172.21.15.176:3300/0 conn(0x7f61f806ccd0 0x7f61f8164040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fd32c640 1 -- 172.21.15.67:0/2265002642 >> v1:172.21.15.67:6789/0 conn(0x7f61f815abd0 legacy=0x7f61f806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61f77fe640 1 -- 172.21.15.67:0/2265002642 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1039117772 0 0) 0x7f61f8164db0 con 0x7f61f815abd0 2024-04-25T15:28:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61fd32c640 1 -- 172.21.15.67:0/2265002642 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f8161010 con 0x7f61f8164580 2024-04-25T15:28:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.573+0000 7f61f7fff640 1 --2- 172.21.15.67:0/2265002642 >> v2:172.21.15.176:3300/0 conn(0x7f61f806ccd0 0x7f61f8164040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.574+0000 7f61fd32c640 1 --2- 172.21.15.67:0/2265002642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f8164580 0x7f61f80d4100 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f61e80097a0 tx=0x7f61e802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d88cc296a2fe6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.574+0000 7f61f77fe640 1 -- 172.21.15.67:0/2265002642 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8037070 con 0x7f61f8164580 2024-04-25T15:28:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.574+0000 7f61f77fe640 1 -- 172.21.15.67:0/2265002642 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e8032440 con 0x7f61f8164580 2024-04-25T15:28:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.574+0000 7f61f77fe640 1 -- 172.21.15.67:0/2265002642 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8032810 con 0x7f61f8164580 2024-04-25T15:28:53.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 -- 172.21.15.67:0/2265002642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f8164580 msgr2=0x7f61f80d4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 --2- 172.21.15.67:0/2265002642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f8164580 0x7f61f80d4100 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f61e80097a0 tx=0x7f61e802a1e0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 -- 172.21.15.67:0/2265002642 shutdown_connections 2024-04-25T15:28:53.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 --2- 172.21.15.67:0/2265002642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f8164580 0x7f61f80d4100 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 --2- 172.21.15.67:0/2265002642 >> v2:172.21.15.176:3300/0 conn(0x7f61f806ccd0 0x7f61f8164040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 -- 172.21.15.67:0/2265002642 >> 172.21.15.67:0/2265002642 conn(0x7f61f80d5df0 msgr2=0x7f61f80d6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:53.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 -- 172.21.15.67:0/2265002642 shutdown_connections 2024-04-25T15:28:53.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.575+0000 7f61fed8f640 1 -- 172.21.15.67:0/2265002642 wait complete. 2024-04-25T15:28:53.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.577+0000 7f61fed8f640 1 Processor -- start 2024-04-25T15:28:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.577+0000 7f61fed8f640 1 -- start start 2024-04-25T15:28:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fed8f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61f815abd0 0x7f61f815fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fed8f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61f8164580 0x7f61f80b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fed8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f80b84a0 0x7f61f80b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fed8f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61f8161810 con 0x7f61f80b84a0 2024-04-25T15:28:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fed8f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f61f8164990 con 0x7f61f8164580 2024-04-25T15:28:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fed8f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f61f80b6570 con 0x7f61f815abd0 2024-04-25T15:28:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fd32c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f80b84a0 0x7f61f80b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fd32c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f80b84a0 0x7f61f80b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42082/0 (socket says 172.21.15.67:42082) 2024-04-25T15:28:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.578+0000 7f61fd32c640 1 -- 172.21.15.67:0/3895955253 learned_addr learned my addr 172.21.15.67:0/3895955253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fcb2b640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61f815abd0 0x7f61f815fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fd32c640 1 -- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61f815abd0 msgr2=0x7f61f815fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61f7fff640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61f8164580 0x7f61f80b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fd32c640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61f815abd0 0x7f61f815fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fd32c640 1 -- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61f8164580 msgr2=0x7f61f80b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fd32c640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61f8164580 0x7f61f80b7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fd32c640 1 -- 172.21.15.67:0/3895955253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f81f6d90 con 0x7f61f80b84a0 2024-04-25T15:28:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fd32c640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f80b84a0 0x7f61f80b5850 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f61e8035040 tx=0x7f61e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61f57fa640 1 -- 172.21.15.67:0/3895955253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8037040 con 0x7f61f80b84a0 2024-04-25T15:28:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61f57fa640 1 -- 172.21.15.67:0/3895955253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e8042070 con 0x7f61f80b84a0 2024-04-25T15:28:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f815eff0 con 0x7f61f80b84a0 2024-04-25T15:28:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61f57fa640 1 -- 172.21.15.67:0/3895955253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8030b50 con 0x7f61f80b84a0 2024-04-25T15:28:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.579+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f815f5b0 con 0x7f61f80b84a0 2024-04-25T15:28:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.580+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f8050740 con 0x7f61f80b84a0 2024-04-25T15:28:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.581+0000 7f61f57fa640 1 -- 172.21.15.67:0/3895955253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f61e8030cf0 con 0x7f61f80b84a0 2024-04-25T15:28:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.581+0000 7f61f57fa640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61d80744b0 0x7f61d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.582+0000 7f61f57fa640 1 -- 172.21.15.67:0/3895955253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7f61e80b5b50 con 0x7f61f80b84a0 2024-04-25T15:28:53.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.582+0000 7f61fcb2b640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61d80744b0 0x7f61d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.582+0000 7f61fcb2b640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61d80744b0 0x7f61d80769c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f61f00012e0 tx=0x7f61f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.586+0000 7f61f57fa640 1 -- 172.21.15.67:0/3895955253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61e8032e20 con 0x7f61f80b84a0 2024-04-25T15:28:53.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.772+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f61f80bbd70 con 0x7f61f80b84a0 2024-04-25T15:28:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.772+0000 7f61f57fa640 1 -- 172.21.15.67:0/3895955253 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v88) ==== 84+0+65 (secure 0 0 0) 0x7f61e8080070 con 0x7f61f80b84a0 2024-04-25T15:28:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:53.774 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:28:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:28:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:28:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:28:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:28:53.775 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61d80744b0 msgr2=0x7f61d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61d80744b0 0x7f61d80769c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f61f00012e0 tx=0x7f61f0009040 comp rx=0 tx=0).stop 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f80b84a0 msgr2=0x7f61f80b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f80b84a0 0x7f61f80b5850 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f61e8035040 tx=0x7f61e80046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 shutdown_connections 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61d80744b0 0x7f61d80769c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61f80b84a0 0x7f61f80b5850 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61f8164580 0x7f61f80b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 --2- 172.21.15.67:0/3895955253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61f815abd0 0x7f61f815fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 >> 172.21.15.67:0/3895955253 conn(0x7f61f80d5df0 msgr2=0x7f61f80d4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:53.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 shutdown_connections 2024-04-25T15:28:53.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.774+0000 7f61fed8f640 1 -- 172.21.15.67:0/3895955253 wait complete. 2024-04-25T15:28:53.787 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:28:53.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:28:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.902+0000 7f5b6c5f8640 1 Processor -- start 2024-04-25T15:28:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.902+0000 7f5b6c5f8640 1 -- start start 2024-04-25T15:28:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.903+0000 7f5b6c5f8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 0x7f5b640d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.903+0000 7f5b6c5f8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b640d1850 con 0x7f5b6406c5c0 2024-04-25T15:28:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.903+0000 7f5b6c5f8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b64160de0 con 0x7f5b640d4a50 2024-04-25T15:28:53.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.903+0000 7f5b6c5f8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5b641610c0 con 0x7f5b640d0e00 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69b93640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b6406c5c0 0x7f5b6406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54592/0 (socket says 172.21.15.67:54592) 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69b93640 1 -- 172.21.15.67:0/2289975992 learned_addr learned my addr 172.21.15.67:0/2289975992 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b6a394640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 0x7f5b640d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3657014078 0 0) 0x7f5b640d1850 con 0x7f5b6406c5c0 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b50003520 con 0x7f5b6406c5c0 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607967307 0 0) 0x7f5b64160de0 con 0x7f5b640d4a50 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b640d1850 con 0x7f5b640d4a50 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2295162731 0 0) 0x7f5b50003520 con 0x7f5b6406c5c0 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 msgr2=0x7f5b640d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 0x7f5b640d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 >> v1:172.21.15.176:6789/0 conn(0x7f5b640d4a50 legacy=0x7f5b640d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b641613a0 con 0x7f5b6406c5c0 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f5b54001670 con 0x7f5b6406c5c0 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b6a394640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 0x7f5b640d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 >> v1:172.21.15.67:6789/0 conn(0x7f5b6406c5c0 legacy=0x7f5b6406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50003b70 0x7f5b50003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b50004800 0x7f5b50004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 0x7f5b500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b641610c0 con 0x7f5b50004800 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5000baa0 con 0x7f5b50003b70 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5b5000bca0 con 0x7f5b640d0e00 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.904+0000 7f5b69b93640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b50004800 0x7f5b50004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b6a394640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50003b70 0x7f5b50003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b6ab95640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 0x7f5b500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b69b93640 1 -- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 msgr2=0x7f5b500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b69b93640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640d0e00 0x7f5b500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b69b93640 1 -- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50003b70 msgr2=0x7f5b50003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b69b93640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50003b70 0x7f5b50003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b69b93640 1 -- 172.21.15.67:0/2289975992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b50003520 con 0x7f5b50004800 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.905+0000 7f5b69b93640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b50004800 0x7f5b50004c10 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f5b54001eb0 tx=0x7f5b50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f06a0677c010e8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54007ce0 con 0x7f5b50004800 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b54005ad0 con 0x7f5b50004800 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b69392640 1 -- 172.21.15.67:0/2289975992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54007070 con 0x7f5b50004800 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/2289975992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b50004800 msgr2=0x7f5b50004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b6c5f8640 1 --2- 172.21.15.67:0/2289975992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b50004800 0x7f5b50004c10 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f5b54001eb0 tx=0x7f5b50005830 comp rx=0 tx=0).stop 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b6ab95640 1 -- 172.21.15.67:0/2289975992 reap_dead start 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/2289975992 shutdown_connections 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/2289975992 >> 172.21.15.67:0/2289975992 conn(0x7f5b640d5d90 msgr2=0x7f5b640d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/2289975992 shutdown_connections 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.906+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/2289975992 wait complete. 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.908+0000 7f5b6c5f8640 1 Processor -- start 2024-04-25T15:28:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.908+0000 7f5b6c5f8640 1 -- start start 2024-04-25T15:28:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.909+0000 7f5b6c5f8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50004800 0x7f5b640cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.909+0000 7f5b6c5f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641554d0 0x7f5b64154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.909+0000 7f5b6c5f8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b6415ef00 0x7f5b6415f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.909+0000 7f5b6c5f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b64164e00 con 0x7f5b641554d0 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.909+0000 7f5b6c5f8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5b640b5320 con 0x7f5b50004800 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.909+0000 7f5b6c5f8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5b64164f80 con 0x7f5b6415ef00 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641554d0 0x7f5b64154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641554d0 0x7f5b64154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42100/0 (socket says 172.21.15.67:42100) 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 -- 172.21.15.67:0/4219405032 learned_addr learned my addr 172.21.15.67:0/4219405032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b6a394640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50004800 0x7f5b640cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b6ab95640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b6415ef00 0x7f5b6415f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 -- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b6415ef00 msgr2=0x7f5b6415f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b6415ef00 0x7f5b6415f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 -- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50004800 msgr2=0x7f5b640cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50004800 0x7f5b640cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 -- 172.21.15.67:0/4219405032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6415f850 con 0x7f5b641554d0 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b69b93640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641554d0 0x7f5b64154d40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f5b5402ad60 tx=0x7f5b64161570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b5affd640 1 -- 172.21.15.67:0/4219405032 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54001670 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b5affd640 1 -- 172.21.15.67:0/4219405032 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b54042070 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b64162ff0 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b5affd640 1 -- 172.21.15.67:0/4219405032 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b540072b0 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.910+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b641635b0 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.911+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b640b8ff0 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.912+0000 7f5b5affd640 1 -- 172.21.15.67:0/4219405032 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5b54007450 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.912+0000 7f5b5affd640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5b480744b0 0x7f5b480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.913+0000 7f5b5affd640 1 -- 172.21.15.67:0/4219405032 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7f5b540b4e40 con 0x7f5b641554d0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.913+0000 7f5b6a394640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5b480744b0 0x7f5b480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:53.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.913+0000 7f5b6a394640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5b480744b0 0x7f5b480769c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009560 tx=0x7f5b60008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:53.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:53.917+0000 7f5b5affd640 1 -- 172.21.15.67:0/4219405032 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b640b8ff0 con 0x7f5b641554d0 2024-04-25T15:28:54.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:54.102+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5b64050740 con 0x7f5b641554d0 2024-04-25T15:28:55.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.027+0000 7f5b5affd640 1 -- 172.21.15.67:0/4219405032 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v89) ==== 138+0+0 (secure 0 0 0) 0x7f5b6415f850 con 0x7f5b641554d0 2024-04-25T15:28:55.029 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:28:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.032+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5b480744b0 msgr2=0x7f5b480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.032+0000 7f5b6c5f8640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5b480744b0 0x7f5b480769c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009560 tx=0x7f5b60008040 comp rx=0 tx=0).stop 2024-04-25T15:28:55.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.032+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641554d0 msgr2=0x7f5b64154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.032+0000 7f5b6c5f8640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641554d0 0x7f5b64154d40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f5b5402ad60 tx=0x7f5b64161570 comp rx=0 tx=0).stop 2024-04-25T15:28:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 shutdown_connections 2024-04-25T15:28:55.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5b480744b0 0x7f5b480769c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b6415ef00 0x7f5b6415f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641554d0 0x7f5b64154d40 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 --2- 172.21.15.67:0/4219405032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b50004800 0x7f5b640cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 >> 172.21.15.67:0/4219405032 conn(0x7f5b640d5d90 msgr2=0x7f5b640b5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 shutdown_connections 2024-04-25T15:28:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.035+0000 7f5b6c5f8640 1 -- 172.21.15.67:0/4219405032 wait complete. 2024-04-25T15:28:55.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:28:55.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.175+0000 7fc0ca24c640 1 Processor -- start 2024-04-25T15:28:55.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.175+0000 7fc0ca24c640 1 -- start start 2024-04-25T15:28:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0ca24c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c4164640 0x7fc0c40b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0ca24c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c40b0a70 0x7fc0c40b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0ca24c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c40d3170 con 0x7fc0c4164ad0 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0ca24c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc0c40d2f40 con 0x7fc0c4164640 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0ca24c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c4164ee0 con 0x7fc0c40b0a70 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c4164640 0x7fc0c40b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc0c4164ad0 0x7fc0c4163f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54600/0 (socket says 172.21.15.67:54600) 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c37fe640 1 -- 172.21.15.67:0/2441278757 learned_addr learned my addr 172.21.15.67:0/2441278757 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c3fff640 1 --2- 172.21.15.67:0/2441278757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c40b0a70 0x7fc0c40b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c2ffd640 1 -- 172.21.15.67:0/2441278757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c40b0a70 msgr2=0x7fc0c40b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c2ffd640 1 --2- 172.21.15.67:0/2441278757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c40b0a70 0x7fc0c40b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c2ffd640 1 -- 172.21.15.67:0/2441278757 >> v1:172.21.15.67:6789/0 conn(0x7fc0c4164ad0 legacy=0x7fc0c4163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c2ffd640 1 -- 172.21.15.67:0/2441278757 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c40d3450 con 0x7fc0c4164640 2024-04-25T15:28:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.176+0000 7fc0c3fff640 1 --2- 172.21.15.67:0/2441278757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c40b0a70 0x7fc0c40b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:55.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.177+0000 7fc0c2ffd640 1 --2- 172.21.15.67:0/2441278757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c4164640 0x7fc0c40b2ca0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80097a0 tx=0x7fc0b802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa5b324fe931b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:55.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.177+0000 7fc0c27fc640 1 -- 172.21.15.67:0/2441278757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8037070 con 0x7fc0c4164640 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.177+0000 7fc0c27fc640 1 -- 172.21.15.67:0/2441278757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b8032440 con 0x7fc0c4164640 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.177+0000 7fc0c27fc640 1 -- 172.21.15.67:0/2441278757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8032810 con 0x7fc0c4164640 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 -- 172.21.15.67:0/2441278757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c4164640 msgr2=0x7fc0c40b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/2441278757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c4164640 0x7fc0c40b2ca0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80097a0 tx=0x7fc0b802a180 comp rx=0 tx=0).stop 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 -- 172.21.15.67:0/2441278757 shutdown_connections 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/2441278757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c40b0a70 0x7fc0c40b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/2441278757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c4164640 0x7fc0c40b2ca0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 -- 172.21.15.67:0/2441278757 >> 172.21.15.67:0/2441278757 conn(0x7fc0c40d3aa0 msgr2=0x7fc0c40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 -- 172.21.15.67:0/2441278757 shutdown_connections 2024-04-25T15:28:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.178+0000 7fc0ca24c640 1 -- 172.21.15.67:0/2441278757 wait complete. 2024-04-25T15:28:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.180+0000 7fc0ca24c640 1 Processor -- start 2024-04-25T15:28:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.180+0000 7fc0ca24c640 1 -- start start 2024-04-25T15:28:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.181+0000 7fc0ca24c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c40b0a70 0x7fc0c40c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.181+0000 7fc0ca24c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c4164ad0 0x7fc0c40c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.181+0000 7fc0ca24c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c4151860 0x7fc0c40ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.181+0000 7fc0ca24c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c406d9b0 con 0x7fc0c40b0a70 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.181+0000 7fc0ca24c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c40b31e0 con 0x7fc0c4164ad0 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.181+0000 7fc0ca24c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c40b9430 con 0x7fc0c4151860 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c40b0a70 0x7fc0c40c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c40b0a70 0x7fc0c40c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42108/0 (socket says 172.21.15.67:42108) 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 -- 172.21.15.67:0/1860430428 learned_addr learned my addr 172.21.15.67:0/1860430428 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c3fff640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c4151860 0x7fc0c40ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c2ffd640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c4164ad0 0x7fc0c40c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 -- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c4151860 msgr2=0x7fc0c40ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c4151860 0x7fc0c40ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 -- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c4164ad0 msgr2=0x7fc0c40c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c4164ad0 0x7fc0c40c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 -- 172.21.15.67:0/1860430428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c41ea740 con 0x7fc0c40b0a70 2024-04-25T15:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0c37fe640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c40b0a70 0x7fc0c40c7540 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c3c0 tx=0x7fc0b400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0abfff640 1 -- 172.21.15.67:0/1860430428 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4017070 con 0x7fc0c40b0a70 2024-04-25T15:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0abfff640 1 -- 172.21.15.67:0/1860430428 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b4005ce0 con 0x7fc0c40b0a70 2024-04-25T15:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0abfff640 1 -- 172.21.15.67:0/1860430428 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b40072f0 con 0x7fc0c40b0a70 2024-04-25T15:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c41eaa20 con 0x7fc0c40b0a70 2024-04-25T15:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.182+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c40ca080 con 0x7fc0c40b0a70 2024-04-25T15:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.184+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c40d4770 con 0x7fc0c40b0a70 2024-04-25T15:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.184+0000 7fc0abfff640 1 -- 172.21.15.67:0/1860430428 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc0b401a070 con 0x7fc0c40b0a70 2024-04-25T15:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.185+0000 7fc0abfff640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc0b00744e0 0x7fc0b00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.185+0000 7fc0abfff640 1 -- 172.21.15.67:0/1860430428 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5227+0+0 (secure 0 0 0) 0x7fc0b4094340 con 0x7fc0c40b0a70 2024-04-25T15:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.185+0000 7fc0c2ffd640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc0b00744e0 0x7fc0b00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.185+0000 7fc0c2ffd640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc0b00744e0 0x7fc0b00769f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8034000 tx=0x7fc0b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:55.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.190+0000 7fc0abfff640 1 -- 172.21.15.67:0/1860430428 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0b405edd0 con 0x7fc0c40b0a70 2024-04-25T15:28:55.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:55.380+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc0c4068bd0 con 0x7fc0c40b0a70 2024-04-25T15:28:56.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.030+0000 7fc0abfff640 1 -- 172.21.15.67:0/1860430428 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v90) ==== 170+0+0 (secure 0 0 0) 0x7fc0b405e6d0 con 0x7fc0c40b0a70 2024-04-25T15:28:56.031 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:28:56.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc0b00744e0 msgr2=0x7fc0b00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:56.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc0b00744e0 0x7fc0b00769f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8034000 tx=0x7fc0b802e000 comp rx=0 tx=0).stop 2024-04-25T15:28:56.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c40b0a70 msgr2=0x7fc0c40c7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c40b0a70 0x7fc0c40c7540 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c3c0 tx=0x7fc0b400c7c0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 shutdown_connections 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc0b00744e0 0x7fc0b00769f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c4151860 0x7fc0c40ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c4164ad0 0x7fc0c40c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 --2- 172.21.15.67:0/1860430428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c40b0a70 0x7fc0c40c7540 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.032+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 >> 172.21.15.67:0/1860430428 conn(0x7fc0c40d3aa0 msgr2=0x7fc0c40d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.033+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 shutdown_connections 2024-04-25T15:28:56.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.033+0000 7fc0ca24c640 1 -- 172.21.15.67:0/1860430428 wait complete. 2024-04-25T15:28:56.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:28:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.177+0000 7f440c630640 1 Processor -- start 2024-04-25T15:28:56.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.177+0000 7f440c630640 1 -- start start 2024-04-25T15:28:56.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f440c630640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f440406ccd0 0x7f440415e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:56.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f440c630640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f44041643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:56.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f440c630640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44041609b0 con 0x7f4404161710 2024-04-25T15:28:56.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f440c630640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4404164cd0 con 0x7f440406ccd0 2024-04-25T15:28:56.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f440c630640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4404160c30 con 0x7f4404163fa0 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f4409bcb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f440406ccd0 0x7f440415e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f440a3cc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4404161710 0x7f440406c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54602/0 (socket says 172.21.15.67:54602) 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.178+0000 7f440a3cc640 1 -- 172.21.15.67:0/3694882220 learned_addr learned my addr 172.21.15.67:0/3694882220 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.179+0000 7f440abcd640 1 --2- 172.21.15.67:0/3694882220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f44041643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.179+0000 7f4409bcb640 1 -- 172.21.15.67:0/3694882220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 msgr2=0x7f44041643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.179+0000 7f4409bcb640 1 --2- 172.21.15.67:0/3694882220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f44041643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.179+0000 7f4409bcb640 1 -- 172.21.15.67:0/3694882220 >> v1:172.21.15.67:6789/0 conn(0x7f4404161710 legacy=0x7f440406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.179+0000 7f4409bcb640 1 -- 172.21.15.67:0/3694882220 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4404160f10 con 0x7f440406ccd0 2024-04-25T15:28:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.179+0000 7f44093ca640 1 -- 172.21.15.67:0/3694882220 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4140087306 0 0) 0x7f4400001110 con 0x7f4404161710 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.179+0000 7f4409bcb640 1 --2- 172.21.15.67:0/3694882220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440406ccd0 0x7f440415e080 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f43f40097a0 tx=0x7f43f402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cedefea1e9570b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f44093ca640 1 -- 172.21.15.67:0/3694882220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4037070 con 0x7f440406ccd0 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f44093ca640 1 -- 172.21.15.67:0/3694882220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f402ae50 con 0x7f440406ccd0 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f44093ca640 1 -- 172.21.15.67:0/3694882220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4032660 con 0x7f440406ccd0 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 -- 172.21.15.67:0/3694882220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440406ccd0 msgr2=0x7f440415e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 --2- 172.21.15.67:0/3694882220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440406ccd0 0x7f440415e080 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f43f40097a0 tx=0x7f43f402a100 comp rx=0 tx=0).stop 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 -- 172.21.15.67:0/3694882220 shutdown_connections 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 --2- 172.21.15.67:0/3694882220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f44041643b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 --2- 172.21.15.67:0/3694882220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440406ccd0 0x7f440415e080 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 -- 172.21.15.67:0/3694882220 >> 172.21.15.67:0/3694882220 conn(0x7f44040d5d90 msgr2=0x7f44040d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 -- 172.21.15.67:0/3694882220 shutdown_connections 2024-04-25T15:28:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.180+0000 7f440c630640 1 -- 172.21.15.67:0/3694882220 wait complete. 2024-04-25T15:28:56.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.182+0000 7f440c630640 1 Processor -- start 2024-04-25T15:28:56.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.182+0000 7f440c630640 1 -- start start 2024-04-25T15:28:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.183+0000 7f440c630640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4404161710 0x7f4404158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.183+0000 7f440c630640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f4404153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.183+0000 7f440c630640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44041576f0 0x7f4404161fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.183+0000 7f440c630640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4404164a70 con 0x7f44041576f0 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.183+0000 7f440c630640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f44041648f0 con 0x7f4404161710 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.183+0000 7f440c630640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44040d49f0 con 0x7f4404163fa0 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44041576f0 0x7f4404161fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44041576f0 0x7f4404161fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42132/0 (socket says 172.21.15.67:42132) 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 -- 172.21.15.67:0/2252675299 learned_addr learned my addr 172.21.15.67:0/2252675299 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f4409bcb640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f4404153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440a3cc640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4404161710 0x7f4404158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 -- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 msgr2=0x7f4404153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f4404153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 -- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4404161710 msgr2=0x7f4404158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4404161710 0x7f4404158550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 -- 172.21.15.67:0/2252675299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44040cd0b0 con 0x7f44041576f0 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f4409bcb640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f4404153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440abcd640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44041576f0 0x7f4404161fb0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f43fc007d70 tx=0x7f43fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f43faffd640 1 -- 172.21.15.67:0/2252675299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc014420 con 0x7f44041576f0 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f43faffd640 1 -- 172.21.15.67:0/2252675299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43fc019070 con 0x7f44041576f0 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f43faffd640 1 -- 172.21.15.67:0/2252675299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc014820 con 0x7f44041576f0 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44040cd390 con 0x7f44041576f0 2024-04-25T15:28:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.184+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44040cd950 con 0x7f44041576f0 2024-04-25T15:28:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.185+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4404050740 con 0x7f44041576f0 2024-04-25T15:28:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.186+0000 7f43faffd640 1 -- 172.21.15.67:0/2252675299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f43fc0149c0 con 0x7f44041576f0 2024-04-25T15:28:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.187+0000 7f43faffd640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43e80744e0 0x7f43e80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.187+0000 7f43faffd640 1 -- 172.21.15.67:0/2252675299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5609+0+0 (secure 0 0 0) 0x7f43fc096780 con 0x7f44041576f0 2024-04-25T15:28:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.187+0000 7f440a3cc640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43e80744e0 0x7f43e80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.187+0000 7f440a3cc640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43e80744e0 0x7f43e80769f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4400000f60 tx=0x7f4400009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.191+0000 7f43faffd640 1 -- 172.21.15.67:0/2252675299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43fc061050 con 0x7f44041576f0 2024-04-25T15:28:56.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:56.416+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f44040cce20 con 0x7f44041576f0 2024-04-25T15:28:57.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.061+0000 7f43faffd640 1 -- 172.21.15.67:0/2252675299 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v53) ==== 174+0+0 (secure 0 0 0) 0x7f43fc060950 con 0x7f44041576f0 2024-04-25T15:28:57.063 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 14 and data pool 15 2024-04-25T15:28:57.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.064+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43e80744e0 msgr2=0x7f43e80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.064+0000 7f440c630640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43e80744e0 0x7f43e80769f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4400000f60 tx=0x7f4400009040 comp rx=0 tx=0).stop 2024-04-25T15:28:57.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.064+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44041576f0 msgr2=0x7f4404161fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.064+0000 7f440c630640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44041576f0 0x7f4404161fb0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f43fc007d70 tx=0x7f43fc00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.068+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 shutdown_connections 2024-04-25T15:28:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.068+0000 7f440c630640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f43e80744e0 0x7f43e80769f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.068+0000 7f440c630640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44041576f0 0x7f4404161fb0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.068+0000 7f440c630640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4404163fa0 0x7f4404153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.068+0000 7f440c630640 1 --2- 172.21.15.67:0/2252675299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4404161710 0x7f4404158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.068+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 >> 172.21.15.67:0/2252675299 conn(0x7f44040d5d90 msgr2=0x7f440415fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:57.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.069+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 shutdown_connections 2024-04-25T15:28:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.070+0000 7f440c630640 1 -- 172.21.15.67:0/2252675299 wait complete. 2024-04-25T15:28:57.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.207+0000 7ff5a3f84640 1 Processor -- start 2024-04-25T15:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.207+0000 7ff5a3f84640 1 -- start start 2024-04-25T15:28:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.208+0000 7ff5a3f84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 0x7ff59c0d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.208+0000 7ff5a3f84640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59c06f3e0 con 0x7ff59c1434d0 2024-04-25T15:28:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.208+0000 7ff5a3f84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59c06f6c0 con 0x7ff59c160be0 2024-04-25T15:28:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.208+0000 7ff5a3f84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff59c1438e0 con 0x7ff59c0d0ed0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.208+0000 7ff5a1d20640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff59c1434d0 0x7ff59c070480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54616/0 (socket says 172.21.15.67:54616) 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.208+0000 7ff5a1d20640 1 -- 172.21.15.67:0/686133893 learned_addr learned my addr 172.21.15.67:0/686133893 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a151f640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 0x7ff59c0d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1977107367 0 0) 0x7ff59c06f3e0 con 0x7ff59c1434d0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff588003520 con 0x7ff59c1434d0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2234314420 0 0) 0x7ff59c06f6c0 con 0x7ff59c160be0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff59c06f3e0 con 0x7ff59c160be0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2272610089 0 0) 0x7ff588003520 con 0x7ff59c1434d0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 msgr2=0x7ff59c0d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 0x7ff59c0d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 >> v1:172.21.15.176:6789/0 conn(0x7ff59c160be0 legacy=0x7ff59c160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59c06f9a0 con 0x7ff59c1434d0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7ff598001670 con 0x7ff59c1434d0 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 >> v1:172.21.15.67:6789/0 conn(0x7ff59c1434d0 legacy=0x7ff59c070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff588003b70 0x7ff588003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff588004800 0x7ff588004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 0x7ff588005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c1438e0 con 0x7ff588004800 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff58800bb50 con 0x7ff588003b70 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff58800bd00 con 0x7ff59c0d0ed0 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a151f640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff588004800 0x7ff588004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a1d20640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff588003b70 0x7ff588003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.209+0000 7ff5a2521640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 0x7ff588005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a151f640 1 -- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 msgr2=0x7ff588005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a151f640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 0x7ff588005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a151f640 1 -- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff588003b70 msgr2=0x7ff588003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a151f640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff588003b70 0x7ff588003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a151f640 1 -- 172.21.15.67:0/686133893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff588003520 con 0x7ff588004800 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a2521640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff59c0d0ed0 0x7ff588005240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a151f640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff588004800 0x7ff588004c10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009a90 tx=0x7ff588005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede4bfb4523c8dd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c036070 con 0x7ff588004800 2024-04-25T15:28:57.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff58c031400 con 0x7ff588004800 2024-04-25T15:28:57.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.210+0000 7ff5a0d1e640 1 -- 172.21.15.67:0/686133893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c009690 con 0x7ff588004800 2024-04-25T15:28:57.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.211+0000 7ff5a3f84640 1 -- 172.21.15.67:0/686133893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff588004800 msgr2=0x7ff588004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.211+0000 7ff5a3f84640 1 --2- 172.21.15.67:0/686133893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff588004800 0x7ff588004c10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009a90 tx=0x7ff588005830 comp rx=0 tx=0).stop 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.211+0000 7ff5a2521640 1 -- 172.21.15.67:0/686133893 reap_dead start 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.211+0000 7ff5a3f84640 1 -- 172.21.15.67:0/686133893 shutdown_connections 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.211+0000 7ff5a3f84640 1 -- 172.21.15.67:0/686133893 >> 172.21.15.67:0/686133893 conn(0x7ff59c0d1ae0 msgr2=0x7ff59c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.211+0000 7ff5a3f84640 1 -- 172.21.15.67:0/686133893 shutdown_connections 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.211+0000 7ff5a3f84640 1 -- 172.21.15.67:0/686133893 wait complete. 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.213+0000 7ff5a3f84640 1 Processor -- start 2024-04-25T15:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.213+0000 7ff5a3f84640 1 -- start start 2024-04-25T15:28:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.214+0000 7ff5a3f84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff588004800 0x7ff59c15a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.214+0000 7ff5a3f84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff59c15ca70 0x7ff59c1e81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.214+0000 7ff5a3f84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff59c1384f0 0x7ff59c13ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.214+0000 7ff5a3f84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c0d12e0 con 0x7ff59c1384f0 2024-04-25T15:28:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.214+0000 7ff5a3f84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c06f090 con 0x7ff59c15ca70 2024-04-25T15:28:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.214+0000 7ff5a3f84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff59c0d1460 con 0x7ff588004800 2024-04-25T15:28:57.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff59c1384f0 0x7ff59c13ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff59c1384f0 0x7ff59c13ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42154/0 (socket says 172.21.15.67:42154) 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 -- 172.21.15.67:0/3721898906 learned_addr learned my addr 172.21.15.67:0/3721898906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a151f640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff59c15ca70 0x7ff59c1e81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a1d20640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff588004800 0x7ff59c15a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 -- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff588004800 msgr2=0x7ff59c15a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff588004800 0x7ff59c15a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 -- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff59c15ca70 msgr2=0x7ff59c1e81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff59c15ca70 0x7ff59c1e81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 -- 172.21.15.67:0/3721898906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59c15ad50 con 0x7ff59c1384f0 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a1d20640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff588004800 0x7ff59c15a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a151f640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff59c15ca70 0x7ff59c1e81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a2521640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff59c1384f0 0x7ff59c13ac70 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff59c072e50 tx=0x7ff59400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5927fc640 1 -- 172.21.15.67:0/3721898906 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594019070 con 0x7ff59c1384f0 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5927fc640 1 -- 172.21.15.67:0/3721898906 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff59400ee50 con 0x7ff59c1384f0 2024-04-25T15:28:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5927fc640 1 -- 172.21.15.67:0/3721898906 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594003dc0 con 0x7ff59c1384f0 2024-04-25T15:28:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59c15b030 con 0x7ff59c1384f0 2024-04-25T15:28:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.215+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59c15b5f0 con 0x7ff59c1384f0 2024-04-25T15:28:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.216+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59c15aac0 con 0x7ff59c1384f0 2024-04-25T15:28:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.217+0000 7ff5927fc640 1 -- 172.21.15.67:0/3721898906 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff59401c070 con 0x7ff59c1384f0 2024-04-25T15:28:57.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.218+0000 7ff5927fc640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5800744b0 0x7ff5800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.218+0000 7ff5927fc640 1 -- 172.21.15.67:0/3721898906 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5709+0+0 (secure 0 0 0) 0x7ff594094410 con 0x7ff59c1384f0 2024-04-25T15:28:57.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.218+0000 7ff5a1d20640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5800744b0 0x7ff5800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.218+0000 7ff5a1d20640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5800744b0 0x7ff5800769c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff598007c20 tx=0x7ff598009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.223+0000 7ff5927fc640 1 -- 172.21.15.67:0/3721898906 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff59405e5e0 con 0x7ff59c1384f0 2024-04-25T15:28:57.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.410+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff59c050740 con 0x7ff59c1384f0 2024-04-25T15:28:57.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.411+0000 7ff5927fc640 1 -- 172.21.15.67:0/3721898906 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16585 (secure 0 0 0) 0x7ff5940628e0 con 0x7ff59c1384f0 2024-04-25T15:28:57.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:57.412 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":91,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:56.417940+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.413+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5800744b0 msgr2=0x7ff5800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5800744b0 0x7ff5800769c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff598007c20 tx=0x7ff598009040 comp rx=0 tx=0).stop 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff59c1384f0 msgr2=0x7ff59c13ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff59c1384f0 0x7ff59c13ac70 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff59c072e50 tx=0x7ff59400ec00 comp rx=0 tx=0).stop 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 shutdown_connections 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff5800744b0 0x7ff5800769c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff59c1384f0 0x7ff59c13ac70 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff59c15ca70 0x7ff59c1e81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 --2- 172.21.15.67:0/3721898906 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff588004800 0x7ff59c15a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 >> 172.21.15.67:0/3721898906 conn(0x7ff59c0d1ae0 msgr2=0x7ff59c0d0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 shutdown_connections 2024-04-25T15:28:57.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.414+0000 7ff5a3f84640 1 -- 172.21.15.67:0/3721898906 wait complete. 2024-04-25T15:28:57.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:28:57.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.537+0000 7faac0b72640 1 Processor -- start 2024-04-25T15:28:57.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.537+0000 7faac0b72640 1 -- start start 2024-04-25T15:28:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faac0b72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faac0b72640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7faabc0b7f50 0x7faabc0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faac0b72640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faabc0d1c10 con 0x7faabc0b3e00 2024-04-25T15:28:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faac0b72640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7faabc0d27e0 con 0x7faabc0b7f50 2024-04-25T15:28:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faac0b72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faabc0d1eb0 con 0x7faabc0cfd40 2024-04-25T15:28:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faaba59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faabad9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7faabc0b7f50 0x7faabc0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faaba59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc0b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:33838/0 (socket says 172.21.15.67:33838) 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faaba59c640 1 -- 172.21.15.67:0/2257473964 learned_addr learned my addr 172.21.15.67:0/2257473964 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.539+0000 7faaba59c640 1 -- 172.21.15.67:0/2257473964 >> v2:172.21.15.176:3300/0 conn(0x7faabc0b7f50 msgr2=0x7faabc0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.540+0000 7faaba59c640 1 --2- 172.21.15.67:0/2257473964 >> v2:172.21.15.176:3300/0 conn(0x7faabc0b7f50 0x7faabc0ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.540+0000 7faaba59c640 1 -- 172.21.15.67:0/2257473964 >> v1:172.21.15.176:6789/0 conn(0x7faabc0b3e00 legacy=0x7faabc0b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.540+0000 7faaba59c640 1 -- 172.21.15.67:0/2257473964 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc0d2190 con 0x7faabc0cfd40 2024-04-25T15:28:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.540+0000 7faabad9d640 1 --2- 172.21.15.67:0/2257473964 >> v2:172.21.15.176:3300/0 conn(0x7faabc0b7f50 0x7faabc0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.540+0000 7faaba59c640 1 --2- 172.21.15.67:0/2257473964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc0b21a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faabc0d2960 tx=0x7faab402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b924cd7759d2cc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.540+0000 7faab959a640 1 -- 172.21.15.67:0/2257473964 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4037070 con 0x7faabc0cfd40 2024-04-25T15:28:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.540+0000 7faab959a640 1 -- 172.21.15.67:0/2257473964 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab4032440 con 0x7faabc0cfd40 2024-04-25T15:28:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faab959a640 1 -- 172.21.15.67:0/2257473964 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4032810 con 0x7faabc0cfd40 2024-04-25T15:28:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 -- 172.21.15.67:0/2257473964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 msgr2=0x7faabc0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 --2- 172.21.15.67:0/2257473964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc0b21a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faabc0d2960 tx=0x7faab402a250 comp rx=0 tx=0).stop 2024-04-25T15:28:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 -- 172.21.15.67:0/2257473964 shutdown_connections 2024-04-25T15:28:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 --2- 172.21.15.67:0/2257473964 >> v2:172.21.15.176:3300/0 conn(0x7faabc0b7f50 0x7faabc0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 --2- 172.21.15.67:0/2257473964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc0b21a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 -- 172.21.15.67:0/2257473964 >> 172.21.15.67:0/2257473964 conn(0x7faabc09b7d0 msgr2=0x7faabc09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 -- 172.21.15.67:0/2257473964 shutdown_connections 2024-04-25T15:28:57.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.541+0000 7faac0b72640 1 -- 172.21.15.67:0/2257473964 wait complete. 2024-04-25T15:28:57.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.543+0000 7faac0b72640 1 Processor -- start 2024-04-25T15:28:57.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.543+0000 7faac0b72640 1 -- start start 2024-04-25T15:28:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.544+0000 7faac0b72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faabc0b3e00 0x7faabc147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.544+0000 7faac0b72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.544+0000 7faac0b72640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faabc153bb0 0x7faabc160be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.544+0000 7faac0b72640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faabc0b4210 con 0x7faabc0b3e00 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.544+0000 7faac0b72640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faabc0ce650 con 0x7faabc153bb0 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.544+0000 7faac0b72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faabc0b4390 con 0x7faabc0cfd40 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faabc0b3e00 0x7faabc147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faabc0b3e00 0x7faabc147230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42158/0 (socket says 172.21.15.67:42158) 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 -- 172.21.15.67:0/3395255424 learned_addr learned my addr 172.21.15.67:0/3395255424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faabad9d640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faabc153bb0 0x7faabc160be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faab9d9b640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 -- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 msgr2=0x7faabc153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 -- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faabc153bb0 msgr2=0x7faabc160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faabc153bb0 0x7faabc160be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 -- 172.21.15.67:0/3395255424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc0ca430 con 0x7faabc0b3e00 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faaba59c640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faabc0b3e00 0x7faabc147230 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7faab402a290 tx=0x7faab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faab2ffd640 1 -- 172.21.15.67:0/3395255424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4037040 con 0x7faabc0b3e00 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faab2ffd640 1 -- 172.21.15.67:0/3395255424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab4042070 con 0x7faabc0b3e00 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faabc0ca710 con 0x7faabc0b3e00 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faab2ffd640 1 -- 172.21.15.67:0/3395255424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4030880 con 0x7faabc0b3e00 2024-04-25T15:28:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.545+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faabc0cacd0 con 0x7faabc0b3e00 2024-04-25T15:28:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.547+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faabc050740 con 0x7faabc0b3e00 2024-04-25T15:28:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.547+0000 7faab2ffd640 1 -- 172.21.15.67:0/3395255424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7faab4030a20 con 0x7faabc0b3e00 2024-04-25T15:28:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.548+0000 7faab2ffd640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faa9c0744e0 0x7faa9c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.548+0000 7faab2ffd640 1 -- 172.21.15.67:0/3395255424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5709+0+0 (secure 0 0 0) 0x7faab40b59f0 con 0x7faabc0b3e00 2024-04-25T15:28:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.548+0000 7faab9d9b640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faa9c0744e0 0x7faa9c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.548+0000 7faab9d9b640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faa9c0744e0 0x7faa9c0769f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faaa4007fd0 tx=0x7faaa4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.552+0000 7faab2ffd640 1 -- 172.21.15.67:0/3395255424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faab4080220 con 0x7faabc0b3e00 2024-04-25T15:28:57.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.744+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faabc1eabf0 con 0x7faabc0b3e00 2024-04-25T15:28:57.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.745+0000 7faab2ffd640 1 -- 172.21.15.67:0/3395255424 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16585 (secure 0 0 0) 0x7faab407fb20 con 0x7faabc0b3e00 2024-04-25T15:28:57.746 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:28:57.746 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":91,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:56.417940+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:28:57.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faa9c0744e0 msgr2=0x7faa9c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faa9c0744e0 0x7faa9c0769f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7faaa4007fd0 tx=0x7faaa4009040 comp rx=0 tx=0).stop 2024-04-25T15:28:57.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faabc0b3e00 msgr2=0x7faabc147230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faabc0b3e00 0x7faabc147230 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7faab402a290 tx=0x7faab40046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 shutdown_connections 2024-04-25T15:28:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faa9c0744e0 0x7faa9c0769f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faabc153bb0 0x7faabc160be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faabc0cfd40 0x7faabc153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 --2- 172.21.15.67:0/3395255424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faabc0b3e00 0x7faabc147230 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 >> 172.21.15.67:0/3395255424 conn(0x7faabc09b7d0 msgr2=0x7faabc0b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.747+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 shutdown_connections 2024-04-25T15:28:57.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.748+0000 7faac0b72640 1 -- 172.21.15.67:0/3395255424 wait complete. 2024-04-25T15:28:57.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:28:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.874+0000 7f651ed3f640 1 Processor -- start 2024-04-25T15:28:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.874+0000 7f651ed3f640 1 -- start start 2024-04-25T15:28:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651ed3f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518070820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651ed3f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f651806d5d0 0x7f651806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651ed3f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f651806df20 con 0x7f651815e780 2024-04-25T15:28:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651ed3f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6518158c60 con 0x7f651806d5d0 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651ed3f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f651815ef90 con 0x7f6518158850 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651d2dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f651806d5d0 0x7f651806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651d2dc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f651806d5d0 0x7f651806d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57684/0 (socket says 172.21.15.67:57684) 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651d2dc640 1 -- 172.21.15.67:0/3718823507 learned_addr learned my addr 172.21.15.67:0/3718823507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.875+0000 7f651cadb640 1 --2- 172.21.15.67:0/3718823507 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.876+0000 7f651d2dc640 1 -- 172.21.15.67:0/3718823507 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 msgr2=0x7f6518070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.876+0000 7f651d2dc640 1 --2- 172.21.15.67:0/3718823507 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.876+0000 7f651d2dc640 1 -- 172.21.15.67:0/3718823507 >> v1:172.21.15.176:6789/0 conn(0x7f651815e780 legacy=0x7f651815f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.876+0000 7f651d2dc640 1 -- 172.21.15.67:0/3718823507 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651815f270 con 0x7f651806d5d0 2024-04-25T15:28:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.876+0000 7f651d2dc640 1 --2- 172.21.15.67:0/3718823507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651806d5d0 0x7f651806d9e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443902f4e4ff44c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.876+0000 7f65177fe640 1 -- 172.21.15.67:0/3718823507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508037070 con 0x7f651806d5d0 2024-04-25T15:28:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f65177fe640 1 -- 172.21.15.67:0/3718823507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6508032440 con 0x7f651806d5d0 2024-04-25T15:28:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f65177fe640 1 -- 172.21.15.67:0/3718823507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508032810 con 0x7f651806d5d0 2024-04-25T15:28:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 -- 172.21.15.67:0/3718823507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651806d5d0 msgr2=0x7f651806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 --2- 172.21.15.67:0/3718823507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651806d5d0 0x7f651806d9e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650802a270 comp rx=0 tx=0).stop 2024-04-25T15:28:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 -- 172.21.15.67:0/3718823507 shutdown_connections 2024-04-25T15:28:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 --2- 172.21.15.67:0/3718823507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651806d5d0 0x7f651806d9e0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 --2- 172.21.15.67:0/3718823507 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518070820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 -- 172.21.15.67:0/3718823507 >> 172.21.15.67:0/3718823507 conn(0x7f65180730b0 msgr2=0x7f65180734c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 -- 172.21.15.67:0/3718823507 shutdown_connections 2024-04-25T15:28:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.877+0000 7f651ed3f640 1 -- 172.21.15.67:0/3718823507 wait complete. 2024-04-25T15:28:57.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.879+0000 7f651ed3f640 1 Processor -- start 2024-04-25T15:28:57.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.879+0000 7f651ed3f640 1 -- start start 2024-04-25T15:28:57.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.880+0000 7f651ed3f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f651806d5d0 0x7f651812f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.880+0000 7f651ed3f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651ed3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651815e780 0x7f6518142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651ed3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6518160010 con 0x7f651815e780 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651ed3f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f651806f040 con 0x7f651806d5d0 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651ed3f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f651815eb90 con 0x7f6518158850 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651815e780 0x7f6518142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651815e780 0x7f6518142c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57686/0 (socket says 172.21.15.67:57686) 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 -- 172.21.15.67:0/4264230041 learned_addr learned my addr 172.21.15.67:0/4264230041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f6517fff640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651cadb640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f651806d5d0 0x7f651812f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 -- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 msgr2=0x7f6518142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518142700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 -- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f651806d5d0 msgr2=0x7f651812f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f651806d5d0 0x7f651812f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 -- 172.21.15.67:0/4264230041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6518141140 con 0x7f651815e780 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651cadb640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f651806d5d0 0x7f651812f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651d2dc640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651815e780 0x7f6518142c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f65080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f65157fa640 1 -- 172.21.15.67:0/4264230041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508041020 con 0x7f651815e780 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f65157fa640 1 -- 172.21.15.67:0/4264230041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65080306c0 con 0x7f651815e780 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f65157fa640 1 -- 172.21.15.67:0/4264230041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508037040 con 0x7f651815e780 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.881+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651813cd30 con 0x7f651815e780 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.882+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65181413e0 con 0x7f651815e780 2024-04-25T15:28:57.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.882+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65180761c0 con 0x7f651815e780 2024-04-25T15:28:57.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.883+0000 7f65157fa640 1 -- 172.21.15.67:0/4264230041 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6508044020 con 0x7f651815e780 2024-04-25T15:28:57.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.884+0000 7f65157fa640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64f80744e0 0x7f64f80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:57.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.884+0000 7f65157fa640 1 -- 172.21.15.67:0/4264230041 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5709+0+0 (secure 0 0 0) 0x7f65080b5a00 con 0x7f651815e780 2024-04-25T15:28:57.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.884+0000 7f651cadb640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64f80744e0 0x7f64f80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:57.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.884+0000 7f651cadb640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64f80744e0 0x7f64f80769f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6518147ee0 tx=0x7f65100039d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:57.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:57.888+0000 7f65157fa640 1 -- 172.21.15.67:0/4264230041 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f650804b030 con 0x7f651815e780 2024-04-25T15:28:58.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:58.119+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6518147980 con 0x7f651815e780 2024-04-25T15:28:59.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.083+0000 7f65157fa640 1 -- 172.21.15.67:0/4264230041 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) ==== 118+0+0 (secure 0 0 0) 0x7f6508080190 con 0x7f651815e780 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64f80744e0 msgr2=0x7f64f80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64f80744e0 0x7f64f80769f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6518147ee0 tx=0x7f65100039d0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651815e780 msgr2=0x7f6518142c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651815e780 0x7f6518142c40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f65080046c0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 shutdown_connections 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f64f80744e0 0x7f64f80769f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f651815e780 0x7f6518142c40 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6518158850 0x7f6518142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 --2- 172.21.15.67:0/4264230041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f651806d5d0 0x7f651812f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 >> 172.21.15.67:0/4264230041 conn(0x7f65180730b0 msgr2=0x7f651815d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:59.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 shutdown_connections 2024-04-25T15:28:59.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.087+0000 7f651ed3f640 1 -- 172.21.15.67:0/4264230041 wait complete. 2024-04-25T15:28:59.101 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T15:28:59.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:28:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.222+0000 7f7dd3b7a640 1 Processor -- start 2024-04-25T15:28:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.223+0000 7f7dd3b7a640 1 -- start start 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd3b7a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc070630 0x7f7dcc0cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd3b7a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7dcc06dce0 0x7f7dcc160d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd3b7a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dcc071110 con 0x7f7dcc15adf0 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd3b7a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7dcc0713f0 con 0x7f7dcc06dce0 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd3b7a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7dcc0701a0 con 0x7f7dcc070630 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd2117640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7dcc06dce0 0x7f7dcc160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd1916640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc070630 0x7f7dcc0cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd2117640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7dcc06dce0 0x7f7dcc160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:43470/0 (socket says 172.21.15.67:43470) 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd1115640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7dcc15adf0 0x7f7dcc06d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:42942/0 (socket says 172.21.15.67:42942) 2024-04-25T15:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.224+0000 7f7dd2117640 1 -- 172.21.15.67:0/628838630 learned_addr learned my addr 172.21.15.67:0/628838630 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.225+0000 7f7dd0914640 1 -- 172.21.15.67:0/628838630 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449839285 0 0) 0x7f7dcc071110 con 0x7f7dcc15adf0 2024-04-25T15:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.225+0000 7f7dd0914640 1 -- 172.21.15.67:0/628838630 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7db4005730 con 0x7f7dcc15adf0 2024-04-25T15:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.225+0000 7f7dd1916640 1 -- 172.21.15.67:0/628838630 >> v2:172.21.15.176:3300/0 conn(0x7f7dcc06dce0 msgr2=0x7f7dcc160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.225+0000 7f7dd1916640 1 --2- 172.21.15.67:0/628838630 >> v2:172.21.15.176:3300/0 conn(0x7f7dcc06dce0 0x7f7dcc160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.225+0000 7f7dd1916640 1 -- 172.21.15.67:0/628838630 >> v1:172.21.15.176:6789/0 conn(0x7f7dcc15adf0 legacy=0x7f7dcc06d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.225+0000 7f7dd1916640 1 -- 172.21.15.67:0/628838630 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dcc06fb50 con 0x7f7dcc070630 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.225+0000 7f7dd2117640 1 --2- 172.21.15.67:0/628838630 >> v2:172.21.15.176:3300/0 conn(0x7f7dcc06dce0 0x7f7dcc160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd1916640 1 --2- 172.21.15.67:0/628838630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc070630 0x7f7dcc0cec50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40097a0 tx=0x7f7dc402a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28dbb5fd3da3bf57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd0914640 1 -- 172.21.15.67:0/628838630 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4037070 con 0x7f7dcc070630 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd0914640 1 -- 172.21.15.67:0/628838630 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc4032440 con 0x7f7dcc070630 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd0914640 1 -- 172.21.15.67:0/628838630 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc402ae60 con 0x7f7dcc070630 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/628838630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc070630 msgr2=0x7f7dcc0cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/628838630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc070630 0x7f7dcc0cec50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40097a0 tx=0x7f7dc402a110 comp rx=0 tx=0).stop 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/628838630 shutdown_connections 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/628838630 >> v2:172.21.15.176:3300/0 conn(0x7f7dcc06dce0 0x7f7dcc160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/628838630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc070630 0x7f7dcc0cec50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.226+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/628838630 >> 172.21.15.67:0/628838630 conn(0x7f7dcc073090 msgr2=0x7f7dcc0734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.227+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/628838630 shutdown_connections 2024-04-25T15:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.227+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/628838630 wait complete. 2024-04-25T15:28:59.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.228+0000 7f7dd3b7a640 1 Processor -- start 2024-04-25T15:28:59.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.228+0000 7f7dd3b7a640 1 -- start start 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd3b7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dcc070630 0x7f7dcc155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd3b7a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc15adf0 0x7f7dcc157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd3b7a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dcc14aae0 0x7f7dcc14aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd3b7a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc0b0e80 con 0x7f7dcc070630 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd3b7a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc0b0d00 con 0x7f7dcc14aae0 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd3b7a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7dcc070c80 con 0x7f7dcc15adf0 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dcc070630 0x7f7dcc155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dcc070630 0x7f7dcc155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57688/0 (socket says 172.21.15.67:57688) 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 -- 172.21.15.67:0/2257775795 learned_addr learned my addr 172.21.15.67:0/2257775795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1115640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc15adf0 0x7f7dcc157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd2117640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dcc14aae0 0x7f7dcc14aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 -- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc15adf0 msgr2=0x7f7dcc157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc15adf0 0x7f7dcc157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 -- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dcc14aae0 msgr2=0x7f7dcc14aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dcc14aae0 0x7f7dcc14aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 -- 172.21.15.67:0/2257775795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dcc1586e0 con 0x7f7dcc070630 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dd1916640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dcc070630 0x7f7dcc155af0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4000c00 tx=0x7f7dc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.230+0000 7f7dca7fc640 1 -- 172.21.15.67:0/2257775795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4041070 con 0x7f7dcc070630 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.231+0000 7f7dca7fc640 1 -- 172.21.15.67:0/2257775795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc403c890 con 0x7f7dcc070630 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.231+0000 7f7dca7fc640 1 -- 172.21.15.67:0/2257775795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4037040 con 0x7f7dcc070630 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.231+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dcc1589c0 con 0x7f7dcc070630 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.231+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dcc15ec40 con 0x7f7dcc070630 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.232+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dcc050740 con 0x7f7dcc070630 2024-04-25T15:28:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.232+0000 7f7dca7fc640 1 -- 172.21.15.67:0/2257775795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7dc4044070 con 0x7f7dcc070630 2024-04-25T15:28:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.233+0000 7f7dca7fc640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7db80744b0 0x7f7db80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:28:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.233+0000 7f7dca7fc640 1 -- 172.21.15.67:0/2257775795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f7dc40b5ba0 con 0x7f7dcc070630 2024-04-25T15:28:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.233+0000 7f7dd1115640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7db80744b0 0x7f7db80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:28:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.233+0000 7f7dd1115640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7db80744b0 0x7f7db80769c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0000f60 tx=0x7f7dc0007cf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:28:59.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.237+0000 7f7dca7fc640 1 -- 172.21.15.67:0/2257775795 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7dc407fce0 con 0x7f7dcc070630 2024-04-25T15:28:59.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:28:59.468+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f7dcc06c8a0 con 0x7f7dcc070630 2024-04-25T15:28:59.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:28:59.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:29:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.084+0000 7f7dca7fc640 1 -- 172.21.15.67:0/2257775795 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) ==== 122+0+0 (secure 0 0 0) 0x7f7dc4083fe0 con 0x7f7dcc070630 2024-04-25T15:29:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7db80744b0 msgr2=0x7f7db80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7db80744b0 0x7f7db80769c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0000f60 tx=0x7f7dc0007cf0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dcc070630 msgr2=0x7f7dcc155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dcc070630 0x7f7dcc155af0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4000c00 tx=0x7f7dc40046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 shutdown_connections 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7db80744b0 0x7f7db80769c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dcc14aae0 0x7f7dcc14aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dcc15adf0 0x7f7dcc157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 --2- 172.21.15.67:0/2257775795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dcc070630 0x7f7dcc155af0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 >> 172.21.15.67:0/2257775795 conn(0x7f7dcc073090 msgr2=0x7f7dcc0ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 shutdown_connections 2024-04-25T15:29:00.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.086+0000 7f7dd3b7a640 1 -- 172.21.15.67:0/2257775795 wait complete. 2024-04-25T15:29:00.099 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:29:00.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:29:00.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.213+0000 7f3a86b34640 1 Processor -- start 2024-04-25T15:29:00.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.213+0000 7f3a86b34640 1 -- start start 2024-04-25T15:29:00.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a86b34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 0x7f3a800d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a86b34640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3a8006dcf0 0x7f3a800d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a86b34640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a800d04a0 0x7f3a800d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a86b34640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3a800b9520 con 0x7f3a800d04a0 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a86b34640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3a800b4820 con 0x7f3a8006dcf0 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a86b34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a800b3c10 con 0x7f3a8006d8e0 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a800d04a0 0x7f3a800d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3a800d04a0 0x7f3a800d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57698/0 (socket says 172.21.15.67:57698) 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 -- 172.21.15.67:0/2463158647 learned_addr learned my addr 172.21.15.67:0/2463158647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a7ffff640 1 --2- 172.21.15.67:0/2463158647 >> v2:172.21.15.176:3300/0 conn(0x7f3a8006dcf0 0x7f3a800d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a848d0640 1 --2- 172.21.15.67:0/2463158647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 0x7f3a800d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 -- 172.21.15.67:0/2463158647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 msgr2=0x7f3a800d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 --2- 172.21.15.67:0/2463158647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 0x7f3a800d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 -- 172.21.15.67:0/2463158647 >> v2:172.21.15.176:3300/0 conn(0x7f3a8006dcf0 msgr2=0x7f3a800d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 --2- 172.21.15.67:0/2463158647 >> v2:172.21.15.176:3300/0 conn(0x7f3a8006dcf0 0x7f3a800d3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a850d1640 1 -- 172.21.15.67:0/2463158647 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a800b3ef0 con 0x7f3a800d04a0 2024-04-25T15:29:00.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.215+0000 7f3a7ffff640 1 --2- 172.21.15.67:0/2463158647 >> v2:172.21.15.176:3300/0 conn(0x7f3a8006dcf0 0x7f3a800d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.216+0000 7f3a850d1640 1 --2- 172.21.15.67:0/2463158647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a800d04a0 0x7f3a800d08b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3a800d1580 tx=0x7f3a6802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6097bba110e14add server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.216+0000 7f3a7f7fe640 1 -- 172.21.15.67:0/2463158647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68037070 con 0x7f3a800d04a0 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.216+0000 7f3a7f7fe640 1 -- 172.21.15.67:0/2463158647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a68032440 con 0x7f3a800d04a0 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.216+0000 7f3a7f7fe640 1 -- 172.21.15.67:0/2463158647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68032810 con 0x7f3a800d04a0 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 -- 172.21.15.67:0/2463158647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a800d04a0 msgr2=0x7f3a800d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 --2- 172.21.15.67:0/2463158647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a800d04a0 0x7f3a800d08b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3a800d1580 tx=0x7f3a6802a140 comp rx=0 tx=0).stop 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 -- 172.21.15.67:0/2463158647 shutdown_connections 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 --2- 172.21.15.67:0/2463158647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a800d04a0 0x7f3a800d08b0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 --2- 172.21.15.67:0/2463158647 >> v2:172.21.15.176:3300/0 conn(0x7f3a8006dcf0 0x7f3a800d3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 --2- 172.21.15.67:0/2463158647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 0x7f3a800d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 -- 172.21.15.67:0/2463158647 >> 172.21.15.67:0/2463158647 conn(0x7f3a8008d110 msgr2=0x7f3a8008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 -- 172.21.15.67:0/2463158647 shutdown_connections 2024-04-25T15:29:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.217+0000 7f3a86b34640 1 -- 172.21.15.67:0/2463158647 wait complete. 2024-04-25T15:29:00.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.219+0000 7f3a86b34640 1 Processor -- start 2024-04-25T15:29:00.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.219+0000 7f3a86b34640 1 -- start start 2024-04-25T15:29:00.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a86b34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 0x7f3a801f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a86b34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006dcf0 0x7f3a801e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a86b34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a800d04a0 0x7f3a800a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a86b34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3a800b38e0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a86b34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3a800b44b0 con 0x7f3a800d04a0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a86b34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a800d0df0 con 0x7f3a8006d8e0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006dcf0 0x7f3a801e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006dcf0 0x7f3a801e6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57708/0 (socket says 172.21.15.67:57708) 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.220+0000 7f3a7ffff640 1 -- 172.21.15.67:0/3757746957 learned_addr learned my addr 172.21.15.67:0/3757746957 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7ffff640 1 -- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 msgr2=0x7f3a801f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7ffff640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 0x7f3a801f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7ffff640 1 -- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a800d04a0 msgr2=0x7f3a800a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7ffff640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a800d04a0 0x7f3a800a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7ffff640 1 -- 172.21.15.67:0/3757746957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a801e76f0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7ffff640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006dcf0 0x7f3a801e6a30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000d7d0 tx=0x7f3a7000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7d7fa640 1 -- 172.21.15.67:0/3757746957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70004030 con 0x7f3a8006dcf0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7d7fa640 1 -- 172.21.15.67:0/3757746957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a700041d0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a801e79d0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a7d7fa640 1 -- 172.21.15.67:0/3757746957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a700096a0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.221+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a801e7f90 con 0x7f3a8006dcf0 2024-04-25T15:29:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.222+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a80050740 con 0x7f3a8006dcf0 2024-04-25T15:29:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.223+0000 7f3a7d7fa640 1 -- 172.21.15.67:0/3757746957 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3a70009840 con 0x7f3a8006dcf0 2024-04-25T15:29:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.223+0000 7f3a7d7fa640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a600744b0 0x7f3a600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.224+0000 7f3a7d7fa640 1 -- 172.21.15.67:0/3757746957 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f3a700954a0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.224+0000 7f3a848d0640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a600744b0 0x7f3a600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.224+0000 7f3a848d0640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a600744b0 0x7f3a600769c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3a800744c0 tx=0x7f3a78009470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.228+0000 7f3a7d7fa640 1 -- 172.21.15.67:0/3757746957 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a7005fcc0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.452+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a8009eb60 con 0x7f3a8006dcf0 2024-04-25T15:29:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.453+0000 7f3a7d7fa640 1 -- 172.21.15.67:0/3757746957 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5189 (secure 0 0 0) 0x7f3a7005f5c0 con 0x7f3a8006dcf0 2024-04-25T15:29:00.454 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:00.454 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2709900765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2709900765},{"type":"v1","addr":"172.21.15.67:6834","nonce":2709900765}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4104906051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4104906051},{"type":"v1","addr":"172.21.15.176:6836","nonce":4104906051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6655,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2981673460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2981673460},{"type":"v1","addr":"172.21.15.176:6837","nonce":2981673460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:56.417898+0000","modified":"2024-04-25T15:28:59.469250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7128":{"gid":7128,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3429198629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3429198629},{"type":"v1","addr":"172.21.15.67:6835","nonce":3429198629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7128,"qdb_cluster":[7128]},"id":7}]} 2024-04-25T15:29:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 57 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a600744b0 msgr2=0x7f3a600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a600744b0 0x7f3a600769c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3a800744c0 tx=0x7f3a78009470 comp rx=0 tx=0).stop 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006dcf0 msgr2=0x7f3a801e6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006dcf0 0x7f3a801e6a30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000d7d0 tx=0x7f3a7000dc90 comp rx=0 tx=0).stop 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 shutdown_connections 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a600744b0 0x7f3a600769c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a800d04a0 0x7f3a800a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006dcf0 0x7f3a801e6a30 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 --2- 172.21.15.67:0/3757746957 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a8006d8e0 0x7f3a801f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 >> 172.21.15.67:0/3757746957 conn(0x7f3a8008d110 msgr2=0x7f3a80071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 shutdown_connections 2024-04-25T15:29:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.455+0000 7f3a86b34640 1 -- 172.21.15.67:0/3757746957 wait complete. 2024-04-25T15:29:00.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:29:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.579+0000 7f8dcccd1640 1 Processor -- start 2024-04-25T15:29:00.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.579+0000 7f8dcccd1640 1 -- start start 2024-04-25T15:29:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dcccd1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8dc80d0e00 0x7f8dc80d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dcccd1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8dc806c5c0 0x7f8dc806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dcccd1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc80b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dcccd1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8dc806cf10 con 0x7f8dc806c5c0 2024-04-25T15:29:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dcccd1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8dc80d1850 con 0x7f8dc80d0e00 2024-04-25T15:29:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dcccd1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc8160de0 con 0x7f8dc80b7810 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dc5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8dc806c5c0 0x7f8dc806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dc5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8dc806c5c0 0x7f8dc806c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57720/0 (socket says 172.21.15.67:57720) 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dc5d9b640 1 -- 172.21.15.67:0/3194612827 learned_addr learned my addr 172.21.15.67:0/3194612827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dc659c640 1 --2- 172.21.15.67:0/3194612827 >> v2:172.21.15.176:3300/0 conn(0x7f8dc80d0e00 0x7f8dc80d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.580+0000 7f8dc6d9d640 1 --2- 172.21.15.67:0/3194612827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc80b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc5d9b640 1 -- 172.21.15.67:0/3194612827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 msgr2=0x7f8dc80b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc5d9b640 1 --2- 172.21.15.67:0/3194612827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc80b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc5d9b640 1 -- 172.21.15.67:0/3194612827 >> v2:172.21.15.176:3300/0 conn(0x7f8dc80d0e00 msgr2=0x7f8dc80d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc5d9b640 1 --2- 172.21.15.67:0/3194612827 >> v2:172.21.15.176:3300/0 conn(0x7f8dc80d0e00 0x7f8dc80d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc5d9b640 1 -- 172.21.15.67:0/3194612827 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc81610c0 con 0x7f8dc806c5c0 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc6d9d640 1 --2- 172.21.15.67:0/3194612827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc80b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc5d9b640 1 --2- 172.21.15.67:0/3194612827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc806c9d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80d3e60 tx=0x7f8db002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688220bfcc61294c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.581+0000 7f8dc559a640 1 -- 172.21.15.67:0/3194612827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0037070 con 0x7f8dc806c5c0 2024-04-25T15:29:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dc559a640 1 -- 172.21.15.67:0/3194612827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db0032440 con 0x7f8dc806c5c0 2024-04-25T15:29:00.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dc559a640 1 -- 172.21.15.67:0/3194612827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0032810 con 0x7f8dc806c5c0 2024-04-25T15:29:00.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 -- 172.21.15.67:0/3194612827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 msgr2=0x7f8dc806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/3194612827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc806c9d0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80d3e60 tx=0x7f8db002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 -- 172.21.15.67:0/3194612827 shutdown_connections 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/3194612827 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc80b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/3194612827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc806c9d0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/3194612827 >> v2:172.21.15.176:3300/0 conn(0x7f8dc80d0e00 0x7f8dc80d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 -- 172.21.15.67:0/3194612827 >> 172.21.15.67:0/3194612827 conn(0x7f8dc80d5d90 msgr2=0x7f8dc80d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 -- 172.21.15.67:0/3194612827 shutdown_connections 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.582+0000 7f8dcccd1640 1 -- 172.21.15.67:0/3194612827 wait complete. 2024-04-25T15:29:00.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.584+0000 7f8dcccd1640 1 Processor -- start 2024-04-25T15:29:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.584+0000 7f8dcccd1640 1 -- start start 2024-04-25T15:29:00.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.585+0000 7f8dcccd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc81fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.585+0000 7f8dcccd1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc81f9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dcccd1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8dc80d0e00 0x7f8dc81585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dcccd1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc80b5320 con 0x7f8dc806c5c0 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dcccd1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc8160ad0 con 0x7f8dc80d0e00 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dcccd1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc8164e00 con 0x7f8dc80b7810 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc81fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc81fa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57724/0 (socket says 172.21.15.67:57724) 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 -- 172.21.15.67:0/968002989 learned_addr learned my addr 172.21.15.67:0/968002989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc6d9d640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8dc80d0e00 0x7f8dc81585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc5d9b640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc81f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 -- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 msgr2=0x7f8dc81f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc81f9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 -- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8dc80d0e00 msgr2=0x7f8dc81585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8dc80d0e00 0x7f8dc81585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 -- 172.21.15.67:0/968002989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc81fac90 con 0x7f8dc806c5c0 2024-04-25T15:29:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dc659c640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc81fa1d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e8a0 tx=0x7f8dc000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dbeffd640 1 -- 172.21.15.67:0/968002989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0017070 con 0x7f8dc806c5c0 2024-04-25T15:29:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dbeffd640 1 -- 172.21.15.67:0/968002989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc000c940 con 0x7f8dc806c5c0 2024-04-25T15:29:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc81faf70 con 0x7f8dc806c5c0 2024-04-25T15:29:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dbeffd640 1 -- 172.21.15.67:0/968002989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc000cd30 con 0x7f8dc806c5c0 2024-04-25T15:29:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.586+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc81fa710 con 0x7f8dc806c5c0 2024-04-25T15:29:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.587+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc8050740 con 0x7f8dc806c5c0 2024-04-25T15:29:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.588+0000 7f8dbeffd640 1 -- 172.21.15.67:0/968002989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8dc001a070 con 0x7f8dc806c5c0 2024-04-25T15:29:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.588+0000 7f8dbeffd640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8da80744b0 0x7f8da80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.589+0000 7f8dbeffd640 1 -- 172.21.15.67:0/968002989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f8dc00954c0 con 0x7f8dc806c5c0 2024-04-25T15:29:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.589+0000 7f8dc5d9b640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8da80744b0 0x7f8da80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.589+0000 7f8dc5d9b640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8da80744b0 0x7f8da80769c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8db0034000 tx=0x7f8db002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.593+0000 7f8dbeffd640 1 -- 172.21.15.67:0/968002989 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8dc005fc50 con 0x7f8dc806c5c0 2024-04-25T15:29:00.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.780+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8dc80d4320 con 0x7f8dc806c5c0 2024-04-25T15:29:00.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.781+0000 7f8dbeffd640 1 -- 172.21.15.67:0/968002989 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16572 (secure 0 0 0) 0x7f8dc005f550 con 0x7f8dc806c5c0 2024-04-25T15:29:00.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:00.782 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:58.058412+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8da80744b0 msgr2=0x7f8da80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8da80744b0 0x7f8da80769c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8db0034000 tx=0x7f8db002e000 comp rx=0 tx=0).stop 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 msgr2=0x7f8dc81fa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc81fa1d0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000e8a0 tx=0x7f8dc000ed60 comp rx=0 tx=0).stop 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 shutdown_connections 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8da80744b0 0x7f8da80769c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8dc80d0e00 0x7f8dc81585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8dc80b7810 0x7f8dc81f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 --2- 172.21.15.67:0/968002989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8dc806c5c0 0x7f8dc81fa1d0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 >> 172.21.15.67:0/968002989 conn(0x7f8dc80d5d90 msgr2=0x7f8dc815f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 shutdown_connections 2024-04-25T15:29:00.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.784+0000 7f8dcccd1640 1 -- 172.21.15.67:0/968002989 wait complete. 2024-04-25T15:29:00.796 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:29:00.796 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:00.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.908+0000 7f3a4079d640 1 Processor -- start 2024-04-25T15:29:00.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.908+0000 7f3a4079d640 1 -- start start 2024-04-25T15:29:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.909+0000 7f3a4079d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3a380d11f0 0x7f3a380d5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.909+0000 7f3a4079d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a380d1830 con 0x7f3a380d0de0 2024-04-25T15:29:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.909+0000 7f3a4079d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a38160dc0 con 0x7f3a3806ccd0 2024-04-25T15:29:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.909+0000 7f3a4079d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3a380d1600 con 0x7f3a380d11f0 2024-04-25T15:29:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.909+0000 7f3a3e539640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a380d0de0 0x7f3a3806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40788/0 (socket says 172.21.15.67:40788) 2024-04-25T15:29:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.909+0000 7f3a3e539640 1 -- 172.21.15.67:0/1666756893 learned_addr learned my addr 172.21.15.67:0/1666756893 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1666756893 >> v2:172.21.15.176:3300/0 conn(0x7f3a380d11f0 0x7f3a380d5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033785458 0 0) 0x7f3a380d1830 con 0x7f3a380d0de0 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a24003520 con 0x7f3a380d0de0 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1471211566 0 0) 0x7f3a24003520 con 0x7f3a380d0de0 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 >> v2:172.21.15.176:3300/0 conn(0x7f3a380d11f0 msgr2=0x7f3a380d5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 --2- 172.21.15.67:0/1666756893 >> v2:172.21.15.176:3300/0 conn(0x7f3a380d11f0 0x7f3a380d5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 >> v1:172.21.15.176:6789/0 conn(0x7f3a3806ccd0 legacy=0x7f3a380d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a381610a0 con 0x7f3a380d0de0 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f3a34001670 con 0x7f3a380d0de0 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 >> v1:172.21.15.67:6789/0 conn(0x7f3a380d0de0 legacy=0x7f3a3806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a240039b0 0x7f3a24003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a24004800 0x7f3a24004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a24005250 0x7f3a2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2400c1b0 con 0x7f3a24005250 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2400c360 con 0x7f3a24004800 2024-04-25T15:29:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a2400c560 con 0x7f3a240039b0 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a24005250 0x7f3a2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3e539640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a240039b0 0x7f3a24003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3dd38640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a24004800 0x7f3a24004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a240039b0 msgr2=0x7f3a24003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a240039b0 0x7f3a24003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a24004800 msgr2=0x7f3a24004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a24004800 0x7f3a24004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.910+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1666756893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a24003520 con 0x7f3a24005250 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.911+0000 7f3a3e539640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a240039b0 0x7f3a24003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.911+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1666756893 reap_dead start 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.911+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a24005250 0x7f3a2400b9c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009fc0 tx=0x7f3a24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d9b4d30c7a0e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.911+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30035020 con 0x7f3a24005250 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.911+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a30003b90 con 0x7f3a24005250 2024-04-25T15:29:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.911+0000 7f3a3d537640 1 -- 172.21.15.67:0/1666756893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30009c10 con 0x7f3a24005250 2024-04-25T15:29:00.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.912+0000 7f3a4079d640 1 -- 172.21.15.67:0/1666756893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a24005250 msgr2=0x7f3a2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.912+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a24005250 0x7f3a2400b9c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009fc0 tx=0x7f3a24005830 comp rx=0 tx=0).stop 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.912+0000 7f3a4079d640 1 -- 172.21.15.67:0/1666756893 shutdown_connections 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.912+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1666756893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a24005250 0x7f3a2400b9c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.912+0000 7f3a4079d640 1 -- 172.21.15.67:0/1666756893 >> 172.21.15.67:0/1666756893 conn(0x7f3a380d5d70 msgr2=0x7f3a380d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.912+0000 7f3a4079d640 1 -- 172.21.15.67:0/1666756893 shutdown_connections 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.912+0000 7f3a4079d640 1 -- 172.21.15.67:0/1666756893 wait complete. 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.914+0000 7f3a4079d640 1 Processor -- start 2024-04-25T15:29:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.914+0000 7f3a4079d640 1 -- start start 2024-04-25T15:29:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.915+0000 7f3a4079d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a24005250 0x7f3a380cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.915+0000 7f3a4079d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38151130 0x7f3a380b98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.915+0000 7f3a4079d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380b9e10 0x7f3a380b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.915+0000 7f3a4079d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3a380b5300 con 0x7f3a380b9e10 2024-04-25T15:29:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.915+0000 7f3a4079d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3a38160ab0 con 0x7f3a38151130 2024-04-25T15:29:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.915+0000 7f3a4079d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a38164de0 con 0x7f3a24005250 2024-04-25T15:29:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380b9e10 0x7f3a380b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380b9e10 0x7f3a380b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57752/0 (socket says 172.21.15.67:57752) 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1288499436 learned_addr learned my addr 172.21.15.67:0/1288499436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a24005250 msgr2=0x7f3a380cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3e539640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a24005250 0x7f3a380cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3dd38640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38151130 0x7f3a380b98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a24005250 0x7f3a380cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38151130 msgr2=0x7f3a380b98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38151130 0x7f3a380b98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 -- 172.21.15.67:0/1288499436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a380ba220 con 0x7f3a380b9e10 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3e539640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a24005250 0x7f3a380cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3dd38640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38151130 0x7f3a380b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a3ed3a640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380b9e10 0x7f3a380b5b20 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000e230 tx=0x7f3a30009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a2effd640 1 -- 172.21.15.67:0/1288499436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3003f030 con 0x7f3a380b9e10 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a2effd640 1 -- 172.21.15.67:0/1288499436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a30003990 con 0x7f3a380b9e10 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a381f9b50 con 0x7f3a380b9e10 2024-04-25T15:29:00.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a2effd640 1 -- 172.21.15.67:0/1288499436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30035070 con 0x7f3a380b9e10 2024-04-25T15:29:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.916+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a381fa110 con 0x7f3a380b9e10 2024-04-25T15:29:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.917+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a38050740 con 0x7f3a380b9e10 2024-04-25T15:29:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.918+0000 7f3a2effd640 1 -- 172.21.15.67:0/1288499436 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3a30042030 con 0x7f3a380b9e10 2024-04-25T15:29:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.918+0000 7f3a2effd640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a1c0744e0 0x7f3a1c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.919+0000 7f3a2effd640 1 -- 172.21.15.67:0/1288499436 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f3a300b5b80 con 0x7f3a380b9e10 2024-04-25T15:29:00.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.919+0000 7f3a3e539640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a1c0744e0 0x7f3a1c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:00.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.919+0000 7f3a3e539640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a1c0744e0 0x7f3a1c0769f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3a34000fd0 tx=0x7f3a34001170 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:00.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:00.923+0000 7f3a2effd640 1 -- 172.21.15.67:0/1288499436 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a380ba220 con 0x7f3a380b9e10 2024-04-25T15:29:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.106+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3a380d4810 con 0x7f3a1c0744e0 2024-04-25T15:29:01.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.113+0000 7f3a2effd640 1 -- 172.21.15.67:0/1288499436 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246754 (secure 0 0 0) 0x7f3a380d4810 con 0x7f3a1c0744e0 2024-04-25T15:29:01.116 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:01.117 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a1c0744e0 msgr2=0x7f3a1c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a1c0744e0 0x7f3a1c0769f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3a34000fd0 tx=0x7f3a34001170 comp rx=0 tx=0).stop 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380b9e10 msgr2=0x7f3a380b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380b9e10 0x7f3a380b5b20 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000e230 tx=0x7f3a30009750 comp rx=0 tx=0).stop 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 shutdown_connections 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3a1c0744e0 0x7f3a1c0769f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a380b9e10 0x7f3a380b5b20 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a38151130 0x7f3a380b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 --2- 172.21.15.67:0/1288499436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a24005250 0x7f3a380cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 >> 172.21.15.67:0/1288499436 conn(0x7f3a380d5d70 msgr2=0x7f3a38161cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:01.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 shutdown_connections 2024-04-25T15:29:01.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.116+0000 7f3a4079d640 1 -- 172.21.15.67:0/1288499436 wait complete. 2024-04-25T15:29:01.128 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-25T15:29:00.847967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242652,"kb_used_data":9840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754732068,"statfs":{"total":773094113280,"available":772845637632,"internally_reserved":0,"allocated":10076160,"data_stored":4946690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.818182"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052980+0000","last_change":"2024-04-25T15:28:56.052980+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052980+0000","last_undegraded":"2024-04-25T15:28:56.052980+0000","last_fullsized":"2024-04-25T15:28:56.052980+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040106+0000","last_change":"2024-04-25T15:28:56.040106+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040106+0000","last_undegraded":"2024-04-25T15:28:56.040106+0000","last_fullsized":"2024-04-25T15:28:56.040106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044401+0000","last_change":"2024-04-25T15:28:56.044401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044401+0000","last_undegraded":"2024-04-25T15:28:56.044401+0000","last_fullsized":"2024-04-25T15:28:56.044401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044161+0000","last_change":"2024-04-25T15:28:56.044161+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044161+0000","last_undegraded":"2024-04-25T15:28:56.044161+0000","last_fullsized":"2024-04-25T15:28:56.044161+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040956+0000","last_change":"2024-04-25T15:28:56.040956+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040956+0000","last_undegraded":"2024-04-25T15:28:56.040956+0000","last_fullsized":"2024-04-25T15:28:56.040956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043783+0000","last_change":"2024-04-25T15:28:56.043783+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043783+0000","last_undegraded":"2024-04-25T15:28:56.043783+0000","last_fullsized":"2024-04-25T15:28:56.043783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043768+0000","last_change":"2024-04-25T15:28:56.043768+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043768+0000","last_undegraded":"2024-04-25T15:28:56.043768+0000","last_fullsized":"2024-04-25T15:28:56.043768+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038877+0000","last_change":"2024-04-25T15:28:56.038877+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038877+0000","last_undegraded":"2024-04-25T15:28:56.038877+0000","last_fullsized":"2024-04-25T15:28:56.038877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037775+0000","last_change":"2024-04-25T15:28:56.037775+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037775+0000","last_undegraded":"2024-04-25T15:28:56.037775+0000","last_fullsized":"2024-04-25T15:28:56.037775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041034+0000","last_change":"2024-04-25T15:28:56.041034+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041034+0000","last_undegraded":"2024-04-25T15:28:56.041034+0000","last_fullsized":"2024-04-25T15:28:56.041034+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046085+0000","last_change":"2024-04-25T15:28:56.046085+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046085+0000","last_undegraded":"2024-04-25T15:28:56.046085+0000","last_fullsized":"2024-04-25T15:28:56.046085+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039309+0000","last_change":"2024-04-25T15:28:56.039309+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039309+0000","last_undegraded":"2024-04-25T15:28:56.039309+0000","last_fullsized":"2024-04-25T15:28:56.039309+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043640+0000","last_change":"2024-04-25T15:28:56.043640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043640+0000","last_undegraded":"2024-04-25T15:28:56.043640+0000","last_fullsized":"2024-04-25T15:28:56.043640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047935+0000","last_change":"2024-04-25T15:28:56.047935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047935+0000","last_undegraded":"2024-04-25T15:28:56.047935+0000","last_fullsized":"2024-04-25T15:28:56.047935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043253+0000","last_change":"2024-04-25T15:28:56.043253+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043253+0000","last_undegraded":"2024-04-25T15:28:56.043253+0000","last_fullsized":"2024-04-25T15:28:56.043253+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043312+0000","last_change":"2024-04-25T15:28:56.043312+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043312+0000","last_undegraded":"2024-04-25T15:28:56.043312+0000","last_fullsized":"2024-04-25T15:28:56.043312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038794+0000","last_change":"2024-04-25T15:28:56.038794+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038794+0000","last_undegraded":"2024-04-25T15:28:56.038794+0000","last_fullsized":"2024-04-25T15:28:56.038794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043094+0000","last_change":"2024-04-25T15:28:56.043094+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043094+0000","last_undegraded":"2024-04-25T15:28:56.043094+0000","last_fullsized":"2024-04-25T15:28:56.043094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047300+0000","last_change":"2024-04-25T15:28:56.047300+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047300+0000","last_undegraded":"2024-04-25T15:28:56.047300+0000","last_fullsized":"2024-04-25T15:28:56.047300+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037516+0000","last_change":"2024-04-25T15:28:56.037516+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037516+0000","last_undegraded":"2024-04-25T15:28:56.037516+0000","last_fullsized":"2024-04-25T15:28:56.037516+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046746+0000","last_change":"2024-04-25T15:28:56.046746+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046746+0000","last_undegraded":"2024-04-25T15:28:56.046746+0000","last_fullsized":"2024-04-25T15:28:56.046746+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036501+0000","last_change":"2024-04-25T15:28:56.036501+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036501+0000","last_undegraded":"2024-04-25T15:28:56.036501+0000","last_fullsized":"2024-04-25T15:28:56.036501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036507+0000","last_change":"2024-04-25T15:28:56.036507+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036507+0000","last_undegraded":"2024-04-25T15:28:56.036507+0000","last_fullsized":"2024-04-25T15:28:56.036507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042774+0000","last_change":"2024-04-25T15:28:56.042774+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042774+0000","last_undegraded":"2024-04-25T15:28:56.042774+0000","last_fullsized":"2024-04-25T15:28:56.042774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043848+0000","last_change":"2024-04-25T15:28:56.043848+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043848+0000","last_undegraded":"2024-04-25T15:28:56.043848+0000","last_fullsized":"2024-04-25T15:28:56.043848+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039503+0000","last_change":"2024-04-25T15:28:56.039503+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039503+0000","last_undegraded":"2024-04-25T15:28:56.039503+0000","last_fullsized":"2024-04-25T15:28:56.039503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045584+0000","last_change":"2024-04-25T15:28:56.045584+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045584+0000","last_undegraded":"2024-04-25T15:28:56.045584+0000","last_fullsized":"2024-04-25T15:28:56.045584+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045164+0000","last_change":"2024-04-25T15:28:56.045164+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045164+0000","last_undegraded":"2024-04-25T15:28:56.045164+0000","last_fullsized":"2024-04-25T15:28:56.045164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039640+0000","last_change":"2024-04-25T15:28:56.039640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039640+0000","last_undegraded":"2024-04-25T15:28:56.039640+0000","last_fullsized":"2024-04-25T15:28:56.039640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046096+0000","last_change":"2024-04-25T15:28:56.046096+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046096+0000","last_undegraded":"2024-04-25T15:28:56.046096+0000","last_fullsized":"2024-04-25T15:28:56.046096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042753+0000","last_change":"2024-04-25T15:28:56.042753+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042753+0000","last_undegraded":"2024-04-25T15:28:56.042753+0000","last_fullsized":"2024-04-25T15:28:56.042753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.049201+0000","last_change":"2024-04-25T15:28:56.049201+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.049201+0000","last_undegraded":"2024-04-25T15:28:56.049201+0000","last_fullsized":"2024-04-25T15:28:56.049201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045572+0000","last_change":"2024-04-25T15:28:56.045572+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045572+0000","last_undegraded":"2024-04-25T15:28:56.045572+0000","last_fullsized":"2024-04-25T15:28:56.045572+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060682+0000","last_change":"2024-04-25T15:28:56.059439+0000","last_active":"2024-04-25T15:28:56.060682+0000","last_peered":"2024-04-25T15:28:56.060682+0000","last_clean":"2024-04-25T15:28:56.060682+0000","last_became_active":"2024-04-25T15:28:56.059082+0000","last_became_peered":"2024-04-25T15:28:56.059082+0000","last_unstale":"2024-04-25T15:28:56.060682+0000","last_undegraded":"2024-04-25T15:28:56.060682+0000","last_fullsized":"2024-04-25T15:28:56.060682+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042730+0000","last_change":"2024-04-25T15:28:56.042730+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042730+0000","last_undegraded":"2024-04-25T15:28:56.042730+0000","last_fullsized":"2024-04-25T15:28:56.042730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059155+0000","last_change":"2024-04-25T15:28:56.055939+0000","last_active":"2024-04-25T15:28:56.059155+0000","last_peered":"2024-04-25T15:28:56.059155+0000","last_clean":"2024-04-25T15:28:56.059155+0000","last_became_active":"2024-04-25T15:28:56.055541+0000","last_became_peered":"2024-04-25T15:28:56.055541+0000","last_unstale":"2024-04-25T15:28:56.059155+0000","last_undegraded":"2024-04-25T15:28:56.059155+0000","last_fullsized":"2024-04-25T15:28:56.059155+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046120+0000","last_change":"2024-04-25T15:28:56.046120+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046120+0000","last_undegraded":"2024-04-25T15:28:56.046120+0000","last_fullsized":"2024-04-25T15:28:56.046120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060146+0000","last_change":"2024-04-25T15:28:56.056907+0000","last_active":"2024-04-25T15:28:56.060146+0000","last_peered":"2024-04-25T15:28:56.060146+0000","last_clean":"2024-04-25T15:28:56.060146+0000","last_became_active":"2024-04-25T15:28:56.056549+0000","last_became_peered":"2024-04-25T15:28:56.056549+0000","last_unstale":"2024-04-25T15:28:56.060146+0000","last_undegraded":"2024-04-25T15:28:56.060146+0000","last_fullsized":"2024-04-25T15:28:56.060146+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037437+0000","last_change":"2024-04-25T15:28:56.037437+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037437+0000","last_undegraded":"2024-04-25T15:28:56.037437+0000","last_fullsized":"2024-04-25T15:28:56.037437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059710+0000","last_change":"2024-04-25T15:28:56.057126+0000","last_active":"2024-04-25T15:28:56.059710+0000","last_peered":"2024-04-25T15:28:56.059710+0000","last_clean":"2024-04-25T15:28:56.059710+0000","last_became_active":"2024-04-25T15:28:56.056619+0000","last_became_peered":"2024-04-25T15:28:56.056619+0000","last_unstale":"2024-04-25T15:28:56.059710+0000","last_undegraded":"2024-04-25T15:28:56.059710+0000","last_fullsized":"2024-04-25T15:28:56.059710+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:50.204782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040410+0000","last_change":"2024-04-25T15:28:56.040410+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040410+0000","last_undegraded":"2024-04-25T15:28:56.040410+0000","last_fullsized":"2024-04-25T15:28:56.040410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062504+0000","last_change":"2024-04-25T15:28:56.061172+0000","last_active":"2024-04-25T15:28:56.062504+0000","last_peered":"2024-04-25T15:28:56.062504+0000","last_clean":"2024-04-25T15:28:56.062504+0000","last_became_active":"2024-04-25T15:28:56.060821+0000","last_became_peered":"2024-04-25T15:28:56.060821+0000","last_unstale":"2024-04-25T15:28:56.062504+0000","last_undegraded":"2024-04-25T15:28:56.062504+0000","last_fullsized":"2024-04-25T15:28:56.062504+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:58.403814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044043+0000","last_change":"2024-04-25T15:28:56.044043+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044043+0000","last_undegraded":"2024-04-25T15:28:56.044043+0000","last_fullsized":"2024-04-25T15:28:56.044043+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056409+0000","last_change":"2024-04-25T15:28:56.053676+0000","last_active":"2024-04-25T15:28:56.056409+0000","last_peered":"2024-04-25T15:28:56.056409+0000","last_clean":"2024-04-25T15:28:56.056409+0000","last_became_active":"2024-04-25T15:28:56.053355+0000","last_became_peered":"2024-04-25T15:28:56.053355+0000","last_unstale":"2024-04-25T15:28:56.056409+0000","last_undegraded":"2024-04-25T15:28:56.056409+0000","last_fullsized":"2024-04-25T15:28:56.056409+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042179+0000","last_change":"2024-04-25T15:28:56.042179+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042179+0000","last_undegraded":"2024-04-25T15:28:56.042179+0000","last_fullsized":"2024-04-25T15:28:56.042179+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056491+0000","last_change":"2024-04-25T15:28:56.051321+0000","last_active":"2024-04-25T15:28:56.056491+0000","last_peered":"2024-04-25T15:28:56.056491+0000","last_clean":"2024-04-25T15:28:56.056491+0000","last_became_active":"2024-04-25T15:28:56.050541+0000","last_became_peered":"2024-04-25T15:28:56.050541+0000","last_unstale":"2024-04-25T15:28:56.056491+0000","last_undegraded":"2024-04-25T15:28:56.056491+0000","last_fullsized":"2024-04-25T15:28:56.056491+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036701+0000","last_change":"2024-04-25T15:28:56.036701+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036701+0000","last_undegraded":"2024-04-25T15:28:56.036701+0000","last_fullsized":"2024-04-25T15:28:56.036701+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062513+0000","last_change":"2024-04-25T15:28:56.061338+0000","last_active":"2024-04-25T15:28:56.062513+0000","last_peered":"2024-04-25T15:28:56.062513+0000","last_clean":"2024-04-25T15:28:56.062513+0000","last_became_active":"2024-04-25T15:28:56.061028+0000","last_became_peered":"2024-04-25T15:28:56.061028+0000","last_unstale":"2024-04-25T15:28:56.062513+0000","last_undegraded":"2024-04-25T15:28:56.062513+0000","last_fullsized":"2024-04-25T15:28:56.062513+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:59.070550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038961+0000","last_change":"2024-04-25T15:28:56.038961+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038961+0000","last_undegraded":"2024-04-25T15:28:56.038961+0000","last_fullsized":"2024-04-25T15:28:56.038961+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062170+0000","last_change":"2024-04-25T15:28:56.058667+0000","last_active":"2024-04-25T15:28:56.062170+0000","last_peered":"2024-04-25T15:28:56.062170+0000","last_clean":"2024-04-25T15:28:56.062170+0000","last_became_active":"2024-04-25T15:28:56.057323+0000","last_became_peered":"2024-04-25T15:28:56.057323+0000","last_unstale":"2024-04-25T15:28:56.062170+0000","last_undegraded":"2024-04-25T15:28:56.062170+0000","last_fullsized":"2024-04-25T15:28:56.062170+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:06.756381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037203+0000","last_change":"2024-04-25T15:28:56.037203+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037203+0000","last_undegraded":"2024-04-25T15:28:56.037203+0000","last_fullsized":"2024-04-25T15:28:56.037203+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.053076+0000","last_change":"2024-04-25T15:28:56.046616+0000","last_active":"2024-04-25T15:28:56.053076+0000","last_peered":"2024-04-25T15:28:56.053076+0000","last_clean":"2024-04-25T15:28:56.053076+0000","last_became_active":"2024-04-25T15:28:56.046311+0000","last_became_peered":"2024-04-25T15:28:56.046311+0000","last_unstale":"2024-04-25T15:28:56.053076+0000","last_undegraded":"2024-04-25T15:28:56.053076+0000","last_fullsized":"2024-04-25T15:28:56.053076+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:21:45.358976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044011+0000","last_change":"2024-04-25T15:28:56.044011+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044011+0000","last_undegraded":"2024-04-25T15:28:56.044011+0000","last_fullsized":"2024-04-25T15:28:56.044011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059069+0000","last_change":"2024-04-25T15:28:56.053865+0000","last_active":"2024-04-25T15:28:56.059069+0000","last_peered":"2024-04-25T15:28:56.059069+0000","last_clean":"2024-04-25T15:28:56.059069+0000","last_became_active":"2024-04-25T15:28:56.053511+0000","last_became_peered":"2024-04-25T15:28:56.053511+0000","last_unstale":"2024-04-25T15:28:56.059069+0000","last_undegraded":"2024-04-25T15:28:56.059069+0000","last_fullsized":"2024-04-25T15:28:56.059069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044913+0000","last_change":"2024-04-25T15:28:56.044913+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044913+0000","last_undegraded":"2024-04-25T15:28:56.044913+0000","last_fullsized":"2024-04-25T15:28:56.044913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.050295+0000","last_change":"2024-04-25T15:28:56.044068+0000","last_active":"2024-04-25T15:28:56.050295+0000","last_peered":"2024-04-25T15:28:56.050295+0000","last_clean":"2024-04-25T15:28:56.050295+0000","last_became_active":"2024-04-25T15:28:56.043670+0000","last_became_peered":"2024-04-25T15:28:56.043670+0000","last_unstale":"2024-04-25T15:28:56.050295+0000","last_undegraded":"2024-04-25T15:28:56.050295+0000","last_fullsized":"2024-04-25T15:28:56.050295+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041617+0000","last_change":"2024-04-25T15:28:56.041617+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041617+0000","last_undegraded":"2024-04-25T15:28:56.041617+0000","last_fullsized":"2024-04-25T15:28:56.041617+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056097+0000","last_change":"2024-04-25T15:28:56.051968+0000","last_active":"2024-04-25T15:28:56.056097+0000","last_peered":"2024-04-25T15:28:56.056097+0000","last_clean":"2024-04-25T15:28:56.056097+0000","last_became_active":"2024-04-25T15:28:56.051236+0000","last_became_peered":"2024-04-25T15:28:56.051236+0000","last_unstale":"2024-04-25T15:28:56.056097+0000","last_undegraded":"2024-04-25T15:28:56.056097+0000","last_fullsized":"2024-04-25T15:28:56.056097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:45.569280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045084+0000","last_change":"2024-04-25T15:28:56.045084+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045084+0000","last_undegraded":"2024-04-25T15:28:56.045084+0000","last_fullsized":"2024-04-25T15:28:56.045084+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057007+0000","last_change":"2024-04-25T15:28:56.053779+0000","last_active":"2024-04-25T15:28:56.057007+0000","last_peered":"2024-04-25T15:28:56.057007+0000","last_clean":"2024-04-25T15:28:56.057007+0000","last_became_active":"2024-04-25T15:28:56.053471+0000","last_became_peered":"2024-04-25T15:28:56.053471+0000","last_unstale":"2024-04-25T15:28:56.057007+0000","last_undegraded":"2024-04-25T15:28:56.057007+0000","last_fullsized":"2024-04-25T15:28:56.057007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:25:01.410150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037150+0000","last_change":"2024-04-25T15:28:56.037150+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037150+0000","last_undegraded":"2024-04-25T15:28:56.037150+0000","last_fullsized":"2024-04-25T15:28:56.037150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059300+0000","last_change":"2024-04-25T15:28:56.052681+0000","last_active":"2024-04-25T15:28:56.059300+0000","last_peered":"2024-04-25T15:28:56.059300+0000","last_clean":"2024-04-25T15:28:56.059300+0000","last_became_active":"2024-04-25T15:28:56.052385+0000","last_became_peered":"2024-04-25T15:28:56.052385+0000","last_unstale":"2024-04-25T15:28:56.059300+0000","last_undegraded":"2024-04-25T15:28:56.059300+0000","last_fullsized":"2024-04-25T15:28:56.059300+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:45.314705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044935+0000","last_change":"2024-04-25T15:28:56.044935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044935+0000","last_undegraded":"2024-04-25T15:28:56.044935+0000","last_fullsized":"2024-04-25T15:28:56.044935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062215+0000","last_change":"2024-04-25T15:28:56.060187+0000","last_active":"2024-04-25T15:28:56.062215+0000","last_peered":"2024-04-25T15:28:56.062215+0000","last_clean":"2024-04-25T15:28:56.062215+0000","last_became_active":"2024-04-25T15:28:56.059867+0000","last_became_peered":"2024-04-25T15:28:56.059867+0000","last_unstale":"2024-04-25T15:28:56.062215+0000","last_undegraded":"2024-04-25T15:28:56.062215+0000","last_fullsized":"2024-04-25T15:28:56.062215+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:59.420234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043391+0000","last_change":"2024-04-25T15:28:56.043391+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043391+0000","last_undegraded":"2024-04-25T15:28:56.043391+0000","last_fullsized":"2024-04-25T15:28:56.043391+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059189+0000","last_change":"2024-04-25T15:28:56.052624+0000","last_active":"2024-04-25T15:28:56.059189+0000","last_peered":"2024-04-25T15:28:56.059189+0000","last_clean":"2024-04-25T15:28:56.059189+0000","last_became_active":"2024-04-25T15:28:56.052283+0000","last_became_peered":"2024-04-25T15:28:56.052283+0000","last_unstale":"2024-04-25T15:28:56.059189+0000","last_undegraded":"2024-04-25T15:28:56.059189+0000","last_fullsized":"2024-04-25T15:28:56.059189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:37.819844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041305+0000","last_change":"2024-04-25T15:28:56.041305+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041305+0000","last_undegraded":"2024-04-25T15:28:56.041305+0000","last_fullsized":"2024-04-25T15:28:56.041305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056432+0000","last_change":"2024-04-25T15:28:56.050894+0000","last_active":"2024-04-25T15:28:56.056432+0000","last_peered":"2024-04-25T15:28:56.056432+0000","last_clean":"2024-04-25T15:28:56.056432+0000","last_became_active":"2024-04-25T15:28:56.050570+0000","last_became_peered":"2024-04-25T15:28:56.050570+0000","last_unstale":"2024-04-25T15:28:56.056432+0000","last_undegraded":"2024-04-25T15:28:56.056432+0000","last_fullsized":"2024-04-25T15:28:56.056432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044386+0000","last_change":"2024-04-25T15:28:56.044386+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044386+0000","last_undegraded":"2024-04-25T15:28:56.044386+0000","last_fullsized":"2024-04-25T15:28:56.044386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.049680+0000","last_change":"2024-04-25T15:28:56.045642+0000","last_active":"2024-04-25T15:28:56.049680+0000","last_peered":"2024-04-25T15:28:56.049680+0000","last_clean":"2024-04-25T15:28:56.049680+0000","last_became_active":"2024-04-25T15:28:56.045273+0000","last_became_peered":"2024-04-25T15:28:56.045273+0000","last_unstale":"2024-04-25T15:28:56.049680+0000","last_undegraded":"2024-04-25T15:28:56.049680+0000","last_fullsized":"2024-04-25T15:28:56.049680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:21.737399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044193+0000","last_change":"2024-04-25T15:28:56.044193+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044193+0000","last_undegraded":"2024-04-25T15:28:56.044193+0000","last_fullsized":"2024-04-25T15:28:56.044193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062036+0000","last_change":"2024-04-25T15:28:56.059690+0000","last_active":"2024-04-25T15:28:56.062036+0000","last_peered":"2024-04-25T15:28:56.062036+0000","last_clean":"2024-04-25T15:28:56.062036+0000","last_became_active":"2024-04-25T15:28:56.059372+0000","last_became_peered":"2024-04-25T15:28:56.059372+0000","last_unstale":"2024-04-25T15:28:56.062036+0000","last_undegraded":"2024-04-25T15:28:56.062036+0000","last_fullsized":"2024-04-25T15:28:56.062036+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:03.773654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044493+0000","last_change":"2024-04-25T15:28:56.044493+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044493+0000","last_undegraded":"2024-04-25T15:28:56.044493+0000","last_fullsized":"2024-04-25T15:28:56.044493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056674+0000","last_change":"2024-04-25T15:28:56.053555+0000","last_active":"2024-04-25T15:28:56.056674+0000","last_peered":"2024-04-25T15:28:56.056674+0000","last_clean":"2024-04-25T15:28:56.056674+0000","last_became_active":"2024-04-25T15:28:56.053235+0000","last_became_peered":"2024-04-25T15:28:56.053235+0000","last_unstale":"2024-04-25T15:28:56.056674+0000","last_undegraded":"2024-04-25T15:28:56.056674+0000","last_fullsized":"2024-04-25T15:28:56.056674+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043401+0000","last_change":"2024-04-25T15:28:56.043401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043401+0000","last_undegraded":"2024-04-25T15:28:56.043401+0000","last_fullsized":"2024-04-25T15:28:56.043401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.054456+0000","last_change":"2024-04-25T15:28:56.049930+0000","last_active":"2024-04-25T15:28:56.054456+0000","last_peered":"2024-04-25T15:28:56.054456+0000","last_clean":"2024-04-25T15:28:56.054456+0000","last_became_active":"2024-04-25T15:28:56.049617+0000","last_became_peered":"2024-04-25T15:28:56.049617+0000","last_unstale":"2024-04-25T15:28:56.054456+0000","last_undegraded":"2024-04-25T15:28:56.054456+0000","last_fullsized":"2024-04-25T15:28:56.054456+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:29.266322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042363+0000","last_change":"2024-04-25T15:28:56.042363+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042363+0000","last_undegraded":"2024-04-25T15:28:56.042363+0000","last_fullsized":"2024-04-25T15:28:56.042363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057275+0000","last_change":"2024-04-25T15:28:56.054470+0000","last_active":"2024-04-25T15:28:56.057275+0000","last_peered":"2024-04-25T15:28:56.057275+0000","last_clean":"2024-04-25T15:28:56.057275+0000","last_became_active":"2024-04-25T15:28:56.054148+0000","last_became_peered":"2024-04-25T15:28:56.054148+0000","last_unstale":"2024-04-25T15:28:56.057275+0000","last_undegraded":"2024-04-25T15:28:56.057275+0000","last_fullsized":"2024-04-25T15:28:56.057275+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037520+0000","last_change":"2024-04-25T15:28:56.037520+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037520+0000","last_undegraded":"2024-04-25T15:28:56.037520+0000","last_fullsized":"2024-04-25T15:28:56.037520+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056659+0000","last_change":"2024-04-25T15:28:56.051763+0000","last_active":"2024-04-25T15:28:56.056659+0000","last_peered":"2024-04-25T15:28:56.056659+0000","last_clean":"2024-04-25T15:28:56.056659+0000","last_became_active":"2024-04-25T15:28:56.051431+0000","last_became_peered":"2024-04-25T15:28:56.051431+0000","last_unstale":"2024-04-25T15:28:56.056659+0000","last_undegraded":"2024-04-25T15:28:56.056659+0000","last_fullsized":"2024-04-25T15:28:56.056659+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:44:13.470300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052252+0000","last_change":"2024-04-25T15:28:56.052252+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052252+0000","last_undegraded":"2024-04-25T15:28:56.052252+0000","last_fullsized":"2024-04-25T15:28:56.052252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060700+0000","last_change":"2024-04-25T15:28:56.059005+0000","last_active":"2024-04-25T15:28:56.060700+0000","last_peered":"2024-04-25T15:28:56.060700+0000","last_clean":"2024-04-25T15:28:56.060700+0000","last_became_active":"2024-04-25T15:28:56.057981+0000","last_became_peered":"2024-04-25T15:28:56.057981+0000","last_unstale":"2024-04-25T15:28:56.060700+0000","last_undegraded":"2024-04-25T15:28:56.060700+0000","last_fullsized":"2024-04-25T15:28:56.060700+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045632+0000","last_change":"2024-04-25T15:28:56.045632+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045632+0000","last_undegraded":"2024-04-25T15:28:56.045632+0000","last_fullsized":"2024-04-25T15:28:56.045632+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056297+0000","last_change":"2024-04-25T15:28:56.048576+0000","last_active":"2024-04-25T15:28:56.056297+0000","last_peered":"2024-04-25T15:28:56.056297+0000","last_clean":"2024-04-25T15:28:56.056297+0000","last_became_active":"2024-04-25T15:28:56.048273+0000","last_became_peered":"2024-04-25T15:28:56.048273+0000","last_unstale":"2024-04-25T15:28:56.056297+0000","last_undegraded":"2024-04-25T15:28:56.056297+0000","last_fullsized":"2024-04-25T15:28:56.056297+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042145+0000","last_change":"2024-04-25T15:28:56.042145+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042145+0000","last_undegraded":"2024-04-25T15:28:56.042145+0000","last_fullsized":"2024-04-25T15:28:56.042145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056488+0000","last_change":"2024-04-25T15:28:56.051473+0000","last_active":"2024-04-25T15:28:56.056488+0000","last_peered":"2024-04-25T15:28:56.056488+0000","last_clean":"2024-04-25T15:28:56.056488+0000","last_became_active":"2024-04-25T15:28:56.051171+0000","last_became_peered":"2024-04-25T15:28:56.051171+0000","last_unstale":"2024-04-25T15:28:56.056488+0000","last_undegraded":"2024-04-25T15:28:56.056488+0000","last_fullsized":"2024-04-25T15:28:56.056488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042897+0000","last_change":"2024-04-25T15:28:56.042897+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042897+0000","last_undegraded":"2024-04-25T15:28:56.042897+0000","last_fullsized":"2024-04-25T15:28:56.042897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060976+0000","last_change":"2024-04-25T15:28:56.058257+0000","last_active":"2024-04-25T15:28:56.060976+0000","last_peered":"2024-04-25T15:28:56.060976+0000","last_clean":"2024-04-25T15:28:56.060976+0000","last_became_active":"2024-04-25T15:28:56.057736+0000","last_became_peered":"2024-04-25T15:28:56.057736+0000","last_unstale":"2024-04-25T15:28:56.060976+0000","last_undegraded":"2024-04-25T15:28:56.060976+0000","last_fullsized":"2024-04-25T15:28:56.060976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036658+0000","last_change":"2024-04-25T15:28:56.036658+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036658+0000","last_undegraded":"2024-04-25T15:28:56.036658+0000","last_fullsized":"2024-04-25T15:28:56.036658+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.061871+0000","last_change":"2024-04-25T15:28:56.056141+0000","last_active":"2024-04-25T15:28:56.061871+0000","last_peered":"2024-04-25T15:28:56.061871+0000","last_clean":"2024-04-25T15:28:56.061871+0000","last_became_active":"2024-04-25T15:28:56.055725+0000","last_became_peered":"2024-04-25T15:28:56.055725+0000","last_unstale":"2024-04-25T15:28:56.061871+0000","last_undegraded":"2024-04-25T15:28:56.061871+0000","last_fullsized":"2024-04-25T15:28:56.061871+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042444+0000","last_change":"2024-04-25T15:28:56.042444+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042444+0000","last_undegraded":"2024-04-25T15:28:56.042444+0000","last_fullsized":"2024-04-25T15:28:56.042444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060626+0000","last_change":"2024-04-25T15:28:56.055311+0000","last_active":"2024-04-25T15:28:56.060626+0000","last_peered":"2024-04-25T15:28:56.060626+0000","last_clean":"2024-04-25T15:28:56.060626+0000","last_became_active":"2024-04-25T15:28:56.054958+0000","last_became_peered":"2024-04-25T15:28:56.054958+0000","last_unstale":"2024-04-25T15:28:56.060626+0000","last_undegraded":"2024-04-25T15:28:56.060626+0000","last_fullsized":"2024-04-25T15:28:56.060626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:04.302144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041713+0000","last_change":"2024-04-25T15:28:56.041713+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041713+0000","last_undegraded":"2024-04-25T15:28:56.041713+0000","last_fullsized":"2024-04-25T15:28:56.041713+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.051579+0000","last_change":"2024-04-25T15:28:56.050355+0000","last_active":"2024-04-25T15:28:56.051579+0000","last_peered":"2024-04-25T15:28:56.051579+0000","last_clean":"2024-04-25T15:28:56.051579+0000","last_became_active":"2024-04-25T15:28:56.050028+0000","last_became_peered":"2024-04-25T15:28:56.050028+0000","last_unstale":"2024-04-25T15:28:56.051579+0000","last_undegraded":"2024-04-25T15:28:56.051579+0000","last_fullsized":"2024-04-25T15:28:56.051579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:18.740971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044259+0000","last_change":"2024-04-25T15:28:56.044259+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044259+0000","last_undegraded":"2024-04-25T15:28:56.044259+0000","last_fullsized":"2024-04-25T15:28:56.044259+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059127+0000","last_change":"2024-04-25T15:28:56.054041+0000","last_active":"2024-04-25T15:28:56.059127+0000","last_peered":"2024-04-25T15:28:56.059127+0000","last_clean":"2024-04-25T15:28:56.059127+0000","last_became_active":"2024-04-25T15:28:56.053693+0000","last_became_peered":"2024-04-25T15:28:56.053693+0000","last_unstale":"2024-04-25T15:28:56.059127+0000","last_undegraded":"2024-04-25T15:28:56.059127+0000","last_fullsized":"2024-04-25T15:28:56.059127+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":235,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.043621+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:56.043621+0000","last_peered":"2024-04-25T15:28:56.043621+0000","last_clean":"2024-04-25T15:28:56.043621+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:56.043621+0000","last_undegraded":"2024-04-25T15:28:56.043621+0000","last_fullsized":"2024-04-25T15:28:56.043621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:01.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:01.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.279+0000 7f2c6bb8c640 1 Processor -- start 2024-04-25T15:29:01.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.279+0000 7f2c6bb8c640 1 -- start start 2024-04-25T15:29:01.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6bb8c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2c640cfd40 0x7f2c640b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6bb8c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c640b7f50 0x7f2c640ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6bb8c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c640d1c10 con 0x7f2c640b3e00 2024-04-25T15:29:01.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6bb8c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2c640d27e0 con 0x7f2c640b7f50 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6bb8c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2c640d1eb0 con 0x7f2c640cfd40 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6a129640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c640b7f50 0x7f2c640ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c69127640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2c640b3e00 0x7f2c640b7840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40798/0 (socket says 172.21.15.67:40798) 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c69127640 1 -- 172.21.15.67:0/3091174570 learned_addr learned my addr 172.21.15.67:0/3091174570 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c69928640 1 --2- 172.21.15.67:0/3091174570 >> v2:172.21.15.176:3300/0 conn(0x7f2c640cfd40 0x7f2c640b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6a129640 1 -- 172.21.15.67:0/3091174570 >> v2:172.21.15.176:3300/0 conn(0x7f2c640cfd40 msgr2=0x7f2c640b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6a129640 1 --2- 172.21.15.67:0/3091174570 >> v2:172.21.15.176:3300/0 conn(0x7f2c640cfd40 0x7f2c640b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6a129640 1 -- 172.21.15.67:0/3091174570 >> v1:172.21.15.67:6789/0 conn(0x7f2c640b3e00 legacy=0x7f2c640b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c6a129640 1 -- 172.21.15.67:0/3091174570 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c640d2190 con 0x7f2c640b7f50 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c68926640 1 -- 172.21.15.67:0/3091174570 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036191414 0 0) 0x7f2c640d1c10 con 0x7f2c640b3e00 2024-04-25T15:29:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.281+0000 7f2c69928640 1 --2- 172.21.15.67:0/3091174570 >> v2:172.21.15.176:3300/0 conn(0x7f2c640cfd40 0x7f2c640b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:01.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.282+0000 7f2c6a129640 1 --2- 172.21.15.67:0/3091174570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640b7f50 0x7f2c640ce110 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2c640b8620 tx=0x7f2c6002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd2820eeb68ae39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:01.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.282+0000 7f2c68926640 1 -- 172.21.15.67:0/3091174570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60007740 con 0x7f2c640b7f50 2024-04-25T15:29:01.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.282+0000 7f2c68926640 1 -- 172.21.15.67:0/3091174570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c600078e0 con 0x7f2c640b7f50 2024-04-25T15:29:01.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.282+0000 7f2c68926640 1 -- 172.21.15.67:0/3091174570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60007cb0 con 0x7f2c640b7f50 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/3091174570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640b7f50 msgr2=0x7f2c640ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/3091174570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640b7f50 0x7f2c640ce110 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2c640b8620 tx=0x7f2c6002a5f0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/3091174570 shutdown_connections 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/3091174570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640b7f50 0x7f2c640ce110 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/3091174570 >> v2:172.21.15.176:3300/0 conn(0x7f2c640cfd40 0x7f2c640b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/3091174570 >> 172.21.15.67:0/3091174570 conn(0x7f2c6409b7d0 msgr2=0x7f2c6409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/3091174570 shutdown_connections 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.283+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/3091174570 wait complete. 2024-04-25T15:29:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.285+0000 7f2c6bb8c640 1 Processor -- start 2024-04-25T15:29:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.285+0000 7f2c6bb8c640 1 -- start start 2024-04-25T15:29:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6bb8c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c640b3e00 0x7f2c640b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6bb8c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c640cfd40 0x7f2c640cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6bb8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640cbac0 0x7f2c64070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6bb8c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c640b4210 con 0x7f2c640cbac0 2024-04-25T15:29:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6bb8c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2c640ce650 con 0x7f2c640cfd40 2024-04-25T15:29:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6bb8c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2c640b4390 con 0x7f2c640b3e00 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6a129640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640cbac0 0x7f2c64070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6a129640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640cbac0 0x7f2c64070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57766/0 (socket says 172.21.15.67:57766) 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.286+0000 7f2c6a129640 1 -- 172.21.15.67:0/794332882 learned_addr learned my addr 172.21.15.67:0/794332882 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c69127640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c640cfd40 0x7f2c640cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c69928640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c640b3e00 0x7f2c640b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6a129640 1 -- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c640b3e00 msgr2=0x7f2c640b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6a129640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c640b3e00 0x7f2c640b13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6a129640 1 -- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c640cfd40 msgr2=0x7f2c640cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6a129640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c640cfd40 0x7f2c640cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6a129640 1 -- 172.21.15.67:0/794332882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c641f9500 con 0x7f2c640cbac0 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c69127640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c640cfd40 0x7f2c640cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6a129640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640cbac0 0x7f2c64070b00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009e50 tx=0x7f2c60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c5a7fc640 1 -- 172.21.15.67:0/794332882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6003f070 con 0x7f2c640cbac0 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c5a7fc640 1 -- 172.21.15.67:0/794332882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c60031bf0 con 0x7f2c640cbac0 2024-04-25T15:29:01.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c5a7fc640 1 -- 172.21.15.67:0/794332882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c60044460 con 0x7f2c640cbac0 2024-04-25T15:29:01.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c641f97e0 con 0x7f2c640cbac0 2024-04-25T15:29:01.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.287+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c641f9da0 con 0x7f2c640cbac0 2024-04-25T15:29:01.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.288+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c64050740 con 0x7f2c640cbac0 2024-04-25T15:29:01.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.289+0000 7f2c5a7fc640 1 -- 172.21.15.67:0/794332882 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2c60007790 con 0x7f2c640cbac0 2024-04-25T15:29:01.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.289+0000 7f2c5a7fc640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c480744b0 0x7f2c480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.289+0000 7f2c5a7fc640 1 -- 172.21.15.67:0/794332882 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f2c600b6ab0 con 0x7f2c640cbac0 2024-04-25T15:29:01.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.290+0000 7f2c69928640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c480744b0 0x7f2c480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.290+0000 7f2c69928640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c480744b0 0x7f2c480769c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0096d0 tx=0x7f2c5c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:01.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.294+0000 7f2c5a7fc640 1 -- 172.21.15.67:0/794332882 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c600812d0 con 0x7f2c640cbac0 2024-04-25T15:29:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.476+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2c6409bd80 con 0x7f2c480744b0 2024-04-25T15:29:01.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.482+0000 7f2c5a7fc640 1 -- 172.21.15.67:0/794332882 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246754 (secure 0 0 0) 0x7f2c6409bd80 con 0x7f2c480744b0 2024-04-25T15:29:01.485 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c480744b0 msgr2=0x7f2c480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c480744b0 0x7f2c480769c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0096d0 tx=0x7f2c5c008040 comp rx=0 tx=0).stop 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640cbac0 msgr2=0x7f2c64070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640cbac0 0x7f2c64070b00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009e50 tx=0x7f2c60004a40 comp rx=0 tx=0).stop 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 shutdown_connections 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2c480744b0 0x7f2c480769c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c640cbac0 0x7f2c64070b00 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c640cfd40 0x7f2c640cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 --2- 172.21.15.67:0/794332882 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c640b3e00 0x7f2c640b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 >> 172.21.15.67:0/794332882 conn(0x7f2c6409b7d0 msgr2=0x7f2c640b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:01.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 shutdown_connections 2024-04-25T15:29:01.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.485+0000 7f2c6bb8c640 1 -- 172.21.15.67:0/794332882 wait complete. 2024-04-25T15:29:01.496 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-25T15:29:00.847967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242652,"kb_used_data":9840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754732068,"statfs":{"total":773094113280,"available":772845637632,"internally_reserved":0,"allocated":10076160,"data_stored":4946690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.818182"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052980+0000","last_change":"2024-04-25T15:28:56.052980+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052980+0000","last_undegraded":"2024-04-25T15:28:56.052980+0000","last_fullsized":"2024-04-25T15:28:56.052980+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040106+0000","last_change":"2024-04-25T15:28:56.040106+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040106+0000","last_undegraded":"2024-04-25T15:28:56.040106+0000","last_fullsized":"2024-04-25T15:28:56.040106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044401+0000","last_change":"2024-04-25T15:28:56.044401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044401+0000","last_undegraded":"2024-04-25T15:28:56.044401+0000","last_fullsized":"2024-04-25T15:28:56.044401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044161+0000","last_change":"2024-04-25T15:28:56.044161+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044161+0000","last_undegraded":"2024-04-25T15:28:56.044161+0000","last_fullsized":"2024-04-25T15:28:56.044161+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040956+0000","last_change":"2024-04-25T15:28:56.040956+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040956+0000","last_undegraded":"2024-04-25T15:28:56.040956+0000","last_fullsized":"2024-04-25T15:28:56.040956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043783+0000","last_change":"2024-04-25T15:28:56.043783+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043783+0000","last_undegraded":"2024-04-25T15:28:56.043783+0000","last_fullsized":"2024-04-25T15:28:56.043783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043768+0000","last_change":"2024-04-25T15:28:56.043768+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043768+0000","last_undegraded":"2024-04-25T15:28:56.043768+0000","last_fullsized":"2024-04-25T15:28:56.043768+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038877+0000","last_change":"2024-04-25T15:28:56.038877+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038877+0000","last_undegraded":"2024-04-25T15:28:56.038877+0000","last_fullsized":"2024-04-25T15:28:56.038877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037775+0000","last_change":"2024-04-25T15:28:56.037775+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037775+0000","last_undegraded":"2024-04-25T15:28:56.037775+0000","last_fullsized":"2024-04-25T15:28:56.037775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041034+0000","last_change":"2024-04-25T15:28:56.041034+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041034+0000","last_undegraded":"2024-04-25T15:28:56.041034+0000","last_fullsized":"2024-04-25T15:28:56.041034+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046085+0000","last_change":"2024-04-25T15:28:56.046085+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046085+0000","last_undegraded":"2024-04-25T15:28:56.046085+0000","last_fullsized":"2024-04-25T15:28:56.046085+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039309+0000","last_change":"2024-04-25T15:28:56.039309+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039309+0000","last_undegraded":"2024-04-25T15:28:56.039309+0000","last_fullsized":"2024-04-25T15:28:56.039309+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043640+0000","last_change":"2024-04-25T15:28:56.043640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043640+0000","last_undegraded":"2024-04-25T15:28:56.043640+0000","last_fullsized":"2024-04-25T15:28:56.043640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047935+0000","last_change":"2024-04-25T15:28:56.047935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047935+0000","last_undegraded":"2024-04-25T15:28:56.047935+0000","last_fullsized":"2024-04-25T15:28:56.047935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043253+0000","last_change":"2024-04-25T15:28:56.043253+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043253+0000","last_undegraded":"2024-04-25T15:28:56.043253+0000","last_fullsized":"2024-04-25T15:28:56.043253+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043312+0000","last_change":"2024-04-25T15:28:56.043312+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043312+0000","last_undegraded":"2024-04-25T15:28:56.043312+0000","last_fullsized":"2024-04-25T15:28:56.043312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038794+0000","last_change":"2024-04-25T15:28:56.038794+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038794+0000","last_undegraded":"2024-04-25T15:28:56.038794+0000","last_fullsized":"2024-04-25T15:28:56.038794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043094+0000","last_change":"2024-04-25T15:28:56.043094+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043094+0000","last_undegraded":"2024-04-25T15:28:56.043094+0000","last_fullsized":"2024-04-25T15:28:56.043094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047300+0000","last_change":"2024-04-25T15:28:56.047300+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047300+0000","last_undegraded":"2024-04-25T15:28:56.047300+0000","last_fullsized":"2024-04-25T15:28:56.047300+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037516+0000","last_change":"2024-04-25T15:28:56.037516+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037516+0000","last_undegraded":"2024-04-25T15:28:56.037516+0000","last_fullsized":"2024-04-25T15:28:56.037516+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046746+0000","last_change":"2024-04-25T15:28:56.046746+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046746+0000","last_undegraded":"2024-04-25T15:28:56.046746+0000","last_fullsized":"2024-04-25T15:28:56.046746+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036501+0000","last_change":"2024-04-25T15:28:56.036501+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036501+0000","last_undegraded":"2024-04-25T15:28:56.036501+0000","last_fullsized":"2024-04-25T15:28:56.036501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036507+0000","last_change":"2024-04-25T15:28:56.036507+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036507+0000","last_undegraded":"2024-04-25T15:28:56.036507+0000","last_fullsized":"2024-04-25T15:28:56.036507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042774+0000","last_change":"2024-04-25T15:28:56.042774+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042774+0000","last_undegraded":"2024-04-25T15:28:56.042774+0000","last_fullsized":"2024-04-25T15:28:56.042774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043848+0000","last_change":"2024-04-25T15:28:56.043848+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043848+0000","last_undegraded":"2024-04-25T15:28:56.043848+0000","last_fullsized":"2024-04-25T15:28:56.043848+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039503+0000","last_change":"2024-04-25T15:28:56.039503+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039503+0000","last_undegraded":"2024-04-25T15:28:56.039503+0000","last_fullsized":"2024-04-25T15:28:56.039503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045584+0000","last_change":"2024-04-25T15:28:56.045584+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045584+0000","last_undegraded":"2024-04-25T15:28:56.045584+0000","last_fullsized":"2024-04-25T15:28:56.045584+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045164+0000","last_change":"2024-04-25T15:28:56.045164+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045164+0000","last_undegraded":"2024-04-25T15:28:56.045164+0000","last_fullsized":"2024-04-25T15:28:56.045164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039640+0000","last_change":"2024-04-25T15:28:56.039640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039640+0000","last_undegraded":"2024-04-25T15:28:56.039640+0000","last_fullsized":"2024-04-25T15:28:56.039640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046096+0000","last_change":"2024-04-25T15:28:56.046096+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046096+0000","last_undegraded":"2024-04-25T15:28:56.046096+0000","last_fullsized":"2024-04-25T15:28:56.046096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042753+0000","last_change":"2024-04-25T15:28:56.042753+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042753+0000","last_undegraded":"2024-04-25T15:28:56.042753+0000","last_fullsized":"2024-04-25T15:28:56.042753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.049201+0000","last_change":"2024-04-25T15:28:56.049201+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.049201+0000","last_undegraded":"2024-04-25T15:28:56.049201+0000","last_fullsized":"2024-04-25T15:28:56.049201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045572+0000","last_change":"2024-04-25T15:28:56.045572+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045572+0000","last_undegraded":"2024-04-25T15:28:56.045572+0000","last_fullsized":"2024-04-25T15:28:56.045572+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060682+0000","last_change":"2024-04-25T15:28:56.059439+0000","last_active":"2024-04-25T15:28:56.060682+0000","last_peered":"2024-04-25T15:28:56.060682+0000","last_clean":"2024-04-25T15:28:56.060682+0000","last_became_active":"2024-04-25T15:28:56.059082+0000","last_became_peered":"2024-04-25T15:28:56.059082+0000","last_unstale":"2024-04-25T15:28:56.060682+0000","last_undegraded":"2024-04-25T15:28:56.060682+0000","last_fullsized":"2024-04-25T15:28:56.060682+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042730+0000","last_change":"2024-04-25T15:28:56.042730+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042730+0000","last_undegraded":"2024-04-25T15:28:56.042730+0000","last_fullsized":"2024-04-25T15:28:56.042730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059155+0000","last_change":"2024-04-25T15:28:56.055939+0000","last_active":"2024-04-25T15:28:56.059155+0000","last_peered":"2024-04-25T15:28:56.059155+0000","last_clean":"2024-04-25T15:28:56.059155+0000","last_became_active":"2024-04-25T15:28:56.055541+0000","last_became_peered":"2024-04-25T15:28:56.055541+0000","last_unstale":"2024-04-25T15:28:56.059155+0000","last_undegraded":"2024-04-25T15:28:56.059155+0000","last_fullsized":"2024-04-25T15:28:56.059155+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046120+0000","last_change":"2024-04-25T15:28:56.046120+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046120+0000","last_undegraded":"2024-04-25T15:28:56.046120+0000","last_fullsized":"2024-04-25T15:28:56.046120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060146+0000","last_change":"2024-04-25T15:28:56.056907+0000","last_active":"2024-04-25T15:28:56.060146+0000","last_peered":"2024-04-25T15:28:56.060146+0000","last_clean":"2024-04-25T15:28:56.060146+0000","last_became_active":"2024-04-25T15:28:56.056549+0000","last_became_peered":"2024-04-25T15:28:56.056549+0000","last_unstale":"2024-04-25T15:28:56.060146+0000","last_undegraded":"2024-04-25T15:28:56.060146+0000","last_fullsized":"2024-04-25T15:28:56.060146+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037437+0000","last_change":"2024-04-25T15:28:56.037437+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037437+0000","last_undegraded":"2024-04-25T15:28:56.037437+0000","last_fullsized":"2024-04-25T15:28:56.037437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059710+0000","last_change":"2024-04-25T15:28:56.057126+0000","last_active":"2024-04-25T15:28:56.059710+0000","last_peered":"2024-04-25T15:28:56.059710+0000","last_clean":"2024-04-25T15:28:56.059710+0000","last_became_active":"2024-04-25T15:28:56.056619+0000","last_became_peered":"2024-04-25T15:28:56.056619+0000","last_unstale":"2024-04-25T15:28:56.059710+0000","last_undegraded":"2024-04-25T15:28:56.059710+0000","last_fullsized":"2024-04-25T15:28:56.059710+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:50.204782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040410+0000","last_change":"2024-04-25T15:28:56.040410+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040410+0000","last_undegraded":"2024-04-25T15:28:56.040410+0000","last_fullsized":"2024-04-25T15:28:56.040410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062504+0000","last_change":"2024-04-25T15:28:56.061172+0000","last_active":"2024-04-25T15:28:56.062504+0000","last_peered":"2024-04-25T15:28:56.062504+0000","last_clean":"2024-04-25T15:28:56.062504+0000","last_became_active":"2024-04-25T15:28:56.060821+0000","last_became_peered":"2024-04-25T15:28:56.060821+0000","last_unstale":"2024-04-25T15:28:56.062504+0000","last_undegraded":"2024-04-25T15:28:56.062504+0000","last_fullsized":"2024-04-25T15:28:56.062504+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:58.403814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044043+0000","last_change":"2024-04-25T15:28:56.044043+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044043+0000","last_undegraded":"2024-04-25T15:28:56.044043+0000","last_fullsized":"2024-04-25T15:28:56.044043+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056409+0000","last_change":"2024-04-25T15:28:56.053676+0000","last_active":"2024-04-25T15:28:56.056409+0000","last_peered":"2024-04-25T15:28:56.056409+0000","last_clean":"2024-04-25T15:28:56.056409+0000","last_became_active":"2024-04-25T15:28:56.053355+0000","last_became_peered":"2024-04-25T15:28:56.053355+0000","last_unstale":"2024-04-25T15:28:56.056409+0000","last_undegraded":"2024-04-25T15:28:56.056409+0000","last_fullsized":"2024-04-25T15:28:56.056409+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042179+0000","last_change":"2024-04-25T15:28:56.042179+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042179+0000","last_undegraded":"2024-04-25T15:28:56.042179+0000","last_fullsized":"2024-04-25T15:28:56.042179+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056491+0000","last_change":"2024-04-25T15:28:56.051321+0000","last_active":"2024-04-25T15:28:56.056491+0000","last_peered":"2024-04-25T15:28:56.056491+0000","last_clean":"2024-04-25T15:28:56.056491+0000","last_became_active":"2024-04-25T15:28:56.050541+0000","last_became_peered":"2024-04-25T15:28:56.050541+0000","last_unstale":"2024-04-25T15:28:56.056491+0000","last_undegraded":"2024-04-25T15:28:56.056491+0000","last_fullsized":"2024-04-25T15:28:56.056491+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036701+0000","last_change":"2024-04-25T15:28:56.036701+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036701+0000","last_undegraded":"2024-04-25T15:28:56.036701+0000","last_fullsized":"2024-04-25T15:28:56.036701+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062513+0000","last_change":"2024-04-25T15:28:56.061338+0000","last_active":"2024-04-25T15:28:56.062513+0000","last_peered":"2024-04-25T15:28:56.062513+0000","last_clean":"2024-04-25T15:28:56.062513+0000","last_became_active":"2024-04-25T15:28:56.061028+0000","last_became_peered":"2024-04-25T15:28:56.061028+0000","last_unstale":"2024-04-25T15:28:56.062513+0000","last_undegraded":"2024-04-25T15:28:56.062513+0000","last_fullsized":"2024-04-25T15:28:56.062513+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:59.070550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038961+0000","last_change":"2024-04-25T15:28:56.038961+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038961+0000","last_undegraded":"2024-04-25T15:28:56.038961+0000","last_fullsized":"2024-04-25T15:28:56.038961+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062170+0000","last_change":"2024-04-25T15:28:56.058667+0000","last_active":"2024-04-25T15:28:56.062170+0000","last_peered":"2024-04-25T15:28:56.062170+0000","last_clean":"2024-04-25T15:28:56.062170+0000","last_became_active":"2024-04-25T15:28:56.057323+0000","last_became_peered":"2024-04-25T15:28:56.057323+0000","last_unstale":"2024-04-25T15:28:56.062170+0000","last_undegraded":"2024-04-25T15:28:56.062170+0000","last_fullsized":"2024-04-25T15:28:56.062170+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:06.756381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037203+0000","last_change":"2024-04-25T15:28:56.037203+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037203+0000","last_undegraded":"2024-04-25T15:28:56.037203+0000","last_fullsized":"2024-04-25T15:28:56.037203+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.053076+0000","last_change":"2024-04-25T15:28:56.046616+0000","last_active":"2024-04-25T15:28:56.053076+0000","last_peered":"2024-04-25T15:28:56.053076+0000","last_clean":"2024-04-25T15:28:56.053076+0000","last_became_active":"2024-04-25T15:28:56.046311+0000","last_became_peered":"2024-04-25T15:28:56.046311+0000","last_unstale":"2024-04-25T15:28:56.053076+0000","last_undegraded":"2024-04-25T15:28:56.053076+0000","last_fullsized":"2024-04-25T15:28:56.053076+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:21:45.358976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044011+0000","last_change":"2024-04-25T15:28:56.044011+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044011+0000","last_undegraded":"2024-04-25T15:28:56.044011+0000","last_fullsized":"2024-04-25T15:28:56.044011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059069+0000","last_change":"2024-04-25T15:28:56.053865+0000","last_active":"2024-04-25T15:28:56.059069+0000","last_peered":"2024-04-25T15:28:56.059069+0000","last_clean":"2024-04-25T15:28:56.059069+0000","last_became_active":"2024-04-25T15:28:56.053511+0000","last_became_peered":"2024-04-25T15:28:56.053511+0000","last_unstale":"2024-04-25T15:28:56.059069+0000","last_undegraded":"2024-04-25T15:28:56.059069+0000","last_fullsized":"2024-04-25T15:28:56.059069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044913+0000","last_change":"2024-04-25T15:28:56.044913+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044913+0000","last_undegraded":"2024-04-25T15:28:56.044913+0000","last_fullsized":"2024-04-25T15:28:56.044913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.050295+0000","last_change":"2024-04-25T15:28:56.044068+0000","last_active":"2024-04-25T15:28:56.050295+0000","last_peered":"2024-04-25T15:28:56.050295+0000","last_clean":"2024-04-25T15:28:56.050295+0000","last_became_active":"2024-04-25T15:28:56.043670+0000","last_became_peered":"2024-04-25T15:28:56.043670+0000","last_unstale":"2024-04-25T15:28:56.050295+0000","last_undegraded":"2024-04-25T15:28:56.050295+0000","last_fullsized":"2024-04-25T15:28:56.050295+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041617+0000","last_change":"2024-04-25T15:28:56.041617+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041617+0000","last_undegraded":"2024-04-25T15:28:56.041617+0000","last_fullsized":"2024-04-25T15:28:56.041617+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056097+0000","last_change":"2024-04-25T15:28:56.051968+0000","last_active":"2024-04-25T15:28:56.056097+0000","last_peered":"2024-04-25T15:28:56.056097+0000","last_clean":"2024-04-25T15:28:56.056097+0000","last_became_active":"2024-04-25T15:28:56.051236+0000","last_became_peered":"2024-04-25T15:28:56.051236+0000","last_unstale":"2024-04-25T15:28:56.056097+0000","last_undegraded":"2024-04-25T15:28:56.056097+0000","last_fullsized":"2024-04-25T15:28:56.056097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:45.569280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045084+0000","last_change":"2024-04-25T15:28:56.045084+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045084+0000","last_undegraded":"2024-04-25T15:28:56.045084+0000","last_fullsized":"2024-04-25T15:28:56.045084+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057007+0000","last_change":"2024-04-25T15:28:56.053779+0000","last_active":"2024-04-25T15:28:56.057007+0000","last_peered":"2024-04-25T15:28:56.057007+0000","last_clean":"2024-04-25T15:28:56.057007+0000","last_became_active":"2024-04-25T15:28:56.053471+0000","last_became_peered":"2024-04-25T15:28:56.053471+0000","last_unstale":"2024-04-25T15:28:56.057007+0000","last_undegraded":"2024-04-25T15:28:56.057007+0000","last_fullsized":"2024-04-25T15:28:56.057007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:25:01.410150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037150+0000","last_change":"2024-04-25T15:28:56.037150+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037150+0000","last_undegraded":"2024-04-25T15:28:56.037150+0000","last_fullsized":"2024-04-25T15:28:56.037150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059300+0000","last_change":"2024-04-25T15:28:56.052681+0000","last_active":"2024-04-25T15:28:56.059300+0000","last_peered":"2024-04-25T15:28:56.059300+0000","last_clean":"2024-04-25T15:28:56.059300+0000","last_became_active":"2024-04-25T15:28:56.052385+0000","last_became_peered":"2024-04-25T15:28:56.052385+0000","last_unstale":"2024-04-25T15:28:56.059300+0000","last_undegraded":"2024-04-25T15:28:56.059300+0000","last_fullsized":"2024-04-25T15:28:56.059300+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:45.314705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044935+0000","last_change":"2024-04-25T15:28:56.044935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044935+0000","last_undegraded":"2024-04-25T15:28:56.044935+0000","last_fullsized":"2024-04-25T15:28:56.044935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062215+0000","last_change":"2024-04-25T15:28:56.060187+0000","last_active":"2024-04-25T15:28:56.062215+0000","last_peered":"2024-04-25T15:28:56.062215+0000","last_clean":"2024-04-25T15:28:56.062215+0000","last_became_active":"2024-04-25T15:28:56.059867+0000","last_became_peered":"2024-04-25T15:28:56.059867+0000","last_unstale":"2024-04-25T15:28:56.062215+0000","last_undegraded":"2024-04-25T15:28:56.062215+0000","last_fullsized":"2024-04-25T15:28:56.062215+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:59.420234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043391+0000","last_change":"2024-04-25T15:28:56.043391+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043391+0000","last_undegraded":"2024-04-25T15:28:56.043391+0000","last_fullsized":"2024-04-25T15:28:56.043391+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059189+0000","last_change":"2024-04-25T15:28:56.052624+0000","last_active":"2024-04-25T15:28:56.059189+0000","last_peered":"2024-04-25T15:28:56.059189+0000","last_clean":"2024-04-25T15:28:56.059189+0000","last_became_active":"2024-04-25T15:28:56.052283+0000","last_became_peered":"2024-04-25T15:28:56.052283+0000","last_unstale":"2024-04-25T15:28:56.059189+0000","last_undegraded":"2024-04-25T15:28:56.059189+0000","last_fullsized":"2024-04-25T15:28:56.059189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:37.819844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041305+0000","last_change":"2024-04-25T15:28:56.041305+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041305+0000","last_undegraded":"2024-04-25T15:28:56.041305+0000","last_fullsized":"2024-04-25T15:28:56.041305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056432+0000","last_change":"2024-04-25T15:28:56.050894+0000","last_active":"2024-04-25T15:28:56.056432+0000","last_peered":"2024-04-25T15:28:56.056432+0000","last_clean":"2024-04-25T15:28:56.056432+0000","last_became_active":"2024-04-25T15:28:56.050570+0000","last_became_peered":"2024-04-25T15:28:56.050570+0000","last_unstale":"2024-04-25T15:28:56.056432+0000","last_undegraded":"2024-04-25T15:28:56.056432+0000","last_fullsized":"2024-04-25T15:28:56.056432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044386+0000","last_change":"2024-04-25T15:28:56.044386+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044386+0000","last_undegraded":"2024-04-25T15:28:56.044386+0000","last_fullsized":"2024-04-25T15:28:56.044386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.049680+0000","last_change":"2024-04-25T15:28:56.045642+0000","last_active":"2024-04-25T15:28:56.049680+0000","last_peered":"2024-04-25T15:28:56.049680+0000","last_clean":"2024-04-25T15:28:56.049680+0000","last_became_active":"2024-04-25T15:28:56.045273+0000","last_became_peered":"2024-04-25T15:28:56.045273+0000","last_unstale":"2024-04-25T15:28:56.049680+0000","last_undegraded":"2024-04-25T15:28:56.049680+0000","last_fullsized":"2024-04-25T15:28:56.049680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:21.737399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044193+0000","last_change":"2024-04-25T15:28:56.044193+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044193+0000","last_undegraded":"2024-04-25T15:28:56.044193+0000","last_fullsized":"2024-04-25T15:28:56.044193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062036+0000","last_change":"2024-04-25T15:28:56.059690+0000","last_active":"2024-04-25T15:28:56.062036+0000","last_peered":"2024-04-25T15:28:56.062036+0000","last_clean":"2024-04-25T15:28:56.062036+0000","last_became_active":"2024-04-25T15:28:56.059372+0000","last_became_peered":"2024-04-25T15:28:56.059372+0000","last_unstale":"2024-04-25T15:28:56.062036+0000","last_undegraded":"2024-04-25T15:28:56.062036+0000","last_fullsized":"2024-04-25T15:28:56.062036+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:03.773654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044493+0000","last_change":"2024-04-25T15:28:56.044493+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044493+0000","last_undegraded":"2024-04-25T15:28:56.044493+0000","last_fullsized":"2024-04-25T15:28:56.044493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056674+0000","last_change":"2024-04-25T15:28:56.053555+0000","last_active":"2024-04-25T15:28:56.056674+0000","last_peered":"2024-04-25T15:28:56.056674+0000","last_clean":"2024-04-25T15:28:56.056674+0000","last_became_active":"2024-04-25T15:28:56.053235+0000","last_became_peered":"2024-04-25T15:28:56.053235+0000","last_unstale":"2024-04-25T15:28:56.056674+0000","last_undegraded":"2024-04-25T15:28:56.056674+0000","last_fullsized":"2024-04-25T15:28:56.056674+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043401+0000","last_change":"2024-04-25T15:28:56.043401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043401+0000","last_undegraded":"2024-04-25T15:28:56.043401+0000","last_fullsized":"2024-04-25T15:28:56.043401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.054456+0000","last_change":"2024-04-25T15:28:56.049930+0000","last_active":"2024-04-25T15:28:56.054456+0000","last_peered":"2024-04-25T15:28:56.054456+0000","last_clean":"2024-04-25T15:28:56.054456+0000","last_became_active":"2024-04-25T15:28:56.049617+0000","last_became_peered":"2024-04-25T15:28:56.049617+0000","last_unstale":"2024-04-25T15:28:56.054456+0000","last_undegraded":"2024-04-25T15:28:56.054456+0000","last_fullsized":"2024-04-25T15:28:56.054456+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:29.266322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042363+0000","last_change":"2024-04-25T15:28:56.042363+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042363+0000","last_undegraded":"2024-04-25T15:28:56.042363+0000","last_fullsized":"2024-04-25T15:28:56.042363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057275+0000","last_change":"2024-04-25T15:28:56.054470+0000","last_active":"2024-04-25T15:28:56.057275+0000","last_peered":"2024-04-25T15:28:56.057275+0000","last_clean":"2024-04-25T15:28:56.057275+0000","last_became_active":"2024-04-25T15:28:56.054148+0000","last_became_peered":"2024-04-25T15:28:56.054148+0000","last_unstale":"2024-04-25T15:28:56.057275+0000","last_undegraded":"2024-04-25T15:28:56.057275+0000","last_fullsized":"2024-04-25T15:28:56.057275+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037520+0000","last_change":"2024-04-25T15:28:56.037520+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037520+0000","last_undegraded":"2024-04-25T15:28:56.037520+0000","last_fullsized":"2024-04-25T15:28:56.037520+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056659+0000","last_change":"2024-04-25T15:28:56.051763+0000","last_active":"2024-04-25T15:28:56.056659+0000","last_peered":"2024-04-25T15:28:56.056659+0000","last_clean":"2024-04-25T15:28:56.056659+0000","last_became_active":"2024-04-25T15:28:56.051431+0000","last_became_peered":"2024-04-25T15:28:56.051431+0000","last_unstale":"2024-04-25T15:28:56.056659+0000","last_undegraded":"2024-04-25T15:28:56.056659+0000","last_fullsized":"2024-04-25T15:28:56.056659+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:44:13.470300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052252+0000","last_change":"2024-04-25T15:28:56.052252+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052252+0000","last_undegraded":"2024-04-25T15:28:56.052252+0000","last_fullsized":"2024-04-25T15:28:56.052252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060700+0000","last_change":"2024-04-25T15:28:56.059005+0000","last_active":"2024-04-25T15:28:56.060700+0000","last_peered":"2024-04-25T15:28:56.060700+0000","last_clean":"2024-04-25T15:28:56.060700+0000","last_became_active":"2024-04-25T15:28:56.057981+0000","last_became_peered":"2024-04-25T15:28:56.057981+0000","last_unstale":"2024-04-25T15:28:56.060700+0000","last_undegraded":"2024-04-25T15:28:56.060700+0000","last_fullsized":"2024-04-25T15:28:56.060700+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045632+0000","last_change":"2024-04-25T15:28:56.045632+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045632+0000","last_undegraded":"2024-04-25T15:28:56.045632+0000","last_fullsized":"2024-04-25T15:28:56.045632+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056297+0000","last_change":"2024-04-25T15:28:56.048576+0000","last_active":"2024-04-25T15:28:56.056297+0000","last_peered":"2024-04-25T15:28:56.056297+0000","last_clean":"2024-04-25T15:28:56.056297+0000","last_became_active":"2024-04-25T15:28:56.048273+0000","last_became_peered":"2024-04-25T15:28:56.048273+0000","last_unstale":"2024-04-25T15:28:56.056297+0000","last_undegraded":"2024-04-25T15:28:56.056297+0000","last_fullsized":"2024-04-25T15:28:56.056297+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042145+0000","last_change":"2024-04-25T15:28:56.042145+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042145+0000","last_undegraded":"2024-04-25T15:28:56.042145+0000","last_fullsized":"2024-04-25T15:28:56.042145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056488+0000","last_change":"2024-04-25T15:28:56.051473+0000","last_active":"2024-04-25T15:28:56.056488+0000","last_peered":"2024-04-25T15:28:56.056488+0000","last_clean":"2024-04-25T15:28:56.056488+0000","last_became_active":"2024-04-25T15:28:56.051171+0000","last_became_peered":"2024-04-25T15:28:56.051171+0000","last_unstale":"2024-04-25T15:28:56.056488+0000","last_undegraded":"2024-04-25T15:28:56.056488+0000","last_fullsized":"2024-04-25T15:28:56.056488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042897+0000","last_change":"2024-04-25T15:28:56.042897+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042897+0000","last_undegraded":"2024-04-25T15:28:56.042897+0000","last_fullsized":"2024-04-25T15:28:56.042897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060976+0000","last_change":"2024-04-25T15:28:56.058257+0000","last_active":"2024-04-25T15:28:56.060976+0000","last_peered":"2024-04-25T15:28:56.060976+0000","last_clean":"2024-04-25T15:28:56.060976+0000","last_became_active":"2024-04-25T15:28:56.057736+0000","last_became_peered":"2024-04-25T15:28:56.057736+0000","last_unstale":"2024-04-25T15:28:56.060976+0000","last_undegraded":"2024-04-25T15:28:56.060976+0000","last_fullsized":"2024-04-25T15:28:56.060976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036658+0000","last_change":"2024-04-25T15:28:56.036658+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036658+0000","last_undegraded":"2024-04-25T15:28:56.036658+0000","last_fullsized":"2024-04-25T15:28:56.036658+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.061871+0000","last_change":"2024-04-25T15:28:56.056141+0000","last_active":"2024-04-25T15:28:56.061871+0000","last_peered":"2024-04-25T15:28:56.061871+0000","last_clean":"2024-04-25T15:28:56.061871+0000","last_became_active":"2024-04-25T15:28:56.055725+0000","last_became_peered":"2024-04-25T15:28:56.055725+0000","last_unstale":"2024-04-25T15:28:56.061871+0000","last_undegraded":"2024-04-25T15:28:56.061871+0000","last_fullsized":"2024-04-25T15:28:56.061871+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042444+0000","last_change":"2024-04-25T15:28:56.042444+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042444+0000","last_undegraded":"2024-04-25T15:28:56.042444+0000","last_fullsized":"2024-04-25T15:28:56.042444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060626+0000","last_change":"2024-04-25T15:28:56.055311+0000","last_active":"2024-04-25T15:28:56.060626+0000","last_peered":"2024-04-25T15:28:56.060626+0000","last_clean":"2024-04-25T15:28:56.060626+0000","last_became_active":"2024-04-25T15:28:56.054958+0000","last_became_peered":"2024-04-25T15:28:56.054958+0000","last_unstale":"2024-04-25T15:28:56.060626+0000","last_undegraded":"2024-04-25T15:28:56.060626+0000","last_fullsized":"2024-04-25T15:28:56.060626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:04.302144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041713+0000","last_change":"2024-04-25T15:28:56.041713+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041713+0000","last_undegraded":"2024-04-25T15:28:56.041713+0000","last_fullsized":"2024-04-25T15:28:56.041713+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.051579+0000","last_change":"2024-04-25T15:28:56.050355+0000","last_active":"2024-04-25T15:28:56.051579+0000","last_peered":"2024-04-25T15:28:56.051579+0000","last_clean":"2024-04-25T15:28:56.051579+0000","last_became_active":"2024-04-25T15:28:56.050028+0000","last_became_peered":"2024-04-25T15:28:56.050028+0000","last_unstale":"2024-04-25T15:28:56.051579+0000","last_undegraded":"2024-04-25T15:28:56.051579+0000","last_fullsized":"2024-04-25T15:28:56.051579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:18.740971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044259+0000","last_change":"2024-04-25T15:28:56.044259+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044259+0000","last_undegraded":"2024-04-25T15:28:56.044259+0000","last_fullsized":"2024-04-25T15:28:56.044259+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059127+0000","last_change":"2024-04-25T15:28:56.054041+0000","last_active":"2024-04-25T15:28:56.059127+0000","last_peered":"2024-04-25T15:28:56.059127+0000","last_clean":"2024-04-25T15:28:56.059127+0000","last_became_active":"2024-04-25T15:28:56.053693+0000","last_became_peered":"2024-04-25T15:28:56.053693+0000","last_unstale":"2024-04-25T15:28:56.059127+0000","last_undegraded":"2024-04-25T15:28:56.059127+0000","last_fullsized":"2024-04-25T15:28:56.059127+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":235,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.043621+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:56.043621+0000","last_peered":"2024-04-25T15:28:56.043621+0000","last_clean":"2024-04-25T15:28:56.043621+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:56.043621+0000","last_undegraded":"2024-04-25T15:28:56.043621+0000","last_fullsized":"2024-04-25T15:28:56.043621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:01.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.648+0000 7fbe69a20640 1 Processor -- start 2024-04-25T15:29:01.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.648+0000 7fbe69a20640 1 -- start start 2024-04-25T15:29:01.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.649+0000 7fbe69a20640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbe640d0e00 0x7fbe640d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.649+0000 7fbe69a20640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe640d4a50 0x7fbe640d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.649+0000 7fbe69a20640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe640d1850 con 0x7fbe6406c5c0 2024-04-25T15:29:01.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.649+0000 7fbe69a20640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbe64160de0 con 0x7fbe640d4a50 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.649+0000 7fbe69a20640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbe641610c0 con 0x7fbe640d0e00 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.649+0000 7fbe637fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe640d4a50 0x7fbe640d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe627fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbe6406c5c0 0x7fbe6406c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40810/0 (socket says 172.21.15.67:40810) 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe627fc640 1 -- 172.21.15.67:0/613429779 learned_addr learned my addr 172.21.15.67:0/613429779 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe637fe640 1 -- 172.21.15.67:0/613429779 >> v2:172.21.15.176:3300/0 conn(0x7fbe640d0e00 msgr2=0x7fbe640d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe62ffd640 1 --2- 172.21.15.67:0/613429779 >> v2:172.21.15.176:3300/0 conn(0x7fbe640d0e00 0x7fbe640d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe61ffb640 1 -- 172.21.15.67:0/613429779 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066601413 0 0) 0x7fbe640d1850 con 0x7fbe6406c5c0 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe637fe640 1 --2- 172.21.15.67:0/613429779 >> v2:172.21.15.176:3300/0 conn(0x7fbe640d0e00 0x7fbe640d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe637fe640 1 -- 172.21.15.67:0/613429779 >> v1:172.21.15.67:6789/0 conn(0x7fbe6406c5c0 legacy=0x7fbe6406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe637fe640 1 -- 172.21.15.67:0/613429779 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe641613a0 con 0x7fbe640d4a50 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.650+0000 7fbe62ffd640 1 --2- 172.21.15.67:0/613429779 >> v2:172.21.15.176:3300/0 conn(0x7fbe640d0e00 0x7fbe640d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe637fe640 1 --2- 172.21.15.67:0/613429779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640d4e60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fbe640d3e60 tx=0x7fbe5c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24d7fb272c2a94e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe61ffb640 1 -- 172.21.15.67:0/613429779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c007740 con 0x7fbe640d4a50 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe61ffb640 1 -- 172.21.15.67:0/613429779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5c0078e0 con 0x7fbe640d4a50 2024-04-25T15:29:01.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe61ffb640 1 -- 172.21.15.67:0/613429779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c007cb0 con 0x7fbe640d4a50 2024-04-25T15:29:01.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 -- 172.21.15.67:0/613429779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 msgr2=0x7fbe640d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 --2- 172.21.15.67:0/613429779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640d4e60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fbe640d3e60 tx=0x7fbe5c02a5f0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 -- 172.21.15.67:0/613429779 shutdown_connections 2024-04-25T15:29:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 --2- 172.21.15.67:0/613429779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640d4e60 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 --2- 172.21.15.67:0/613429779 >> v2:172.21.15.176:3300/0 conn(0x7fbe640d0e00 0x7fbe640d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 -- 172.21.15.67:0/613429779 >> 172.21.15.67:0/613429779 conn(0x7fbe640d5d90 msgr2=0x7fbe640d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 -- 172.21.15.67:0/613429779 shutdown_connections 2024-04-25T15:29:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.651+0000 7fbe69a20640 1 -- 172.21.15.67:0/613429779 wait complete. 2024-04-25T15:29:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.653+0000 7fbe69a20640 1 Processor -- start 2024-04-25T15:29:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.653+0000 7fbe69a20640 1 -- start start 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe69a20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406c5c0 0x7fbe640caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe69a20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe640d0e00 0x7fbe640c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe69a20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe69a20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe64164e00 con 0x7fbe640d4a50 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe69a20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe640b5320 con 0x7fbe640d0e00 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe69a20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe64164f80 con 0x7fbe6406c5c0 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640cb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57780/0 (socket says 172.21.15.67:57780) 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 -- 172.21.15.67:0/106767968 learned_addr learned my addr 172.21.15.67:0/106767968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe627fc640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe640d0e00 0x7fbe640c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe62ffd640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406c5c0 0x7fbe640caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 -- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406c5c0 msgr2=0x7fbe640caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406c5c0 0x7fbe640caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 -- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe640d0e00 msgr2=0x7fbe640c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe640d0e00 0x7fbe640c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 -- 172.21.15.67:0/106767968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe640c9e00 con 0x7fbe640d4a50 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe637fe640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640cb770 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c02abe0 tx=0x7fbe5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.655+0000 7fbe4bfff640 1 -- 172.21.15.67:0/106767968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c03f070 con 0x7fbe640d4a50 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.656+0000 7fbe4bfff640 1 -- 172.21.15.67:0/106767968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5c031c90 con 0x7fbe640d4a50 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.656+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe640ca0e0 con 0x7fbe640d4a50 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.656+0000 7fbe4bfff640 1 -- 172.21.15.67:0/106767968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c044680 con 0x7fbe640d4a50 2024-04-25T15:29:01.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.656+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe640ca6a0 con 0x7fbe640d4a50 2024-04-25T15:29:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.657+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe64050740 con 0x7fbe640d4a50 2024-04-25T15:29:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.657+0000 7fbe4bfff640 1 -- 172.21.15.67:0/106767968 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbe5c03b410 con 0x7fbe640d4a50 2024-04-25T15:29:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.658+0000 7fbe4bfff640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe500744b0 0x7fbe500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.658+0000 7fbe4bfff640 1 -- 172.21.15.67:0/106767968 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fbe5c0b6b40 con 0x7fbe640d4a50 2024-04-25T15:29:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.658+0000 7fbe62ffd640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe500744b0 0x7fbe500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.659+0000 7fbe62ffd640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe500744b0 0x7fbe500769c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400aac0 tx=0x7fbe54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:01.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.662+0000 7fbe4bfff640 1 -- 172.21.15.67:0/106767968 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe5c080d40 con 0x7fbe640d4a50 2024-04-25T15:29:01.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.844+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe640d1560 con 0x7fbe500744b0 2024-04-25T15:29:01.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.850+0000 7fbe4bfff640 1 -- 172.21.15.67:0/106767968 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246754 (secure 0 0 0) 0x7fbe640d1560 con 0x7fbe500744b0 2024-04-25T15:29:01.854 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:01.855 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe500744b0 msgr2=0x7fbe500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe500744b0 0x7fbe500769c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400aac0 tx=0x7fbe54008040 comp rx=0 tx=0).stop 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 msgr2=0x7fbe640cb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640cb770 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c02abe0 tx=0x7fbe5c004a40 comp rx=0 tx=0).stop 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 shutdown_connections 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fbe500744b0 0x7fbe500769c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe640d4a50 0x7fbe640cb770 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe640d0e00 0x7fbe640c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 --2- 172.21.15.67:0/106767968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406c5c0 0x7fbe640caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.853+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 >> 172.21.15.67:0/106767968 conn(0x7fbe640d5d90 msgr2=0x7fbe641632c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.854+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 shutdown_connections 2024-04-25T15:29:01.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:01.854+0000 7fbe69a20640 1 -- 172.21.15.67:0/106767968 wait complete. 2024-04-25T15:29:01.865 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-25T15:29:00.847967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242652,"kb_used_data":9840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754732068,"statfs":{"total":773094113280,"available":772845637632,"internally_reserved":0,"allocated":10076160,"data_stored":4946690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.818182"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052980+0000","last_change":"2024-04-25T15:28:56.052980+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052980+0000","last_undegraded":"2024-04-25T15:28:56.052980+0000","last_fullsized":"2024-04-25T15:28:56.052980+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040106+0000","last_change":"2024-04-25T15:28:56.040106+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040106+0000","last_undegraded":"2024-04-25T15:28:56.040106+0000","last_fullsized":"2024-04-25T15:28:56.040106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044401+0000","last_change":"2024-04-25T15:28:56.044401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044401+0000","last_undegraded":"2024-04-25T15:28:56.044401+0000","last_fullsized":"2024-04-25T15:28:56.044401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044161+0000","last_change":"2024-04-25T15:28:56.044161+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044161+0000","last_undegraded":"2024-04-25T15:28:56.044161+0000","last_fullsized":"2024-04-25T15:28:56.044161+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040956+0000","last_change":"2024-04-25T15:28:56.040956+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040956+0000","last_undegraded":"2024-04-25T15:28:56.040956+0000","last_fullsized":"2024-04-25T15:28:56.040956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043783+0000","last_change":"2024-04-25T15:28:56.043783+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043783+0000","last_undegraded":"2024-04-25T15:28:56.043783+0000","last_fullsized":"2024-04-25T15:28:56.043783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043768+0000","last_change":"2024-04-25T15:28:56.043768+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043768+0000","last_undegraded":"2024-04-25T15:28:56.043768+0000","last_fullsized":"2024-04-25T15:28:56.043768+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038877+0000","last_change":"2024-04-25T15:28:56.038877+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038877+0000","last_undegraded":"2024-04-25T15:28:56.038877+0000","last_fullsized":"2024-04-25T15:28:56.038877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037775+0000","last_change":"2024-04-25T15:28:56.037775+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037775+0000","last_undegraded":"2024-04-25T15:28:56.037775+0000","last_fullsized":"2024-04-25T15:28:56.037775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041034+0000","last_change":"2024-04-25T15:28:56.041034+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041034+0000","last_undegraded":"2024-04-25T15:28:56.041034+0000","last_fullsized":"2024-04-25T15:28:56.041034+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046085+0000","last_change":"2024-04-25T15:28:56.046085+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046085+0000","last_undegraded":"2024-04-25T15:28:56.046085+0000","last_fullsized":"2024-04-25T15:28:56.046085+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039309+0000","last_change":"2024-04-25T15:28:56.039309+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039309+0000","last_undegraded":"2024-04-25T15:28:56.039309+0000","last_fullsized":"2024-04-25T15:28:56.039309+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043640+0000","last_change":"2024-04-25T15:28:56.043640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043640+0000","last_undegraded":"2024-04-25T15:28:56.043640+0000","last_fullsized":"2024-04-25T15:28:56.043640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047935+0000","last_change":"2024-04-25T15:28:56.047935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047935+0000","last_undegraded":"2024-04-25T15:28:56.047935+0000","last_fullsized":"2024-04-25T15:28:56.047935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043253+0000","last_change":"2024-04-25T15:28:56.043253+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043253+0000","last_undegraded":"2024-04-25T15:28:56.043253+0000","last_fullsized":"2024-04-25T15:28:56.043253+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043312+0000","last_change":"2024-04-25T15:28:56.043312+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043312+0000","last_undegraded":"2024-04-25T15:28:56.043312+0000","last_fullsized":"2024-04-25T15:28:56.043312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038794+0000","last_change":"2024-04-25T15:28:56.038794+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038794+0000","last_undegraded":"2024-04-25T15:28:56.038794+0000","last_fullsized":"2024-04-25T15:28:56.038794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043094+0000","last_change":"2024-04-25T15:28:56.043094+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043094+0000","last_undegraded":"2024-04-25T15:28:56.043094+0000","last_fullsized":"2024-04-25T15:28:56.043094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047300+0000","last_change":"2024-04-25T15:28:56.047300+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047300+0000","last_undegraded":"2024-04-25T15:28:56.047300+0000","last_fullsized":"2024-04-25T15:28:56.047300+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037516+0000","last_change":"2024-04-25T15:28:56.037516+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037516+0000","last_undegraded":"2024-04-25T15:28:56.037516+0000","last_fullsized":"2024-04-25T15:28:56.037516+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046746+0000","last_change":"2024-04-25T15:28:56.046746+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046746+0000","last_undegraded":"2024-04-25T15:28:56.046746+0000","last_fullsized":"2024-04-25T15:28:56.046746+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036501+0000","last_change":"2024-04-25T15:28:56.036501+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036501+0000","last_undegraded":"2024-04-25T15:28:56.036501+0000","last_fullsized":"2024-04-25T15:28:56.036501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036507+0000","last_change":"2024-04-25T15:28:56.036507+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036507+0000","last_undegraded":"2024-04-25T15:28:56.036507+0000","last_fullsized":"2024-04-25T15:28:56.036507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042774+0000","last_change":"2024-04-25T15:28:56.042774+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042774+0000","last_undegraded":"2024-04-25T15:28:56.042774+0000","last_fullsized":"2024-04-25T15:28:56.042774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043848+0000","last_change":"2024-04-25T15:28:56.043848+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043848+0000","last_undegraded":"2024-04-25T15:28:56.043848+0000","last_fullsized":"2024-04-25T15:28:56.043848+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039503+0000","last_change":"2024-04-25T15:28:56.039503+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039503+0000","last_undegraded":"2024-04-25T15:28:56.039503+0000","last_fullsized":"2024-04-25T15:28:56.039503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045584+0000","last_change":"2024-04-25T15:28:56.045584+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045584+0000","last_undegraded":"2024-04-25T15:28:56.045584+0000","last_fullsized":"2024-04-25T15:28:56.045584+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045164+0000","last_change":"2024-04-25T15:28:56.045164+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045164+0000","last_undegraded":"2024-04-25T15:28:56.045164+0000","last_fullsized":"2024-04-25T15:28:56.045164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039640+0000","last_change":"2024-04-25T15:28:56.039640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039640+0000","last_undegraded":"2024-04-25T15:28:56.039640+0000","last_fullsized":"2024-04-25T15:28:56.039640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046096+0000","last_change":"2024-04-25T15:28:56.046096+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046096+0000","last_undegraded":"2024-04-25T15:28:56.046096+0000","last_fullsized":"2024-04-25T15:28:56.046096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042753+0000","last_change":"2024-04-25T15:28:56.042753+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042753+0000","last_undegraded":"2024-04-25T15:28:56.042753+0000","last_fullsized":"2024-04-25T15:28:56.042753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.049201+0000","last_change":"2024-04-25T15:28:56.049201+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.049201+0000","last_undegraded":"2024-04-25T15:28:56.049201+0000","last_fullsized":"2024-04-25T15:28:56.049201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045572+0000","last_change":"2024-04-25T15:28:56.045572+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045572+0000","last_undegraded":"2024-04-25T15:28:56.045572+0000","last_fullsized":"2024-04-25T15:28:56.045572+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060682+0000","last_change":"2024-04-25T15:28:56.059439+0000","last_active":"2024-04-25T15:28:56.060682+0000","last_peered":"2024-04-25T15:28:56.060682+0000","last_clean":"2024-04-25T15:28:56.060682+0000","last_became_active":"2024-04-25T15:28:56.059082+0000","last_became_peered":"2024-04-25T15:28:56.059082+0000","last_unstale":"2024-04-25T15:28:56.060682+0000","last_undegraded":"2024-04-25T15:28:56.060682+0000","last_fullsized":"2024-04-25T15:28:56.060682+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042730+0000","last_change":"2024-04-25T15:28:56.042730+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042730+0000","last_undegraded":"2024-04-25T15:28:56.042730+0000","last_fullsized":"2024-04-25T15:28:56.042730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059155+0000","last_change":"2024-04-25T15:28:56.055939+0000","last_active":"2024-04-25T15:28:56.059155+0000","last_peered":"2024-04-25T15:28:56.059155+0000","last_clean":"2024-04-25T15:28:56.059155+0000","last_became_active":"2024-04-25T15:28:56.055541+0000","last_became_peered":"2024-04-25T15:28:56.055541+0000","last_unstale":"2024-04-25T15:28:56.059155+0000","last_undegraded":"2024-04-25T15:28:56.059155+0000","last_fullsized":"2024-04-25T15:28:56.059155+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046120+0000","last_change":"2024-04-25T15:28:56.046120+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046120+0000","last_undegraded":"2024-04-25T15:28:56.046120+0000","last_fullsized":"2024-04-25T15:28:56.046120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060146+0000","last_change":"2024-04-25T15:28:56.056907+0000","last_active":"2024-04-25T15:28:56.060146+0000","last_peered":"2024-04-25T15:28:56.060146+0000","last_clean":"2024-04-25T15:28:56.060146+0000","last_became_active":"2024-04-25T15:28:56.056549+0000","last_became_peered":"2024-04-25T15:28:56.056549+0000","last_unstale":"2024-04-25T15:28:56.060146+0000","last_undegraded":"2024-04-25T15:28:56.060146+0000","last_fullsized":"2024-04-25T15:28:56.060146+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037437+0000","last_change":"2024-04-25T15:28:56.037437+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037437+0000","last_undegraded":"2024-04-25T15:28:56.037437+0000","last_fullsized":"2024-04-25T15:28:56.037437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059710+0000","last_change":"2024-04-25T15:28:56.057126+0000","last_active":"2024-04-25T15:28:56.059710+0000","last_peered":"2024-04-25T15:28:56.059710+0000","last_clean":"2024-04-25T15:28:56.059710+0000","last_became_active":"2024-04-25T15:28:56.056619+0000","last_became_peered":"2024-04-25T15:28:56.056619+0000","last_unstale":"2024-04-25T15:28:56.059710+0000","last_undegraded":"2024-04-25T15:28:56.059710+0000","last_fullsized":"2024-04-25T15:28:56.059710+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:50.204782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040410+0000","last_change":"2024-04-25T15:28:56.040410+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040410+0000","last_undegraded":"2024-04-25T15:28:56.040410+0000","last_fullsized":"2024-04-25T15:28:56.040410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062504+0000","last_change":"2024-04-25T15:28:56.061172+0000","last_active":"2024-04-25T15:28:56.062504+0000","last_peered":"2024-04-25T15:28:56.062504+0000","last_clean":"2024-04-25T15:28:56.062504+0000","last_became_active":"2024-04-25T15:28:56.060821+0000","last_became_peered":"2024-04-25T15:28:56.060821+0000","last_unstale":"2024-04-25T15:28:56.062504+0000","last_undegraded":"2024-04-25T15:28:56.062504+0000","last_fullsized":"2024-04-25T15:28:56.062504+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:58.403814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044043+0000","last_change":"2024-04-25T15:28:56.044043+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044043+0000","last_undegraded":"2024-04-25T15:28:56.044043+0000","last_fullsized":"2024-04-25T15:28:56.044043+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056409+0000","last_change":"2024-04-25T15:28:56.053676+0000","last_active":"2024-04-25T15:28:56.056409+0000","last_peered":"2024-04-25T15:28:56.056409+0000","last_clean":"2024-04-25T15:28:56.056409+0000","last_became_active":"2024-04-25T15:28:56.053355+0000","last_became_peered":"2024-04-25T15:28:56.053355+0000","last_unstale":"2024-04-25T15:28:56.056409+0000","last_undegraded":"2024-04-25T15:28:56.056409+0000","last_fullsized":"2024-04-25T15:28:56.056409+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042179+0000","last_change":"2024-04-25T15:28:56.042179+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042179+0000","last_undegraded":"2024-04-25T15:28:56.042179+0000","last_fullsized":"2024-04-25T15:28:56.042179+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056491+0000","last_change":"2024-04-25T15:28:56.051321+0000","last_active":"2024-04-25T15:28:56.056491+0000","last_peered":"2024-04-25T15:28:56.056491+0000","last_clean":"2024-04-25T15:28:56.056491+0000","last_became_active":"2024-04-25T15:28:56.050541+0000","last_became_peered":"2024-04-25T15:28:56.050541+0000","last_unstale":"2024-04-25T15:28:56.056491+0000","last_undegraded":"2024-04-25T15:28:56.056491+0000","last_fullsized":"2024-04-25T15:28:56.056491+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036701+0000","last_change":"2024-04-25T15:28:56.036701+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036701+0000","last_undegraded":"2024-04-25T15:28:56.036701+0000","last_fullsized":"2024-04-25T15:28:56.036701+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062513+0000","last_change":"2024-04-25T15:28:56.061338+0000","last_active":"2024-04-25T15:28:56.062513+0000","last_peered":"2024-04-25T15:28:56.062513+0000","last_clean":"2024-04-25T15:28:56.062513+0000","last_became_active":"2024-04-25T15:28:56.061028+0000","last_became_peered":"2024-04-25T15:28:56.061028+0000","last_unstale":"2024-04-25T15:28:56.062513+0000","last_undegraded":"2024-04-25T15:28:56.062513+0000","last_fullsized":"2024-04-25T15:28:56.062513+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:59.070550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038961+0000","last_change":"2024-04-25T15:28:56.038961+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038961+0000","last_undegraded":"2024-04-25T15:28:56.038961+0000","last_fullsized":"2024-04-25T15:28:56.038961+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062170+0000","last_change":"2024-04-25T15:28:56.058667+0000","last_active":"2024-04-25T15:28:56.062170+0000","last_peered":"2024-04-25T15:28:56.062170+0000","last_clean":"2024-04-25T15:28:56.062170+0000","last_became_active":"2024-04-25T15:28:56.057323+0000","last_became_peered":"2024-04-25T15:28:56.057323+0000","last_unstale":"2024-04-25T15:28:56.062170+0000","last_undegraded":"2024-04-25T15:28:56.062170+0000","last_fullsized":"2024-04-25T15:28:56.062170+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:06.756381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037203+0000","last_change":"2024-04-25T15:28:56.037203+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037203+0000","last_undegraded":"2024-04-25T15:28:56.037203+0000","last_fullsized":"2024-04-25T15:28:56.037203+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.053076+0000","last_change":"2024-04-25T15:28:56.046616+0000","last_active":"2024-04-25T15:28:56.053076+0000","last_peered":"2024-04-25T15:28:56.053076+0000","last_clean":"2024-04-25T15:28:56.053076+0000","last_became_active":"2024-04-25T15:28:56.046311+0000","last_became_peered":"2024-04-25T15:28:56.046311+0000","last_unstale":"2024-04-25T15:28:56.053076+0000","last_undegraded":"2024-04-25T15:28:56.053076+0000","last_fullsized":"2024-04-25T15:28:56.053076+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:21:45.358976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044011+0000","last_change":"2024-04-25T15:28:56.044011+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044011+0000","last_undegraded":"2024-04-25T15:28:56.044011+0000","last_fullsized":"2024-04-25T15:28:56.044011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059069+0000","last_change":"2024-04-25T15:28:56.053865+0000","last_active":"2024-04-25T15:28:56.059069+0000","last_peered":"2024-04-25T15:28:56.059069+0000","last_clean":"2024-04-25T15:28:56.059069+0000","last_became_active":"2024-04-25T15:28:56.053511+0000","last_became_peered":"2024-04-25T15:28:56.053511+0000","last_unstale":"2024-04-25T15:28:56.059069+0000","last_undegraded":"2024-04-25T15:28:56.059069+0000","last_fullsized":"2024-04-25T15:28:56.059069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044913+0000","last_change":"2024-04-25T15:28:56.044913+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044913+0000","last_undegraded":"2024-04-25T15:28:56.044913+0000","last_fullsized":"2024-04-25T15:28:56.044913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.050295+0000","last_change":"2024-04-25T15:28:56.044068+0000","last_active":"2024-04-25T15:28:56.050295+0000","last_peered":"2024-04-25T15:28:56.050295+0000","last_clean":"2024-04-25T15:28:56.050295+0000","last_became_active":"2024-04-25T15:28:56.043670+0000","last_became_peered":"2024-04-25T15:28:56.043670+0000","last_unstale":"2024-04-25T15:28:56.050295+0000","last_undegraded":"2024-04-25T15:28:56.050295+0000","last_fullsized":"2024-04-25T15:28:56.050295+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041617+0000","last_change":"2024-04-25T15:28:56.041617+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041617+0000","last_undegraded":"2024-04-25T15:28:56.041617+0000","last_fullsized":"2024-04-25T15:28:56.041617+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056097+0000","last_change":"2024-04-25T15:28:56.051968+0000","last_active":"2024-04-25T15:28:56.056097+0000","last_peered":"2024-04-25T15:28:56.056097+0000","last_clean":"2024-04-25T15:28:56.056097+0000","last_became_active":"2024-04-25T15:28:56.051236+0000","last_became_peered":"2024-04-25T15:28:56.051236+0000","last_unstale":"2024-04-25T15:28:56.056097+0000","last_undegraded":"2024-04-25T15:28:56.056097+0000","last_fullsized":"2024-04-25T15:28:56.056097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:45.569280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045084+0000","last_change":"2024-04-25T15:28:56.045084+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045084+0000","last_undegraded":"2024-04-25T15:28:56.045084+0000","last_fullsized":"2024-04-25T15:28:56.045084+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057007+0000","last_change":"2024-04-25T15:28:56.053779+0000","last_active":"2024-04-25T15:28:56.057007+0000","last_peered":"2024-04-25T15:28:56.057007+0000","last_clean":"2024-04-25T15:28:56.057007+0000","last_became_active":"2024-04-25T15:28:56.053471+0000","last_became_peered":"2024-04-25T15:28:56.053471+0000","last_unstale":"2024-04-25T15:28:56.057007+0000","last_undegraded":"2024-04-25T15:28:56.057007+0000","last_fullsized":"2024-04-25T15:28:56.057007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:25:01.410150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037150+0000","last_change":"2024-04-25T15:28:56.037150+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037150+0000","last_undegraded":"2024-04-25T15:28:56.037150+0000","last_fullsized":"2024-04-25T15:28:56.037150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059300+0000","last_change":"2024-04-25T15:28:56.052681+0000","last_active":"2024-04-25T15:28:56.059300+0000","last_peered":"2024-04-25T15:28:56.059300+0000","last_clean":"2024-04-25T15:28:56.059300+0000","last_became_active":"2024-04-25T15:28:56.052385+0000","last_became_peered":"2024-04-25T15:28:56.052385+0000","last_unstale":"2024-04-25T15:28:56.059300+0000","last_undegraded":"2024-04-25T15:28:56.059300+0000","last_fullsized":"2024-04-25T15:28:56.059300+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:45.314705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044935+0000","last_change":"2024-04-25T15:28:56.044935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044935+0000","last_undegraded":"2024-04-25T15:28:56.044935+0000","last_fullsized":"2024-04-25T15:28:56.044935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062215+0000","last_change":"2024-04-25T15:28:56.060187+0000","last_active":"2024-04-25T15:28:56.062215+0000","last_peered":"2024-04-25T15:28:56.062215+0000","last_clean":"2024-04-25T15:28:56.062215+0000","last_became_active":"2024-04-25T15:28:56.059867+0000","last_became_peered":"2024-04-25T15:28:56.059867+0000","last_unstale":"2024-04-25T15:28:56.062215+0000","last_undegraded":"2024-04-25T15:28:56.062215+0000","last_fullsized":"2024-04-25T15:28:56.062215+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:59.420234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043391+0000","last_change":"2024-04-25T15:28:56.043391+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043391+0000","last_undegraded":"2024-04-25T15:28:56.043391+0000","last_fullsized":"2024-04-25T15:28:56.043391+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059189+0000","last_change":"2024-04-25T15:28:56.052624+0000","last_active":"2024-04-25T15:28:56.059189+0000","last_peered":"2024-04-25T15:28:56.059189+0000","last_clean":"2024-04-25T15:28:56.059189+0000","last_became_active":"2024-04-25T15:28:56.052283+0000","last_became_peered":"2024-04-25T15:28:56.052283+0000","last_unstale":"2024-04-25T15:28:56.059189+0000","last_undegraded":"2024-04-25T15:28:56.059189+0000","last_fullsized":"2024-04-25T15:28:56.059189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:37.819844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041305+0000","last_change":"2024-04-25T15:28:56.041305+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041305+0000","last_undegraded":"2024-04-25T15:28:56.041305+0000","last_fullsized":"2024-04-25T15:28:56.041305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056432+0000","last_change":"2024-04-25T15:28:56.050894+0000","last_active":"2024-04-25T15:28:56.056432+0000","last_peered":"2024-04-25T15:28:56.056432+0000","last_clean":"2024-04-25T15:28:56.056432+0000","last_became_active":"2024-04-25T15:28:56.050570+0000","last_became_peered":"2024-04-25T15:28:56.050570+0000","last_unstale":"2024-04-25T15:28:56.056432+0000","last_undegraded":"2024-04-25T15:28:56.056432+0000","last_fullsized":"2024-04-25T15:28:56.056432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044386+0000","last_change":"2024-04-25T15:28:56.044386+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044386+0000","last_undegraded":"2024-04-25T15:28:56.044386+0000","last_fullsized":"2024-04-25T15:28:56.044386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.049680+0000","last_change":"2024-04-25T15:28:56.045642+0000","last_active":"2024-04-25T15:28:56.049680+0000","last_peered":"2024-04-25T15:28:56.049680+0000","last_clean":"2024-04-25T15:28:56.049680+0000","last_became_active":"2024-04-25T15:28:56.045273+0000","last_became_peered":"2024-04-25T15:28:56.045273+0000","last_unstale":"2024-04-25T15:28:56.049680+0000","last_undegraded":"2024-04-25T15:28:56.049680+0000","last_fullsized":"2024-04-25T15:28:56.049680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:21.737399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044193+0000","last_change":"2024-04-25T15:28:56.044193+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044193+0000","last_undegraded":"2024-04-25T15:28:56.044193+0000","last_fullsized":"2024-04-25T15:28:56.044193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062036+0000","last_change":"2024-04-25T15:28:56.059690+0000","last_active":"2024-04-25T15:28:56.062036+0000","last_peered":"2024-04-25T15:28:56.062036+0000","last_clean":"2024-04-25T15:28:56.062036+0000","last_became_active":"2024-04-25T15:28:56.059372+0000","last_became_peered":"2024-04-25T15:28:56.059372+0000","last_unstale":"2024-04-25T15:28:56.062036+0000","last_undegraded":"2024-04-25T15:28:56.062036+0000","last_fullsized":"2024-04-25T15:28:56.062036+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:03.773654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044493+0000","last_change":"2024-04-25T15:28:56.044493+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044493+0000","last_undegraded":"2024-04-25T15:28:56.044493+0000","last_fullsized":"2024-04-25T15:28:56.044493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056674+0000","last_change":"2024-04-25T15:28:56.053555+0000","last_active":"2024-04-25T15:28:56.056674+0000","last_peered":"2024-04-25T15:28:56.056674+0000","last_clean":"2024-04-25T15:28:56.056674+0000","last_became_active":"2024-04-25T15:28:56.053235+0000","last_became_peered":"2024-04-25T15:28:56.053235+0000","last_unstale":"2024-04-25T15:28:56.056674+0000","last_undegraded":"2024-04-25T15:28:56.056674+0000","last_fullsized":"2024-04-25T15:28:56.056674+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043401+0000","last_change":"2024-04-25T15:28:56.043401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043401+0000","last_undegraded":"2024-04-25T15:28:56.043401+0000","last_fullsized":"2024-04-25T15:28:56.043401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.054456+0000","last_change":"2024-04-25T15:28:56.049930+0000","last_active":"2024-04-25T15:28:56.054456+0000","last_peered":"2024-04-25T15:28:56.054456+0000","last_clean":"2024-04-25T15:28:56.054456+0000","last_became_active":"2024-04-25T15:28:56.049617+0000","last_became_peered":"2024-04-25T15:28:56.049617+0000","last_unstale":"2024-04-25T15:28:56.054456+0000","last_undegraded":"2024-04-25T15:28:56.054456+0000","last_fullsized":"2024-04-25T15:28:56.054456+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:29.266322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042363+0000","last_change":"2024-04-25T15:28:56.042363+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042363+0000","last_undegraded":"2024-04-25T15:28:56.042363+0000","last_fullsized":"2024-04-25T15:28:56.042363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057275+0000","last_change":"2024-04-25T15:28:56.054470+0000","last_active":"2024-04-25T15:28:56.057275+0000","last_peered":"2024-04-25T15:28:56.057275+0000","last_clean":"2024-04-25T15:28:56.057275+0000","last_became_active":"2024-04-25T15:28:56.054148+0000","last_became_peered":"2024-04-25T15:28:56.054148+0000","last_unstale":"2024-04-25T15:28:56.057275+0000","last_undegraded":"2024-04-25T15:28:56.057275+0000","last_fullsized":"2024-04-25T15:28:56.057275+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037520+0000","last_change":"2024-04-25T15:28:56.037520+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037520+0000","last_undegraded":"2024-04-25T15:28:56.037520+0000","last_fullsized":"2024-04-25T15:28:56.037520+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056659+0000","last_change":"2024-04-25T15:28:56.051763+0000","last_active":"2024-04-25T15:28:56.056659+0000","last_peered":"2024-04-25T15:28:56.056659+0000","last_clean":"2024-04-25T15:28:56.056659+0000","last_became_active":"2024-04-25T15:28:56.051431+0000","last_became_peered":"2024-04-25T15:28:56.051431+0000","last_unstale":"2024-04-25T15:28:56.056659+0000","last_undegraded":"2024-04-25T15:28:56.056659+0000","last_fullsized":"2024-04-25T15:28:56.056659+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:44:13.470300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052252+0000","last_change":"2024-04-25T15:28:56.052252+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052252+0000","last_undegraded":"2024-04-25T15:28:56.052252+0000","last_fullsized":"2024-04-25T15:28:56.052252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060700+0000","last_change":"2024-04-25T15:28:56.059005+0000","last_active":"2024-04-25T15:28:56.060700+0000","last_peered":"2024-04-25T15:28:56.060700+0000","last_clean":"2024-04-25T15:28:56.060700+0000","last_became_active":"2024-04-25T15:28:56.057981+0000","last_became_peered":"2024-04-25T15:28:56.057981+0000","last_unstale":"2024-04-25T15:28:56.060700+0000","last_undegraded":"2024-04-25T15:28:56.060700+0000","last_fullsized":"2024-04-25T15:28:56.060700+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045632+0000","last_change":"2024-04-25T15:28:56.045632+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045632+0000","last_undegraded":"2024-04-25T15:28:56.045632+0000","last_fullsized":"2024-04-25T15:28:56.045632+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056297+0000","last_change":"2024-04-25T15:28:56.048576+0000","last_active":"2024-04-25T15:28:56.056297+0000","last_peered":"2024-04-25T15:28:56.056297+0000","last_clean":"2024-04-25T15:28:56.056297+0000","last_became_active":"2024-04-25T15:28:56.048273+0000","last_became_peered":"2024-04-25T15:28:56.048273+0000","last_unstale":"2024-04-25T15:28:56.056297+0000","last_undegraded":"2024-04-25T15:28:56.056297+0000","last_fullsized":"2024-04-25T15:28:56.056297+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042145+0000","last_change":"2024-04-25T15:28:56.042145+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042145+0000","last_undegraded":"2024-04-25T15:28:56.042145+0000","last_fullsized":"2024-04-25T15:28:56.042145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056488+0000","last_change":"2024-04-25T15:28:56.051473+0000","last_active":"2024-04-25T15:28:56.056488+0000","last_peered":"2024-04-25T15:28:56.056488+0000","last_clean":"2024-04-25T15:28:56.056488+0000","last_became_active":"2024-04-25T15:28:56.051171+0000","last_became_peered":"2024-04-25T15:28:56.051171+0000","last_unstale":"2024-04-25T15:28:56.056488+0000","last_undegraded":"2024-04-25T15:28:56.056488+0000","last_fullsized":"2024-04-25T15:28:56.056488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042897+0000","last_change":"2024-04-25T15:28:56.042897+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042897+0000","last_undegraded":"2024-04-25T15:28:56.042897+0000","last_fullsized":"2024-04-25T15:28:56.042897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060976+0000","last_change":"2024-04-25T15:28:56.058257+0000","last_active":"2024-04-25T15:28:56.060976+0000","last_peered":"2024-04-25T15:28:56.060976+0000","last_clean":"2024-04-25T15:28:56.060976+0000","last_became_active":"2024-04-25T15:28:56.057736+0000","last_became_peered":"2024-04-25T15:28:56.057736+0000","last_unstale":"2024-04-25T15:28:56.060976+0000","last_undegraded":"2024-04-25T15:28:56.060976+0000","last_fullsized":"2024-04-25T15:28:56.060976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036658+0000","last_change":"2024-04-25T15:28:56.036658+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036658+0000","last_undegraded":"2024-04-25T15:28:56.036658+0000","last_fullsized":"2024-04-25T15:28:56.036658+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.061871+0000","last_change":"2024-04-25T15:28:56.056141+0000","last_active":"2024-04-25T15:28:56.061871+0000","last_peered":"2024-04-25T15:28:56.061871+0000","last_clean":"2024-04-25T15:28:56.061871+0000","last_became_active":"2024-04-25T15:28:56.055725+0000","last_became_peered":"2024-04-25T15:28:56.055725+0000","last_unstale":"2024-04-25T15:28:56.061871+0000","last_undegraded":"2024-04-25T15:28:56.061871+0000","last_fullsized":"2024-04-25T15:28:56.061871+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042444+0000","last_change":"2024-04-25T15:28:56.042444+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042444+0000","last_undegraded":"2024-04-25T15:28:56.042444+0000","last_fullsized":"2024-04-25T15:28:56.042444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060626+0000","last_change":"2024-04-25T15:28:56.055311+0000","last_active":"2024-04-25T15:28:56.060626+0000","last_peered":"2024-04-25T15:28:56.060626+0000","last_clean":"2024-04-25T15:28:56.060626+0000","last_became_active":"2024-04-25T15:28:56.054958+0000","last_became_peered":"2024-04-25T15:28:56.054958+0000","last_unstale":"2024-04-25T15:28:56.060626+0000","last_undegraded":"2024-04-25T15:28:56.060626+0000","last_fullsized":"2024-04-25T15:28:56.060626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:04.302144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041713+0000","last_change":"2024-04-25T15:28:56.041713+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041713+0000","last_undegraded":"2024-04-25T15:28:56.041713+0000","last_fullsized":"2024-04-25T15:28:56.041713+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.051579+0000","last_change":"2024-04-25T15:28:56.050355+0000","last_active":"2024-04-25T15:28:56.051579+0000","last_peered":"2024-04-25T15:28:56.051579+0000","last_clean":"2024-04-25T15:28:56.051579+0000","last_became_active":"2024-04-25T15:28:56.050028+0000","last_became_peered":"2024-04-25T15:28:56.050028+0000","last_unstale":"2024-04-25T15:28:56.051579+0000","last_undegraded":"2024-04-25T15:28:56.051579+0000","last_fullsized":"2024-04-25T15:28:56.051579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:18.740971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044259+0000","last_change":"2024-04-25T15:28:56.044259+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044259+0000","last_undegraded":"2024-04-25T15:28:56.044259+0000","last_fullsized":"2024-04-25T15:28:56.044259+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059127+0000","last_change":"2024-04-25T15:28:56.054041+0000","last_active":"2024-04-25T15:28:56.059127+0000","last_peered":"2024-04-25T15:28:56.059127+0000","last_clean":"2024-04-25T15:28:56.059127+0000","last_became_active":"2024-04-25T15:28:56.053693+0000","last_became_peered":"2024-04-25T15:28:56.053693+0000","last_unstale":"2024-04-25T15:28:56.059127+0000","last_undegraded":"2024-04-25T15:28:56.059127+0000","last_fullsized":"2024-04-25T15:28:56.059127+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":235,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.043621+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:56.043621+0000","last_peered":"2024-04-25T15:28:56.043621+0000","last_clean":"2024-04-25T15:28:56.043621+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:56.043621+0000","last_undegraded":"2024-04-25T15:28:56.043621+0000","last_fullsized":"2024-04-25T15:28:56.043621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:01.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3f is not active+clean 2024-04-25T15:29:01.869 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.052980+0000', 'last_change': '2024-04-25T15:28:56.052980+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.052980+0000', 'last_undegraded': '2024-04-25T15:28:56.052980+0000', 'last_fullsized': '2024-04-25T15:28:56.052980+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:29:01.869 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3e is not active+clean 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.040106+0000', 'last_change': '2024-04-25T15:28:56.040106+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.040106+0000', 'last_undegraded': '2024-04-25T15:28:56.040106+0000', 'last_fullsized': '2024-04-25T15:28:56.040106+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3d is not active+clean 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044401+0000', 'last_change': '2024-04-25T15:28:56.044401+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044401+0000', 'last_undegraded': '2024-04-25T15:28:56.044401+0000', 'last_fullsized': '2024-04-25T15:28:56.044401+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3c is not active+clean 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044161+0000', 'last_change': '2024-04-25T15:28:56.044161+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044161+0000', 'last_undegraded': '2024-04-25T15:28:56.044161+0000', 'last_fullsized': '2024-04-25T15:28:56.044161+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3b is not active+clean 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.040956+0000', 'last_change': '2024-04-25T15:28:56.040956+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.040956+0000', 'last_undegraded': '2024-04-25T15:28:56.040956+0000', 'last_fullsized': '2024-04-25T15:28:56.040956+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3a is not active+clean 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043783+0000', 'last_change': '2024-04-25T15:28:56.043783+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043783+0000', 'last_undegraded': '2024-04-25T15:28:56.043783+0000', 'last_fullsized': '2024-04-25T15:28:56.043783+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.870 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.39 is not active+clean 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043768+0000', 'last_change': '2024-04-25T15:28:56.043768+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043768+0000', 'last_undegraded': '2024-04-25T15:28:56.043768+0000', 'last_fullsized': '2024-04-25T15:28:56.043768+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.38 is not active+clean 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.038877+0000', 'last_change': '2024-04-25T15:28:56.038877+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.038877+0000', 'last_undegraded': '2024-04-25T15:28:56.038877+0000', 'last_fullsized': '2024-04-25T15:28:56.038877+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.37 is not active+clean 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.037775+0000', 'last_change': '2024-04-25T15:28:56.037775+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.037775+0000', 'last_undegraded': '2024-04-25T15:28:56.037775+0000', 'last_fullsized': '2024-04-25T15:28:56.037775+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.36 is not active+clean 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.041034+0000', 'last_change': '2024-04-25T15:28:56.041034+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.041034+0000', 'last_undegraded': '2024-04-25T15:28:56.041034+0000', 'last_fullsized': '2024-04-25T15:28:56.041034+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.35 is not active+clean 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.046085+0000', 'last_change': '2024-04-25T15:28:56.046085+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.046085+0000', 'last_undegraded': '2024-04-25T15:28:56.046085+0000', 'last_fullsized': '2024-04-25T15:28:56.046085+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.34 is not active+clean 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.039309+0000', 'last_change': '2024-04-25T15:28:56.039309+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.039309+0000', 'last_undegraded': '2024-04-25T15:28:56.039309+0000', 'last_fullsized': '2024-04-25T15:28:56.039309+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.871 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.33 is not active+clean 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043640+0000', 'last_change': '2024-04-25T15:28:56.043640+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043640+0000', 'last_undegraded': '2024-04-25T15:28:56.043640+0000', 'last_fullsized': '2024-04-25T15:28:56.043640+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.32 is not active+clean 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.047935+0000', 'last_change': '2024-04-25T15:28:56.047935+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.047935+0000', 'last_undegraded': '2024-04-25T15:28:56.047935+0000', 'last_fullsized': '2024-04-25T15:28:56.047935+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.31 is not active+clean 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043253+0000', 'last_change': '2024-04-25T15:28:56.043253+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043253+0000', 'last_undegraded': '2024-04-25T15:28:56.043253+0000', 'last_fullsized': '2024-04-25T15:28:56.043253+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.30 is not active+clean 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043312+0000', 'last_change': '2024-04-25T15:28:56.043312+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043312+0000', 'last_undegraded': '2024-04-25T15:28:56.043312+0000', 'last_fullsized': '2024-04-25T15:28:56.043312+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2f is not active+clean 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.038794+0000', 'last_change': '2024-04-25T15:28:56.038794+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.038794+0000', 'last_undegraded': '2024-04-25T15:28:56.038794+0000', 'last_fullsized': '2024-04-25T15:28:56.038794+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2e is not active+clean 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043094+0000', 'last_change': '2024-04-25T15:28:56.043094+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043094+0000', 'last_undegraded': '2024-04-25T15:28:56.043094+0000', 'last_fullsized': '2024-04-25T15:28:56.043094+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2d is not active+clean 2024-04-25T15:29:01.872 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.047300+0000', 'last_change': '2024-04-25T15:28:56.047300+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.047300+0000', 'last_undegraded': '2024-04-25T15:28:56.047300+0000', 'last_fullsized': '2024-04-25T15:28:56.047300+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2c is not active+clean 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.037516+0000', 'last_change': '2024-04-25T15:28:56.037516+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.037516+0000', 'last_undegraded': '2024-04-25T15:28:56.037516+0000', 'last_fullsized': '2024-04-25T15:28:56.037516+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2b is not active+clean 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.046746+0000', 'last_change': '2024-04-25T15:28:56.046746+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.046746+0000', 'last_undegraded': '2024-04-25T15:28:56.046746+0000', 'last_fullsized': '2024-04-25T15:28:56.046746+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2a is not active+clean 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.036501+0000', 'last_change': '2024-04-25T15:28:56.036501+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.036501+0000', 'last_undegraded': '2024-04-25T15:28:56.036501+0000', 'last_fullsized': '2024-04-25T15:28:56.036501+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.29 is not active+clean 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.036507+0000', 'last_change': '2024-04-25T15:28:56.036507+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.036507+0000', 'last_undegraded': '2024-04-25T15:28:56.036507+0000', 'last_fullsized': '2024-04-25T15:28:56.036507+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.28 is not active+clean 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042774+0000', 'last_change': '2024-04-25T15:28:56.042774+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042774+0000', 'last_undegraded': '2024-04-25T15:28:56.042774+0000', 'last_fullsized': '2024-04-25T15:28:56.042774+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.873 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.27 is not active+clean 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043848+0000', 'last_change': '2024-04-25T15:28:56.043848+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043848+0000', 'last_undegraded': '2024-04-25T15:28:56.043848+0000', 'last_fullsized': '2024-04-25T15:28:56.043848+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.26 is not active+clean 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.039503+0000', 'last_change': '2024-04-25T15:28:56.039503+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.039503+0000', 'last_undegraded': '2024-04-25T15:28:56.039503+0000', 'last_fullsized': '2024-04-25T15:28:56.039503+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.25 is not active+clean 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.045584+0000', 'last_change': '2024-04-25T15:28:56.045584+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.045584+0000', 'last_undegraded': '2024-04-25T15:28:56.045584+0000', 'last_fullsized': '2024-04-25T15:28:56.045584+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.24 is not active+clean 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.045164+0000', 'last_change': '2024-04-25T15:28:56.045164+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.045164+0000', 'last_undegraded': '2024-04-25T15:28:56.045164+0000', 'last_fullsized': '2024-04-25T15:28:56.045164+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.874 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.23 is not active+clean 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.039640+0000', 'last_change': '2024-04-25T15:28:56.039640+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.039640+0000', 'last_undegraded': '2024-04-25T15:28:56.039640+0000', 'last_fullsized': '2024-04-25T15:28:56.039640+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.22 is not active+clean 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.046096+0000', 'last_change': '2024-04-25T15:28:56.046096+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.046096+0000', 'last_undegraded': '2024-04-25T15:28:56.046096+0000', 'last_fullsized': '2024-04-25T15:28:56.046096+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.21 is not active+clean 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042753+0000', 'last_change': '2024-04-25T15:28:56.042753+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042753+0000', 'last_undegraded': '2024-04-25T15:28:56.042753+0000', 'last_fullsized': '2024-04-25T15:28:56.042753+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.20 is not active+clean 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.049201+0000', 'last_change': '2024-04-25T15:28:56.049201+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.049201+0000', 'last_undegraded': '2024-04-25T15:28:56.049201+0000', 'last_fullsized': '2024-04-25T15:28:56.049201+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1e is not active+clean 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.045572+0000', 'last_change': '2024-04-25T15:28:56.045572+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.045572+0000', 'last_undegraded': '2024-04-25T15:28:56.045572+0000', 'last_fullsized': '2024-04-25T15:28:56.045572+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1f is not active+clean 2024-04-25T15:29:01.875 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042730+0000', 'last_change': '2024-04-25T15:28:56.042730+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042730+0000', 'last_undegraded': '2024-04-25T15:28:56.042730+0000', 'last_fullsized': '2024-04-25T15:28:56.042730+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1c is not active+clean 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.046120+0000', 'last_change': '2024-04-25T15:28:56.046120+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.046120+0000', 'last_undegraded': '2024-04-25T15:28:56.046120+0000', 'last_fullsized': '2024-04-25T15:28:56.046120+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1d is not active+clean 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.037437+0000', 'last_change': '2024-04-25T15:28:56.037437+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.037437+0000', 'last_undegraded': '2024-04-25T15:28:56.037437+0000', 'last_fullsized': '2024-04-25T15:28:56.037437+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1a is not active+clean 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.040410+0000', 'last_change': '2024-04-25T15:28:56.040410+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.040410+0000', 'last_undegraded': '2024-04-25T15:28:56.040410+0000', 'last_fullsized': '2024-04-25T15:28:56.040410+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1b is not active+clean 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044043+0000', 'last_change': '2024-04-25T15:28:56.044043+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044043+0000', 'last_undegraded': '2024-04-25T15:28:56.044043+0000', 'last_fullsized': '2024-04-25T15:28:56.044043+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.18 is not active+clean 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042179+0000', 'last_change': '2024-04-25T15:28:56.042179+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042179+0000', 'last_undegraded': '2024-04-25T15:28:56.042179+0000', 'last_fullsized': '2024-04-25T15:28:56.042179+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.19 is not active+clean 2024-04-25T15:29:01.876 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.036701+0000', 'last_change': '2024-04-25T15:28:56.036701+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.036701+0000', 'last_undegraded': '2024-04-25T15:28:56.036701+0000', 'last_fullsized': '2024-04-25T15:28:56.036701+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.16 is not active+clean 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.038961+0000', 'last_change': '2024-04-25T15:28:56.038961+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.038961+0000', 'last_undegraded': '2024-04-25T15:28:56.038961+0000', 'last_fullsized': '2024-04-25T15:28:56.038961+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.17 is not active+clean 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.037203+0000', 'last_change': '2024-04-25T15:28:56.037203+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.037203+0000', 'last_undegraded': '2024-04-25T15:28:56.037203+0000', 'last_fullsized': '2024-04-25T15:28:56.037203+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.14 is not active+clean 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044011+0000', 'last_change': '2024-04-25T15:28:56.044011+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044011+0000', 'last_undegraded': '2024-04-25T15:28:56.044011+0000', 'last_fullsized': '2024-04-25T15:28:56.044011+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.15 is not active+clean 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044913+0000', 'last_change': '2024-04-25T15:28:56.044913+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044913+0000', 'last_undegraded': '2024-04-25T15:28:56.044913+0000', 'last_fullsized': '2024-04-25T15:28:56.044913+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.12 is not active+clean 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.041617+0000', 'last_change': '2024-04-25T15:28:56.041617+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.041617+0000', 'last_undegraded': '2024-04-25T15:28:56.041617+0000', 'last_fullsized': '2024-04-25T15:28:56.041617+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.13 is not active+clean 2024-04-25T15:29:01.877 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.045084+0000', 'last_change': '2024-04-25T15:28:56.045084+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.045084+0000', 'last_undegraded': '2024-04-25T15:28:56.045084+0000', 'last_fullsized': '2024-04-25T15:28:56.045084+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.10 is not active+clean 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.037150+0000', 'last_change': '2024-04-25T15:28:56.037150+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.037150+0000', 'last_undegraded': '2024-04-25T15:28:56.037150+0000', 'last_fullsized': '2024-04-25T15:28:56.037150+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.11 is not active+clean 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044935+0000', 'last_change': '2024-04-25T15:28:56.044935+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044935+0000', 'last_undegraded': '2024-04-25T15:28:56.044935+0000', 'last_fullsized': '2024-04-25T15:28:56.044935+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.f is not active+clean 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043391+0000', 'last_change': '2024-04-25T15:28:56.043391+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043391+0000', 'last_undegraded': '2024-04-25T15:28:56.043391+0000', 'last_fullsized': '2024-04-25T15:28:56.043391+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.c is not active+clean 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.041305+0000', 'last_change': '2024-04-25T15:28:56.041305+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.041305+0000', 'last_undegraded': '2024-04-25T15:28:56.041305+0000', 'last_fullsized': '2024-04-25T15:28:56.041305+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.d is not active+clean 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044386+0000', 'last_change': '2024-04-25T15:28:56.044386+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044386+0000', 'last_undegraded': '2024-04-25T15:28:56.044386+0000', 'last_fullsized': '2024-04-25T15:28:56.044386+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T15:29:01.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.a is not active+clean 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044193+0000', 'last_change': '2024-04-25T15:28:56.044193+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044193+0000', 'last_undegraded': '2024-04-25T15:28:56.044193+0000', 'last_fullsized': '2024-04-25T15:28:56.044193+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.b is not active+clean 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044493+0000', 'last_change': '2024-04-25T15:28:56.044493+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044493+0000', 'last_undegraded': '2024-04-25T15:28:56.044493+0000', 'last_fullsized': '2024-04-25T15:28:56.044493+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.8 is not active+clean 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.043401+0000', 'last_change': '2024-04-25T15:28:56.043401+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.043401+0000', 'last_undegraded': '2024-04-25T15:28:56.043401+0000', 'last_fullsized': '2024-04-25T15:28:56.043401+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.9 is not active+clean 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042363+0000', 'last_change': '2024-04-25T15:28:56.042363+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042363+0000', 'last_undegraded': '2024-04-25T15:28:56.042363+0000', 'last_fullsized': '2024-04-25T15:28:56.042363+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.6 is not active+clean 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.037520+0000', 'last_change': '2024-04-25T15:28:56.037520+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.037520+0000', 'last_undegraded': '2024-04-25T15:28:56.037520+0000', 'last_fullsized': '2024-04-25T15:28:56.037520+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.7 is not active+clean 2024-04-25T15:29:01.879 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.052252+0000', 'last_change': '2024-04-25T15:28:56.052252+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.052252+0000', 'last_undegraded': '2024-04-25T15:28:56.052252+0000', 'last_fullsized': '2024-04-25T15:28:56.052252+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.4 is not active+clean 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.045632+0000', 'last_change': '2024-04-25T15:28:56.045632+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.045632+0000', 'last_undegraded': '2024-04-25T15:28:56.045632+0000', 'last_fullsized': '2024-04-25T15:28:56.045632+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.5 is not active+clean 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042145+0000', 'last_change': '2024-04-25T15:28:56.042145+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042145+0000', 'last_undegraded': '2024-04-25T15:28:56.042145+0000', 'last_fullsized': '2024-04-25T15:28:56.042145+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2 is not active+clean 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042897+0000', 'last_change': '2024-04-25T15:28:56.042897+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042897+0000', 'last_undegraded': '2024-04-25T15:28:56.042897+0000', 'last_fullsized': '2024-04-25T15:28:56.042897+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3 is not active+clean 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.036658+0000', 'last_change': '2024-04-25T15:28:56.036658+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.036658+0000', 'last_undegraded': '2024-04-25T15:28:56.036658+0000', 'last_fullsized': '2024-04-25T15:28:56.036658+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.0 is not active+clean 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.042444+0000', 'last_change': '2024-04-25T15:28:56.042444+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.042444+0000', 'last_undegraded': '2024-04-25T15:28:56.042444+0000', 'last_fullsized': '2024-04-25T15:28:56.042444+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1 is not active+clean 2024-04-25T15:29:01.880 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.041713+0000', 'last_change': '2024-04-25T15:28:56.041713+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.041713+0000', 'last_undegraded': '2024-04-25T15:28:56.041713+0000', 'last_fullsized': '2024-04-25T15:28:56.041713+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T15:29:01.881 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.e is not active+clean 2024-04-25T15:29:01.881 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 90, 'state': 'creating+peering', 'last_fresh': '2024-04-25T15:28:56.044259+0000', 'last_change': '2024-04-25T15:28:56.044259+0000', 'last_active': '2024-04-25T15:28:56.028812+0000', 'last_peered': '2024-04-25T15:28:56.028812+0000', 'last_clean': '2024-04-25T15:28:56.028812+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T15:28:56.044259+0000', 'last_undegraded': '2024-04-25T15:28:56.044259+0000', 'last_fullsized': '2024-04-25T15:28:56.044259+0000', 'mapping_epoch': 90, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 90, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'last_clean_scrub_stamp': '2024-04-25T15:28:56.028812+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T15:29:01.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-25T15:29:02.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.015+0000 7f49274b6640 1 Processor -- start 2024-04-25T15:29:02.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.015+0000 7f49274b6640 1 -- start start 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f49274b6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49200b4830 0x7f492006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f49274b6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f492006d870 con 0x7f49200d4570 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f49274b6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f492006db10 con 0x7f49200b2dc0 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f49274b6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f49200b4c40 con 0x7f49200b4830 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4925252640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49200d4570 0x7f49200d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40814/0 (socket says 172.21.15.67:40814) 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4925252640 1 -- 172.21.15.67:0/4017616764 learned_addr learned my addr 172.21.15.67:0/4017616764 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4924a51640 1 --2- 172.21.15.67:0/4017616764 >> v2:172.21.15.67:3300/0 conn(0x7f49200b4830 0x7f492006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4917fff640 1 -- 172.21.15.67:0/4017616764 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182253255 0 0) 0x7f492006d870 con 0x7f49200d4570 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4917fff640 1 -- 172.21.15.67:0/4017616764 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f490c003520 con 0x7f49200d4570 2024-04-25T15:29:02.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4924a51640 1 -- 172.21.15.67:0/4017616764 >> v1:172.21.15.176:6789/0 conn(0x7f49200b2dc0 legacy=0x7f49200b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4924a51640 1 -- 172.21.15.67:0/4017616764 >> v1:172.21.15.67:6789/0 conn(0x7f49200d4570 legacy=0x7f49200d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.017+0000 7f4924a51640 1 -- 172.21.15.67:0/4017616764 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492006e0f0 con 0x7f49200b4830 2024-04-25T15:29:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.018+0000 7f4924a51640 1 --2- 172.21.15.67:0/4017616764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b4830 0x7f492006e8f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f49100097a0 tx=0x7f491002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee1dfdb22da86f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.018+0000 7f4917fff640 1 -- 172.21.15.67:0/4017616764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910037070 con 0x7f49200b4830 2024-04-25T15:29:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.018+0000 7f4917fff640 1 -- 172.21.15.67:0/4017616764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4910032440 con 0x7f49200b4830 2024-04-25T15:29:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.018+0000 7f4917fff640 1 -- 172.21.15.67:0/4017616764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910032810 con 0x7f49200b4830 2024-04-25T15:29:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.019+0000 7f49274b6640 1 -- 172.21.15.67:0/4017616764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b4830 msgr2=0x7f492006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.019+0000 7f49274b6640 1 --2- 172.21.15.67:0/4017616764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b4830 0x7f492006e8f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f49100097a0 tx=0x7f491002a1a0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.019+0000 7f49274b6640 1 -- 172.21.15.67:0/4017616764 shutdown_connections 2024-04-25T15:29:02.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.019+0000 7f49274b6640 1 --2- 172.21.15.67:0/4017616764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b4830 0x7f492006e8f0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.019+0000 7f49274b6640 1 -- 172.21.15.67:0/4017616764 >> 172.21.15.67:0/4017616764 conn(0x7f49200d3a50 msgr2=0x7f49200d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:02.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.019+0000 7f49274b6640 1 -- 172.21.15.67:0/4017616764 shutdown_connections 2024-04-25T15:29:02.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.019+0000 7f49274b6640 1 -- 172.21.15.67:0/4017616764 wait complete. 2024-04-25T15:29:02.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.021+0000 7f49274b6640 1 Processor -- start 2024-04-25T15:29:02.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.021+0000 7f49274b6640 1 -- start start 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f49274b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b2dc0 0x7f492009ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f49274b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49200b4830 0x7f49201632e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f49274b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49200d4570 0x7f492009f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f49274b6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49200d0fb0 con 0x7f49200b2dc0 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f49274b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f492006e460 con 0x7f49200b4830 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f49274b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f49200b38e0 con 0x7f49200d4570 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b2dc0 0x7f492009ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b2dc0 0x7f492009ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57792/0 (socket says 172.21.15.67:57792) 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 -- 172.21.15.67:0/705881360 learned_addr learned my addr 172.21.15.67:0/705881360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 -- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49200d4570 msgr2=0x7f492009f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49200d4570 0x7f492009f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 -- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49200b4830 msgr2=0x7f49201632e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4924a51640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49200b4830 0x7f49201632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49200b4830 0x7f49201632e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.022+0000 7f4925252640 1 -- 172.21.15.67:0/705881360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492009d9f0 con 0x7f49200b2dc0 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.023+0000 7f4925252640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b2dc0 0x7f492009ea90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c840 tx=0x7f491c00cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.023+0000 7f4915ffb640 1 -- 172.21.15.67:0/705881360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c017070 con 0x7f49200b2dc0 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.023+0000 7f4915ffb640 1 -- 172.21.15.67:0/705881360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f491c00a640 con 0x7f49200b2dc0 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.023+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492009dcd0 con 0x7f49200b2dc0 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.023+0000 7f4915ffb640 1 -- 172.21.15.67:0/705881360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c00aa30 con 0x7f49200b2dc0 2024-04-25T15:29:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.023+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492009e290 con 0x7f49200b2dc0 2024-04-25T15:29:02.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.024+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4920073120 con 0x7f49200b2dc0 2024-04-25T15:29:02.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.024+0000 7f4915ffb640 1 -- 172.21.15.67:0/705881360 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f491c01a070 con 0x7f49200b2dc0 2024-04-25T15:29:02.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.025+0000 7f4915ffb640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49040744e0 0x7f49040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.025+0000 7f4915ffb640 1 -- 172.21.15.67:0/705881360 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f491c094900 con 0x7f49200b2dc0 2024-04-25T15:29:02.026 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-25T15:29:02.025+0000 7fbb82118640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:29:02.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.025+0000 7f4924a51640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49040744e0 0x7f49040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.026+0000 7f4924a51640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49040744e0 0x7f49040769f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4910034000 tx=0x7f491002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:02.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.029+0000 7f4915ffb640 1 -- 172.21.15.67:0/705881360 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4920073120 con 0x7f49200b2dc0 2024-04-25T15:29:02.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.284+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f492006eeb0 con 0x7f49200b2dc0 2024-04-25T15:29:02.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.285+0000 7f4915ffb640 1 -- 172.21.15.67:0/705881360 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1453 (secure 0 0 0) 0x7f492006eeb0 con 0x7f49200b2dc0 2024-04-25T15:29:02.286 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:02.286 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":237,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":92,"num_osds":8,"num_up_osds":8,"osd_up_since":1714058715,"num_in_osds":8,"osd_in_since":1714058706,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":248475648,"bytes_avail":772845637632,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":57,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":7128}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-25T15:28:44.844048+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49040744e0 msgr2=0x7f49040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49040744e0 0x7f49040769f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4910034000 tx=0x7f491002e000 comp rx=0 tx=0).stop 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b2dc0 msgr2=0x7f492009ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b2dc0 0x7f492009ea90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c840 tx=0x7f491c00cd00 comp rx=0 tx=0).stop 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 shutdown_connections 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f49040744e0 0x7f49040769f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49200d4570 0x7f492009f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49200b4830 0x7f49201632e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 --2- 172.21.15.67:0/705881360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49200b2dc0 0x7f492009ea90 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 >> 172.21.15.67:0/705881360 conn(0x7f49200d3a50 msgr2=0x7f49200b8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 shutdown_connections 2024-04-25T15:29:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.287+0000 7f49274b6640 1 -- 172.21.15.67:0/705881360 wait complete. 2024-04-25T15:29:02.299 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-25T15:29:02.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:02.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.410+0000 7fd93d196640 1 Processor -- start 2024-04-25T15:29:02.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.410+0000 7fd93d196640 1 -- start start 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.411+0000 7fd93d196640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9380b4f40 0x7fd9380b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.411+0000 7fd93d196640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd9380d3860 0x7fd9380d3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.411+0000 7fd93d196640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9380d41b0 con 0x7fd9380b2cc0 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.411+0000 7fd93d196640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd9380b4ab0 con 0x7fd9380b4f40 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.411+0000 7fd93d196640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd9380b5350 con 0x7fd9380d3860 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9380b4f40 0x7fd9380b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd9380b4f40 0x7fd9380b9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57810/0 (socket says 172.21.15.67:57810) 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93659c640 1 -- 172.21.15.67:0/2434483979 learned_addr learned my addr 172.21.15.67:0/2434483979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93759e640 1 --2- 172.21.15.67:0/2434483979 >> v2:172.21.15.176:3300/0 conn(0x7fd9380d3860 0x7fd9380d3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93659c640 1 -- 172.21.15.67:0/2434483979 >> v2:172.21.15.176:3300/0 conn(0x7fd9380d3860 msgr2=0x7fd9380d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93659c640 1 --2- 172.21.15.67:0/2434483979 >> v2:172.21.15.176:3300/0 conn(0x7fd9380d3860 0x7fd9380d3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93659c640 1 -- 172.21.15.67:0/2434483979 >> v1:172.21.15.176:6789/0 conn(0x7fd9380b2cc0 legacy=0x7fd9380d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:02.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93659c640 1 -- 172.21.15.67:0/2434483979 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9380b4180 con 0x7fd9380b4f40 2024-04-25T15:29:02.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.412+0000 7fd93759e640 1 --2- 172.21.15.67:0/2434483979 >> v2:172.21.15.176:3300/0 conn(0x7fd9380d3860 0x7fd9380d3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd93659c640 1 --2- 172.21.15.67:0/2434483979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b4f40 0x7fd9380b9160 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd9380ce070 tx=0x7fd92c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc77e0d272b61692 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd935d9b640 1 -- 172.21.15.67:0/2434483979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c037070 con 0x7fd9380b4f40 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd935d9b640 1 -- 172.21.15.67:0/2434483979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd92c032440 con 0x7fd9380b4f40 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd935d9b640 1 -- 172.21.15.67:0/2434483979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c032810 con 0x7fd9380b4f40 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd93d196640 1 -- 172.21.15.67:0/2434483979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b4f40 msgr2=0x7fd9380b9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd93d196640 1 --2- 172.21.15.67:0/2434483979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b4f40 0x7fd9380b9160 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd9380ce070 tx=0x7fd92c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd93d196640 1 -- 172.21.15.67:0/2434483979 shutdown_connections 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd93d196640 1 --2- 172.21.15.67:0/2434483979 >> v2:172.21.15.176:3300/0 conn(0x7fd9380d3860 0x7fd9380d3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd93d196640 1 --2- 172.21.15.67:0/2434483979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b4f40 0x7fd9380b9160 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.413+0000 7fd93d196640 1 -- 172.21.15.67:0/2434483979 >> 172.21.15.67:0/2434483979 conn(0x7fd93808d110 msgr2=0x7fd93808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:02.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.414+0000 7fd93d196640 1 -- 172.21.15.67:0/2434483979 shutdown_connections 2024-04-25T15:29:02.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.414+0000 7fd93d196640 1 -- 172.21.15.67:0/2434483979 wait complete. 2024-04-25T15:29:02.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.416+0000 7fd93d196640 1 Processor -- start 2024-04-25T15:29:02.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.416+0000 7fd93d196640 1 -- start start 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd93d196640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b2cc0 0x7fd938150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd93d196640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9380b4f40 0x7fd93815e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd93d196640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9380d3860 0x7fd93815ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd93d196640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd9380b05c0 con 0x7fd9380b2cc0 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd93d196640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd9380b4740 con 0x7fd9380d3860 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd93d196640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd9380b0740 con 0x7fd9380b4f40 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b2cc0 0x7fd938150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b2cc0 0x7fd938150e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57820/0 (socket says 172.21.15.67:57820) 2024-04-25T15:29:02.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 -- 172.21.15.67:0/41409848 learned_addr learned my addr 172.21.15.67:0/41409848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd93759e640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9380d3860 0x7fd93815ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 -- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9380b4f40 msgr2=0x7fd93815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9380b4f40 0x7fd93815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 -- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9380d3860 msgr2=0x7fd93815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9380d3860 0x7fd93815ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.417+0000 7fd936d9d640 1 -- 172.21.15.67:0/41409848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd938151380 con 0x7fd9380b2cc0 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.418+0000 7fd936d9d640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b2cc0 0x7fd938150e40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b3f0 tx=0x7fd92800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.418+0000 7fd91f7fe640 1 -- 172.21.15.67:0/41409848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928015020 con 0x7fd9380b2cc0 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.418+0000 7fd91f7fe640 1 -- 172.21.15.67:0/41409848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd928011830 con 0x7fd9380b2cc0 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.418+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd93815a8c0 con 0x7fd9380b2cc0 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.418+0000 7fd91f7fe640 1 -- 172.21.15.67:0/41409848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928011c20 con 0x7fd9380b2cc0 2024-04-25T15:29:02.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.418+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9380ccb10 con 0x7fd9380b2cc0 2024-04-25T15:29:02.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.419+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd938050740 con 0x7fd9380b2cc0 2024-04-25T15:29:02.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.420+0000 7fd91f7fe640 1 -- 172.21.15.67:0/41409848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd928018020 con 0x7fd9380b2cc0 2024-04-25T15:29:02.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.420+0000 7fd91f7fe640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd9240744e0 0x7fd9240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:02.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.420+0000 7fd91f7fe640 1 -- 172.21.15.67:0/41409848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fd928095150 con 0x7fd9380b2cc0 2024-04-25T15:29:02.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.420+0000 7fd93659c640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd9240744e0 0x7fd9240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:02.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.421+0000 7fd93659c640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd9240744e0 0x7fd9240769f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd92c034000 tx=0x7fd92c02c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:02.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.425+0000 7fd91f7fe640 1 -- 172.21.15.67:0/41409848 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd92801f020 con 0x7fd9380b2cc0 2024-04-25T15:29:02.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.605+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd93808d770 con 0x7fd9240744e0 2024-04-25T15:29:02.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.612+0000 7fd91f7fe640 1 -- 172.21.15.67:0/41409848 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246754 (secure 0 0 0) 0x7fd93808d770 con 0x7fd9240744e0 2024-04-25T15:29:02.615 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:02.616 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.614+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd9240744e0 msgr2=0x7fd9240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.614+0000 7fd93d196640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd9240744e0 0x7fd9240769f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd92c034000 tx=0x7fd92c02c000 comp rx=0 tx=0).stop 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b2cc0 msgr2=0x7fd938150e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b2cc0 0x7fd938150e40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b3f0 tx=0x7fd92800b8b0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 shutdown_connections 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd9240744e0 0x7fd9240769f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9380d3860 0x7fd93815ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9380b4f40 0x7fd93815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 --2- 172.21.15.67:0/41409848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9380b2cc0 0x7fd938150e40 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 >> 172.21.15.67:0/41409848 conn(0x7fd93808d110 msgr2=0x7fd9380b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 shutdown_connections 2024-04-25T15:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:02.615+0000 7fd93d196640 1 -- 172.21.15.67:0/41409848 wait complete. 2024-04-25T15:29:02.627 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-25T15:29:00.847967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242652,"kb_used_data":9840,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754732068,"statfs":{"total":773094113280,"available":772845637632,"internally_reserved":0,"allocated":10076160,"data_stored":4946690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.818182"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052980+0000","last_change":"2024-04-25T15:28:56.052980+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052980+0000","last_undegraded":"2024-04-25T15:28:56.052980+0000","last_fullsized":"2024-04-25T15:28:56.052980+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040106+0000","last_change":"2024-04-25T15:28:56.040106+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040106+0000","last_undegraded":"2024-04-25T15:28:56.040106+0000","last_fullsized":"2024-04-25T15:28:56.040106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044401+0000","last_change":"2024-04-25T15:28:56.044401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044401+0000","last_undegraded":"2024-04-25T15:28:56.044401+0000","last_fullsized":"2024-04-25T15:28:56.044401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044161+0000","last_change":"2024-04-25T15:28:56.044161+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044161+0000","last_undegraded":"2024-04-25T15:28:56.044161+0000","last_fullsized":"2024-04-25T15:28:56.044161+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040956+0000","last_change":"2024-04-25T15:28:56.040956+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040956+0000","last_undegraded":"2024-04-25T15:28:56.040956+0000","last_fullsized":"2024-04-25T15:28:56.040956+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043783+0000","last_change":"2024-04-25T15:28:56.043783+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043783+0000","last_undegraded":"2024-04-25T15:28:56.043783+0000","last_fullsized":"2024-04-25T15:28:56.043783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043768+0000","last_change":"2024-04-25T15:28:56.043768+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043768+0000","last_undegraded":"2024-04-25T15:28:56.043768+0000","last_fullsized":"2024-04-25T15:28:56.043768+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038877+0000","last_change":"2024-04-25T15:28:56.038877+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038877+0000","last_undegraded":"2024-04-25T15:28:56.038877+0000","last_fullsized":"2024-04-25T15:28:56.038877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037775+0000","last_change":"2024-04-25T15:28:56.037775+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037775+0000","last_undegraded":"2024-04-25T15:28:56.037775+0000","last_fullsized":"2024-04-25T15:28:56.037775+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041034+0000","last_change":"2024-04-25T15:28:56.041034+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041034+0000","last_undegraded":"2024-04-25T15:28:56.041034+0000","last_fullsized":"2024-04-25T15:28:56.041034+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046085+0000","last_change":"2024-04-25T15:28:56.046085+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046085+0000","last_undegraded":"2024-04-25T15:28:56.046085+0000","last_fullsized":"2024-04-25T15:28:56.046085+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039309+0000","last_change":"2024-04-25T15:28:56.039309+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039309+0000","last_undegraded":"2024-04-25T15:28:56.039309+0000","last_fullsized":"2024-04-25T15:28:56.039309+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043640+0000","last_change":"2024-04-25T15:28:56.043640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043640+0000","last_undegraded":"2024-04-25T15:28:56.043640+0000","last_fullsized":"2024-04-25T15:28:56.043640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047935+0000","last_change":"2024-04-25T15:28:56.047935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047935+0000","last_undegraded":"2024-04-25T15:28:56.047935+0000","last_fullsized":"2024-04-25T15:28:56.047935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043253+0000","last_change":"2024-04-25T15:28:56.043253+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043253+0000","last_undegraded":"2024-04-25T15:28:56.043253+0000","last_fullsized":"2024-04-25T15:28:56.043253+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043312+0000","last_change":"2024-04-25T15:28:56.043312+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043312+0000","last_undegraded":"2024-04-25T15:28:56.043312+0000","last_fullsized":"2024-04-25T15:28:56.043312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038794+0000","last_change":"2024-04-25T15:28:56.038794+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038794+0000","last_undegraded":"2024-04-25T15:28:56.038794+0000","last_fullsized":"2024-04-25T15:28:56.038794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043094+0000","last_change":"2024-04-25T15:28:56.043094+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043094+0000","last_undegraded":"2024-04-25T15:28:56.043094+0000","last_fullsized":"2024-04-25T15:28:56.043094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.047300+0000","last_change":"2024-04-25T15:28:56.047300+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.047300+0000","last_undegraded":"2024-04-25T15:28:56.047300+0000","last_fullsized":"2024-04-25T15:28:56.047300+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037516+0000","last_change":"2024-04-25T15:28:56.037516+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037516+0000","last_undegraded":"2024-04-25T15:28:56.037516+0000","last_fullsized":"2024-04-25T15:28:56.037516+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046746+0000","last_change":"2024-04-25T15:28:56.046746+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046746+0000","last_undegraded":"2024-04-25T15:28:56.046746+0000","last_fullsized":"2024-04-25T15:28:56.046746+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036501+0000","last_change":"2024-04-25T15:28:56.036501+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036501+0000","last_undegraded":"2024-04-25T15:28:56.036501+0000","last_fullsized":"2024-04-25T15:28:56.036501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036507+0000","last_change":"2024-04-25T15:28:56.036507+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036507+0000","last_undegraded":"2024-04-25T15:28:56.036507+0000","last_fullsized":"2024-04-25T15:28:56.036507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042774+0000","last_change":"2024-04-25T15:28:56.042774+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042774+0000","last_undegraded":"2024-04-25T15:28:56.042774+0000","last_fullsized":"2024-04-25T15:28:56.042774+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043848+0000","last_change":"2024-04-25T15:28:56.043848+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043848+0000","last_undegraded":"2024-04-25T15:28:56.043848+0000","last_fullsized":"2024-04-25T15:28:56.043848+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039503+0000","last_change":"2024-04-25T15:28:56.039503+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039503+0000","last_undegraded":"2024-04-25T15:28:56.039503+0000","last_fullsized":"2024-04-25T15:28:56.039503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045584+0000","last_change":"2024-04-25T15:28:56.045584+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045584+0000","last_undegraded":"2024-04-25T15:28:56.045584+0000","last_fullsized":"2024-04-25T15:28:56.045584+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045164+0000","last_change":"2024-04-25T15:28:56.045164+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045164+0000","last_undegraded":"2024-04-25T15:28:56.045164+0000","last_fullsized":"2024-04-25T15:28:56.045164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.039640+0000","last_change":"2024-04-25T15:28:56.039640+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.039640+0000","last_undegraded":"2024-04-25T15:28:56.039640+0000","last_fullsized":"2024-04-25T15:28:56.039640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046096+0000","last_change":"2024-04-25T15:28:56.046096+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046096+0000","last_undegraded":"2024-04-25T15:28:56.046096+0000","last_fullsized":"2024-04-25T15:28:56.046096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042753+0000","last_change":"2024-04-25T15:28:56.042753+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042753+0000","last_undegraded":"2024-04-25T15:28:56.042753+0000","last_fullsized":"2024-04-25T15:28:56.042753+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.049201+0000","last_change":"2024-04-25T15:28:56.049201+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.049201+0000","last_undegraded":"2024-04-25T15:28:56.049201+0000","last_fullsized":"2024-04-25T15:28:56.049201+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045572+0000","last_change":"2024-04-25T15:28:56.045572+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045572+0000","last_undegraded":"2024-04-25T15:28:56.045572+0000","last_fullsized":"2024-04-25T15:28:56.045572+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060682+0000","last_change":"2024-04-25T15:28:56.059439+0000","last_active":"2024-04-25T15:28:56.060682+0000","last_peered":"2024-04-25T15:28:56.060682+0000","last_clean":"2024-04-25T15:28:56.060682+0000","last_became_active":"2024-04-25T15:28:56.059082+0000","last_became_peered":"2024-04-25T15:28:56.059082+0000","last_unstale":"2024-04-25T15:28:56.060682+0000","last_undegraded":"2024-04-25T15:28:56.060682+0000","last_fullsized":"2024-04-25T15:28:56.060682+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042730+0000","last_change":"2024-04-25T15:28:56.042730+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042730+0000","last_undegraded":"2024-04-25T15:28:56.042730+0000","last_fullsized":"2024-04-25T15:28:56.042730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059155+0000","last_change":"2024-04-25T15:28:56.055939+0000","last_active":"2024-04-25T15:28:56.059155+0000","last_peered":"2024-04-25T15:28:56.059155+0000","last_clean":"2024-04-25T15:28:56.059155+0000","last_became_active":"2024-04-25T15:28:56.055541+0000","last_became_peered":"2024-04-25T15:28:56.055541+0000","last_unstale":"2024-04-25T15:28:56.059155+0000","last_undegraded":"2024-04-25T15:28:56.059155+0000","last_fullsized":"2024-04-25T15:28:56.059155+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.046120+0000","last_change":"2024-04-25T15:28:56.046120+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.046120+0000","last_undegraded":"2024-04-25T15:28:56.046120+0000","last_fullsized":"2024-04-25T15:28:56.046120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060146+0000","last_change":"2024-04-25T15:28:56.056907+0000","last_active":"2024-04-25T15:28:56.060146+0000","last_peered":"2024-04-25T15:28:56.060146+0000","last_clean":"2024-04-25T15:28:56.060146+0000","last_became_active":"2024-04-25T15:28:56.056549+0000","last_became_peered":"2024-04-25T15:28:56.056549+0000","last_unstale":"2024-04-25T15:28:56.060146+0000","last_undegraded":"2024-04-25T15:28:56.060146+0000","last_fullsized":"2024-04-25T15:28:56.060146+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037437+0000","last_change":"2024-04-25T15:28:56.037437+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037437+0000","last_undegraded":"2024-04-25T15:28:56.037437+0000","last_fullsized":"2024-04-25T15:28:56.037437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059710+0000","last_change":"2024-04-25T15:28:56.057126+0000","last_active":"2024-04-25T15:28:56.059710+0000","last_peered":"2024-04-25T15:28:56.059710+0000","last_clean":"2024-04-25T15:28:56.059710+0000","last_became_active":"2024-04-25T15:28:56.056619+0000","last_became_peered":"2024-04-25T15:28:56.056619+0000","last_unstale":"2024-04-25T15:28:56.059710+0000","last_undegraded":"2024-04-25T15:28:56.059710+0000","last_fullsized":"2024-04-25T15:28:56.059710+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:50.204782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.040410+0000","last_change":"2024-04-25T15:28:56.040410+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.040410+0000","last_undegraded":"2024-04-25T15:28:56.040410+0000","last_fullsized":"2024-04-25T15:28:56.040410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062504+0000","last_change":"2024-04-25T15:28:56.061172+0000","last_active":"2024-04-25T15:28:56.062504+0000","last_peered":"2024-04-25T15:28:56.062504+0000","last_clean":"2024-04-25T15:28:56.062504+0000","last_became_active":"2024-04-25T15:28:56.060821+0000","last_became_peered":"2024-04-25T15:28:56.060821+0000","last_unstale":"2024-04-25T15:28:56.062504+0000","last_undegraded":"2024-04-25T15:28:56.062504+0000","last_fullsized":"2024-04-25T15:28:56.062504+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:58.403814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044043+0000","last_change":"2024-04-25T15:28:56.044043+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044043+0000","last_undegraded":"2024-04-25T15:28:56.044043+0000","last_fullsized":"2024-04-25T15:28:56.044043+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056409+0000","last_change":"2024-04-25T15:28:56.053676+0000","last_active":"2024-04-25T15:28:56.056409+0000","last_peered":"2024-04-25T15:28:56.056409+0000","last_clean":"2024-04-25T15:28:56.056409+0000","last_became_active":"2024-04-25T15:28:56.053355+0000","last_became_peered":"2024-04-25T15:28:56.053355+0000","last_unstale":"2024-04-25T15:28:56.056409+0000","last_undegraded":"2024-04-25T15:28:56.056409+0000","last_fullsized":"2024-04-25T15:28:56.056409+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042179+0000","last_change":"2024-04-25T15:28:56.042179+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042179+0000","last_undegraded":"2024-04-25T15:28:56.042179+0000","last_fullsized":"2024-04-25T15:28:56.042179+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056491+0000","last_change":"2024-04-25T15:28:56.051321+0000","last_active":"2024-04-25T15:28:56.056491+0000","last_peered":"2024-04-25T15:28:56.056491+0000","last_clean":"2024-04-25T15:28:56.056491+0000","last_became_active":"2024-04-25T15:28:56.050541+0000","last_became_peered":"2024-04-25T15:28:56.050541+0000","last_unstale":"2024-04-25T15:28:56.056491+0000","last_undegraded":"2024-04-25T15:28:56.056491+0000","last_fullsized":"2024-04-25T15:28:56.056491+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:30.501641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036701+0000","last_change":"2024-04-25T15:28:56.036701+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036701+0000","last_undegraded":"2024-04-25T15:28:56.036701+0000","last_fullsized":"2024-04-25T15:28:56.036701+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062513+0000","last_change":"2024-04-25T15:28:56.061338+0000","last_active":"2024-04-25T15:28:56.062513+0000","last_peered":"2024-04-25T15:28:56.062513+0000","last_clean":"2024-04-25T15:28:56.062513+0000","last_became_active":"2024-04-25T15:28:56.061028+0000","last_became_peered":"2024-04-25T15:28:56.061028+0000","last_unstale":"2024-04-25T15:28:56.062513+0000","last_undegraded":"2024-04-25T15:28:56.062513+0000","last_fullsized":"2024-04-25T15:28:56.062513+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:59.070550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.038961+0000","last_change":"2024-04-25T15:28:56.038961+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.038961+0000","last_undegraded":"2024-04-25T15:28:56.038961+0000","last_fullsized":"2024-04-25T15:28:56.038961+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062170+0000","last_change":"2024-04-25T15:28:56.058667+0000","last_active":"2024-04-25T15:28:56.062170+0000","last_peered":"2024-04-25T15:28:56.062170+0000","last_clean":"2024-04-25T15:28:56.062170+0000","last_became_active":"2024-04-25T15:28:56.057323+0000","last_became_peered":"2024-04-25T15:28:56.057323+0000","last_unstale":"2024-04-25T15:28:56.062170+0000","last_undegraded":"2024-04-25T15:28:56.062170+0000","last_fullsized":"2024-04-25T15:28:56.062170+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:06.756381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037203+0000","last_change":"2024-04-25T15:28:56.037203+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037203+0000","last_undegraded":"2024-04-25T15:28:56.037203+0000","last_fullsized":"2024-04-25T15:28:56.037203+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.053076+0000","last_change":"2024-04-25T15:28:56.046616+0000","last_active":"2024-04-25T15:28:56.053076+0000","last_peered":"2024-04-25T15:28:56.053076+0000","last_clean":"2024-04-25T15:28:56.053076+0000","last_became_active":"2024-04-25T15:28:56.046311+0000","last_became_peered":"2024-04-25T15:28:56.046311+0000","last_unstale":"2024-04-25T15:28:56.053076+0000","last_undegraded":"2024-04-25T15:28:56.053076+0000","last_fullsized":"2024-04-25T15:28:56.053076+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:21:45.358976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044011+0000","last_change":"2024-04-25T15:28:56.044011+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044011+0000","last_undegraded":"2024-04-25T15:28:56.044011+0000","last_fullsized":"2024-04-25T15:28:56.044011+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059069+0000","last_change":"2024-04-25T15:28:56.053865+0000","last_active":"2024-04-25T15:28:56.059069+0000","last_peered":"2024-04-25T15:28:56.059069+0000","last_clean":"2024-04-25T15:28:56.059069+0000","last_became_active":"2024-04-25T15:28:56.053511+0000","last_became_peered":"2024-04-25T15:28:56.053511+0000","last_unstale":"2024-04-25T15:28:56.059069+0000","last_undegraded":"2024-04-25T15:28:56.059069+0000","last_fullsized":"2024-04-25T15:28:56.059069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044913+0000","last_change":"2024-04-25T15:28:56.044913+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044913+0000","last_undegraded":"2024-04-25T15:28:56.044913+0000","last_fullsized":"2024-04-25T15:28:56.044913+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.050295+0000","last_change":"2024-04-25T15:28:56.044068+0000","last_active":"2024-04-25T15:28:56.050295+0000","last_peered":"2024-04-25T15:28:56.050295+0000","last_clean":"2024-04-25T15:28:56.050295+0000","last_became_active":"2024-04-25T15:28:56.043670+0000","last_became_peered":"2024-04-25T15:28:56.043670+0000","last_unstale":"2024-04-25T15:28:56.050295+0000","last_undegraded":"2024-04-25T15:28:56.050295+0000","last_fullsized":"2024-04-25T15:28:56.050295+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041617+0000","last_change":"2024-04-25T15:28:56.041617+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041617+0000","last_undegraded":"2024-04-25T15:28:56.041617+0000","last_fullsized":"2024-04-25T15:28:56.041617+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056097+0000","last_change":"2024-04-25T15:28:56.051968+0000","last_active":"2024-04-25T15:28:56.056097+0000","last_peered":"2024-04-25T15:28:56.056097+0000","last_clean":"2024-04-25T15:28:56.056097+0000","last_became_active":"2024-04-25T15:28:56.051236+0000","last_became_peered":"2024-04-25T15:28:56.051236+0000","last_unstale":"2024-04-25T15:28:56.056097+0000","last_undegraded":"2024-04-25T15:28:56.056097+0000","last_fullsized":"2024-04-25T15:28:56.056097+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:45.569280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045084+0000","last_change":"2024-04-25T15:28:56.045084+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045084+0000","last_undegraded":"2024-04-25T15:28:56.045084+0000","last_fullsized":"2024-04-25T15:28:56.045084+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057007+0000","last_change":"2024-04-25T15:28:56.053779+0000","last_active":"2024-04-25T15:28:56.057007+0000","last_peered":"2024-04-25T15:28:56.057007+0000","last_clean":"2024-04-25T15:28:56.057007+0000","last_became_active":"2024-04-25T15:28:56.053471+0000","last_became_peered":"2024-04-25T15:28:56.053471+0000","last_unstale":"2024-04-25T15:28:56.057007+0000","last_undegraded":"2024-04-25T15:28:56.057007+0000","last_fullsized":"2024-04-25T15:28:56.057007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:25:01.410150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037150+0000","last_change":"2024-04-25T15:28:56.037150+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037150+0000","last_undegraded":"2024-04-25T15:28:56.037150+0000","last_fullsized":"2024-04-25T15:28:56.037150+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059300+0000","last_change":"2024-04-25T15:28:56.052681+0000","last_active":"2024-04-25T15:28:56.059300+0000","last_peered":"2024-04-25T15:28:56.059300+0000","last_clean":"2024-04-25T15:28:56.059300+0000","last_became_active":"2024-04-25T15:28:56.052385+0000","last_became_peered":"2024-04-25T15:28:56.052385+0000","last_unstale":"2024-04-25T15:28:56.059300+0000","last_undegraded":"2024-04-25T15:28:56.059300+0000","last_fullsized":"2024-04-25T15:28:56.059300+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:45.314705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044935+0000","last_change":"2024-04-25T15:28:56.044935+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044935+0000","last_undegraded":"2024-04-25T15:28:56.044935+0000","last_fullsized":"2024-04-25T15:28:56.044935+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062215+0000","last_change":"2024-04-25T15:28:56.060187+0000","last_active":"2024-04-25T15:28:56.062215+0000","last_peered":"2024-04-25T15:28:56.062215+0000","last_clean":"2024-04-25T15:28:56.062215+0000","last_became_active":"2024-04-25T15:28:56.059867+0000","last_became_peered":"2024-04-25T15:28:56.059867+0000","last_unstale":"2024-04-25T15:28:56.062215+0000","last_undegraded":"2024-04-25T15:28:56.062215+0000","last_fullsized":"2024-04-25T15:28:56.062215+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:59.420234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043391+0000","last_change":"2024-04-25T15:28:56.043391+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043391+0000","last_undegraded":"2024-04-25T15:28:56.043391+0000","last_fullsized":"2024-04-25T15:28:56.043391+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059189+0000","last_change":"2024-04-25T15:28:56.052624+0000","last_active":"2024-04-25T15:28:56.059189+0000","last_peered":"2024-04-25T15:28:56.059189+0000","last_clean":"2024-04-25T15:28:56.059189+0000","last_became_active":"2024-04-25T15:28:56.052283+0000","last_became_peered":"2024-04-25T15:28:56.052283+0000","last_unstale":"2024-04-25T15:28:56.059189+0000","last_undegraded":"2024-04-25T15:28:56.059189+0000","last_fullsized":"2024-04-25T15:28:56.059189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:37.819844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041305+0000","last_change":"2024-04-25T15:28:56.041305+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041305+0000","last_undegraded":"2024-04-25T15:28:56.041305+0000","last_fullsized":"2024-04-25T15:28:56.041305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056432+0000","last_change":"2024-04-25T15:28:56.050894+0000","last_active":"2024-04-25T15:28:56.056432+0000","last_peered":"2024-04-25T15:28:56.056432+0000","last_clean":"2024-04-25T15:28:56.056432+0000","last_became_active":"2024-04-25T15:28:56.050570+0000","last_became_peered":"2024-04-25T15:28:56.050570+0000","last_unstale":"2024-04-25T15:28:56.056432+0000","last_undegraded":"2024-04-25T15:28:56.056432+0000","last_fullsized":"2024-04-25T15:28:56.056432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044386+0000","last_change":"2024-04-25T15:28:56.044386+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044386+0000","last_undegraded":"2024-04-25T15:28:56.044386+0000","last_fullsized":"2024-04-25T15:28:56.044386+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.049680+0000","last_change":"2024-04-25T15:28:56.045642+0000","last_active":"2024-04-25T15:28:56.049680+0000","last_peered":"2024-04-25T15:28:56.049680+0000","last_clean":"2024-04-25T15:28:56.049680+0000","last_became_active":"2024-04-25T15:28:56.045273+0000","last_became_peered":"2024-04-25T15:28:56.045273+0000","last_unstale":"2024-04-25T15:28:56.049680+0000","last_undegraded":"2024-04-25T15:28:56.049680+0000","last_fullsized":"2024-04-25T15:28:56.049680+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:21.737399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044193+0000","last_change":"2024-04-25T15:28:56.044193+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044193+0000","last_undegraded":"2024-04-25T15:28:56.044193+0000","last_fullsized":"2024-04-25T15:28:56.044193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.062036+0000","last_change":"2024-04-25T15:28:56.059690+0000","last_active":"2024-04-25T15:28:56.062036+0000","last_peered":"2024-04-25T15:28:56.062036+0000","last_clean":"2024-04-25T15:28:56.062036+0000","last_became_active":"2024-04-25T15:28:56.059372+0000","last_became_peered":"2024-04-25T15:28:56.059372+0000","last_unstale":"2024-04-25T15:28:56.062036+0000","last_undegraded":"2024-04-25T15:28:56.062036+0000","last_fullsized":"2024-04-25T15:28:56.062036+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:03.773654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044493+0000","last_change":"2024-04-25T15:28:56.044493+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044493+0000","last_undegraded":"2024-04-25T15:28:56.044493+0000","last_fullsized":"2024-04-25T15:28:56.044493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056674+0000","last_change":"2024-04-25T15:28:56.053555+0000","last_active":"2024-04-25T15:28:56.056674+0000","last_peered":"2024-04-25T15:28:56.056674+0000","last_clean":"2024-04-25T15:28:56.056674+0000","last_became_active":"2024-04-25T15:28:56.053235+0000","last_became_peered":"2024-04-25T15:28:56.053235+0000","last_unstale":"2024-04-25T15:28:56.056674+0000","last_undegraded":"2024-04-25T15:28:56.056674+0000","last_fullsized":"2024-04-25T15:28:56.056674+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:55.012054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.043401+0000","last_change":"2024-04-25T15:28:56.043401+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.043401+0000","last_undegraded":"2024-04-25T15:28:56.043401+0000","last_fullsized":"2024-04-25T15:28:56.043401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.054456+0000","last_change":"2024-04-25T15:28:56.049930+0000","last_active":"2024-04-25T15:28:56.054456+0000","last_peered":"2024-04-25T15:28:56.054456+0000","last_clean":"2024-04-25T15:28:56.054456+0000","last_became_active":"2024-04-25T15:28:56.049617+0000","last_became_peered":"2024-04-25T15:28:56.049617+0000","last_unstale":"2024-04-25T15:28:56.054456+0000","last_undegraded":"2024-04-25T15:28:56.054456+0000","last_fullsized":"2024-04-25T15:28:56.054456+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:29.266322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042363+0000","last_change":"2024-04-25T15:28:56.042363+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042363+0000","last_undegraded":"2024-04-25T15:28:56.042363+0000","last_fullsized":"2024-04-25T15:28:56.042363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.057275+0000","last_change":"2024-04-25T15:28:56.054470+0000","last_active":"2024-04-25T15:28:56.057275+0000","last_peered":"2024-04-25T15:28:56.057275+0000","last_clean":"2024-04-25T15:28:56.057275+0000","last_became_active":"2024-04-25T15:28:56.054148+0000","last_became_peered":"2024-04-25T15:28:56.054148+0000","last_unstale":"2024-04-25T15:28:56.057275+0000","last_undegraded":"2024-04-25T15:28:56.057275+0000","last_fullsized":"2024-04-25T15:28:56.057275+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.037520+0000","last_change":"2024-04-25T15:28:56.037520+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.037520+0000","last_undegraded":"2024-04-25T15:28:56.037520+0000","last_fullsized":"2024-04-25T15:28:56.037520+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056659+0000","last_change":"2024-04-25T15:28:56.051763+0000","last_active":"2024-04-25T15:28:56.056659+0000","last_peered":"2024-04-25T15:28:56.056659+0000","last_clean":"2024-04-25T15:28:56.056659+0000","last_became_active":"2024-04-25T15:28:56.051431+0000","last_became_peered":"2024-04-25T15:28:56.051431+0000","last_unstale":"2024-04-25T15:28:56.056659+0000","last_undegraded":"2024-04-25T15:28:56.056659+0000","last_fullsized":"2024-04-25T15:28:56.056659+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:44:13.470300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.052252+0000","last_change":"2024-04-25T15:28:56.052252+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.052252+0000","last_undegraded":"2024-04-25T15:28:56.052252+0000","last_fullsized":"2024-04-25T15:28:56.052252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060700+0000","last_change":"2024-04-25T15:28:56.059005+0000","last_active":"2024-04-25T15:28:56.060700+0000","last_peered":"2024-04-25T15:28:56.060700+0000","last_clean":"2024-04-25T15:28:56.060700+0000","last_became_active":"2024-04-25T15:28:56.057981+0000","last_became_peered":"2024-04-25T15:28:56.057981+0000","last_unstale":"2024-04-25T15:28:56.060700+0000","last_undegraded":"2024-04-25T15:28:56.060700+0000","last_fullsized":"2024-04-25T15:28:56.060700+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.045632+0000","last_change":"2024-04-25T15:28:56.045632+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.045632+0000","last_undegraded":"2024-04-25T15:28:56.045632+0000","last_fullsized":"2024-04-25T15:28:56.045632+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056297+0000","last_change":"2024-04-25T15:28:56.048576+0000","last_active":"2024-04-25T15:28:56.056297+0000","last_peered":"2024-04-25T15:28:56.056297+0000","last_clean":"2024-04-25T15:28:56.056297+0000","last_became_active":"2024-04-25T15:28:56.048273+0000","last_became_peered":"2024-04-25T15:28:56.048273+0000","last_unstale":"2024-04-25T15:28:56.056297+0000","last_undegraded":"2024-04-25T15:28:56.056297+0000","last_fullsized":"2024-04-25T15:28:56.056297+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:30.682522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042145+0000","last_change":"2024-04-25T15:28:56.042145+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042145+0000","last_undegraded":"2024-04-25T15:28:56.042145+0000","last_fullsized":"2024-04-25T15:28:56.042145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.056488+0000","last_change":"2024-04-25T15:28:56.051473+0000","last_active":"2024-04-25T15:28:56.056488+0000","last_peered":"2024-04-25T15:28:56.056488+0000","last_clean":"2024-04-25T15:28:56.056488+0000","last_became_active":"2024-04-25T15:28:56.051171+0000","last_became_peered":"2024-04-25T15:28:56.051171+0000","last_unstale":"2024-04-25T15:28:56.056488+0000","last_undegraded":"2024-04-25T15:28:56.056488+0000","last_fullsized":"2024-04-25T15:28:56.056488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:03.672980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042897+0000","last_change":"2024-04-25T15:28:56.042897+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042897+0000","last_undegraded":"2024-04-25T15:28:56.042897+0000","last_fullsized":"2024-04-25T15:28:56.042897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060976+0000","last_change":"2024-04-25T15:28:56.058257+0000","last_active":"2024-04-25T15:28:56.060976+0000","last_peered":"2024-04-25T15:28:56.060976+0000","last_clean":"2024-04-25T15:28:56.060976+0000","last_became_active":"2024-04-25T15:28:56.057736+0000","last_became_peered":"2024-04-25T15:28:56.057736+0000","last_unstale":"2024-04-25T15:28:56.060976+0000","last_undegraded":"2024-04-25T15:28:56.060976+0000","last_fullsized":"2024-04-25T15:28:56.060976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.036658+0000","last_change":"2024-04-25T15:28:56.036658+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.036658+0000","last_undegraded":"2024-04-25T15:28:56.036658+0000","last_fullsized":"2024-04-25T15:28:56.036658+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.061871+0000","last_change":"2024-04-25T15:28:56.056141+0000","last_active":"2024-04-25T15:28:56.061871+0000","last_peered":"2024-04-25T15:28:56.061871+0000","last_clean":"2024-04-25T15:28:56.061871+0000","last_became_active":"2024-04-25T15:28:56.055725+0000","last_became_peered":"2024-04-25T15:28:56.055725+0000","last_unstale":"2024-04-25T15:28:56.061871+0000","last_undegraded":"2024-04-25T15:28:56.061871+0000","last_fullsized":"2024-04-25T15:28:56.061871+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:29.578756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.042444+0000","last_change":"2024-04-25T15:28:56.042444+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.042444+0000","last_undegraded":"2024-04-25T15:28:56.042444+0000","last_fullsized":"2024-04-25T15:28:56.042444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.060626+0000","last_change":"2024-04-25T15:28:56.055311+0000","last_active":"2024-04-25T15:28:56.060626+0000","last_peered":"2024-04-25T15:28:56.060626+0000","last_clean":"2024-04-25T15:28:56.060626+0000","last_became_active":"2024-04-25T15:28:56.054958+0000","last_became_peered":"2024-04-25T15:28:56.054958+0000","last_unstale":"2024-04-25T15:28:56.060626+0000","last_undegraded":"2024-04-25T15:28:56.060626+0000","last_fullsized":"2024-04-25T15:28:56.060626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:04.302144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.041713+0000","last_change":"2024-04-25T15:28:56.041713+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.041713+0000","last_undegraded":"2024-04-25T15:28:56.041713+0000","last_fullsized":"2024-04-25T15:28:56.041713+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.051579+0000","last_change":"2024-04-25T15:28:56.050355+0000","last_active":"2024-04-25T15:28:56.051579+0000","last_peered":"2024-04-25T15:28:56.051579+0000","last_clean":"2024-04-25T15:28:56.051579+0000","last_became_active":"2024-04-25T15:28:56.050028+0000","last_became_peered":"2024-04-25T15:28:56.050028+0000","last_unstale":"2024-04-25T15:28:56.051579+0000","last_undegraded":"2024-04-25T15:28:56.051579+0000","last_fullsized":"2024-04-25T15:28:56.051579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:18.740971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":90,"state":"creating+peering","last_fresh":"2024-04-25T15:28:56.044259+0000","last_change":"2024-04-25T15:28:56.044259+0000","last_active":"2024-04-25T15:28:56.028812+0000","last_peered":"2024-04-25T15:28:56.028812+0000","last_clean":"2024-04-25T15:28:56.028812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T15:28:56.044259+0000","last_undegraded":"2024-04-25T15:28:56.044259+0000","last_fullsized":"2024-04-25T15:28:56.044259+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.059127+0000","last_change":"2024-04-25T15:28:56.054041+0000","last_active":"2024-04-25T15:28:56.059127+0000","last_peered":"2024-04-25T15:28:56.059127+0000","last_clean":"2024-04-25T15:28:56.059127+0000","last_became_active":"2024-04-25T15:28:56.053693+0000","last_became_peered":"2024-04-25T15:28:56.053693+0000","last_unstale":"2024-04-25T15:28:56.059127+0000","last_undegraded":"2024-04-25T15:28:56.059127+0000","last_fullsized":"2024-04-25T15:28:56.059127+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":235,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:28:56.043621+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:56.043621+0000","last_peered":"2024-04-25T15:28:56.043621+0000","last_clean":"2024-04-25T15:28:56.043621+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:56.043621+0000","last_undegraded":"2024-04-25T15:28:56.043621+0000","last_fullsized":"2024-04-25T15:28:56.043621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1114112,"data_stored":472313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1101824,"data_stored":466038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1708032,"data_stored":1062681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:05.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.756+0000 7fe8ca734640 1 Processor -- start 2024-04-25T15:29:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.756+0000 7fe8ca734640 1 -- start start 2024-04-25T15:29:05.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8ca734640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8c40cfd40 0x7fe8c40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:05.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8ca734640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b7f50 0x7fe8c40ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:05.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8ca734640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8c40d1c10 con 0x7fe8c40b3e00 2024-04-25T15:29:05.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8ca734640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c40d27e0 con 0x7fe8c40cfd40 2024-04-25T15:29:05.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8ca734640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c40d1eb0 con 0x7fe8c40b7f50 2024-04-25T15:29:05.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8c40cfd40 0x7fe8c40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe8c40cfd40 0x7fe8c40b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57834/0 (socket says 172.21.15.67:57834) 2024-04-25T15:29:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c3fff640 1 -- 172.21.15.67:0/1808842349 learned_addr learned my addr 172.21.15.67:0/1808842349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c8cd1640 1 --2- 172.21.15.67:0/1808842349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b7f50 0x7fe8c40ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c3fff640 1 -- 172.21.15.67:0/1808842349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b7f50 msgr2=0x7fe8c40ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c3fff640 1 --2- 172.21.15.67:0/1808842349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b7f50 0x7fe8c40ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c3fff640 1 -- 172.21.15.67:0/1808842349 >> v1:172.21.15.176:6789/0 conn(0x7fe8c40b3e00 legacy=0x7fe8c40b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.758+0000 7fe8c3fff640 1 -- 172.21.15.67:0/1808842349 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c40d2190 con 0x7fe8c40cfd40 2024-04-25T15:29:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.759+0000 7fe8c3fff640 1 --2- 172.21.15.67:0/1808842349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40cfd40 0x7fe8c40b21a0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009870 tx=0x7fe8ac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4c87b5d592106d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.759+0000 7fe8c2ffd640 1 -- 172.21.15.67:0/1808842349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac037070 con 0x7fe8c40cfd40 2024-04-25T15:29:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.759+0000 7fe8c2ffd640 1 -- 172.21.15.67:0/1808842349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac032440 con 0x7fe8c40cfd40 2024-04-25T15:29:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.759+0000 7fe8c2ffd640 1 -- 172.21.15.67:0/1808842349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac032810 con 0x7fe8c40cfd40 2024-04-25T15:29:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 -- 172.21.15.67:0/1808842349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40cfd40 msgr2=0x7fe8c40b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 --2- 172.21.15.67:0/1808842349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40cfd40 0x7fe8c40b21a0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009870 tx=0x7fe8ac02a270 comp rx=0 tx=0).stop 2024-04-25T15:29:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 -- 172.21.15.67:0/1808842349 shutdown_connections 2024-04-25T15:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 --2- 172.21.15.67:0/1808842349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b7f50 0x7fe8c40ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 --2- 172.21.15.67:0/1808842349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40cfd40 0x7fe8c40b21a0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 -- 172.21.15.67:0/1808842349 >> 172.21.15.67:0/1808842349 conn(0x7fe8c409b7d0 msgr2=0x7fe8c409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 -- 172.21.15.67:0/1808842349 shutdown_connections 2024-04-25T15:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.760+0000 7fe8ca734640 1 -- 172.21.15.67:0/1808842349 wait complete. 2024-04-25T15:29:05.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.762+0000 7fe8ca734640 1 Processor -- start 2024-04-25T15:29:05.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.762+0000 7fe8ca734640 1 -- start start 2024-04-25T15:29:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.763+0000 7fe8ca734640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b3e00 0x7fe8c4159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.763+0000 7fe8ca734640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8c40cfd40 0x7fe8c4162670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.763+0000 7fe8ca734640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40b1a40 0x7fe8c406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.763+0000 7fe8ca734640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c40b4210 con 0x7fe8c40b1a40 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.763+0000 7fe8ca734640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c40ce650 con 0x7fe8c40cfd40 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.763+0000 7fe8ca734640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c40b4390 con 0x7fe8c40b3e00 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40b1a40 0x7fe8c406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40b1a40 0x7fe8c406dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57850/0 (socket says 172.21.15.67:57850) 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 -- 172.21.15.67:0/3420011908 learned_addr learned my addr 172.21.15.67:0/3420011908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c3fff640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b3e00 0x7fe8c4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c37fe640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8c40cfd40 0x7fe8c4162670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 -- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b3e00 msgr2=0x7fe8c4159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b3e00 0x7fe8c4159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 -- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8c40cfd40 msgr2=0x7fe8c4162670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8c40cfd40 0x7fe8c4162670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 -- 172.21.15.67:0/3420011908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c4162fc0 con 0x7fe8c40b1a40 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c8cd1640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40b1a40 0x7fe8c406dc40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009d30 tx=0x7fe8bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c0ff9640 1 -- 172.21.15.67:0/3420011908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc018020 con 0x7fe8c40b1a40 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c0ff9640 1 -- 172.21.15.67:0/3420011908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8bc014480 con 0x7fe8c40b1a40 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c41632a0 con 0x7fe8c40b1a40 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8c0ff9640 1 -- 172.21.15.67:0/3420011908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc014870 con 0x7fe8c40b1a40 2024-04-25T15:29:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.764+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c4160cc0 con 0x7fe8c40b1a40 2024-04-25T15:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.765+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c4050740 con 0x7fe8c40b1a40 2024-04-25T15:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.766+0000 7fe8c0ff9640 1 -- 172.21.15.67:0/3420011908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe8bc005d10 con 0x7fe8c40b1a40 2024-04-25T15:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.766+0000 7fe8c0ff9640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8a40744b0 0x7fe8a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.767+0000 7fe8c0ff9640 1 -- 172.21.15.67:0/3420011908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fe8bc095b20 con 0x7fe8c40b1a40 2024-04-25T15:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.767+0000 7fe8c3fff640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8a40744b0 0x7fe8a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:05.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.767+0000 7fe8c3fff640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8a40744b0 0x7fe8a40769c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac034000 tx=0x7fe8ac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:05.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.771+0000 7fe8c0ff9640 1 -- 172.21.15.67:0/3420011908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8bc022070 con 0x7fe8c40b1a40 2024-04-25T15:29:05.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.950+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8c40cf940 con 0x7fe8a40744b0 2024-04-25T15:29:05.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.958+0000 7fe8c0ff9640 1 -- 172.21.15.67:0/3420011908 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252075 (secure 0 0 0) 0x7fe8c40cf940 con 0x7fe8a40744b0 2024-04-25T15:29:05.961 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:05.962 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:05.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.960+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8a40744b0 msgr2=0x7fe8a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:05.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.960+0000 7fe8ca734640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8a40744b0 0x7fe8a40769c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac034000 tx=0x7fe8ac02e000 comp rx=0 tx=0).stop 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.960+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40b1a40 msgr2=0x7fe8c406dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.960+0000 7fe8ca734640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40b1a40 0x7fe8c406dc40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009d30 tx=0x7fe8bc00c490 comp rx=0 tx=0).stop 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 shutdown_connections 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe8a40744b0 0x7fe8a40769c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8c40b1a40 0x7fe8c406dc40 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8c40cfd40 0x7fe8c4162670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 --2- 172.21.15.67:0/3420011908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8c40b3e00 0x7fe8c4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 >> 172.21.15.67:0/3420011908 conn(0x7fe8c409b7d0 msgr2=0x7fe8c415f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 shutdown_connections 2024-04-25T15:29:05.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:05.961+0000 7fe8ca734640 1 -- 172.21.15.67:0/3420011908 wait complete. 2024-04-25T15:29:05.973 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-04-25T15:29:04.849073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":251284,"kb_used_data":10120,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754723436,"statfs":{"total":773094113280,"available":772836798464,"internally_reserved":0,"allocated":10362880,"data_stored":5068660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002125"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093754+0000","last_change":"2024-04-25T15:28:57.074275+0000","last_active":"2024-04-25T15:28:58.093754+0000","last_peered":"2024-04-25T15:28:58.093754+0000","last_clean":"2024-04-25T15:28:58.093754+0000","last_became_active":"2024-04-25T15:28:57.073671+0000","last_became_peered":"2024-04-25T15:28:57.073671+0000","last_unstale":"2024-04-25T15:28:58.093754+0000","last_undegraded":"2024-04-25T15:28:58.093754+0000","last_fullsized":"2024-04-25T15:28:58.093754+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:00.622735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062816+0000","last_change":"2024-04-25T15:28:57.072577+0000","last_active":"2024-04-25T15:28:58.062816+0000","last_peered":"2024-04-25T15:28:58.062816+0000","last_clean":"2024-04-25T15:28:58.062816+0000","last_became_active":"2024-04-25T15:28:57.072255+0000","last_became_peered":"2024-04-25T15:28:57.072255+0000","last_unstale":"2024-04-25T15:28:58.062816+0000","last_undegraded":"2024-04-25T15:28:58.062816+0000","last_fullsized":"2024-04-25T15:28:58.062816+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:13.476136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093316+0000","last_change":"2024-04-25T15:28:57.073966+0000","last_active":"2024-04-25T15:28:58.093316+0000","last_peered":"2024-04-25T15:28:58.093316+0000","last_clean":"2024-04-25T15:28:58.093316+0000","last_became_active":"2024-04-25T15:28:57.073515+0000","last_became_peered":"2024-04-25T15:28:57.073515+0000","last_unstale":"2024-04-25T15:28:58.093316+0000","last_undegraded":"2024-04-25T15:28:58.093316+0000","last_fullsized":"2024-04-25T15:28:58.093316+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:44:14.475757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062963+0000","last_change":"2024-04-25T15:28:57.068480+0000","last_active":"2024-04-25T15:28:58.062963+0000","last_peered":"2024-04-25T15:28:58.062963+0000","last_clean":"2024-04-25T15:28:58.062963+0000","last_became_active":"2024-04-25T15:28:57.067988+0000","last_became_peered":"2024-04-25T15:28:57.067988+0000","last_unstale":"2024-04-25T15:28:58.062963+0000","last_undegraded":"2024-04-25T15:28:58.062963+0000","last_fullsized":"2024-04-25T15:28:58.062963+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:45.476286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154743+0000","last_change":"2024-04-25T15:28:57.070838+0000","last_active":"2024-04-25T15:28:59.154743+0000","last_peered":"2024-04-25T15:28:59.154743+0000","last_clean":"2024-04-25T15:28:59.154743+0000","last_became_active":"2024-04-25T15:28:57.070526+0000","last_became_peered":"2024-04-25T15:28:57.070526+0000","last_unstale":"2024-04-25T15:28:59.154743+0000","last_undegraded":"2024-04-25T15:28:59.154743+0000","last_fullsized":"2024-04-25T15:28:59.154743+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:09.401534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063003+0000","last_change":"2024-04-25T15:28:57.069701+0000","last_active":"2024-04-25T15:28:58.063003+0000","last_peered":"2024-04-25T15:28:58.063003+0000","last_clean":"2024-04-25T15:28:58.063003+0000","last_became_active":"2024-04-25T15:28:57.069386+0000","last_became_peered":"2024-04-25T15:28:57.069386+0000","last_unstale":"2024-04-25T15:28:58.063003+0000","last_undegraded":"2024-04-25T15:28:58.063003+0000","last_fullsized":"2024-04-25T15:28:58.063003+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:03.577565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062903+0000","last_change":"2024-04-25T15:28:57.072396+0000","last_active":"2024-04-25T15:28:58.062903+0000","last_peered":"2024-04-25T15:28:58.062903+0000","last_clean":"2024-04-25T15:28:58.062903+0000","last_became_active":"2024-04-25T15:28:57.069489+0000","last_became_peered":"2024-04-25T15:28:57.069489+0000","last_unstale":"2024-04-25T15:28:58.062903+0000","last_undegraded":"2024-04-25T15:28:58.062903+0000","last_fullsized":"2024-04-25T15:28:58.062903+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:16.364472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062499+0000","last_change":"2024-04-25T15:28:57.068624+0000","last_active":"2024-04-25T15:28:58.062499+0000","last_peered":"2024-04-25T15:28:58.062499+0000","last_clean":"2024-04-25T15:28:58.062499+0000","last_became_active":"2024-04-25T15:28:57.068177+0000","last_became_peered":"2024-04-25T15:28:57.068177+0000","last_unstale":"2024-04-25T15:28:58.062499+0000","last_undegraded":"2024-04-25T15:28:58.062499+0000","last_fullsized":"2024-04-25T15:28:58.062499+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:22.651126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986658+0000","last_change":"2024-04-25T15:28:57.075552+0000","last_active":"2024-04-25T15:28:58.986658+0000","last_peered":"2024-04-25T15:28:58.986658+0000","last_clean":"2024-04-25T15:28:58.986658+0000","last_became_active":"2024-04-25T15:28:57.073980+0000","last_became_peered":"2024-04-25T15:28:57.073980+0000","last_unstale":"2024-04-25T15:28:58.986658+0000","last_undegraded":"2024-04-25T15:28:58.986658+0000","last_fullsized":"2024-04-25T15:28:58.986658+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:34:44.423535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154749+0000","last_change":"2024-04-25T15:28:57.078243+0000","last_active":"2024-04-25T15:28:59.154749+0000","last_peered":"2024-04-25T15:28:59.154749+0000","last_clean":"2024-04-25T15:28:59.154749+0000","last_became_active":"2024-04-25T15:28:57.077907+0000","last_became_peered":"2024-04-25T15:28:57.077907+0000","last_unstale":"2024-04-25T15:28:59.154749+0000","last_undegraded":"2024-04-25T15:28:59.154749+0000","last_fullsized":"2024-04-25T15:28:59.154749+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:16.364472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063217+0000","last_change":"2024-04-25T15:28:57.075777+0000","last_active":"2024-04-25T15:28:58.063217+0000","last_peered":"2024-04-25T15:28:58.063217+0000","last_clean":"2024-04-25T15:28:58.063217+0000","last_became_active":"2024-04-25T15:28:57.075049+0000","last_became_peered":"2024-04-25T15:28:57.075049+0000","last_unstale":"2024-04-25T15:28:58.063217+0000","last_undegraded":"2024-04-25T15:28:58.063217+0000","last_fullsized":"2024-04-25T15:28:58.063217+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:03.566264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986507+0000","last_change":"2024-04-25T15:28:57.078221+0000","last_active":"2024-04-25T15:28:58.986507+0000","last_peered":"2024-04-25T15:28:58.986507+0000","last_clean":"2024-04-25T15:28:58.986507+0000","last_became_active":"2024-04-25T15:28:57.077870+0000","last_became_peered":"2024-04-25T15:28:57.077870+0000","last_unstale":"2024-04-25T15:28:58.986507+0000","last_undegraded":"2024-04-25T15:28:58.986507+0000","last_fullsized":"2024-04-25T15:28:58.986507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:20.121011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062413+0000","last_change":"2024-04-25T15:28:57.074911+0000","last_active":"2024-04-25T15:28:58.062413+0000","last_peered":"2024-04-25T15:28:58.062413+0000","last_clean":"2024-04-25T15:28:58.062413+0000","last_became_active":"2024-04-25T15:28:57.074584+0000","last_became_peered":"2024-04-25T15:28:57.074584+0000","last_unstale":"2024-04-25T15:28:58.062413+0000","last_undegraded":"2024-04-25T15:28:58.062413+0000","last_fullsized":"2024-04-25T15:28:58.062413+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:03.566264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066695+0000","last_change":"2024-04-25T15:28:57.058638+0000","last_active":"2024-04-25T15:28:59.066695+0000","last_peered":"2024-04-25T15:28:59.066695+0000","last_clean":"2024-04-25T15:28:59.066695+0000","last_became_active":"2024-04-25T15:28:57.058261+0000","last_became_peered":"2024-04-25T15:28:57.058261+0000","last_unstale":"2024-04-25T15:28:59.066695+0000","last_undegraded":"2024-04-25T15:28:59.066695+0000","last_fullsized":"2024-04-25T15:28:59.066695+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:52.749048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062317+0000","last_change":"2024-04-25T15:28:57.078116+0000","last_active":"2024-04-25T15:28:58.062317+0000","last_peered":"2024-04-25T15:28:58.062317+0000","last_clean":"2024-04-25T15:28:58.062317+0000","last_became_active":"2024-04-25T15:28:57.077804+0000","last_became_peered":"2024-04-25T15:28:57.077804+0000","last_unstale":"2024-04-25T15:28:58.062317+0000","last_undegraded":"2024-04-25T15:28:58.062317+0000","last_fullsized":"2024-04-25T15:28:58.062317+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:19:55.184936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062933+0000","last_change":"2024-04-25T15:28:57.068910+0000","last_active":"2024-04-25T15:28:58.062933+0000","last_peered":"2024-04-25T15:28:58.062933+0000","last_clean":"2024-04-25T15:28:58.062933+0000","last_became_active":"2024-04-25T15:28:57.068578+0000","last_became_peered":"2024-04-25T15:28:57.068578+0000","last_unstale":"2024-04-25T15:28:58.062933+0000","last_undegraded":"2024-04-25T15:28:58.062933+0000","last_fullsized":"2024-04-25T15:28:58.062933+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:36.737991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062950+0000","last_change":"2024-04-25T15:28:57.076611+0000","last_active":"2024-04-25T15:28:58.062950+0000","last_peered":"2024-04-25T15:28:58.062950+0000","last_clean":"2024-04-25T15:28:58.062950+0000","last_became_active":"2024-04-25T15:28:57.076283+0000","last_became_peered":"2024-04-25T15:28:57.076283+0000","last_unstale":"2024-04-25T15:28:58.062950+0000","last_undegraded":"2024-04-25T15:28:58.062950+0000","last_fullsized":"2024-04-25T15:28:58.062950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:57.206526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063137+0000","last_change":"2024-04-25T15:28:57.067367+0000","last_active":"2024-04-25T15:28:58.063137+0000","last_peered":"2024-04-25T15:28:58.063137+0000","last_clean":"2024-04-25T15:28:58.063137+0000","last_became_active":"2024-04-25T15:28:57.067008+0000","last_became_peered":"2024-04-25T15:28:57.067008+0000","last_unstale":"2024-04-25T15:28:58.063137+0000","last_undegraded":"2024-04-25T15:28:58.063137+0000","last_fullsized":"2024-04-25T15:28:58.063137+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:46.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066985+0000","last_change":"2024-04-25T15:28:57.080579+0000","last_active":"2024-04-25T15:28:59.066985+0000","last_peered":"2024-04-25T15:28:59.066985+0000","last_clean":"2024-04-25T15:28:59.066985+0000","last_became_active":"2024-04-25T15:28:57.080275+0000","last_became_peered":"2024-04-25T15:28:57.080275+0000","last_unstale":"2024-04-25T15:28:59.066985+0000","last_undegraded":"2024-04-25T15:28:59.066985+0000","last_fullsized":"2024-04-25T15:28:59.066985+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:00.425691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986523+0000","last_change":"2024-04-25T15:28:57.074131+0000","last_active":"2024-04-25T15:28:58.986523+0000","last_peered":"2024-04-25T15:28:58.986523+0000","last_clean":"2024-04-25T15:28:58.986523+0000","last_became_active":"2024-04-25T15:28:57.073413+0000","last_became_peered":"2024-04-25T15:28:57.073413+0000","last_unstale":"2024-04-25T15:28:58.986523+0000","last_undegraded":"2024-04-25T15:28:58.986523+0000","last_fullsized":"2024-04-25T15:28:58.986523+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:05.029043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.067073+0000","last_change":"2024-04-25T15:28:57.069156+0000","last_active":"2024-04-25T15:28:59.067073+0000","last_peered":"2024-04-25T15:28:59.067073+0000","last_clean":"2024-04-25T15:28:59.067073+0000","last_became_active":"2024-04-25T15:28:57.068809+0000","last_became_peered":"2024-04-25T15:28:57.068809+0000","last_unstale":"2024-04-25T15:28:59.067073+0000","last_undegraded":"2024-04-25T15:28:59.067073+0000","last_fullsized":"2024-04-25T15:28:59.067073+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:05.729295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062261+0000","last_change":"2024-04-25T15:28:57.069408+0000","last_active":"2024-04-25T15:28:58.062261+0000","last_peered":"2024-04-25T15:28:58.062261+0000","last_clean":"2024-04-25T15:28:58.062261+0000","last_became_active":"2024-04-25T15:28:57.069074+0000","last_became_peered":"2024-04-25T15:28:57.069074+0000","last_unstale":"2024-04-25T15:28:58.062261+0000","last_undegraded":"2024-04-25T15:28:58.062261+0000","last_fullsized":"2024-04-25T15:28:58.062261+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:12.190330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062239+0000","last_change":"2024-04-25T15:28:57.070083+0000","last_active":"2024-04-25T15:28:58.062239+0000","last_peered":"2024-04-25T15:28:58.062239+0000","last_clean":"2024-04-25T15:28:58.062239+0000","last_became_active":"2024-04-25T15:28:57.069675+0000","last_became_peered":"2024-04-25T15:28:57.069675+0000","last_unstale":"2024-04-25T15:28:58.062239+0000","last_undegraded":"2024-04-25T15:28:58.062239+0000","last_fullsized":"2024-04-25T15:28:58.062239+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:37.997830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063134+0000","last_change":"2024-04-25T15:28:57.078148+0000","last_active":"2024-04-25T15:28:58.063134+0000","last_peered":"2024-04-25T15:28:58.063134+0000","last_clean":"2024-04-25T15:28:58.063134+0000","last_became_active":"2024-04-25T15:28:57.077795+0000","last_became_peered":"2024-04-25T15:28:57.077795+0000","last_unstale":"2024-04-25T15:28:58.063134+0000","last_undegraded":"2024-04-25T15:28:58.063134+0000","last_fullsized":"2024-04-25T15:28:58.063134+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.607419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154906+0000","last_change":"2024-04-25T15:28:57.068484+0000","last_active":"2024-04-25T15:28:59.154906+0000","last_peered":"2024-04-25T15:28:59.154906+0000","last_clean":"2024-04-25T15:28:59.154906+0000","last_became_active":"2024-04-25T15:28:57.068145+0000","last_became_peered":"2024-04-25T15:28:57.068145+0000","last_unstale":"2024-04-25T15:28:59.154906+0000","last_undegraded":"2024-04-25T15:28:59.154906+0000","last_fullsized":"2024-04-25T15:28:59.154906+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.607419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063007+0000","last_change":"2024-04-25T15:28:57.076966+0000","last_active":"2024-04-25T15:28:58.063007+0000","last_peered":"2024-04-25T15:28:58.063007+0000","last_clean":"2024-04-25T15:28:58.063007+0000","last_became_active":"2024-04-25T15:28:57.076451+0000","last_became_peered":"2024-04-25T15:28:57.076451+0000","last_unstale":"2024-04-25T15:28:58.063007+0000","last_undegraded":"2024-04-25T15:28:58.063007+0000","last_fullsized":"2024-04-25T15:28:58.063007+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:54:08.657543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063379+0000","last_change":"2024-04-25T15:28:57.066970+0000","last_active":"2024-04-25T15:28:58.063379+0000","last_peered":"2024-04-25T15:28:58.063379+0000","last_clean":"2024-04-25T15:28:58.063379+0000","last_became_active":"2024-04-25T15:28:57.066298+0000","last_became_peered":"2024-04-25T15:28:57.066298+0000","last_unstale":"2024-04-25T15:28:58.063379+0000","last_undegraded":"2024-04-25T15:28:58.063379+0000","last_fullsized":"2024-04-25T15:28:58.063379+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:33.403927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154820+0000","last_change":"2024-04-25T15:28:57.070935+0000","last_active":"2024-04-25T15:28:59.154820+0000","last_peered":"2024-04-25T15:28:59.154820+0000","last_clean":"2024-04-25T15:28:59.154820+0000","last_became_active":"2024-04-25T15:28:57.070595+0000","last_became_peered":"2024-04-25T15:28:57.070595+0000","last_unstale":"2024-04-25T15:28:59.154820+0000","last_undegraded":"2024-04-25T15:28:59.154820+0000","last_fullsized":"2024-04-25T15:28:59.154820+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:03.577565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062896+0000","last_change":"2024-04-25T15:28:57.078175+0000","last_active":"2024-04-25T15:28:58.062896+0000","last_peered":"2024-04-25T15:28:58.062896+0000","last_clean":"2024-04-25T15:28:58.062896+0000","last_became_active":"2024-04-25T15:28:57.077845+0000","last_became_peered":"2024-04-25T15:28:57.077845+0000","last_unstale":"2024-04-25T15:28:58.062896+0000","last_undegraded":"2024-04-25T15:28:58.062896+0000","last_fullsized":"2024-04-25T15:28:58.062896+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:20.839955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066713+0000","last_change":"2024-04-25T15:28:57.078649+0000","last_active":"2024-04-25T15:28:59.066713+0000","last_peered":"2024-04-25T15:28:59.066713+0000","last_clean":"2024-04-25T15:28:59.066713+0000","last_became_active":"2024-04-25T15:28:57.078271+0000","last_became_peered":"2024-04-25T15:28:57.078271+0000","last_unstale":"2024-04-25T15:28:59.066713+0000","last_undegraded":"2024-04-25T15:28:59.066713+0000","last_fullsized":"2024-04-25T15:28:59.066713+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:34:15.731973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062458+0000","last_change":"2024-04-25T15:28:57.065777+0000","last_active":"2024-04-25T15:28:58.062458+0000","last_peered":"2024-04-25T15:28:58.062458+0000","last_clean":"2024-04-25T15:28:58.062458+0000","last_became_active":"2024-04-25T15:28:57.065526+0000","last_became_peered":"2024-04-25T15:28:57.065526+0000","last_unstale":"2024-04-25T15:28:58.062458+0000","last_undegraded":"2024-04-25T15:28:58.062458+0000","last_fullsized":"2024-04-25T15:28:58.062458+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:33.403927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093123+0000","last_change":"2024-04-25T15:28:57.079398+0000","last_active":"2024-04-25T15:28:58.093123+0000","last_peered":"2024-04-25T15:28:58.093123+0000","last_clean":"2024-04-25T15:28:58.093123+0000","last_became_active":"2024-04-25T15:28:57.079087+0000","last_became_peered":"2024-04-25T15:28:57.079087+0000","last_unstale":"2024-04-25T15:28:58.093123+0000","last_undegraded":"2024-04-25T15:28:58.093123+0000","last_fullsized":"2024-04-25T15:28:58.093123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:30.271779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.067320+0000","last_change":"2024-04-25T15:28:57.070383+0000","last_active":"2024-04-25T15:28:59.067320+0000","last_peered":"2024-04-25T15:28:59.067320+0000","last_clean":"2024-04-25T15:28:59.067320+0000","last_became_active":"2024-04-25T15:28:57.069997+0000","last_became_peered":"2024-04-25T15:28:57.069997+0000","last_unstale":"2024-04-25T15:28:59.067320+0000","last_undegraded":"2024-04-25T15:28:59.067320+0000","last_fullsized":"2024-04-25T15:28:59.067320+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:02:04.767441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063064+0000","last_change":"2024-04-25T15:28:56.059439+0000","last_active":"2024-04-25T15:28:58.063064+0000","last_peered":"2024-04-25T15:28:58.063064+0000","last_clean":"2024-04-25T15:28:58.063064+0000","last_became_active":"2024-04-25T15:28:56.059082+0000","last_became_peered":"2024-04-25T15:28:56.059082+0000","last_unstale":"2024-04-25T15:28:58.063064+0000","last_undegraded":"2024-04-25T15:28:58.063064+0000","last_fullsized":"2024-04-25T15:28:58.063064+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:32.909975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063667+0000","last_change":"2024-04-25T15:28:57.066571+0000","last_active":"2024-04-25T15:28:58.063667+0000","last_peered":"2024-04-25T15:28:58.063667+0000","last_clean":"2024-04-25T15:28:58.063667+0000","last_became_active":"2024-04-25T15:28:57.066255+0000","last_became_peered":"2024-04-25T15:28:57.066255+0000","last_unstale":"2024-04-25T15:28:58.063667+0000","last_undegraded":"2024-04-25T15:28:58.063667+0000","last_fullsized":"2024-04-25T15:28:58.063667+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:37.997830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062954+0000","last_change":"2024-04-25T15:28:56.055939+0000","last_active":"2024-04-25T15:28:58.062954+0000","last_peered":"2024-04-25T15:28:58.062954+0000","last_clean":"2024-04-25T15:28:58.062954+0000","last_became_active":"2024-04-25T15:28:56.055541+0000","last_became_peered":"2024-04-25T15:28:56.055541+0000","last_unstale":"2024-04-25T15:28:58.062954+0000","last_undegraded":"2024-04-25T15:28:58.062954+0000","last_fullsized":"2024-04-25T15:28:58.062954+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:21:45.358976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093322+0000","last_change":"2024-04-25T15:28:57.076997+0000","last_active":"2024-04-25T15:28:58.093322+0000","last_peered":"2024-04-25T15:28:58.093322+0000","last_clean":"2024-04-25T15:28:58.093322+0000","last_became_active":"2024-04-25T15:28:57.076625+0000","last_became_peered":"2024-04-25T15:28:57.076625+0000","last_unstale":"2024-04-25T15:28:58.093322+0000","last_undegraded":"2024-04-25T15:28:58.093322+0000","last_fullsized":"2024-04-25T15:28:58.093322+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:56.017511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062501+0000","last_change":"2024-04-25T15:28:56.056907+0000","last_active":"2024-04-25T15:28:58.062501+0000","last_peered":"2024-04-25T15:28:58.062501+0000","last_clean":"2024-04-25T15:28:58.062501+0000","last_became_active":"2024-04-25T15:28:56.056549+0000","last_became_peered":"2024-04-25T15:28:56.056549+0000","last_unstale":"2024-04-25T15:28:58.062501+0000","last_undegraded":"2024-04-25T15:28:58.062501+0000","last_fullsized":"2024-04-25T15:28:58.062501+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:12.512283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986634+0000","last_change":"2024-04-25T15:28:57.069089+0000","last_active":"2024-04-25T15:28:58.986634+0000","last_peered":"2024-04-25T15:28:58.986634+0000","last_clean":"2024-04-25T15:28:58.986634+0000","last_became_active":"2024-04-25T15:28:57.068764+0000","last_became_peered":"2024-04-25T15:28:57.068764+0000","last_unstale":"2024-04-25T15:28:58.986634+0000","last_undegraded":"2024-04-25T15:28:58.986634+0000","last_fullsized":"2024-04-25T15:28:58.986634+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:45.973985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062819+0000","last_change":"2024-04-25T15:28:56.057126+0000","last_active":"2024-04-25T15:28:58.062819+0000","last_peered":"2024-04-25T15:28:58.062819+0000","last_clean":"2024-04-25T15:28:58.062819+0000","last_became_active":"2024-04-25T15:28:56.056619+0000","last_became_peered":"2024-04-25T15:28:56.056619+0000","last_unstale":"2024-04-25T15:28:58.062819+0000","last_undegraded":"2024-04-25T15:28:58.062819+0000","last_fullsized":"2024-04-25T15:28:58.062819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:19.115555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.178799+0000","last_change":"2024-04-25T15:28:57.075923+0000","last_active":"2024-04-25T15:28:58.178799+0000","last_peered":"2024-04-25T15:28:58.178799+0000","last_clean":"2024-04-25T15:28:58.178799+0000","last_became_active":"2024-04-25T15:28:57.075504+0000","last_became_peered":"2024-04-25T15:28:57.075504+0000","last_unstale":"2024-04-25T15:28:58.178799+0000","last_undegraded":"2024-04-25T15:28:58.178799+0000","last_fullsized":"2024-04-25T15:28:58.178799+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:02:04.678437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093239+0000","last_change":"2024-04-25T15:28:56.061172+0000","last_active":"2024-04-25T15:28:58.093239+0000","last_peered":"2024-04-25T15:28:58.093239+0000","last_clean":"2024-04-25T15:28:58.093239+0000","last_became_active":"2024-04-25T15:28:56.060821+0000","last_became_peered":"2024-04-25T15:28:56.060821+0000","last_unstale":"2024-04-25T15:28:58.093239+0000","last_undegraded":"2024-04-25T15:28:58.093239+0000","last_fullsized":"2024-04-25T15:28:58.093239+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:28.779955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063189+0000","last_change":"2024-04-25T15:28:57.067194+0000","last_active":"2024-04-25T15:28:58.063189+0000","last_peered":"2024-04-25T15:28:58.063189+0000","last_clean":"2024-04-25T15:28:58.063189+0000","last_became_active":"2024-04-25T15:28:57.066803+0000","last_became_peered":"2024-04-25T15:28:57.066803+0000","last_unstale":"2024-04-25T15:28:58.063189+0000","last_undegraded":"2024-04-25T15:28:58.063189+0000","last_fullsized":"2024-04-25T15:28:58.063189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:19:55.184936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062254+0000","last_change":"2024-04-25T15:28:56.053676+0000","last_active":"2024-04-25T15:28:58.062254+0000","last_peered":"2024-04-25T15:28:58.062254+0000","last_clean":"2024-04-25T15:28:58.062254+0000","last_became_active":"2024-04-25T15:28:56.053355+0000","last_became_peered":"2024-04-25T15:28:56.053355+0000","last_unstale":"2024-04-25T15:28:58.062254+0000","last_undegraded":"2024-04-25T15:28:58.062254+0000","last_fullsized":"2024-04-25T15:28:58.062254+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.366420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062122+0000","last_change":"2024-04-25T15:28:57.079797+0000","last_active":"2024-04-25T15:28:58.062122+0000","last_peered":"2024-04-25T15:28:58.062122+0000","last_clean":"2024-04-25T15:28:58.062122+0000","last_became_active":"2024-04-25T15:28:57.078394+0000","last_became_peered":"2024-04-25T15:28:57.078394+0000","last_unstale":"2024-04-25T15:28:58.062122+0000","last_undegraded":"2024-04-25T15:28:58.062122+0000","last_fullsized":"2024-04-25T15:28:58.062122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:09:03.577565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154695+0000","last_change":"2024-04-25T15:28:56.051321+0000","last_active":"2024-04-25T15:28:59.154695+0000","last_peered":"2024-04-25T15:28:59.154695+0000","last_clean":"2024-04-25T15:28:59.154695+0000","last_became_active":"2024-04-25T15:28:56.050541+0000","last_became_peered":"2024-04-25T15:28:56.050541+0000","last_unstale":"2024-04-25T15:28:59.154695+0000","last_undegraded":"2024-04-25T15:28:59.154695+0000","last_fullsized":"2024-04-25T15:28:59.154695+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:50.204782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986118+0000","last_change":"2024-04-25T15:28:57.074356+0000","last_active":"2024-04-25T15:28:58.986118+0000","last_peered":"2024-04-25T15:28:58.986118+0000","last_clean":"2024-04-25T15:28:58.986118+0000","last_became_active":"2024-04-25T15:28:57.073404+0000","last_became_peered":"2024-04-25T15:28:57.073404+0000","last_unstale":"2024-04-25T15:28:58.986118+0000","last_undegraded":"2024-04-25T15:28:58.986118+0000","last_fullsized":"2024-04-25T15:28:58.986118+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:05.307601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093045+0000","last_change":"2024-04-25T15:28:56.061338+0000","last_active":"2024-04-25T15:28:58.093045+0000","last_peered":"2024-04-25T15:28:58.093045+0000","last_clean":"2024-04-25T15:28:58.093045+0000","last_became_active":"2024-04-25T15:28:56.061028+0000","last_became_peered":"2024-04-25T15:28:56.061028+0000","last_unstale":"2024-04-25T15:28:58.093045+0000","last_undegraded":"2024-04-25T15:28:58.093045+0000","last_fullsized":"2024-04-25T15:28:58.093045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:04.723839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063308+0000","last_change":"2024-04-25T15:28:57.077369+0000","last_active":"2024-04-25T15:28:58.063308+0000","last_peered":"2024-04-25T15:28:58.063308+0000","last_clean":"2024-04-25T15:28:58.063308+0000","last_became_active":"2024-04-25T15:28:57.076819+0000","last_became_peered":"2024-04-25T15:28:57.076819+0000","last_unstale":"2024-04-25T15:28:58.063308+0000","last_undegraded":"2024-04-25T15:28:58.063308+0000","last_fullsized":"2024-04-25T15:28:58.063308+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:26.209795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093874+0000","last_change":"2024-04-25T15:28:56.058667+0000","last_active":"2024-04-25T15:28:58.093874+0000","last_peered":"2024-04-25T15:28:58.093874+0000","last_clean":"2024-04-25T15:28:58.093874+0000","last_became_active":"2024-04-25T15:28:56.057323+0000","last_became_peered":"2024-04-25T15:28:56.057323+0000","last_unstale":"2024-04-25T15:28:58.093874+0000","last_undegraded":"2024-04-25T15:28:58.093874+0000","last_fullsized":"2024-04-25T15:28:58.093874+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:54.744798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986670+0000","last_change":"2024-04-25T15:28:57.075835+0000","last_active":"2024-04-25T15:28:58.986670+0000","last_peered":"2024-04-25T15:28:58.986670+0000","last_clean":"2024-04-25T15:28:58.986670+0000","last_became_active":"2024-04-25T15:28:57.073584+0000","last_became_peered":"2024-04-25T15:28:57.073584+0000","last_unstale":"2024-04-25T15:28:58.986670+0000","last_undegraded":"2024-04-25T15:28:58.986670+0000","last_fullsized":"2024-04-25T15:28:58.986670+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:11.137240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063147+0000","last_change":"2024-04-25T15:28:56.046616+0000","last_active":"2024-04-25T15:28:58.063147+0000","last_peered":"2024-04-25T15:28:58.063147+0000","last_clean":"2024-04-25T15:28:58.063147+0000","last_became_active":"2024-04-25T15:28:56.046311+0000","last_became_peered":"2024-04-25T15:28:56.046311+0000","last_unstale":"2024-04-25T15:28:58.063147+0000","last_undegraded":"2024-04-25T15:28:58.063147+0000","last_fullsized":"2024-04-25T15:28:58.063147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:07:19.115555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154921+0000","last_change":"2024-04-25T15:28:57.068560+0000","last_active":"2024-04-25T15:28:59.154921+0000","last_peered":"2024-04-25T15:28:59.154921+0000","last_clean":"2024-04-25T15:28:59.154921+0000","last_became_active":"2024-04-25T15:28:57.068249+0000","last_became_peered":"2024-04-25T15:28:57.068249+0000","last_unstale":"2024-04-25T15:28:59.154921+0000","last_undegraded":"2024-04-25T15:28:59.154921+0000","last_fullsized":"2024-04-25T15:28:59.154921+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:19:55.184936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063527+0000","last_change":"2024-04-25T15:28:56.053865+0000","last_active":"2024-04-25T15:28:58.063527+0000","last_peered":"2024-04-25T15:28:58.063527+0000","last_clean":"2024-04-25T15:28:58.063527+0000","last_became_active":"2024-04-25T15:28:56.053511+0000","last_became_peered":"2024-04-25T15:28:56.053511+0000","last_unstale":"2024-04-25T15:28:58.063527+0000","last_undegraded":"2024-04-25T15:28:58.063527+0000","last_fullsized":"2024-04-25T15:28:58.063527+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:04.302144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063410+0000","last_change":"2024-04-25T15:28:57.070798+0000","last_active":"2024-04-25T15:28:58.063410+0000","last_peered":"2024-04-25T15:28:58.063410+0000","last_clean":"2024-04-25T15:28:58.063410+0000","last_became_active":"2024-04-25T15:28:57.070476+0000","last_became_peered":"2024-04-25T15:28:57.070476+0000","last_unstale":"2024-04-25T15:28:58.063410+0000","last_undegraded":"2024-04-25T15:28:58.063410+0000","last_fullsized":"2024-04-25T15:28:58.063410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:56:45.476286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986632+0000","last_change":"2024-04-25T15:28:56.044068+0000","last_active":"2024-04-25T15:28:58.986632+0000","last_peered":"2024-04-25T15:28:58.986632+0000","last_clean":"2024-04-25T15:28:58.986632+0000","last_became_active":"2024-04-25T15:28:56.043670+0000","last_became_peered":"2024-04-25T15:28:56.043670+0000","last_unstale":"2024-04-25T15:28:58.986632+0000","last_undegraded":"2024-04-25T15:28:58.986632+0000","last_fullsized":"2024-04-25T15:28:58.986632+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:38:59.617279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062233+0000","last_change":"2024-04-25T15:28:57.067461+0000","last_active":"2024-04-25T15:28:58.062233+0000","last_peered":"2024-04-25T15:28:58.062233+0000","last_clean":"2024-04-25T15:28:58.062233+0000","last_became_active":"2024-04-25T15:28:57.067117+0000","last_became_peered":"2024-04-25T15:28:57.067117+0000","last_unstale":"2024-04-25T15:28:58.062233+0000","last_undegraded":"2024-04-25T15:28:58.062233+0000","last_fullsized":"2024-04-25T15:28:58.062233+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.607419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.067230+0000","last_change":"2024-04-25T15:28:56.051968+0000","last_active":"2024-04-25T15:28:59.067230+0000","last_peered":"2024-04-25T15:28:59.067230+0000","last_clean":"2024-04-25T15:28:59.067230+0000","last_became_active":"2024-04-25T15:28:56.051236+0000","last_became_peered":"2024-04-25T15:28:56.051236+0000","last_unstale":"2024-04-25T15:28:59.067230+0000","last_undegraded":"2024-04-25T15:28:59.067230+0000","last_fullsized":"2024-04-25T15:28:59.067230+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:04:47.179433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093494+0000","last_change":"2024-04-25T15:28:57.074968+0000","last_active":"2024-04-25T15:28:58.093494+0000","last_peered":"2024-04-25T15:28:58.093494+0000","last_clean":"2024-04-25T15:28:58.093494+0000","last_became_active":"2024-04-25T15:28:57.074660+0000","last_became_peered":"2024-04-25T15:28:57.074660+0000","last_unstale":"2024-04-25T15:28:58.093494+0000","last_undegraded":"2024-04-25T15:28:58.093494+0000","last_fullsized":"2024-04-25T15:28:58.093494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:46.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.067114+0000","last_change":"2024-04-25T15:28:56.053779+0000","last_active":"2024-04-25T15:28:59.067114+0000","last_peered":"2024-04-25T15:28:59.067114+0000","last_clean":"2024-04-25T15:28:59.067114+0000","last_became_active":"2024-04-25T15:28:56.053471+0000","last_became_peered":"2024-04-25T15:28:56.053471+0000","last_unstale":"2024-04-25T15:28:59.067114+0000","last_undegraded":"2024-04-25T15:28:59.067114+0000","last_fullsized":"2024-04-25T15:28:59.067114+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:33.669214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986076+0000","last_change":"2024-04-25T15:28:57.069931+0000","last_active":"2024-04-25T15:28:58.986076+0000","last_peered":"2024-04-25T15:28:58.986076+0000","last_clean":"2024-04-25T15:28:58.986076+0000","last_became_active":"2024-04-25T15:28:57.069591+0000","last_became_peered":"2024-04-25T15:28:57.069591+0000","last_unstale":"2024-04-25T15:28:58.986076+0000","last_undegraded":"2024-04-25T15:28:58.986076+0000","last_fullsized":"2024-04-25T15:28:58.986076+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:21:46.364432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986137+0000","last_change":"2024-04-25T15:28:56.052681+0000","last_active":"2024-04-25T15:28:58.986137+0000","last_peered":"2024-04-25T15:28:58.986137+0000","last_clean":"2024-04-25T15:28:58.986137+0000","last_became_active":"2024-04-25T15:28:56.052385+0000","last_became_peered":"2024-04-25T15:28:56.052385+0000","last_unstale":"2024-04-25T15:28:58.986137+0000","last_undegraded":"2024-04-25T15:28:58.986137+0000","last_fullsized":"2024-04-25T15:28:58.986137+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:38.449071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066634+0000","last_change":"2024-04-25T15:28:57.074803+0000","last_active":"2024-04-25T15:28:59.066634+0000","last_peered":"2024-04-25T15:28:59.066634+0000","last_clean":"2024-04-25T15:28:59.066634+0000","last_became_active":"2024-04-25T15:28:57.074504+0000","last_became_peered":"2024-04-25T15:28:57.074504+0000","last_unstale":"2024-04-25T15:28:59.066634+0000","last_undegraded":"2024-04-25T15:28:59.066634+0000","last_fullsized":"2024-04-25T15:28:59.066634+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:26.913195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":22,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093172+0000","last_change":"2024-04-25T15:28:56.060187+0000","last_active":"2024-04-25T15:28:58.093172+0000","last_peered":"2024-04-25T15:28:58.093172+0000","last_clean":"2024-04-25T15:28:58.093172+0000","last_became_active":"2024-04-25T15:28:56.059867+0000","last_became_peered":"2024-04-25T15:28:56.059867+0000","last_unstale":"2024-04-25T15:28:58.093172+0000","last_undegraded":"2024-04-25T15:28:58.093172+0000","last_fullsized":"2024-04-25T15:28:58.093172+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:02:03.761984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154976+0000","last_change":"2024-04-25T15:28:57.075122+0000","last_active":"2024-04-25T15:28:59.154976+0000","last_peered":"2024-04-25T15:28:59.154976+0000","last_clean":"2024-04-25T15:28:59.154976+0000","last_became_active":"2024-04-25T15:28:57.074784+0000","last_became_peered":"2024-04-25T15:28:57.074784+0000","last_unstale":"2024-04-25T15:28:59.154976+0000","last_undegraded":"2024-04-25T15:28:59.154976+0000","last_fullsized":"2024-04-25T15:28:59.154976+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:03.566264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986408+0000","last_change":"2024-04-25T15:28:56.052624+0000","last_active":"2024-04-25T15:28:58.986408+0000","last_peered":"2024-04-25T15:28:58.986408+0000","last_clean":"2024-04-25T15:28:58.986408+0000","last_became_active":"2024-04-25T15:28:56.052283+0000","last_became_peered":"2024-04-25T15:28:56.052283+0000","last_unstale":"2024-04-25T15:28:58.986408+0000","last_undegraded":"2024-04-25T15:28:58.986408+0000","last_fullsized":"2024-04-25T15:28:58.986408+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:47:21.737399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.178815+0000","last_change":"2024-04-25T15:28:57.068347+0000","last_active":"2024-04-25T15:28:58.178815+0000","last_peered":"2024-04-25T15:28:58.178815+0000","last_clean":"2024-04-25T15:28:58.178815+0000","last_became_active":"2024-04-25T15:28:57.067837+0000","last_became_peered":"2024-04-25T15:28:57.067837+0000","last_unstale":"2024-04-25T15:28:58.178815+0000","last_undegraded":"2024-04-25T15:28:58.178815+0000","last_fullsized":"2024-04-25T15:28:58.178815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:30.584212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154622+0000","last_change":"2024-04-25T15:28:56.050894+0000","last_active":"2024-04-25T15:28:59.154622+0000","last_peered":"2024-04-25T15:28:59.154622+0000","last_clean":"2024-04-25T15:28:59.154622+0000","last_became_active":"2024-04-25T15:28:56.050570+0000","last_became_peered":"2024-04-25T15:28:56.050570+0000","last_unstale":"2024-04-25T15:28:59.154622+0000","last_undegraded":"2024-04-25T15:28:59.154622+0000","last_fullsized":"2024-04-25T15:28:59.154622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:00:41.311781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063485+0000","last_change":"2024-04-25T15:28:57.076076+0000","last_active":"2024-04-25T15:28:58.063485+0000","last_peered":"2024-04-25T15:28:58.063485+0000","last_clean":"2024-04-25T15:28:58.063485+0000","last_became_active":"2024-04-25T15:28:57.075448+0000","last_became_peered":"2024-04-25T15:28:57.075448+0000","last_unstale":"2024-04-25T15:28:58.063485+0000","last_undegraded":"2024-04-25T15:28:58.063485+0000","last_fullsized":"2024-04-25T15:28:58.063485+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:46.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062725+0000","last_change":"2024-04-25T15:28:56.045642+0000","last_active":"2024-04-25T15:28:58.062725+0000","last_peered":"2024-04-25T15:28:58.062725+0000","last_clean":"2024-04-25T15:28:58.062725+0000","last_became_active":"2024-04-25T15:28:56.045273+0000","last_became_peered":"2024-04-25T15:28:56.045273+0000","last_unstale":"2024-04-25T15:28:58.062725+0000","last_undegraded":"2024-04-25T15:28:58.062725+0000","last_fullsized":"2024-04-25T15:28:58.062725+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:04.023586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066974+0000","last_change":"2024-04-25T15:28:57.076140+0000","last_active":"2024-04-25T15:28:59.066974+0000","last_peered":"2024-04-25T15:28:59.066974+0000","last_clean":"2024-04-25T15:28:59.066974+0000","last_became_active":"2024-04-25T15:28:57.075778+0000","last_became_peered":"2024-04-25T15:28:57.075778+0000","last_unstale":"2024-04-25T15:28:59.066974+0000","last_undegraded":"2024-04-25T15:28:59.066974+0000","last_fullsized":"2024-04-25T15:28:59.066974+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:07.761838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063006+0000","last_change":"2024-04-25T15:28:56.059690+0000","last_active":"2024-04-25T15:28:58.063006+0000","last_peered":"2024-04-25T15:28:58.063006+0000","last_clean":"2024-04-25T15:28:58.063006+0000","last_became_active":"2024-04-25T15:28:56.059372+0000","last_became_peered":"2024-04-25T15:28:56.059372+0000","last_unstale":"2024-04-25T15:28:58.063006+0000","last_undegraded":"2024-04-25T15:28:58.063006+0000","last_fullsized":"2024-04-25T15:28:58.063006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:44.968528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.067001+0000","last_change":"2024-04-25T15:28:57.068250+0000","last_active":"2024-04-25T15:28:59.067001+0000","last_peered":"2024-04-25T15:28:59.067001+0000","last_clean":"2024-04-25T15:28:59.067001+0000","last_became_active":"2024-04-25T15:28:57.067797+0000","last_became_peered":"2024-04-25T15:28:57.067797+0000","last_unstale":"2024-04-25T15:28:59.067001+0000","last_undegraded":"2024-04-25T15:28:59.067001+0000","last_fullsized":"2024-04-25T15:28:59.067001+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:59.409270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.178841+0000","last_change":"2024-04-25T15:28:56.053555+0000","last_active":"2024-04-25T15:28:58.178841+0000","last_peered":"2024-04-25T15:28:58.178841+0000","last_clean":"2024-04-25T15:28:58.178841+0000","last_became_active":"2024-04-25T15:28:56.053235+0000","last_became_peered":"2024-04-25T15:28:56.053235+0000","last_unstale":"2024-04-25T15:28:58.178841+0000","last_undegraded":"2024-04-25T15:28:58.178841+0000","last_fullsized":"2024-04-25T15:28:58.178841+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:38.449071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066631+0000","last_change":"2024-04-25T15:28:57.072008+0000","last_active":"2024-04-25T15:28:59.066631+0000","last_peered":"2024-04-25T15:28:59.066631+0000","last_clean":"2024-04-25T15:28:59.066631+0000","last_became_active":"2024-04-25T15:28:57.071648+0000","last_became_peered":"2024-04-25T15:28:57.071648+0000","last_unstale":"2024-04-25T15:28:59.066631+0000","last_undegraded":"2024-04-25T15:28:59.066631+0000","last_fullsized":"2024-04-25T15:28:59.066631+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:54.925066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.178642+0000","last_change":"2024-04-25T15:28:56.049930+0000","last_active":"2024-04-25T15:28:58.178642+0000","last_peered":"2024-04-25T15:28:58.178642+0000","last_clean":"2024-04-25T15:28:58.178642+0000","last_became_active":"2024-04-25T15:28:56.049617+0000","last_became_peered":"2024-04-25T15:28:56.049617+0000","last_unstale":"2024-04-25T15:28:58.178642+0000","last_undegraded":"2024-04-25T15:28:58.178642+0000","last_fullsized":"2024-04-25T15:28:58.178642+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:45.314705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154439+0000","last_change":"2024-04-25T15:28:57.072416+0000","last_active":"2024-04-25T15:28:59.154439+0000","last_peered":"2024-04-25T15:28:59.154439+0000","last_clean":"2024-04-25T15:28:59.154439+0000","last_became_active":"2024-04-25T15:28:57.072063+0000","last_became_peered":"2024-04-25T15:28:57.072063+0000","last_unstale":"2024-04-25T15:28:59.154439+0000","last_undegraded":"2024-04-25T15:28:59.154439+0000","last_fullsized":"2024-04-25T15:28:59.154439+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.915432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062611+0000","last_change":"2024-04-25T15:28:56.054470+0000","last_active":"2024-04-25T15:28:58.062611+0000","last_peered":"2024-04-25T15:28:58.062611+0000","last_clean":"2024-04-25T15:28:58.062611+0000","last_became_active":"2024-04-25T15:28:56.054148+0000","last_became_peered":"2024-04-25T15:28:56.054148+0000","last_unstale":"2024-04-25T15:28:58.062611+0000","last_undegraded":"2024-04-25T15:28:58.062611+0000","last_fullsized":"2024-04-25T15:28:58.062611+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:34:43.418078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063346+0000","last_change":"2024-04-25T15:28:57.069551+0000","last_active":"2024-04-25T15:28:58.063346+0000","last_peered":"2024-04-25T15:28:58.063346+0000","last_clean":"2024-04-25T15:28:58.063346+0000","last_became_active":"2024-04-25T15:28:57.069220+0000","last_became_peered":"2024-04-25T15:28:57.069220+0000","last_unstale":"2024-04-25T15:28:58.063346+0000","last_undegraded":"2024-04-25T15:28:58.063346+0000","last_fullsized":"2024-04-25T15:28:58.063346+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:43.494932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.179003+0000","last_change":"2024-04-25T15:28:56.051763+0000","last_active":"2024-04-25T15:28:58.179003+0000","last_peered":"2024-04-25T15:28:58.179003+0000","last_clean":"2024-04-25T15:28:58.179003+0000","last_became_active":"2024-04-25T15:28:56.051431+0000","last_became_peered":"2024-04-25T15:28:56.051431+0000","last_unstale":"2024-04-25T15:28:58.179003+0000","last_undegraded":"2024-04-25T15:28:58.179003+0000","last_fullsized":"2024-04-25T15:28:58.179003+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:37.819844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093755+0000","last_change":"2024-04-25T15:28:57.074966+0000","last_active":"2024-04-25T15:28:58.093755+0000","last_peered":"2024-04-25T15:28:58.093755+0000","last_clean":"2024-04-25T15:28:58.093755+0000","last_became_active":"2024-04-25T15:28:57.074588+0000","last_became_peered":"2024-04-25T15:28:57.074588+0000","last_unstale":"2024-04-25T15:28:58.093755+0000","last_undegraded":"2024-04-25T15:28:58.093755+0000","last_fullsized":"2024-04-25T15:28:58.093755+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:39.454528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063503+0000","last_change":"2024-04-25T15:28:56.059005+0000","last_active":"2024-04-25T15:28:58.063503+0000","last_peered":"2024-04-25T15:28:58.063503+0000","last_clean":"2024-04-25T15:28:58.063503+0000","last_became_active":"2024-04-25T15:28:56.057981+0000","last_became_peered":"2024-04-25T15:28:56.057981+0000","last_unstale":"2024-04-25T15:28:58.063503+0000","last_undegraded":"2024-04-25T15:28:58.063503+0000","last_fullsized":"2024-04-25T15:28:58.063503+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:08.396077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.093437+0000","last_change":"2024-04-25T15:28:57.070267+0000","last_active":"2024-04-25T15:28:58.093437+0000","last_peered":"2024-04-25T15:28:58.093437+0000","last_clean":"2024-04-25T15:28:58.093437+0000","last_became_active":"2024-04-25T15:28:57.069955+0000","last_became_peered":"2024-04-25T15:28:57.069955+0000","last_unstale":"2024-04-25T15:28:58.093437+0000","last_undegraded":"2024-04-25T15:28:58.093437+0000","last_fullsized":"2024-04-25T15:28:58.093437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:57:38.825301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062507+0000","last_change":"2024-04-25T15:28:56.048576+0000","last_active":"2024-04-25T15:28:58.062507+0000","last_peered":"2024-04-25T15:28:58.062507+0000","last_clean":"2024-04-25T15:28:58.062507+0000","last_became_active":"2024-04-25T15:28:56.048273+0000","last_became_peered":"2024-04-25T15:28:56.048273+0000","last_unstale":"2024-04-25T15:28:58.062507+0000","last_undegraded":"2024-04-25T15:28:58.062507+0000","last_fullsized":"2024-04-25T15:28:58.062507+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:04.302144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154769+0000","last_change":"2024-04-25T15:28:57.069733+0000","last_active":"2024-04-25T15:28:59.154769+0000","last_peered":"2024-04-25T15:28:59.154769+0000","last_clean":"2024-04-25T15:28:59.154769+0000","last_became_active":"2024-04-25T15:28:57.069426+0000","last_became_peered":"2024-04-25T15:28:57.069426+0000","last_unstale":"2024-04-25T15:28:59.154769+0000","last_undegraded":"2024-04-25T15:28:59.154769+0000","last_fullsized":"2024-04-25T15:28:59.154769+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:36.737991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154935+0000","last_change":"2024-04-25T15:28:56.051473+0000","last_active":"2024-04-25T15:28:59.154935+0000","last_peered":"2024-04-25T15:28:59.154935+0000","last_clean":"2024-04-25T15:28:59.154935+0000","last_became_active":"2024-04-25T15:28:56.051171+0000","last_became_peered":"2024-04-25T15:28:56.051171+0000","last_unstale":"2024-04-25T15:28:59.154935+0000","last_undegraded":"2024-04-25T15:28:59.154935+0000","last_fullsized":"2024-04-25T15:28:59.154935+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:54.601681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.067003+0000","last_change":"2024-04-25T15:28:57.069112+0000","last_active":"2024-04-25T15:28:59.067003+0000","last_peered":"2024-04-25T15:28:59.067003+0000","last_clean":"2024-04-25T15:28:59.067003+0000","last_became_active":"2024-04-25T15:28:57.068759+0000","last_became_peered":"2024-04-25T15:28:57.068759+0000","last_unstale":"2024-04-25T15:28:59.067003+0000","last_undegraded":"2024-04-25T15:28:59.067003+0000","last_fullsized":"2024-04-25T15:28:59.067003+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.076007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063060+0000","last_change":"2024-04-25T15:28:56.058257+0000","last_active":"2024-04-25T15:28:58.063060+0000","last_peered":"2024-04-25T15:28:58.063060+0000","last_clean":"2024-04-25T15:28:58.063060+0000","last_became_active":"2024-04-25T15:28:56.057736+0000","last_became_peered":"2024-04-25T15:28:56.057736+0000","last_unstale":"2024-04-25T15:28:58.063060+0000","last_undegraded":"2024-04-25T15:28:58.063060+0000","last_fullsized":"2024-04-25T15:28:58.063060+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:10.131783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.986276+0000","last_change":"2024-04-25T15:28:57.073858+0000","last_active":"2024-04-25T15:28:58.986276+0000","last_peered":"2024-04-25T15:28:58.986276+0000","last_clean":"2024-04-25T15:28:58.986276+0000","last_became_active":"2024-04-25T15:28:57.073183+0000","last_became_peered":"2024-04-25T15:28:57.073183+0000","last_unstale":"2024-04-25T15:28:58.986276+0000","last_undegraded":"2024-04-25T15:28:58.986276+0000","last_fullsized":"2024-04-25T15:28:58.986276+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:13.517740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154430+0000","last_change":"2024-04-25T15:28:56.056141+0000","last_active":"2024-04-25T15:28:59.154430+0000","last_peered":"2024-04-25T15:28:59.154430+0000","last_clean":"2024-04-25T15:28:59.154430+0000","last_became_active":"2024-04-25T15:28:56.055725+0000","last_became_peered":"2024-04-25T15:28:56.055725+0000","last_unstale":"2024-04-25T15:28:59.154430+0000","last_undegraded":"2024-04-25T15:28:59.154430+0000","last_fullsized":"2024-04-25T15:28:59.154430+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.393216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066640+0000","last_change":"2024-04-25T15:28:57.074252+0000","last_active":"2024-04-25T15:28:59.066640+0000","last_peered":"2024-04-25T15:28:59.066640+0000","last_clean":"2024-04-25T15:28:59.066640+0000","last_became_active":"2024-04-25T15:28:57.073934+0000","last_became_peered":"2024-04-25T15:28:57.073934+0000","last_unstale":"2024-04-25T15:28:59.066640+0000","last_undegraded":"2024-04-25T15:28:59.066640+0000","last_fullsized":"2024-04-25T15:28:59.066640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:43:37.964393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.062612+0000","last_change":"2024-04-25T15:28:56.055311+0000","last_active":"2024-04-25T15:28:58.062612+0000","last_peered":"2024-04-25T15:28:58.062612+0000","last_clean":"2024-04-25T15:28:58.062612+0000","last_became_active":"2024-04-25T15:28:56.054958+0000","last_became_peered":"2024-04-25T15:28:56.054958+0000","last_unstale":"2024-04-25T15:28:58.062612+0000","last_undegraded":"2024-04-25T15:28:58.062612+0000","last_fullsized":"2024-04-25T15:28:58.062612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:12.512283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.154546+0000","last_change":"2024-04-25T15:28:57.069870+0000","last_active":"2024-04-25T15:28:59.154546+0000","last_peered":"2024-04-25T15:28:59.154546+0000","last_clean":"2024-04-25T15:28:59.154546+0000","last_became_active":"2024-04-25T15:28:57.069553+0000","last_became_peered":"2024-04-25T15:28:57.069553+0000","last_unstale":"2024-04-25T15:28:59.154546+0000","last_undegraded":"2024-04-25T15:28:59.154546+0000","last_fullsized":"2024-04-25T15:28:59.154546+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:47.371877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.066804+0000","last_change":"2024-04-25T15:28:56.050355+0000","last_active":"2024-04-25T15:28:59.066804+0000","last_peered":"2024-04-25T15:28:59.066804+0000","last_clean":"2024-04-25T15:28:59.066804+0000","last_became_active":"2024-04-25T15:28:56.050028+0000","last_became_peered":"2024-04-25T15:28:56.050028+0000","last_unstale":"2024-04-25T15:28:59.066804+0000","last_undegraded":"2024-04-25T15:28:59.066804+0000","last_fullsized":"2024-04-25T15:28:59.066804+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:19:10.508578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:59.067358+0000","last_change":"2024-04-25T15:28:57.077459+0000","last_active":"2024-04-25T15:28:59.067358+0000","last_peered":"2024-04-25T15:28:59.067358+0000","last_clean":"2024-04-25T15:28:59.067358+0000","last_became_active":"2024-04-25T15:28:57.073147+0000","last_became_peered":"2024-04-25T15:28:57.073147+0000","last_unstale":"2024-04-25T15:28:59.067358+0000","last_undegraded":"2024-04-25T15:28:59.067358+0000","last_fullsized":"2024-04-25T15:28:59.067358+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:56.028812+0000","last_clean_scrub_stamp":"2024-04-25T15:28:56.028812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:29.785412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063014+0000","last_change":"2024-04-25T15:28:56.054041+0000","last_active":"2024-04-25T15:28:58.063014+0000","last_peered":"2024-04-25T15:28:58.063014+0000","last_clean":"2024-04-25T15:28:58.063014+0000","last_became_active":"2024-04-25T15:28:56.053693+0000","last_became_peered":"2024-04-25T15:28:56.053693+0000","last_unstale":"2024-04-25T15:28:58.063014+0000","last_undegraded":"2024-04-25T15:28:58.063014+0000","last_fullsized":"2024-04-25T15:28:58.063014+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:28:55.023356+0000","last_clean_scrub_stamp":"2024-04-25T15:28:55.023356+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:03.773654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":239,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T15:28:58.063264+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:28:58.063264+0000","last_peered":"2024-04-25T15:28:58.063264+0000","last_clean":"2024-04-25T15:28:58.063264+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:28:58.063264+0000","last_undegraded":"2024-04-25T15:28:58.063264+0000","last_fullsized":"2024-04-25T15:28:58.063264+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32888,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338952,"statfs":{"total":96636764160,"available":96603086848,"internally_reserved":0,"allocated":1740800,"data_stored":1075868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1146880,"data_stored":486040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1142784,"data_stored":485458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1159168,"data_stored":487124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574899,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":487022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1142784,"data_stored":485422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28136,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343704,"statfs":{"total":96636764160,"available":96607952896,"internally_reserved":0,"allocated":1138688,"data_stored":485388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1736704,"data_stored":1076338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:05.976 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:29:05.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T15:29:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.131+0000 7f8b8fae0640 1 Processor -- start 2024-04-25T15:29:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.131+0000 7f8b8fae0640 1 -- start start 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8fae0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b8ab0 0x7f8b880cee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8fae0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b880cf350 0x7f8b880cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8fae0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b880cfca0 con 0x7f8b880b6900 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8fae0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8b8808d770 con 0x7f8b880cf350 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8fae0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8b880ce980 con 0x7f8b880b8ab0 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8e07d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b880cf350 0x7f8b880cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8d87c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8b880b6900 0x7f8b880b4e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40820/0 (socket says 172.21.15.67:40820) 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8d87c640 1 -- 172.21.15.67:0/1201781757 learned_addr learned my addr 172.21.15.67:0/1201781757 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8d07b640 1 --2- 172.21.15.67:0/1201781757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b8ab0 0x7f8b880cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8e07d640 1 -- 172.21.15.67:0/1201781757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b8ab0 msgr2=0x7f8b880cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:06.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8e07d640 1 --2- 172.21.15.67:0/1201781757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b8ab0 0x7f8b880cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8e07d640 1 -- 172.21.15.67:0/1201781757 >> v1:172.21.15.67:6789/0 conn(0x7f8b880b6900 legacy=0x7f8b880b4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8c87a640 1 -- 172.21.15.67:0/1201781757 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3376794755 0 0) 0x7f8b880cfca0 con 0x7f8b880b6900 2024-04-25T15:29:06.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.133+0000 7f8b8e07d640 1 -- 172.21.15.67:0/1201781757 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b880ce670 con 0x7f8b880cf350 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.134+0000 7f8b8e07d640 1 --2- 172.21.15.67:0/1201781757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cf760 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f8b800097a0 tx=0x7f8b8002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01e4bea499d6112 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.134+0000 7f8b8c87a640 1 -- 172.21.15.67:0/1201781757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80037070 con 0x7f8b880cf350 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.134+0000 7f8b8c87a640 1 -- 172.21.15.67:0/1201781757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b80032440 con 0x7f8b880cf350 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.134+0000 7f8b8c87a640 1 -- 172.21.15.67:0/1201781757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80032810 con 0x7f8b880cf350 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1201781757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 msgr2=0x7f8b880cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1201781757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cf760 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f8b800097a0 tx=0x7f8b8002a180 comp rx=0 tx=0).stop 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1201781757 shutdown_connections 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1201781757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cf760 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1201781757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b8ab0 0x7f8b880cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1201781757 >> 172.21.15.67:0/1201781757 conn(0x7f8b880d0530 msgr2=0x7f8b880d0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:06.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1201781757 shutdown_connections 2024-04-25T15:29:06.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.135+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1201781757 wait complete. 2024-04-25T15:29:06.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.136+0000 7f8b8fae0640 1 Processor -- start 2024-04-25T15:29:06.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.137+0000 7f8b8fae0640 1 -- start start 2024-04-25T15:29:06.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8fae0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b6900 0x7f8b8815b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8fae0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b880b8ab0 0x7f8b8809f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8fae0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8fae0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8808df40 con 0x7f8b880cf350 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8fae0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8808dc60 con 0x7f8b880b8ab0 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8fae0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8b88161df0 con 0x7f8b880b6900 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57866/0 (socket says 172.21.15.67:57866) 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 -- 172.21.15.67:0/1834654823 learned_addr learned my addr 172.21.15.67:0/1834654823 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8d87c640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b6900 0x7f8b8815b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8d07b640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b880b8ab0 0x7f8b8809f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 -- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b6900 msgr2=0x7f8b8815b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b6900 0x7f8b8815b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 -- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b880b8ab0 msgr2=0x7f8b8809f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b880b8ab0 0x7f8b8809f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 -- 172.21.15.67:0/1834654823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b880a05d0 con 0x7f8b880cf350 2024-04-25T15:29:06.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.138+0000 7f8b8e07d640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cadd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f8b80035040 tx=0x7f8b800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:06.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.139+0000 7f8b867fc640 1 -- 172.21.15.67:0/1834654823 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80037050 con 0x7f8b880cf350 2024-04-25T15:29:06.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.139+0000 7f8b867fc640 1 -- 172.21.15.67:0/1834654823 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b80038040 con 0x7f8b880cf350 2024-04-25T15:29:06.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.139+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b880a08b0 con 0x7f8b880cf350 2024-04-25T15:29:06.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.139+0000 7f8b867fc640 1 -- 172.21.15.67:0/1834654823 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80006e10 con 0x7f8b880cf350 2024-04-25T15:29:06.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.139+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b880a0e70 con 0x7f8b880cf350 2024-04-25T15:29:06.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.140+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b880a0340 con 0x7f8b880cf350 2024-04-25T15:29:06.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.140+0000 7f8b867fc640 1 -- 172.21.15.67:0/1834654823 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8b80030980 con 0x7f8b880cf350 2024-04-25T15:29:06.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.141+0000 7f8b867fc640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b740744e0 0x7f8b740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.141+0000 7f8b867fc640 1 -- 172.21.15.67:0/1834654823 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f8b800b5b00 con 0x7f8b880cf350 2024-04-25T15:29:06.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.141+0000 7f8b8d87c640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b740744e0 0x7f8b740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.142+0000 7f8b8d87c640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b740744e0 0x7f8b740769f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8b78007ed0 tx=0x7f8b78007e60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.146+0000 7f8b867fc640 1 -- 172.21.15.67:0/1834654823 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b8004c020 con 0x7f8b880cf350 2024-04-25T15:29:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.404+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f8b88050740 con 0x7f8b880cf350 2024-04-25T15:29:06.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.407+0000 7f8b867fc640 1 -- 172.21.15.67:0/1834654823 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7f8b880ce670 con 0x7f8b880cf350 2024-04-25T15:29:06.408 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:29:06.408 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvdSpmWNndExAAxD6TZjIefGBeka1PUQOC7Q== 2024-04-25T15:29:06.408 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-25T15:29:06.408 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-25T15:29:06.408 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T15:29:06.408 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-25T15:29:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b740744e0 msgr2=0x7f8b740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b740744e0 0x7f8b740769f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8b78007ed0 tx=0x7f8b78007e60 comp rx=0 tx=0).stop 2024-04-25T15:29:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 msgr2=0x7f8b880cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cadd0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f8b80035040 tx=0x7f8b800046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 shutdown_connections 2024-04-25T15:29:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b740744e0 0x7f8b740769f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b880cf350 0x7f8b880cadd0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b880b8ab0 0x7f8b8809f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 --2- 172.21.15.67:0/1834654823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b880b6900 0x7f8b8815b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 >> 172.21.15.67:0/1834654823 conn(0x7f8b880d0530 msgr2=0x7f8b8815de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 shutdown_connections 2024-04-25T15:29:06.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.409+0000 7f8b8fae0640 1 -- 172.21.15.67:0/1834654823 wait complete. 2024-04-25T15:29:06.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:29:06.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.532+0000 7fefb3842640 1 Processor -- start 2024-04-25T15:29:06.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.533+0000 7fefb3842640 1 -- start start 2024-04-25T15:29:06.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefb3842640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fefac0d1210 0x7fefac0d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefb3842640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefac0d1850 con 0x7fefac06ccd0 2024-04-25T15:29:06.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefb3842640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefac160de0 con 0x7fefac0d0e00 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefb3842640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fefac0d1620 con 0x7fefac0d1210 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefb0ddd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fefac06ccd0 0x7fefac0d4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40824/0 (socket says 172.21.15.67:40824) 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefb0ddd640 1 -- 172.21.15.67:0/4281095286 learned_addr learned my addr 172.21.15.67:0/4281095286 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefb1ddf640 1 --2- 172.21.15.67:0/4281095286 >> v2:172.21.15.176:3300/0 conn(0x7fefac0d1210 0x7fefac0d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefa3fff640 1 -- 172.21.15.67:0/4281095286 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553769910 0 0) 0x7fefac0d1850 con 0x7fefac06ccd0 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.534+0000 7fefa3fff640 1 -- 172.21.15.67:0/4281095286 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef98003520 con 0x7fefac06ccd0 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.535+0000 7fefa3fff640 1 -- 172.21.15.67:0/4281095286 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1578729079 0 0) 0x7fef98003520 con 0x7fefac06ccd0 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.535+0000 7fefb1ddf640 1 -- 172.21.15.67:0/4281095286 >> v1:172.21.15.176:6789/0 conn(0x7fefac0d0e00 legacy=0x7fefac06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.535+0000 7fefb1ddf640 1 -- 172.21.15.67:0/4281095286 >> v1:172.21.15.67:6789/0 conn(0x7fefac06ccd0 legacy=0x7fefac0d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.535+0000 7fefb1ddf640 1 -- 172.21.15.67:0/4281095286 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac1610c0 con 0x7fefac0d1210 2024-04-25T15:29:06.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.536+0000 7fefb1ddf640 1 --2- 172.21.15.67:0/4281095286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac0d1210 0x7fefac0d5160 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fefa80097a0 tx=0x7fefa802a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d71a21a08e037f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:06.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.536+0000 7fefa3fff640 1 -- 172.21.15.67:0/4281095286 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8037070 con 0x7fefac0d1210 2024-04-25T15:29:06.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.536+0000 7fefa3fff640 1 -- 172.21.15.67:0/4281095286 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa802abc0 con 0x7fefac0d1210 2024-04-25T15:29:06.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.536+0000 7fefa3fff640 1 -- 172.21.15.67:0/4281095286 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c001670 con 0x7fefac0d1210 2024-04-25T15:29:06.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.536+0000 7fefb3842640 1 -- 172.21.15.67:0/4281095286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac0d1210 msgr2=0x7fefac0d5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.536+0000 7fefb3842640 1 --2- 172.21.15.67:0/4281095286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac0d1210 0x7fefac0d5160 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fefa80097a0 tx=0x7fefa802a130 comp rx=0 tx=0).stop 2024-04-25T15:29:06.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.537+0000 7fefb3842640 1 -- 172.21.15.67:0/4281095286 shutdown_connections 2024-04-25T15:29:06.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.537+0000 7fefb3842640 1 --2- 172.21.15.67:0/4281095286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac0d1210 0x7fefac0d5160 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.537+0000 7fefb3842640 1 -- 172.21.15.67:0/4281095286 >> 172.21.15.67:0/4281095286 conn(0x7fefac0d5d90 msgr2=0x7fefac0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:06.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.537+0000 7fefb3842640 1 -- 172.21.15.67:0/4281095286 shutdown_connections 2024-04-25T15:29:06.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.537+0000 7fefb3842640 1 -- 172.21.15.67:0/4281095286 wait complete. 2024-04-25T15:29:06.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.538+0000 7fefb3842640 1 Processor -- start 2024-04-25T15:29:06.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.538+0000 7fefb3842640 1 -- start start 2024-04-25T15:29:06.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb3842640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefac0d0e00 0x7fefac1588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb3842640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefac0d1210 0x7fefac0cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb3842640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac1fab60 0x7fefac156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb3842640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fefac0b5320 con 0x7fefac0d0e00 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb3842640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fefac160ad0 con 0x7fefac1fab60 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb3842640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fefac164e00 con 0x7fefac0d1210 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefac0d0e00 0x7fefac1588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefac0d0e00 0x7fefac1588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57878/0 (socket says 172.21.15.67:57878) 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 -- 172.21.15.67:0/1939707375 learned_addr learned my addr 172.21.15.67:0/1939707375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb0ddd640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefac0d1210 0x7fefac0cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb1ddf640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac1fab60 0x7fefac156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 -- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefac0d1210 msgr2=0x7fefac0cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefac0d1210 0x7fefac0cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 -- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac1fab60 msgr2=0x7fefac156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac1fab60 0x7fefac156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 -- 172.21.15.67:0/1939707375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac0ce0c0 con 0x7fefac0d0e00 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb1ddf640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac1fab60 0x7fefac156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.540+0000 7fefb15de640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefac0d0e00 0x7fefac1588e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c730 tx=0x7fefa400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.541+0000 7fefa1ffb640 1 -- 172.21.15.67:0/1939707375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa4003b70 con 0x7fefac0d0e00 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.541+0000 7fefa1ffb640 1 -- 172.21.15.67:0/1939707375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa4003d30 con 0x7fefac0d0e00 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.541+0000 7fefa1ffb640 1 -- 172.21.15.67:0/1939707375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa400aa90 con 0x7fefac0d0e00 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.541+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefac0c9a40 con 0x7fefac0d0e00 2024-04-25T15:29:06.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.541+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefac1f9f50 con 0x7fefac0d0e00 2024-04-25T15:29:06.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.542+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefac0c9830 con 0x7fefac0d0e00 2024-04-25T15:29:06.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.542+0000 7fefa1ffb640 1 -- 172.21.15.67:0/1939707375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fefa400ac30 con 0x7fefac0d0e00 2024-04-25T15:29:06.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.543+0000 7fefa1ffb640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef900744b0 0x7fef900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:06.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.543+0000 7fefa1ffb640 1 -- 172.21.15.67:0/1939707375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fefa40950a0 con 0x7fefac0d0e00 2024-04-25T15:29:06.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.543+0000 7fefb0ddd640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef900744b0 0x7fef900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:06.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.544+0000 7fefb0ddd640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef900744b0 0x7fef900769c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fefac1f9eb0 tx=0x7fef9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:06.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.548+0000 7fefa1ffb640 1 -- 172.21.15.67:0/1939707375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fefa405f8c0 con 0x7fefac0d0e00 2024-04-25T15:29:06.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.777+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefac050740 con 0x7fefac0d0e00 2024-04-25T15:29:06.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.778+0000 7fefa1ffb640 1 -- 172.21.15.67:0/1939707375 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5189 (secure 0 0 0) 0x7fefa405f1c0 con 0x7fefac0d0e00 2024-04-25T15:29:06.779 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:06.779 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2709900765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2709900765},{"type":"v1","addr":"172.21.15.67:6834","nonce":2709900765}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4104906051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4104906051},{"type":"v1","addr":"172.21.15.176:6836","nonce":4104906051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6655,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2981673460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2981673460},{"type":"v1","addr":"172.21.15.176:6837","nonce":2981673460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:56.417898+0000","modified":"2024-04-25T15:28:59.469250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7128":{"gid":7128,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3429198629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3429198629},{"type":"v1","addr":"172.21.15.67:6835","nonce":3429198629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7128,"qdb_cluster":[7128]},"id":7}]} 2024-04-25T15:29:06.779 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 57 2024-04-25T15:29:06.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef900744b0 msgr2=0x7fef900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef900744b0 0x7fef900769c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fefac1f9eb0 tx=0x7fef9c009040 comp rx=0 tx=0).stop 2024-04-25T15:29:06.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefac0d0e00 msgr2=0x7fefac1588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:06.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefac0d0e00 0x7fefac1588e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c730 tx=0x7fefa400cbf0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 shutdown_connections 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef900744b0 0x7fef900769c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefac1fab60 0x7fefac156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefac0d1210 0x7fefac0cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 --2- 172.21.15.67:0/1939707375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefac0d0e00 0x7fefac1588e0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 >> 172.21.15.67:0/1939707375 conn(0x7fefac0d5d90 msgr2=0x7fefac164470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 shutdown_connections 2024-04-25T15:29:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:06.780+0000 7fefb3842640 1 -- 172.21.15.67:0/1939707375 wait complete. 2024-04-25T15:29:07.194 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:28:56.417898+0000', 'modified': '2024-04-25T15:28:59.469250+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7128}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7128': {'gid': 7128, 'name': 'a', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/3429198629', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 3429198629}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3429198629}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7128, 'qdb_cluster': [7128]} 2024-04-25T15:29:07.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T15:29:07.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T15:29:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.309+0000 7fd5181d7640 1 Processor -- start 2024-04-25T15:29:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.309+0000 7fd5181d7640 1 -- start start 2024-04-25T15:29:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.310+0000 7fd5181d7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5100b4830 0x7fd51006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd5181d7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51006d870 con 0x7fd5100d4570 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd5181d7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51006db10 con 0x7fd5100b2dc0 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd5181d7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd5100b4c40 con 0x7fd5100b4830 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd515772640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5100b4830 0x7fd51006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd515f73640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd5100d4570 0x7fd5100d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40830/0 (socket says 172.21.15.67:40830) 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd515f73640 1 -- 172.21.15.67:0/2365751614 learned_addr learned my addr 172.21.15.67:0/2365751614 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd515772640 1 -- 172.21.15.67:0/2365751614 >> v1:172.21.15.176:6789/0 conn(0x7fd5100b2dc0 legacy=0x7fd5100b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd515772640 1 -- 172.21.15.67:0/2365751614 >> v1:172.21.15.67:6789/0 conn(0x7fd5100d4570 legacy=0x7fd5100d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.311+0000 7fd515772640 1 -- 172.21.15.67:0/2365751614 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51006e0f0 con 0x7fd5100b4830 2024-04-25T15:29:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.312+0000 7fd515772640 1 --2- 172.21.15.67:0/2365751614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100b4830 0x7fd51006e8f0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd5000097a0 tx=0x7fd50002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237288e05b26d868 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.312+0000 7fd514f71640 1 -- 172.21.15.67:0/2365751614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500037070 con 0x7fd5100b4830 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.312+0000 7fd514f71640 1 -- 172.21.15.67:0/2365751614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50002ae50 con 0x7fd5100b4830 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.312+0000 7fd514f71640 1 -- 172.21.15.67:0/2365751614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500032660 con 0x7fd5100b4830 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.313+0000 7fd5181d7640 1 -- 172.21.15.67:0/2365751614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100b4830 msgr2=0x7fd51006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.313+0000 7fd5181d7640 1 --2- 172.21.15.67:0/2365751614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100b4830 0x7fd51006e8f0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd5000097a0 tx=0x7fd50002a140 comp rx=0 tx=0).stop 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.313+0000 7fd5181d7640 1 -- 172.21.15.67:0/2365751614 shutdown_connections 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.313+0000 7fd5181d7640 1 --2- 172.21.15.67:0/2365751614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100b4830 0x7fd51006e8f0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.313+0000 7fd5181d7640 1 -- 172.21.15.67:0/2365751614 >> 172.21.15.67:0/2365751614 conn(0x7fd5100d3a50 msgr2=0x7fd5100d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.313+0000 7fd5181d7640 1 -- 172.21.15.67:0/2365751614 shutdown_connections 2024-04-25T15:29:07.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.313+0000 7fd5181d7640 1 -- 172.21.15.67:0/2365751614 wait complete. 2024-04-25T15:29:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.315+0000 7fd5181d7640 1 Processor -- start 2024-04-25T15:29:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.315+0000 7fd5181d7640 1 -- start start 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd5181d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5100b2dc0 0x7fd510162040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd5181d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5100b4830 0x7fd5100a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd5181d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100d4570 0x7fd5100ce730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd5181d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5100d0fb0 con 0x7fd5100d4570 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd5181d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd51006e460 con 0x7fd5100b4830 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd5181d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd5100b38e0 con 0x7fd5100b2dc0 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100d4570 0x7fd5100ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100d4570 0x7fd5100ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57894/0 (socket says 172.21.15.67:57894) 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 -- 172.21.15.67:0/564946928 learned_addr learned my addr 172.21.15.67:0/564946928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd515f73640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5100b2dc0 0x7fd510162040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd515772640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5100b4830 0x7fd5100a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 -- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5100b2dc0 msgr2=0x7fd510162040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5100b2dc0 0x7fd510162040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 -- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5100b4830 msgr2=0x7fd5100a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5100b4830 0x7fd5100a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.316+0000 7fd516774640 1 -- 172.21.15.67:0/564946928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd510162580 con 0x7fd5100d4570 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.317+0000 7fd516774640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100d4570 0x7fd5100ce730 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd50800ba10 tx=0x7fd50800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.317+0000 7fd5067fc640 1 -- 172.21.15.67:0/564946928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508019070 con 0x7fd5100d4570 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.317+0000 7fd5067fc640 1 -- 172.21.15.67:0/564946928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50800cb40 con 0x7fd5100d4570 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.317+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5101f9b70 con 0x7fd5100d4570 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.317+0000 7fd5067fc640 1 -- 172.21.15.67:0/564946928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508014440 con 0x7fd5100d4570 2024-04-25T15:29:07.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.317+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5101f7a50 con 0x7fd5100d4570 2024-04-25T15:29:07.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.319+0000 7fd5067fc640 1 -- 172.21.15.67:0/564946928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd50801c070 con 0x7fd5100d4570 2024-04-25T15:29:07.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.320+0000 7fd5067fc640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd4f40744b0 0x7fd4f40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.320+0000 7fd5067fc640 1 -- 172.21.15.67:0/564946928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fd508094cf0 con 0x7fd5100d4570 2024-04-25T15:29:07.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.320+0000 7fd515f73640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd4f40744b0 0x7fd4f40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.321+0000 7fd51f164740 1 Processor -- start 2024-04-25T15:29:07.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.321+0000 7fd51f164740 1 -- start start 2024-04-25T15:29:07.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.321+0000 7fd515f73640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd4f40744b0 0x7fd4f40769c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0010e0 tx=0x7fd50c007d00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd51f164740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9de4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99ef3b0 0x5559f9de0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99f7590 0x5559f9dba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd51f164740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5559f9d1e740 con 0x5559f9a18270 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x5559f9cba3a0 con 0x5559f99f7590 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x5559f9d1a700 con 0x5559f99ef3b0 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9de4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9de4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57902/0 (socket says 172.21.15.67:57902) 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 -- 172.21.15.67:0/706230936 learned_addr learned my addr 172.21.15.67:0/706230936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515772640 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99ef3b0 0x5559f9de0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd516774640 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99f7590 0x5559f9dba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 -- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99ef3b0 msgr2=0x5559f9de0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99ef3b0 0x5559f9de0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 -- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99f7590 msgr2=0x5559f9dba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99f7590 0x5559f9dba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 -- 172.21.15.67:0/706230936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559f9cd6750 con 0x5559f9a18270 2024-04-25T15:29:07.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd516774640 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99f7590 0x5559f9dba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:07.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.322+0000 7fd515f73640 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9de4600 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd50c006f90 tx=0x7fd50c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5021511c38bfe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd4f37fe640 1 -- 172.21.15.67:0/706230936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c025070 con 0x5559f9a18270 2024-04-25T15:29:07.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd4f37fe640 1 -- 172.21.15.67:0/706230936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50c01b050 con 0x5559f9a18270 2024-04-25T15:29:07.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd4f37fe640 1 -- 172.21.15.67:0/706230936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c014bd0 con 0x5559f9a18270 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 -- 172.21.15.67:0/706230936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 msgr2=0x5559f9de4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9de4600 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd50c006f90 tx=0x7fd50c012000 comp rx=0 tx=0).stop 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 -- 172.21.15.67:0/706230936 shutdown_connections 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99f7590 0x5559f9dba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99ef3b0 0x5559f9de0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 --2- 172.21.15.67:0/706230936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9de4600 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 -- 172.21.15.67:0/706230936 >> 172.21.15.67:0/706230936 conn(0x5559f9a0c3c0 msgr2=0x5559f9db99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 -- 172.21.15.67:0/706230936 shutdown_connections 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.323+0000 7fd51f164740 1 -- 172.21.15.67:0/706230936 wait complete. 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.324+0000 7fd51f164740 1 -- start start 2024-04-25T15:29:07.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.324+0000 7fd51f164740 1 Processor -- start 2024-04-25T15:29:07.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9de7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9ddfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd51f164740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9dfe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd51f164740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5559f9ca5fe0 con 0x5559f9a18270 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x5559f9de0530 con 0x5559f99ef3b0 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x5559f9de8510 con 0x5559f99f7590 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd51f164740 10 client.0 _ll_register_callbacks cb 0x5559f9cbc150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9dfe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9dfe6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57906/0 (socket says 172.21.15.67:57906) 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 -- 172.21.15.67:0/3020968238 learned_addr learned my addr 172.21.15.67:0/3020968238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd515f73640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9de7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd515772640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9ddfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 -- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 msgr2=0x5559f9ddfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9ddfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 -- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 msgr2=0x5559f9de7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9de7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd516774640 1 -- 172.21.15.67:0/3020968238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559f9a302d0 con 0x5559f9a18270 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd515772640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9ddfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.326+0000 7fd515f73640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9de7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd516774640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9dfe6b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd50805ee50 tx=0x7fd50805f4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe20f2e551d57da6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd4f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/3020968238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5080631c0 con 0x5559f9a18270 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/3020968238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fd50805e120 con 0x5559f9a18270 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd50c01b050 con 0x5559f9a18270 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/3020968238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fd508067be0 con 0x5559f9a18270 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 10 client.7302 fetch_fsmap learned FSMap version 57 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) -- 0x5559f9afa6b0 con 0x5559f9a18270 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/3020968238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50c01b050 con 0x5559f9a18270 2024-04-25T15:29:07.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/3020968238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 57) ==== 3494+0+0 (secure 0 0 0) 0x7fd5080653d0 con 0x5559f9a18270 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 10 client.7302 fetch_fsmap finished waiting for FSMap version 57 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 10 client.7302 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 20 client.7302 populate_metadata read hostname 'smithi067' 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f99d6e40 0x5559f9e11e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 4 client.7302 mds_command: new command op to 7128 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 --> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559f9d2c350 con 0x5559f99d6e40 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.327+0000 7fd515f73640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f99d6e40 0x5559f9e11e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.328+0000 7fd4f0ff9640 10 client.7302 ms_handle_connect on v2:172.21.15.67:6833/3429198629 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.328+0000 7fd515f73640 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f99d6e40 0x5559f9e11e70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccade4f826727a20 server_cookie=f82747efd52e6c0e in_seq=0 out_seq=0 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.328+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/3020968238 <== mds.0 v2:172.21.15.67:6833/3429198629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5559f9d2c350 con 0x5559f99d6e40 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.328+0000 7fd4f0ff9640 10 client.7302 handle_command_reply: tid=0 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.328+0000 7fd51f164740 1 client.7302 shutdown 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 msgr2=0x5559f9dfe6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9dfe6b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd50805ee50 tx=0x7fd50805f4e0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 shutdown_connections 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9dfe6b0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f99d6e40 0x5559f9e11e70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9ddfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 --2- 172.21.15.67:0/3020968238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9de7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd4f0ff9640 0 client.7302 ms_handle_reset on v2:172.21.15.67:6833/3429198629 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 >> 172.21.15.67:0/3020968238 conn(0x5559f9a0c3c0 msgr2=0x5559f9db8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 shutdown_connections 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 1 -- 172.21.15.67:0/3020968238 wait complete. 2024-04-25T15:29:07.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.329+0000 7fd51f164740 20 client.7302 trim_cache size 0 max 16384 2024-04-25T15:29:07.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.348+0000 7fd51f164740 1 Processor -- start 2024-04-25T15:29:07.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.348+0000 7fd51f164740 1 -- start start 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.349+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9dfe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.349+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9db34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.349+0000 7fd51f164740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9df4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.349+0000 7fd51f164740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd510202e90 con 0x5559f9a18270 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.349+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd5101f98e0 con 0x5559f99ef3b0 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.349+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd5101f8c40 con 0x5559f99f7590 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9df4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9df4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57908/0 (socket says 172.21.15.67:57908) 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 -- 172.21.15.67:0/1701946363 learned_addr learned my addr 172.21.15.67:0/1701946363 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 -- 172.21.15.67:0/1701946363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 msgr2=0x5559f9db34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 --2- 172.21.15.67:0/1701946363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9db34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 -- 172.21.15.67:0/1701946363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 msgr2=0x5559f9dfe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 --2- 172.21.15.67:0/1701946363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9dfe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 -- 172.21.15.67:0/1701946363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559f9e188d0 con 0x5559f9a18270 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd516774640 1 --2- 172.21.15.67:0/1701946363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9df4a90 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd50805b460 tx=0x7fd508061000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c0b6a3272aee8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd4f2ffd640 1 -- 172.21.15.67:0/1701946363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5080631c0 con 0x5559f9a18270 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd4f2ffd640 1 -- 172.21.15.67:0/1701946363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd508080430 con 0x5559f9a18270 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.350+0000 7fd4f2ffd640 1 -- 172.21.15.67:0/1701946363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50805e120 con 0x5559f9a18270 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 -- 172.21.15.67:0/1701946363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 msgr2=0x5559f9df4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 --2- 172.21.15.67:0/1701946363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9df4a90 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd50805b460 tx=0x7fd508061000 comp rx=0 tx=0).stop 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 -- 172.21.15.67:0/1701946363 shutdown_connections 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 --2- 172.21.15.67:0/1701946363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f9a18270 0x5559f9df4a90 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 --2- 172.21.15.67:0/1701946363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f99f7590 0x5559f9db34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 --2- 172.21.15.67:0/1701946363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9dfe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 -- 172.21.15.67:0/1701946363 >> 172.21.15.67:0/1701946363 conn(0x5559f9a0c3c0 msgr2=0x5559f9de7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 -- 172.21.15.67:0/1701946363 shutdown_connections 2024-04-25T15:29:07.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.351+0000 7fd51f164740 1 -- 172.21.15.67:0/1701946363 wait complete. 2024-04-25T15:29:07.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.352+0000 7fd51f164740 1 -- start start 2024-04-25T15:29:07.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.352+0000 7fd51f164740 1 Processor -- start 2024-04-25T15:29:07.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9e02120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd51f164740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f99f7590 0x5559f9e02660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd51f164740 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f9a18270 0x5559f9ddd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd51f164740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5559f9ddac10 con 0x5559f99f7590 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd5101f9290 con 0x5559f99ef3b0 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd51f164740 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x5559f9dffc30 con 0x5559f9a18270 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd51f164740 10 client.0 _ll_register_callbacks cb 0x7fd50c007ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd515772640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f99f7590 0x5559f9e02660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd515772640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f99f7590 0x5559f9e02660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57918/0 (socket says 172.21.15.67:57918) 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.353+0000 7fd515772640 1 -- 172.21.15.67:0/944136991 learned_addr learned my addr 172.21.15.67:0/944136991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd516774640 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f9a18270 0x5559f9ddd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd515f73640 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9e02120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd515772640 1 -- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f9a18270 msgr2=0x5559f9ddd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd515772640 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f9a18270 0x5559f9ddd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd515772640 1 -- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 msgr2=0x5559f9e02120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd515772640 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9e02120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd515772640 1 -- 172.21.15.67:0/944136991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559f9de0ef0 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd515772640 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f99f7590 0x5559f9e02660 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd500000c00 tx=0x7fd500004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe81a4eb68e205c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd4f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/944136991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500037040 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/944136991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fd500048680 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd508080430 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/944136991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fd500050050 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd51f164740 10 client.7308 fetch_fsmap learned FSMap version 57 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) -- 0x5559f9de11e0 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.354+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/944136991 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd508080430 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/944136991 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 57) ==== 3494+0+0 (secure 0 0 0) 0x7fd500053020 con 0x5559f99f7590 2024-04-25T15:29:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd51f164740 10 client.7308 fetch_fsmap finished waiting for FSMap version 57 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd51f164740 10 client.7308 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd51f164740 20 client.7308 populate_metadata read hostname 'smithi067' 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd51f164740 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f9dde4a0 0x5559f9dde8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd51f164740 4 client.7308 mds_command: new command op to 7128 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 --> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] -- command(tid 0: {"prefix": "status"}) -- 0x5559f9db57b0 con 0x5559f9dde4a0 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd515f73640 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f9dde4a0 0x5559f9dde8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd4f0ff9640 10 client.7308 ms_handle_connect on v2:172.21.15.67:6833/3429198629 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd515f73640 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f9dde4a0 0x5559f9dde8b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0c4372fa0f89873 server_cookie=b9a4fe5017d73ebd in_seq=0 out_seq=0 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd4f0ff9640 1 -- 172.21.15.67:0/944136991 <== mds.0 v2:172.21.15.67:6833/3429198629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x5559f9db57b0 con 0x5559f9dde4a0 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd4f0ff9640 10 client.7308 handle_command_reply: tid=0 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.355+0000 7fd51f164740 1 client.7308 shutdown 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f99f7590 msgr2=0x5559f9e02660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f99f7590 0x5559f9e02660 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd500000c00 tx=0x7fd500004730 comp rx=0 tx=0).stop 2024-04-25T15:29:07.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 shutdown_connections 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:6833/3429198629,v1:172.21.15.67:6835/3429198629] conn(0x5559f9dde4a0 0x5559f9dde8b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x5559f9a18270 0x5559f9ddd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd4f0ff9640 0 client.7308 ms_handle_reset on v2:172.21.15.67:6833/3429198629 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5559f99f7590 0x5559f9e02660 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 --2- 172.21.15.67:0/944136991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x5559f99ef3b0 0x5559f9e02120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 >> 172.21.15.67:0/944136991 conn(0x5559f9a0c3c0 msgr2=0x5559f9db34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 shutdown_connections 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 1 -- 172.21.15.67:0/944136991 wait complete. 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd51f164740 20 client.7308 trim_cache size 0 max 16384 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:07.362 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "b43d0acb-252a-4f7a-bf14-34d789b94b95", 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 7128, 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.297303784, 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 57, 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 92, 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 92, 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 16.282206107 2024-04-25T15:29:07.363 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd4f40744b0 msgr2=0x7fd4f40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd4f40744b0 0x7fd4f40769c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0010e0 tx=0x7fd50c007d00 comp rx=0 tx=0).stop 2024-04-25T15:29:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100d4570 msgr2=0x7fd5100ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100d4570 0x7fd5100ce730 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd50800ba10 tx=0x7fd50800bed0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 shutdown_connections 2024-04-25T15:29:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd4f40744b0 0x7fd4f40769c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5100d4570 0x7fd5100ce730 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5100b4830 0x7fd5100a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 --2- 172.21.15.67:0/564946928 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5100b2dc0 0x7fd510162040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 >> 172.21.15.67:0/564946928 conn(0x7fd5100d3a50 msgr2=0x7fd510072770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.356+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 shutdown_connections 2024-04-25T15:29:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.357+0000 7fd5181d7640 1 -- 172.21.15.67:0/564946928 wait complete. 2024-04-25T15:29:07.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:29:07.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.479+0000 7f6b3069a640 1 Processor -- start 2024-04-25T15:29:07.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.479+0000 7f6b3069a640 1 -- start start 2024-04-25T15:29:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.480+0000 7f6b3069a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b280b7f50 0x7f6b2816b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.480+0000 7f6b3069a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b280b2140 con 0x7f6b280b3d60 2024-04-25T15:29:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.480+0000 7f6b3069a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b280cee00 con 0x7f6b280d2570 2024-04-25T15:29:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.480+0000 7f6b3069a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6b280cf0e0 con 0x7f6b280b7f50 2024-04-25T15:29:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.480+0000 7f6b2ec37640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b280b7f50 0x7f6b2816b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.480+0000 7f6b2dc35640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6b280b3d60 0x7f6b280b7840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40848/0 (socket says 172.21.15.67:40848) 2024-04-25T15:29:07.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.480+0000 7f6b2dc35640 1 -- 172.21.15.67:0/1002925116 learned_addr learned my addr 172.21.15.67:0/1002925116 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:07.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.481+0000 7f6b2ec37640 1 -- 172.21.15.67:0/1002925116 >> v1:172.21.15.176:6789/0 conn(0x7f6b280d2570 legacy=0x7f6b280cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:07.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.481+0000 7f6b2ec37640 1 -- 172.21.15.67:0/1002925116 >> v1:172.21.15.67:6789/0 conn(0x7f6b280b3d60 legacy=0x7f6b280b7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.481+0000 7f6b2ec37640 1 -- 172.21.15.67:0/1002925116 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b280cf3c0 con 0x7f6b280b7f50 2024-04-25T15:29:07.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.481+0000 7f6b2ec37640 1 --2- 172.21.15.67:0/1002925116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b280b7f50 0x7f6b2816b1f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6b200097a0 tx=0x7f6b2002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ac8dfa5889ace7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.481+0000 7f6b2d434640 1 -- 172.21.15.67:0/1002925116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20037070 con 0x7f6b280b7f50 2024-04-25T15:29:07.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.481+0000 7f6b2d434640 1 -- 172.21.15.67:0/1002925116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b20032440 con 0x7f6b280b7f50 2024-04-25T15:29:07.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b2d434640 1 -- 172.21.15.67:0/1002925116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20032810 con 0x7f6b280b7f50 2024-04-25T15:29:07.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b3069a640 1 -- 172.21.15.67:0/1002925116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b280b7f50 msgr2=0x7f6b2816b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b3069a640 1 --2- 172.21.15.67:0/1002925116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b280b7f50 0x7f6b2816b1f0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f6b200097a0 tx=0x7f6b2002a1e0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b3069a640 1 -- 172.21.15.67:0/1002925116 shutdown_connections 2024-04-25T15:29:07.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b3069a640 1 --2- 172.21.15.67:0/1002925116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b280b7f50 0x7f6b2816b1f0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b3069a640 1 -- 172.21.15.67:0/1002925116 >> 172.21.15.67:0/1002925116 conn(0x7f6b2809b7d0 msgr2=0x7f6b2809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b3069a640 1 -- 172.21.15.67:0/1002925116 shutdown_connections 2024-04-25T15:29:07.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.482+0000 7f6b3069a640 1 -- 172.21.15.67:0/1002925116 wait complete. 2024-04-25T15:29:07.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.484+0000 7f6b3069a640 1 Processor -- start 2024-04-25T15:29:07.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.484+0000 7f6b3069a640 1 -- start start 2024-04-25T15:29:07.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.485+0000 7f6b3069a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b280b3d60 0x7f6b28159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.485+0000 7f6b3069a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b280d2570 0x7f6b281582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.485+0000 7f6b3069a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b281553d0 0x7f6b28158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.485+0000 7f6b3069a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b280cd2b0 con 0x7f6b281553d0 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.485+0000 7f6b3069a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6b280cd130 con 0x7f6b280d2570 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.485+0000 7f6b3069a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6b2816b730 con 0x7f6b280b3d60 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b281553d0 0x7f6b28158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b281553d0 0x7f6b28158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57932/0 (socket says 172.21.15.67:57932) 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 -- 172.21.15.67:0/2460705120 learned_addr learned my addr 172.21.15.67:0/2460705120 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2e436640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b280b3d60 0x7f6b28159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2dc35640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b280d2570 0x7f6b281582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 -- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b280b3d60 msgr2=0x7f6b28159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b280b3d60 0x7f6b28159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 -- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b280d2570 msgr2=0x7f6b281582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b280d2570 0x7f6b281582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 -- 172.21.15.67:0/2460705120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b28155b80 con 0x7f6b281553d0 2024-04-25T15:29:07.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b2ec37640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b281553d0 0x7f6b28158810 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f6b2002a220 tx=0x7f6b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b26ffd640 1 -- 172.21.15.67:0/2460705120 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20041070 con 0x7f6b281553d0 2024-04-25T15:29:07.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b26ffd640 1 -- 172.21.15.67:0/2460705120 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b2003c890 con 0x7f6b281553d0 2024-04-25T15:29:07.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b28155e60 con 0x7f6b281553d0 2024-04-25T15:29:07.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b26ffd640 1 -- 172.21.15.67:0/2460705120 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20037040 con 0x7f6b281553d0 2024-04-25T15:29:07.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.486+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b28156420 con 0x7f6b281553d0 2024-04-25T15:29:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.487+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b280ca8b0 con 0x7f6b281553d0 2024-04-25T15:29:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.488+0000 7f6b26ffd640 1 -- 172.21.15.67:0/2460705120 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6b20044070 con 0x7f6b281553d0 2024-04-25T15:29:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.488+0000 7f6b26ffd640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b140744e0 0x7f6b140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.489+0000 7f6b26ffd640 1 -- 172.21.15.67:0/2460705120 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f6b200b5e20 con 0x7f6b281553d0 2024-04-25T15:29:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.489+0000 7f6b2e436640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b140744e0 0x7f6b140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.489+0000 7f6b2e436640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b140744e0 0x7f6b140769f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6b2809c5a0 tx=0x7f6b1800d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:07.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.493+0000 7f6b26ffd640 1 -- 172.21.15.67:0/2460705120 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b2007fee0 con 0x7f6b281553d0 2024-04-25T15:29:07.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.718+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6b28050740 con 0x7f6b281553d0 2024-04-25T15:29:07.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.718+0000 7f6b26ffd640 1 -- 172.21.15.67:0/2460705120 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) ==== 53+0+73 (secure 0 0 0) 0x7f6b200841e0 con 0x7f6b281553d0 2024-04-25T15:29:07.719 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b140744e0 msgr2=0x7f6b140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b140744e0 0x7f6b140769f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6b2809c5a0 tx=0x7f6b1800d040 comp rx=0 tx=0).stop 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b281553d0 msgr2=0x7f6b28158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b281553d0 0x7f6b28158810 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f6b2002a220 tx=0x7f6b200046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 shutdown_connections 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f6b140744e0 0x7f6b140769f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b281553d0 0x7f6b28158810 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b280d2570 0x7f6b281582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 --2- 172.21.15.67:0/2460705120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6b280b3d60 0x7f6b28159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 >> 172.21.15.67:0/2460705120 conn(0x7f6b2809b7d0 msgr2=0x7f6b28071a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 shutdown_connections 2024-04-25T15:29:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07.720+0000 7f6b3069a640 1 -- 172.21.15.67:0/2460705120 wait complete. 2024-04-25T15:29:07.732 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:29:07.733 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:29:07.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:07.733 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:29:07.750 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:29:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:29:07.750 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42551sec preferred_lft 42551sec 2024-04-25T15:29:07.751 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2980:e17c:82eb:bd91/64 scope link noprefixroute 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::7482:42ff:fe30:5ffc/64 scope link 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0a:85:0b:04:f6:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::885:bff:fe04:f6cb/64 scope link 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether ca:af:51:1f:d5:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c8af:51ff:fe1f:d5b0/64 scope link 2024-04-25T15:29:07.752 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:07.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:07.753 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:29:07.816 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:29:07.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:07.818 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:29:07.888 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:29:07.888 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:29:07.888 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:29:07.888 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:29:07.888 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:29:07.889 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:29:07.889 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:29:07.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:29:07.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:29:07.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:29:07.889 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:29:07.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:29:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:29:08.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:08.022 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:29:08.041 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:29:08.041 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:29:08.041 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:29:08.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:08.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:29:08.117 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:29:08.118 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:29:08.174 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:29:08.174 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:29:08.174 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:29:08.234 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T15:29:08.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:08.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:29:08.334 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:29:08.334 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:29:08.345 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:29:08.345 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:29:08.345 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:29:08.345 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:29:08.345 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:29:08.346 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T15:29:08.346 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:29:08.346 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:29:08.346 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:29:08.346 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:29:08.347 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:29:08.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:08.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:29:08.374 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:29:08.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:29:08.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:29:08.479 INFO:teuthology.orchestra.run.smithi067.stdout:client7320 2024-04-25T15:29:08.480 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:29:08.480 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:08.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:29:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.648+0000 7f2f9a1f3640 1 Processor -- start 2024-04-25T15:29:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.648+0000 7f2f9a1f3640 1 -- start start 2024-04-25T15:29:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f9a1f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f94164b00 0x7f2f940b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f9a1f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9415df80 0x7f2f940b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f9a1f3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f941615f0 con 0x7f2f94160890 2024-04-25T15:29:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f9a1f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2f941613b0 con 0x7f2f94164b00 2024-04-25T15:29:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f9a1f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f94164670 con 0x7f2f9415df80 2024-04-25T15:29:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f937fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f94164b00 0x7f2f940b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f93fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2f94160890 0x7f2f94160ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38368/0 (socket says 172.21.15.67:38368) 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f937fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f94164b00 0x7f2f940b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57694/0 (socket says 172.21.15.67:57694) 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f93fff640 1 -- 172.21.15.67:0/2731723 learned_addr learned my addr 172.21.15.67:0/2731723 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f92ffd640 1 --2- 172.21.15.67:0/2731723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9415df80 0x7f2f940b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f937fe640 1 -- 172.21.15.67:0/2731723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9415df80 msgr2=0x7f2f940b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f937fe640 1 --2- 172.21.15.67:0/2731723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9415df80 0x7f2f940b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f937fe640 1 -- 172.21.15.67:0/2731723 >> v1:172.21.15.67:6789/0 conn(0x7f2f94160890 legacy=0x7f2f94160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f927fc640 1 -- 172.21.15.67:0/2731723 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030446593 0 0) 0x7f2f941615f0 con 0x7f2f94160890 2024-04-25T15:29:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.650+0000 7f2f937fe640 1 -- 172.21.15.67:0/2731723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f94164020 con 0x7f2f94164b00 2024-04-25T15:29:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.651+0000 7f2f937fe640 1 --2- 172.21.15.67:0/2731723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f94164b00 0x7f2f940b7b80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2f840097a0 tx=0x7f2f8402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423f916eadccdffe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.651+0000 7f2f927fc640 1 -- 172.21.15.67:0/2731723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84037070 con 0x7f2f94164b00 2024-04-25T15:29:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.651+0000 7f2f927fc640 1 -- 172.21.15.67:0/2731723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f84032440 con 0x7f2f94164b00 2024-04-25T15:29:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.651+0000 7f2f927fc640 1 -- 172.21.15.67:0/2731723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84032810 con 0x7f2f94164b00 2024-04-25T15:29:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/2731723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f94164b00 msgr2=0x7f2f940b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/2731723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f94164b00 0x7f2f940b7b80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2f840097a0 tx=0x7f2f8402a180 comp rx=0 tx=0).stop 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/2731723 shutdown_connections 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/2731723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f9415df80 0x7f2f940b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/2731723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f94164b00 0x7f2f940b7b80 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/2731723 >> 172.21.15.67:0/2731723 conn(0x7f2f940d2bc0 msgr2=0x7f2f940d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/2731723 shutdown_connections 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.652+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/2731723 wait complete. 2024-04-25T15:29:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.654+0000 7f2f9a1f3640 1 Processor -- start 2024-04-25T15:29:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.654+0000 7f2f9a1f3640 1 -- start start 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f9a1f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9415df80 0x7f2f941f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f9a1f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f94160890 0x7f2f940b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f9a1f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f94164b00 0x7f2f940b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f9a1f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f940b4a00 con 0x7f2f9415df80 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f9a1f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2f940b4880 con 0x7f2f94160890 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f9a1f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f940b4b80 con 0x7f2f94164b00 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9415df80 0x7f2f941f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9415df80 0x7f2f941f64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57700/0 (socket says 172.21.15.67:57700) 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 -- 172.21.15.67:0/1416342716 learned_addr learned my addr 172.21.15.67:0/1416342716 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 -- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f94164b00 msgr2=0x7f2f940b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f94164b00 0x7f2f940b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 -- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f94160890 msgr2=0x7f2f940b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f94160890 0x7f2f940b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.655+0000 7f2f937fe640 1 -- 172.21.15.67:0/1416342716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f940b6810 con 0x7f2f9415df80 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.656+0000 7f2f937fe640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9415df80 0x7f2f941f64b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f2f840098c0 tx=0x7f2f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.656+0000 7f2f7bfff640 1 -- 172.21.15.67:0/1416342716 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84041070 con 0x7f2f9415df80 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.656+0000 7f2f7bfff640 1 -- 172.21.15.67:0/1416342716 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8403c870 con 0x7f2f9415df80 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.656+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f941545f0 con 0x7f2f9415df80 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.656+0000 7f2f7bfff640 1 -- 172.21.15.67:0/1416342716 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84037040 con 0x7f2f9415df80 2024-04-25T15:29:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.656+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f94142eb0 con 0x7f2f9415df80 2024-04-25T15:29:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.657+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f94050740 con 0x7f2f9415df80 2024-04-25T15:29:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.658+0000 7f2f7bfff640 1 -- 172.21.15.67:0/1416342716 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2f84044070 con 0x7f2f9415df80 2024-04-25T15:29:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.658+0000 7f2f7bfff640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800744e0 0x7f2f800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.658+0000 7f2f7bfff640 1 -- 172.21.15.67:0/1416342716 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f2f840b5980 con 0x7f2f9415df80 2024-04-25T15:29:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.659+0000 7f2f92ffd640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800744e0 0x7f2f800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.659+0000 7f2f92ffd640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800744e0 0x7f2f800769f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2f88006a90 tx=0x7f2f88006a20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.663+0000 7f2f7bfff640 1 -- 172.21.15.67:0/1416342716 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f840801a0 con 0x7f2f9415df80 2024-04-25T15:29:08.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.887+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2f940d4170 con 0x7f2f9415df80 2024-04-25T15:29:08.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.888+0000 7f2f7bfff640 1 -- 172.21.15.67:0/1416342716 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) ==== 53+0+73 (secure 0 0 0) 0x7f2f84088190 con 0x7f2f9415df80 2024-04-25T15:29:08.889 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800744e0 msgr2=0x7f2f800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800744e0 0x7f2f800769f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2f88006a90 tx=0x7f2f88006a20 comp rx=0 tx=0).stop 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9415df80 msgr2=0x7f2f941f64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9415df80 0x7f2f941f64b0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f2f840098c0 tx=0x7f2f840046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 shutdown_connections 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f800744e0 0x7f2f800769f0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f94164b00 0x7f2f940b5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f94160890 0x7f2f940b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 --2- 172.21.15.67:0/1416342716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f9415df80 0x7f2f941f64b0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 >> 172.21.15.67:0/1416342716 conn(0x7f2f940d2bc0 msgr2=0x7f2f94160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 shutdown_connections 2024-04-25T15:29:08.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:08.890+0000 7f2f9a1f3640 1 -- 172.21.15.67:0/1416342716 wait complete. 2024-04-25T15:29:08.901 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:29:08.902 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:29:08.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:08.902 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-04-25T15:29:08.921 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:29:08.921 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:29:08.921 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:29:08.921 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:08.921 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42540sec preferred_lft 42540sec 2024-04-25T15:29:08.922 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ce2d:ab5a:5308:c83d/64 scope link noprefixroute 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::f0b9:caff:fe40:93d7/64 scope link 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether de:24:64:23:dc:ba brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::dc24:64ff:fe23:dcba/64 scope link 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 26:8e:2b:69:7f:63 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::248e:2bff:fe69:7f63/64 scope link 2024-04-25T15:29:08.923 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:29:08.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:08.924 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:29:08.989 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:29:08.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:08.990 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-04-25T15:29:09.049 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-04-25T15:29:09.049 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:29:09.049 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:29:09.049 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:29:09.049 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:29:09.050 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:29:09.050 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-04-25T15:29:09.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:29:09.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:29:09.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:29:09.050 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-04-25T15:29:09.142 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:29:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:29:09.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:29:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:29:09.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.179 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:29:09.194 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:29:09.195 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:29:09.195 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:29:09.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.195 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-04-25T15:29:09.259 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:29:09.259 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T15:29:09.313 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T15:29:09.313 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:29:09.313 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:29:09.368 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T15:29:09.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:29:09.460 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:29:09.461 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:29:09.470 INFO:teuthology.orchestra.run.smithi176.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T15:29:09.470 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T15:29:09.470 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.176:3300,172.21.15.176:3301" 2024-04-25T15:29:09.470 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: trying mount with new device syntax: 1@b43d0acb-252a-4f7a-bf14-34d789b94b95.cephfs=/ 2024-04-25T15:29:09.470 INFO:teuthology.orchestra.run.smithi176.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.176:3300/172.21.15.176:3301" will pass to kernel 2024-04-25T15:29:09.471 INFO:teuthology.orchestra.run.smithi176.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T15:29:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: You just mounted a file system that supports labels which does not 2024-04-25T15:29:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T15:29:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T15:29:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T15:29:09.473 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T15:29:09.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T15:29:09.496 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T15:29:09.584 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:29:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T15:29:09.585 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:29:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:29:09.591 INFO:teuthology.orchestra.run.smithi176.stdout:client6778 2024-04-25T15:29:09.591 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:09.591 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:09.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.607 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir_files) 2024-04-25T15:29:09.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.629 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir_data) 2024-04-25T15:29:09.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.690 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 100 ./subdir_files) 2024-04-25T15:29:09.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:09.751 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 104857600 ./subdir_data) 2024-04-25T15:29:09.811 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:09.811 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> stdin-killer -- python3 -c ' 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> import os 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> import uuid 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> n = 20 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> create_hard_links = False 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> if 0 > 0: 2024-04-25T15:29:09.827 DEBUG:teuthology.orchestra.run.smithi176:> create_hard_links = True 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> path = "/home/ubuntu/cephtest/mnt.1/subdir_files/file" 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> dpath = os.path.dirname(path) 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> fnameprefix = os.path.basename(path) 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> os.makedirs(dpath, exist_ok=True) 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> try: 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> for i in range(n): 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> with open(fpath, '"'"'w'"'"') as f: 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> f.write(f"{i}") 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> f.flush() 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> os.fsync(f.fileno()) 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> os.unlink(fpath) 2024-04-25T15:29:09.828 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> os.fsync(dirfd) 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> if create_hard_links: 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> for j in range(0): 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> os.fsync(dirfd) 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> finally: 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> os.close(dirfd) 2024-04-25T15:29:09.829 DEBUG:teuthology.orchestra.run.smithi176:> ' 2024-04-25T15:29:09.970 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:29:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:29:09.976 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:09.976 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:10.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:10.032 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec dd if=/dev/urandom of=subdir_data/file bs=1M conv=fdatasync count=20 seek=0) 2024-04-25T15:29:10.179 INFO:teuthology.orchestra.run.smithi176.stderr:20+0 records in 2024-04-25T15:29:10.179 INFO:teuthology.orchestra.run.smithi176.stderr:20+0 records out 2024-04-25T15:29:10.179 INFO:teuthology.orchestra.run.smithi176.stderr:20971520 bytes (21 MB, 20 MiB) copied, 0.0903048 s, 232 MB/s 2024-04-25T15:29:10.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:10.180 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 10 ./subdir_files) 2024-04-25T15:29:10.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:10.199 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1048576 ./subdir_data) 2024-04-25T15:29:10.258 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:10.258 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> stdin-killer -- python3 -c ' 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> import os 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> import uuid 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> n = 40 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> create_hard_links = False 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> if 0 > 0: 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> create_hard_links = True 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> path = "/home/ubuntu/cephtest/mnt.1/subdir_files/file" 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> dpath = os.path.dirname(path) 2024-04-25T15:29:10.276 DEBUG:teuthology.orchestra.run.smithi176:> fnameprefix = os.path.basename(path) 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> os.makedirs(dpath, exist_ok=True) 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> try: 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> for i in range(n): 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> with open(fpath, '"'"'w'"'"') as f: 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> f.write(f"{i}") 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:10.277 DEBUG:teuthology.orchestra.run.smithi176:> f.flush() 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> os.fsync(f.fileno()) 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> os.unlink(fpath) 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> os.fsync(dirfd) 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> if create_hard_links: 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> for j in range(0): 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> if False: 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> os.fsync(dirfd) 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> finally: 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> os.close(dirfd) 2024-04-25T15:29:10.278 DEBUG:teuthology.orchestra.run.smithi176:> ' 2024-04-25T15:29:10.512 INFO:teuthology.orchestra.run.smithi176.stderr:Traceback (most recent call last): 2024-04-25T15:29:10.512 INFO:teuthology.orchestra.run.smithi176.stderr: File "", line 20, in 2024-04-25T15:29:10.512 INFO:teuthology.orchestra.run.smithi176.stderr:OSError: [Errno 122] Disk quota exceeded: '/home/ubuntu/cephtest/mnt.1/subdir_files/file_20' 2024-04-25T15:29:10.516 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:29:10 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-04-25T15:29:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:10.522 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:10.522 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:10.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:10.578 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest/mnt.1 && exec dd if=/dev/urandom of=subdir_data/file bs=1M conv=fdatasync count=40 seek=0) 2024-04-25T15:29:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:10.698 INFO:teuthology.orchestra.run.smithi176.stderr:dd: error writing 'subdir_data/file': Disk quota exceeded 2024-04-25T15:29:10.699 INFO:teuthology.orchestra.run.smithi176.stderr:23+0 records in 2024-04-25T15:29:10.699 INFO:teuthology.orchestra.run.smithi176.stderr:22+0 records out 2024-04-25T15:29:10.699 INFO:teuthology.orchestra.run.smithi176.stderr:23068672 bytes (23 MB, 22 MiB) copied, 0.0615405 s, 375 MB/s 2024-04-25T15:29:10.699 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T15:29:10.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:29:10.737 DEBUG:teuthology.orchestra.run.smithi176:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T15:29:10.780 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T15:29:10.780 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:29:10.780 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:29:10.780 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:10.796 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:29:10.796 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:10.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:29:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.003+0000 7fdd0d4e2640 1 Processor -- start 2024-04-25T15:29:11.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.003+0000 7fdd0d4e2640 1 -- start start 2024-04-25T15:29:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd0d4e2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd08164b00 0x7fdd080b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd0d4e2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd08160fa0 0x7fdd080b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd0d4e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd081615f0 con 0x7fdd0815df80 2024-04-25T15:29:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd0d4e2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdd081613b0 con 0x7fdd08160fa0 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd0d4e2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdd08164670 con 0x7fdd08164b00 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd077fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd08160fa0 0x7fdd080b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd077fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd08160fa0 0x7fdd080b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57710/0 (socket says 172.21.15.67:57710) 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd077fe640 1 -- 172.21.15.67:0/14409538 learned_addr learned my addr 172.21.15.67:0/14409538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/14409538 >> v2:172.21.15.176:3300/0 conn(0x7fdd08164b00 0x7fdd080b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd077fe640 1 -- 172.21.15.67:0/14409538 >> v2:172.21.15.176:3300/0 conn(0x7fdd08164b00 msgr2=0x7fdd080b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd077fe640 1 --2- 172.21.15.67:0/14409538 >> v2:172.21.15.176:3300/0 conn(0x7fdd08164b00 0x7fdd080b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd077fe640 1 -- 172.21.15.67:0/14409538 >> v1:172.21.15.176:6789/0 conn(0x7fdd0815df80 legacy=0x7fdd08160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.005+0000 7fdd077fe640 1 -- 172.21.15.67:0/14409538 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd08164020 con 0x7fdd08160fa0 2024-04-25T15:29:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.006+0000 7fdd077fe640 1 --2- 172.21.15.67:0/14409538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd08160fa0 0x7fdd080b80c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fdd0808e410 tx=0x7fdcf802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7594bdf61a358bf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.006+0000 7fdd05ffb640 1 -- 172.21.15.67:0/14409538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8036070 con 0x7fdd08160fa0 2024-04-25T15:29:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.006+0000 7fdd05ffb640 1 -- 172.21.15.67:0/14409538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf8031600 con 0x7fdd08160fa0 2024-04-25T15:29:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.006+0000 7fdd05ffb640 1 -- 172.21.15.67:0/14409538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf80319d0 con 0x7fdd08160fa0 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/14409538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd08160fa0 msgr2=0x7fdd080b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/14409538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd08160fa0 0x7fdd080b80c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fdd0808e410 tx=0x7fdcf802a730 comp rx=0 tx=0).stop 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/14409538 shutdown_connections 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/14409538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd08160fa0 0x7fdd080b80c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/14409538 >> v2:172.21.15.176:3300/0 conn(0x7fdd08164b00 0x7fdd080b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/14409538 >> 172.21.15.67:0/14409538 conn(0x7fdd080d2bc0 msgr2=0x7fdd080d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/14409538 shutdown_connections 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.007+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/14409538 wait complete. 2024-04-25T15:29:11.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.009+0000 7fdd0d4e2640 1 Processor -- start 2024-04-25T15:29:11.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.009+0000 7fdd0d4e2640 1 -- start start 2024-04-25T15:29:11.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd0d4e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd0815df80 0x7fdd080c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd0d4e2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd08160fa0 0x7fdd08157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd0d4e2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd08164b00 0x7fdd0815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd0d4e2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080b4a00 con 0x7fdd0815df80 2024-04-25T15:29:11.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd0d4e2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080b4880 con 0x7fdd08160fa0 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd0d4e2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd080b4b80 con 0x7fdd08164b00 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd0815df80 0x7fdd080c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd06ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd0815df80 0x7fdd080c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57718/0 (socket says 172.21.15.67:57718) 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd06ffd640 1 -- 172.21.15.67:0/4204215797 learned_addr learned my addr 172.21.15.67:0/4204215797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd067fc640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd08160fa0 0x7fdd08157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.010+0000 7fdd077fe640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd08164b00 0x7fdd0815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd06ffd640 1 -- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd08164b00 msgr2=0x7fdd0815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd08164b00 0x7fdd0815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd06ffd640 1 -- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd08160fa0 msgr2=0x7fdd08157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd08160fa0 0x7fdd08157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd06ffd640 1 -- 172.21.15.67:0/4204215797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd080c9300 con 0x7fdd0815df80 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd06ffd640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd0815df80 0x7fdd080c8dc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000e8a0 tx=0x7fdd0000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdcff7fe640 1 -- 172.21.15.67:0/4204215797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00017070 con 0x7fdd0815df80 2024-04-25T15:29:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdcff7fe640 1 -- 172.21.15.67:0/4204215797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0000c940 con 0x7fdd0815df80 2024-04-25T15:29:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd08158ef0 con 0x7fdd0815df80 2024-04-25T15:29:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdcff7fe640 1 -- 172.21.15.67:0/4204215797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0000cd30 con 0x7fdd0815df80 2024-04-25T15:29:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.011+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd08144db0 con 0x7fdd0815df80 2024-04-25T15:29:11.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.012+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd08050740 con 0x7fdd0815df80 2024-04-25T15:29:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.013+0000 7fdcff7fe640 1 -- 172.21.15.67:0/4204215797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdd0001a070 con 0x7fdd0815df80 2024-04-25T15:29:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.013+0000 7fdcff7fe640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdce80744e0 0x7fdce80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.014+0000 7fdcff7fe640 1 -- 172.21.15.67:0/4204215797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fdd000950b0 con 0x7fdd0815df80 2024-04-25T15:29:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.014+0000 7fdd067fc640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdce80744e0 0x7fdce80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.014+0000 7fdd067fc640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdce80744e0 0x7fdce80769f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdd08160ea0 tx=0x7fdcf0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.018+0000 7fdcff7fe640 1 -- 172.21.15.67:0/4204215797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd0005f1f0 con 0x7fdd0815df80 2024-04-25T15:29:11.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.204+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd080d4170 con 0x7fdd0815df80 2024-04-25T15:29:11.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.206+0000 7fdcff7fe640 1 -- 172.21.15.67:0/4204215797 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16572 (secure 0 0 0) 0x7fdd000634f0 con 0x7fdd0815df80 2024-04-25T15:29:11.207 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:11.207 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:58.058412+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:29:11.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdce80744e0 msgr2=0x7fdce80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdce80744e0 0x7fdce80769f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fdd08160ea0 tx=0x7fdcf0009040 comp rx=0 tx=0).stop 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd0815df80 msgr2=0x7fdd080c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd0815df80 0x7fdd080c8dc0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000e8a0 tx=0x7fdd0000ed60 comp rx=0 tx=0).stop 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 shutdown_connections 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fdce80744e0 0x7fdce80769f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd08164b00 0x7fdd0815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd08160fa0 0x7fdd08157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 --2- 172.21.15.67:0/4204215797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd0815df80 0x7fdd080c8dc0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 >> 172.21.15.67:0/4204215797 conn(0x7fdd080d2bc0 msgr2=0x7fdd0815f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 shutdown_connections 2024-04-25T15:29:11.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.208+0000 7fdd0d4e2640 1 -- 172.21.15.67:0/4204215797 wait complete. 2024-04-25T15:29:11.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:28:54.107977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:28:55.386227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:29:11.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:29:11.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.334+0000 7faabc34d640 1 Processor -- start 2024-04-25T15:29:11.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.334+0000 7faabc34d640 1 -- start start 2024-04-25T15:29:11.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faabc34d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7faab4164ad0 0x7faab40b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faabc34d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab4163f30 0x7faab4164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faabc34d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab4164880 con 0x7faab40d1f30 2024-04-25T15:29:11.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faabc34d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7faab4164ee0 con 0x7faab4164ad0 2024-04-25T15:29:11.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faabc34d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faab40b31e0 con 0x7faab4163f30 2024-04-25T15:29:11.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faaba8ea640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faab40d1f30 0x7faab40d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38370/0 (socket says 172.21.15.67:38370) 2024-04-25T15:29:11.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faaba8ea640 1 -- 172.21.15.67:0/1407319111 learned_addr learned my addr 172.21.15.67:0/1407319111 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faab98e8640 1 --2- 172.21.15.67:0/1407319111 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab4163f30 0x7faab4164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faaba0e9640 1 --2- 172.21.15.67:0/1407319111 >> v2:172.21.15.176:3300/0 conn(0x7faab4164ad0 0x7faab40b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faab90e7640 1 -- 172.21.15.67:0/1407319111 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127460420 0 0) 0x7faaa4001490 con 0x7faab40d1f30 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faab90e7640 1 -- 172.21.15.67:0/1407319111 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa9c003520 con 0x7faab40d1f30 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faab98e8640 1 -- 172.21.15.67:0/1407319111 >> v2:172.21.15.176:3300/0 conn(0x7faab4164ad0 msgr2=0x7faab40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faab98e8640 1 --2- 172.21.15.67:0/1407319111 >> v2:172.21.15.176:3300/0 conn(0x7faab4164ad0 0x7faab40b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faab98e8640 1 -- 172.21.15.67:0/1407319111 >> v1:172.21.15.67:6789/0 conn(0x7faab40d1f30 legacy=0x7faab40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.336+0000 7faab98e8640 1 -- 172.21.15.67:0/1407319111 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab40d3170 con 0x7faab4163f30 2024-04-25T15:29:11.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.337+0000 7faab98e8640 1 --2- 172.21.15.67:0/1407319111 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab4163f30 0x7faab4164340 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faaa80097a0 tx=0x7faaa802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3e099c90da1a883 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.337+0000 7faab90e7640 1 -- 172.21.15.67:0/1407319111 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8037070 con 0x7faab4163f30 2024-04-25T15:29:11.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.337+0000 7faab90e7640 1 -- 172.21.15.67:0/1407319111 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa8032440 con 0x7faab4163f30 2024-04-25T15:29:11.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.337+0000 7faab90e7640 1 -- 172.21.15.67:0/1407319111 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80327f0 con 0x7faab4163f30 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 -- 172.21.15.67:0/1407319111 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab4163f30 msgr2=0x7faab4164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 --2- 172.21.15.67:0/1407319111 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab4163f30 0x7faab4164340 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faaa80097a0 tx=0x7faaa802a180 comp rx=0 tx=0).stop 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 -- 172.21.15.67:0/1407319111 shutdown_connections 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 --2- 172.21.15.67:0/1407319111 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab4163f30 0x7faab4164340 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 --2- 172.21.15.67:0/1407319111 >> v2:172.21.15.176:3300/0 conn(0x7faab4164ad0 0x7faab40b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 -- 172.21.15.67:0/1407319111 >> 172.21.15.67:0/1407319111 conn(0x7faab40d3aa0 msgr2=0x7faab40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 -- 172.21.15.67:0/1407319111 shutdown_connections 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.338+0000 7faabc34d640 1 -- 172.21.15.67:0/1407319111 wait complete. 2024-04-25T15:29:11.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.340+0000 7faabc34d640 1 Processor -- start 2024-04-25T15:29:11.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.340+0000 7faabc34d640 1 -- start start 2024-04-25T15:29:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faabc34d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab40d1f30 0x7faab40b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faabc34d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4163f30 0x7faab40c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faabc34d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faab4164ad0 0x7faab40ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faabc34d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faab406d9b0 con 0x7faab4163f30 2024-04-25T15:29:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faabc34d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faab406d6d0 con 0x7faab4164ad0 2024-04-25T15:29:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faabc34d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faab40b9430 con 0x7faab40d1f30 2024-04-25T15:29:11.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faab98e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4163f30 0x7faab40c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faab98e8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4163f30 0x7faab40c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57730/0 (socket says 172.21.15.67:57730) 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faab98e8640 1 -- 172.21.15.67:0/2118974059 learned_addr learned my addr 172.21.15.67:0/2118974059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faaba8ea640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faab4164ad0 0x7faab40ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.341+0000 7faaba0e9640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab40d1f30 0x7faab40b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab98e8640 1 -- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab40d1f30 msgr2=0x7faab40b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab98e8640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab40d1f30 0x7faab40b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab98e8640 1 -- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faab4164ad0 msgr2=0x7faab40ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab98e8640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faab4164ad0 0x7faab40ca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab98e8640 1 -- 172.21.15.67:0/2118974059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab415d4b0 con 0x7faab4163f30 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab98e8640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4163f30 0x7faab40c9af0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7faaa8035040 tx=0x7faaa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab2ffd640 1 -- 172.21.15.67:0/2118974059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8037040 con 0x7faab4163f30 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab2ffd640 1 -- 172.21.15.67:0/2118974059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa8042070 con 0x7faab4163f30 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faab2ffd640 1 -- 172.21.15.67:0/2118974059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa803b930 con 0x7faab4163f30 2024-04-25T15:29:11.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab415d790 con 0x7faab4163f30 2024-04-25T15:29:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.342+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab415dd50 con 0x7faab4163f30 2024-04-25T15:29:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.343+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab4050740 con 0x7faab4163f30 2024-04-25T15:29:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.344+0000 7faab2ffd640 1 -- 172.21.15.67:0/2118974059 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7faaa803bad0 con 0x7faab4163f30 2024-04-25T15:29:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.344+0000 7faab2ffd640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faaa00744b0 0x7faaa00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.345+0000 7faab2ffd640 1 -- 172.21.15.67:0/2118974059 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7faaa80b5830 con 0x7faab4163f30 2024-04-25T15:29:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.345+0000 7faaba0e9640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faaa00744b0 0x7faaa00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.345+0000 7faaba0e9640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faaa00744b0 0x7faaa00769c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faab40746e0 tx=0x7faaac009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.349+0000 7faab2ffd640 1 -- 172.21.15.67:0/2118974059 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faaa807f970 con 0x7faab4163f30 2024-04-25T15:29:11.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.538+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faab40d4770 con 0x7faab4163f30 2024-04-25T15:29:11.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.538+0000 7faab2ffd640 1 -- 172.21.15.67:0/2118974059 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7faaa8083c70 con 0x7faab4163f30 2024-04-25T15:29:11.539 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:29:11.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faaa00744b0 msgr2=0x7faaa00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faaa00744b0 0x7faaa00769c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faab40746e0 tx=0x7faaac009190 comp rx=0 tx=0).stop 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4163f30 msgr2=0x7faab40c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4163f30 0x7faab40c9af0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7faaa8035040 tx=0x7faaa80046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 shutdown_connections 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7faaa00744b0 0x7faaa00769c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faab4164ad0 0x7faab40ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4163f30 0x7faab40c9af0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 --2- 172.21.15.67:0/2118974059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faab40d1f30 0x7faab40b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 >> 172.21.15.67:0/2118974059 conn(0x7faab40d3aa0 msgr2=0x7faab415ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 shutdown_connections 2024-04-25T15:29:11.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.540+0000 7faabc34d640 1 -- 172.21.15.67:0/2118974059 wait complete. 2024-04-25T15:29:11.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:29:11.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.666+0000 7f9b4c827640 1 Processor -- start 2024-04-25T15:29:11.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.666+0000 7f9b4c827640 1 -- start start 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4c827640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 0x7f9b4815e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4c827640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b481609b0 con 0x7f9b48163fa0 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4c827640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b48164cd0 con 0x7f9b4806c5c0 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4c827640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b48160c30 con 0x7f9b48161710 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b46d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9b48163fa0 0x7f9b481643b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38372/0 (socket says 172.21.15.67:38372) 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b46d9d640 1 -- 172.21.15.67:0/642508818 learned_addr learned my addr 172.21.15.67:0/642508818 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572247026 0 0) 0x7f9b38001490 con 0x7f9b48163fa0 2024-04-25T15:29:11.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4659c640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 0x7f9b4815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b34003520 con 0x7f9b48163fa0 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4047823643 0 0) 0x7f9b34003520 con 0x7f9b48163fa0 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 msgr2=0x7f9b4815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 0x7f9b4815e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 >> v1:172.21.15.176:6789/0 conn(0x7f9b4806c5c0 legacy=0x7f9b4806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b48160f10 con 0x7f9b48163fa0 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f9b38001bc0 con 0x7f9b48163fa0 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 >> v1:172.21.15.67:6789/0 conn(0x7f9b48163fa0 legacy=0x7f9b481643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 0x7f9b340039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.668+0000 7f9b4559a640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34003fb0 0x7f9b34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b4559a640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b34004d40 0x7f9b340051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b48160c30 con 0x7f9b34004d40 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3400bc50 con 0x7f9b34003fb0 2024-04-25T15:29:11.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3400be50 con 0x7f9b48161710 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b46d9d640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b34004d40 0x7f9b340051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b4659c640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 0x7f9b340039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b45d9b640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34003fb0 0x7f9b34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b46d9d640 1 -- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 msgr2=0x7f9b340039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b46d9d640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 0x7f9b340039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b46d9d640 1 -- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34003fb0 msgr2=0x7f9b34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b46d9d640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34003fb0 0x7f9b34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b46d9d640 1 -- 172.21.15.67:0/642508818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b34003520 con 0x7f9b34004d40 2024-04-25T15:29:11.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.669+0000 7f9b4659c640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b48161710 0x7f9b340039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:11.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.670+0000 7f9b46d9d640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b34004d40 0x7f9b340051b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f9b480b91d0 tx=0x7f9b34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7e130f6c37091 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.670+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800a580 con 0x7f9b34004d40 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.670+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b3802f070 con 0x7f9b34004d40 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.670+0000 7f9b4559a640 1 -- 172.21.15.67:0/642508818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3802bda0 con 0x7f9b34004d40 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.671+0000 7f9b4c827640 1 -- 172.21.15.67:0/642508818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b34004d40 msgr2=0x7f9b340051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.671+0000 7f9b4c827640 1 --2- 172.21.15.67:0/642508818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b34004d40 0x7f9b340051b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f9b480b91d0 tx=0x7f9b34005830 comp rx=0 tx=0).stop 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.671+0000 7f9b46d9d640 1 -- 172.21.15.67:0/642508818 reap_dead start 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.671+0000 7f9b4c827640 1 -- 172.21.15.67:0/642508818 shutdown_connections 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.671+0000 7f9b4c827640 1 -- 172.21.15.67:0/642508818 >> 172.21.15.67:0/642508818 conn(0x7f9b480d5d90 msgr2=0x7f9b480d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.671+0000 7f9b4c827640 1 -- 172.21.15.67:0/642508818 shutdown_connections 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.671+0000 7f9b4c827640 1 -- 172.21.15.67:0/642508818 wait complete. 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.673+0000 7f9b4c827640 1 Processor -- start 2024-04-25T15:29:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.673+0000 7f9b4c827640 1 -- start start 2024-04-25T15:29:11.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4c827640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34004d40 0x7f9b481fa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4c827640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b480cb9c0 0x7f9b480cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4c827640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b481f3880 0x7f9b481f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4c827640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4815e200 con 0x7f9b480cb9c0 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4c827640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4815e080 con 0x7f9b34004d40 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4c827640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4815e380 con 0x7f9b481f3880 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b480cb9c0 0x7f9b480cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b480cb9c0 0x7f9b480cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57750/0 (socket says 172.21.15.67:57750) 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 -- 172.21.15.67:0/1679793502 learned_addr learned my addr 172.21.15.67:0/1679793502 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4659c640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34004d40 0x7f9b481fa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b46d9d640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b481f3880 0x7f9b481f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 -- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b481f3880 msgr2=0x7f9b481f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b481f3880 0x7f9b481f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 -- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34004d40 msgr2=0x7f9b481fa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34004d40 0x7f9b481fa830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b45d9b640 1 -- 172.21.15.67:0/1679793502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b481f41d0 con 0x7f9b480cb9c0 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.674+0000 7f9b4659c640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34004d40 0x7f9b481fa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.675+0000 7f9b45d9b640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b480cb9c0 0x7f9b480cbdd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000f7f0 tx=0x7f9b3000fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.675+0000 7f9b3effd640 1 -- 172.21.15.67:0/1679793502 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b300050d0 con 0x7f9b480cb9c0 2024-04-25T15:29:11.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.675+0000 7f9b3effd640 1 -- 172.21.15.67:0/1679793502 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b30018070 con 0x7f9b480cb9c0 2024-04-25T15:29:11.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.675+0000 7f9b3effd640 1 -- 172.21.15.67:0/1679793502 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30005490 con 0x7f9b480cb9c0 2024-04-25T15:29:11.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.675+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b481f6ec0 con 0x7f9b480cb9c0 2024-04-25T15:29:11.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.675+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b481f7480 con 0x7f9b480cb9c0 2024-04-25T15:29:11.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.676+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b48050740 con 0x7f9b480cb9c0 2024-04-25T15:29:11.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.677+0000 7f9b3effd640 1 -- 172.21.15.67:0/1679793502 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9b3000d730 con 0x7f9b480cb9c0 2024-04-25T15:29:11.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.677+0000 7f9b3effd640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b280744b0 0x7f9b280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.677+0000 7f9b3effd640 1 -- 172.21.15.67:0/1679793502 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f9b30095030 con 0x7f9b480cb9c0 2024-04-25T15:29:11.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.677+0000 7f9b4659c640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b280744b0 0x7f9b280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.678+0000 7f9b4659c640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b280744b0 0x7f9b280769c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9b4808e250 tx=0x7f9b40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.682+0000 7f9b3effd640 1 -- 172.21.15.67:0/1679793502 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b3005f7d0 con 0x7f9b480cb9c0 2024-04-25T15:29:11.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.866+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9b4806cc90 con 0x7f9b480cb9c0 2024-04-25T15:29:11.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.867+0000 7f9b3effd640 1 -- 172.21.15.67:0/1679793502 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f9b3005f0d0 con 0x7f9b480cb9c0 2024-04-25T15:29:11.868 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:29:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b280744b0 msgr2=0x7f9b280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b280744b0 0x7f9b280769c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9b4808e250 tx=0x7f9b40008040 comp rx=0 tx=0).stop 2024-04-25T15:29:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b480cb9c0 msgr2=0x7f9b480cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b480cb9c0 0x7f9b480cbdd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000f7f0 tx=0x7f9b3000fcb0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 shutdown_connections 2024-04-25T15:29:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9b280744b0 0x7f9b280769c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b481f3880 0x7f9b481f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b480cb9c0 0x7f9b480cbdd0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 --2- 172.21.15.67:0/1679793502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b34004d40 0x7f9b481fa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 >> 172.21.15.67:0/1679793502 conn(0x7f9b480d5d90 msgr2=0x7f9b4815e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 shutdown_connections 2024-04-25T15:29:11.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.869+0000 7f9b4c827640 1 -- 172.21.15.67:0/1679793502 wait complete. 2024-04-25T15:29:11.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:29:11.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.994+0000 7f5609185640 1 Processor -- start 2024-04-25T15:29:11.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.994+0000 7f5609185640 1 -- start start 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.995+0000 7f5609185640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56040d0e00 0x7f56040d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f5609185640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f560406c5c0 0x7f560406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f5609185640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56040b7810 0x7f56040b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f5609185640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f560406cf10 con 0x7f56040b7810 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f5609185640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f56040d1850 con 0x7f560406c5c0 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f5609185640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5604160de0 con 0x7f56040d0e00 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56040b7810 0x7f56040b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56040b7810 0x7f56040b7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57754/0 (socket says 172.21.15.67:57754) 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 -- 172.21.15.67:0/3137557075 learned_addr learned my addr 172.21.15.67:0/3137557075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560259c640 1 --2- 172.21.15.67:0/3137557075 >> v2:172.21.15.176:3300/0 conn(0x7f560406c5c0 0x7f560406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f5602d9d640 1 --2- 172.21.15.67:0/3137557075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56040d0e00 0x7f56040d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 -- 172.21.15.67:0/3137557075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56040d0e00 msgr2=0x7f56040d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 --2- 172.21.15.67:0/3137557075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56040d0e00 0x7f56040d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 -- 172.21.15.67:0/3137557075 >> v2:172.21.15.176:3300/0 conn(0x7f560406c5c0 msgr2=0x7f560406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 --2- 172.21.15.67:0/3137557075 >> v2:172.21.15.176:3300/0 conn(0x7f560406c5c0 0x7f560406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f560359e640 1 -- 172.21.15.67:0/3137557075 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56041610c0 con 0x7f56040b7810 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.996+0000 7f5602d9d640 1 --2- 172.21.15.67:0/3137557075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56040d0e00 0x7f56040d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.997+0000 7f560359e640 1 --2- 172.21.15.67:0/3137557075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040b7c20 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f560406f0a0 tx=0x7f55f402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbbc4594dc4961 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.997+0000 7f5601d9b640 1 -- 172.21.15.67:0/3137557075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4036070 con 0x7f56040b7810 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.997+0000 7f5601d9b640 1 -- 172.21.15.67:0/3137557075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f4031600 con 0x7f56040b7810 2024-04-25T15:29:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.997+0000 7f5601d9b640 1 -- 172.21.15.67:0/3137557075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f40319d0 con 0x7f56040b7810 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 -- 172.21.15.67:0/3137557075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 msgr2=0x7f56040b7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 --2- 172.21.15.67:0/3137557075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040b7c20 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f560406f0a0 tx=0x7f55f402a6a0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 -- 172.21.15.67:0/3137557075 shutdown_connections 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 --2- 172.21.15.67:0/3137557075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040b7c20 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 --2- 172.21.15.67:0/3137557075 >> v2:172.21.15.176:3300/0 conn(0x7f560406c5c0 0x7f560406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 --2- 172.21.15.67:0/3137557075 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56040d0e00 0x7f56040d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 -- 172.21.15.67:0/3137557075 >> 172.21.15.67:0/3137557075 conn(0x7f56040d5d90 msgr2=0x7f56040d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 -- 172.21.15.67:0/3137557075 shutdown_connections 2024-04-25T15:29:12.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:11.998+0000 7f5609185640 1 -- 172.21.15.67:0/3137557075 wait complete. 2024-04-25T15:29:12.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.000+0000 7f5609185640 1 Processor -- start 2024-04-25T15:29:12.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.000+0000 7f5609185640 1 -- start start 2024-04-25T15:29:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f5609185640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f560406c5c0 0x7f56040cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f5609185640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f5609185640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56040d0e00 0x7f56040ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f5609185640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56040b5320 con 0x7f56040b7810 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f5609185640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5604160ad0 con 0x7f56040d0e00 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f5609185640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5604164e00 con 0x7f560406c5c0 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f560259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f560259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57768/0 (socket says 172.21.15.67:57768) 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.001+0000 7f560259c640 1 -- 172.21.15.67:0/2730404722 learned_addr learned my addr 172.21.15.67:0/2730404722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f5602d9d640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f560406c5c0 0x7f56040cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f560359e640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56040d0e00 0x7f56040ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f560259c640 1 -- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f560406c5c0 msgr2=0x7f56040cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f560259c640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f560406c5c0 0x7f56040cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f560259c640 1 -- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56040d0e00 msgr2=0x7f56040ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f560259c640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56040d0e00 0x7f56040ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f560259c640 1 -- 172.21.15.67:0/2730404722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56040cff60 con 0x7f56040b7810 2024-04-25T15:29:12.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f560259c640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040ca9b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00d830 tx=0x7f55ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f55fb7fe640 1 -- 172.21.15.67:0/2730404722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec018030 con 0x7f56040b7810 2024-04-25T15:29:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f55fb7fe640 1 -- 172.21.15.67:0/2730404722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55ec00be10 con 0x7f56040b7810 2024-04-25T15:29:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56040cb5c0 con 0x7f56040b7810 2024-04-25T15:29:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f55fb7fe640 1 -- 172.21.15.67:0/2730404722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec0098b0 con 0x7f56040b7810 2024-04-25T15:29:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.002+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56040cbb80 con 0x7f56040b7810 2024-04-25T15:29:12.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.003+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5604050740 con 0x7f56040b7810 2024-04-25T15:29:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.004+0000 7f55fb7fe640 1 -- 172.21.15.67:0/2730404722 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f55ec01b070 con 0x7f56040b7810 2024-04-25T15:29:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.005+0000 7f55fb7fe640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55e40744b0 0x7f55e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.005+0000 7f55fb7fe640 1 -- 172.21.15.67:0/2730404722 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f55ec0953a0 con 0x7f56040b7810 2024-04-25T15:29:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.005+0000 7f5602d9d640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55e40744b0 0x7f55e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.005+0000 7f5602d9d640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55e40744b0 0x7f55e40769c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00a950 tx=0x7f55fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.009+0000 7f55fb7fe640 1 -- 172.21.15.67:0/2730404722 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55ec05fb30 con 0x7f56040b7810 2024-04-25T15:29:12.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.196+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f56040cb350 con 0x7f56040b7810 2024-04-25T15:29:12.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.196+0000 7f55fb7fe640 1 -- 172.21.15.67:0/2730404722 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f55ec05f430 con 0x7f56040b7810 2024-04-25T15:29:12.197 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55e40744b0 msgr2=0x7f55e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55e40744b0 0x7f55e40769c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00a950 tx=0x7f55fc008040 comp rx=0 tx=0).stop 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 msgr2=0x7f56040ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040ca9b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00d830 tx=0x7f55ec00dcf0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 shutdown_connections 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55e40744b0 0x7f55e40769c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56040d0e00 0x7f56040ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56040b7810 0x7f56040ca9b0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 --2- 172.21.15.67:0/2730404722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f560406c5c0 0x7f56040cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 >> 172.21.15.67:0/2730404722 conn(0x7f56040d5d90 msgr2=0x7f56040b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:12.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 shutdown_connections 2024-04-25T15:29:12.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.198+0000 7f5609185640 1 -- 172.21.15.67:0/2730404722 wait complete. 2024-04-25T15:29:12.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:29:12.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.321+0000 7f61a759e640 1 Processor -- start 2024-04-25T15:29:12.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.321+0000 7f61a759e640 1 -- start start 2024-04-25T15:29:12.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a759e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b5260 0x7f61a80b9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a759e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f61a80b96b0 0x7f61a80b7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a759e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a80d0b30 con 0x7f61a80b4da0 2024-04-25T15:29:12.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a759e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f61a80d4420 con 0x7f61a80b96b0 2024-04-25T15:29:12.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a759e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f61a80d4700 con 0x7f61a80b5260 2024-04-25T15:29:12.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f61a80b4da0 0x7f61a81600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38380/0 (socket says 172.21.15.67:38380) 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a659c640 1 -- 172.21.15.67:0/2054169141 learned_addr learned my addr 172.21.15.67:0/2054169141 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a6d9d640 1 --2- 172.21.15.67:0/2054169141 >> v2:172.21.15.176:3300/0 conn(0x7f61a80b96b0 0x7f61a80b7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a5d9b640 1 --2- 172.21.15.67:0/2054169141 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b5260 0x7f61a80b9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a559a640 1 -- 172.21.15.67:0/2054169141 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707218271 0 0) 0x7f61a80d0b30 con 0x7f61a80b4da0 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a559a640 1 -- 172.21.15.67:0/2054169141 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6194003520 con 0x7f61a80b4da0 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a559a640 1 -- 172.21.15.67:0/2054169141 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4238236061 0 0) 0x7f6194003520 con 0x7f61a80b4da0 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a5d9b640 1 -- 172.21.15.67:0/2054169141 >> v2:172.21.15.176:3300/0 conn(0x7f61a80b96b0 msgr2=0x7f61a80b7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a5d9b640 1 --2- 172.21.15.67:0/2054169141 >> v2:172.21.15.176:3300/0 conn(0x7f61a80b96b0 0x7f61a80b7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a5d9b640 1 -- 172.21.15.67:0/2054169141 >> v1:172.21.15.67:6789/0 conn(0x7f61a80b4da0 legacy=0x7f61a81600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a5d9b640 1 -- 172.21.15.67:0/2054169141 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a80d49e0 con 0x7f61a80b5260 2024-04-25T15:29:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.323+0000 7f61a6d9d640 1 --2- 172.21.15.67:0/2054169141 >> v2:172.21.15.176:3300/0 conn(0x7f61a80b96b0 0x7f61a80b7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.324+0000 7f61a5d9b640 1 --2- 172.21.15.67:0/2054169141 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b5260 0x7f61a80b9170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f61900097a0 tx=0x7f619002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=363fedd6e5308018 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.324+0000 7f61a559a640 1 -- 172.21.15.67:0/2054169141 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6190037070 con 0x7f61a80b5260 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a559a640 1 -- 172.21.15.67:0/2054169141 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f619002ab10 con 0x7f61a80b5260 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a559a640 1 -- 172.21.15.67:0/2054169141 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61980019f0 con 0x7f61a80b5260 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 -- 172.21.15.67:0/2054169141 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b5260 msgr2=0x7f61a80b9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 --2- 172.21.15.67:0/2054169141 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b5260 0x7f61a80b9170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f61900097a0 tx=0x7f619002a080 comp rx=0 tx=0).stop 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 -- 172.21.15.67:0/2054169141 shutdown_connections 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 --2- 172.21.15.67:0/2054169141 >> v2:172.21.15.176:3300/0 conn(0x7f61a80b96b0 0x7f61a80b7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 --2- 172.21.15.67:0/2054169141 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b5260 0x7f61a80b9170 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 -- 172.21.15.67:0/2054169141 >> 172.21.15.67:0/2054169141 conn(0x7f61a80d2bd0 msgr2=0x7f61a80d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 -- 172.21.15.67:0/2054169141 shutdown_connections 2024-04-25T15:29:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.325+0000 7f61a759e640 1 -- 172.21.15.67:0/2054169141 wait complete. 2024-04-25T15:29:12.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.327+0000 7f61a759e640 1 Processor -- start 2024-04-25T15:29:12.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.327+0000 7f61a759e640 1 -- start start 2024-04-25T15:29:12.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a759e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b4da0 0x7f61a80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a759e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61a80b5260 0x7f61a815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a80b96b0 0x7f61a80ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a759e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61a80d08d0 con 0x7f61a80b96b0 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a759e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f61a80d0750 con 0x7f61a80b5260 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a759e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f61a80b8100 con 0x7f61a80b4da0 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a80b96b0 0x7f61a80ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a80b96b0 0x7f61a80ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57776/0 (socket says 172.21.15.67:57776) 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 -- 172.21.15.67:0/1874311536 learned_addr learned my addr 172.21.15.67:0/1874311536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a5d9b640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61a80b5260 0x7f61a815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a659c640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b4da0 0x7f61a80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 -- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b4da0 msgr2=0x7f61a80cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b4da0 0x7f61a80cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 -- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61a80b5260 msgr2=0x7f61a815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61a80b5260 0x7f61a815eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 -- 172.21.15.67:0/1874311536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a80acc20 con 0x7f61a80b96b0 2024-04-25T15:29:12.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.329+0000 7f61a6d9d640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a80b96b0 0x7f61a80ce980 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f61a000e850 tx=0x7f61a000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.330+0000 7f619effd640 1 -- 172.21.15.67:0/1874311536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0017070 con 0x7f61a80b96b0 2024-04-25T15:29:12.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.330+0000 7f619effd640 1 -- 172.21.15.67:0/1874311536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a000c940 con 0x7f61a80b96b0 2024-04-25T15:29:12.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.330+0000 7f619effd640 1 -- 172.21.15.67:0/1874311536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a000ccf0 con 0x7f61a80b96b0 2024-04-25T15:29:12.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.330+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a80cdea0 con 0x7f61a80b96b0 2024-04-25T15:29:12.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.330+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a80cc5a0 con 0x7f61a80b96b0 2024-04-25T15:29:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.331+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a80cdc90 con 0x7f61a80b96b0 2024-04-25T15:29:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.331+0000 7f619effd640 1 -- 172.21.15.67:0/1874311536 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f61a001a070 con 0x7f61a80b96b0 2024-04-25T15:29:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.332+0000 7f619effd640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61880744b0 0x7f61880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.332+0000 7f619effd640 1 -- 172.21.15.67:0/1874311536 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f61a00952b0 con 0x7f61a80b96b0 2024-04-25T15:29:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.332+0000 7f61a659c640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61880744b0 0x7f61880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.333+0000 7f61a659c640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61880744b0 0x7f61880769c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6198005c00 tx=0x7f6198009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.336+0000 7f619effd640 1 -- 172.21.15.67:0/1874311536 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61a005f510 con 0x7f61a80b96b0 2024-04-25T15:29:12.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.534+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f61a8050740 con 0x7f61a80b96b0 2024-04-25T15:29:12.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.535+0000 7f619effd640 1 -- 172.21.15.67:0/1874311536 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v92) ==== 80+0+0 (secure 0 0 0) 0x7f61a00950d0 con 0x7f61a80b96b0 2024-04-25T15:29:12.535 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61880744b0 msgr2=0x7f61880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61880744b0 0x7f61880769c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6198005c00 tx=0x7f6198009040 comp rx=0 tx=0).stop 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a80b96b0 msgr2=0x7f61a80ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a80b96b0 0x7f61a80ce980 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f61a000e850 tx=0x7f61a000ed10 comp rx=0 tx=0).stop 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 shutdown_connections 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f61880744b0 0x7f61880769c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61a80b96b0 0x7f61a80ce980 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61a80b5260 0x7f61a815eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 --2- 172.21.15.67:0/1874311536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61a80b4da0 0x7f61a80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 >> 172.21.15.67:0/1874311536 conn(0x7f61a80d2bd0 msgr2=0x7f61a80b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:12.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 shutdown_connections 2024-04-25T15:29:12.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.537+0000 7f61a759e640 1 -- 172.21.15.67:0/1874311536 wait complete. 2024-04-25T15:29:12.548 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T15:29:12.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:12.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T15:29:12.576 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:29:12.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:12.577 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:29:12.632 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:29:12.632 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:12.632 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:12.651 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:12.651 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:12.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:29:12.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.821+0000 7f89f5fa8640 1 Processor -- start 2024-04-25T15:29:12.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.821+0000 7f89f5fa8640 1 -- start start 2024-04-25T15:29:12.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89f5fa8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f0164b00 0x7f89f00b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89f5fa8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89f015df80 0x7f89f00b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89f5fa8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89f01615f0 con 0x7f89f0160890 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89f5fa8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89f01613b0 con 0x7f89f015df80 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89f5fa8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89f0164670 con 0x7f89f0164b00 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89eeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89f015df80 0x7f89f00b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89eeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89f015df80 0x7f89f00b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57790/0 (socket says 172.21.15.67:57790) 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89eeffd640 1 -- 172.21.15.67:0/2648053684 learned_addr learned my addr 172.21.15.67:0/2648053684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.822+0000 7f89ef7fe640 1 --2- 172.21.15.67:0/2648053684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f0164b00 0x7f89f00b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.823+0000 7f89eeffd640 1 -- 172.21.15.67:0/2648053684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f0164b00 msgr2=0x7f89f00b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.823+0000 7f89eeffd640 1 --2- 172.21.15.67:0/2648053684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f0164b00 0x7f89f00b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.823+0000 7f89eeffd640 1 -- 172.21.15.67:0/2648053684 >> v1:172.21.15.176:6789/0 conn(0x7f89f0160890 legacy=0x7f89f0160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.823+0000 7f89eeffd640 1 -- 172.21.15.67:0/2648053684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f0164020 con 0x7f89f015df80 2024-04-25T15:29:12.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.823+0000 7f89eeffd640 1 --2- 172.21.15.67:0/2648053684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f015df80 0x7f89f00b80c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e761414026845bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.823+0000 7f89ee7fc640 1 -- 172.21.15.67:0/2648053684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4037070 con 0x7f89f015df80 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89ee7fc640 1 -- 172.21.15.67:0/2648053684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89e4032440 con 0x7f89f015df80 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89ee7fc640 1 -- 172.21.15.67:0/2648053684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4032810 con 0x7f89f015df80 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2648053684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f015df80 msgr2=0x7f89f00b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2648053684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f015df80 0x7f89f00b80c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009870 tx=0x7f89e402a210 comp rx=0 tx=0).stop 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2648053684 shutdown_connections 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2648053684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f015df80 0x7f89f00b80c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2648053684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f0164b00 0x7f89f00b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2648053684 >> 172.21.15.67:0/2648053684 conn(0x7f89f00d2bc0 msgr2=0x7f89f00d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2648053684 shutdown_connections 2024-04-25T15:29:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.824+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2648053684 wait complete. 2024-04-25T15:29:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.826+0000 7f89f5fa8640 1 Processor -- start 2024-04-25T15:29:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.826+0000 7f89f5fa8640 1 -- start start 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89f5fa8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f015df80 0x7f89f01536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89f5fa8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89f0160890 0x7f89f0158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89f5fa8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f0164b00 0x7f89f01ea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89f5fa8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89f00b4a00 con 0x7f89f0164b00 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89f5fa8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89f00b4880 con 0x7f89f0160890 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89f5fa8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89f00b4b80 con 0x7f89f015df80 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f0164b00 0x7f89f01ea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f0164b00 0x7f89f01ea2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57812/0 (socket says 172.21.15.67:57812) 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 -- 172.21.15.67:0/2193226861 learned_addr learned my addr 172.21.15.67:0/2193226861 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89eeffd640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89f0160890 0x7f89f0158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89ef7fe640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f015df80 0x7f89f01536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 -- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f015df80 msgr2=0x7f89f01536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f015df80 0x7f89f01536e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 -- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89f0160890 msgr2=0x7f89f0158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89f0160890 0x7f89f0158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 -- 172.21.15.67:0/2193226861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f01628b0 con 0x7f89f0164b00 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.828+0000 7f89effff640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f0164b00 0x7f89f01ea2f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f89e800be60 tx=0x7f89e800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.829+0000 7f89d7fff640 1 -- 172.21.15.67:0/2193226861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8019070 con 0x7f89f0164b00 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.829+0000 7f89d7fff640 1 -- 172.21.15.67:0/2193226861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89e8014440 con 0x7f89f0164b00 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.829+0000 7f89d7fff640 1 -- 172.21.15.67:0/2193226861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8014830 con 0x7f89f0164b00 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.829+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f0162b90 con 0x7f89f0164b00 2024-04-25T15:29:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.829+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f01f6b30 con 0x7f89f0164b00 2024-04-25T15:29:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.830+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89f01f6960 con 0x7f89f0164b00 2024-04-25T15:29:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.830+0000 7f89d7fff640 1 -- 172.21.15.67:0/2193226861 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f89e801c070 con 0x7f89f0164b00 2024-04-25T15:29:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.831+0000 7f89d7fff640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89dc0744e0 0x7f89dc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.831+0000 7f89d7fff640 1 -- 172.21.15.67:0/2193226861 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f89e8095040 con 0x7f89f0164b00 2024-04-25T15:29:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.831+0000 7f89ef7fe640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89dc0744e0 0x7f89dc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.832+0000 7f89ef7fe640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89dc0744e0 0x7f89dc0769f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f89e00079a0 tx=0x7f89e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:12.835+0000 7f89d7fff640 1 -- 172.21.15.67:0/2193226861 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89e805f200 con 0x7f89f0164b00 2024-04-25T15:29:13.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.022+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89f0050740 con 0x7f89f0164b00 2024-04-25T15:29:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.024+0000 7f89d7fff640 1 -- 172.21.15.67:0/2193226861 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16572 (secure 0 0 0) 0x7f89e8063500 con 0x7f89f0164b00 2024-04-25T15:29:13.129 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:13.129 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:58.058412+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.025+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89dc0744e0 msgr2=0x7f89dc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.025+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89dc0744e0 0x7f89dc0769f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f89e00079a0 tx=0x7f89e0008040 comp rx=0 tx=0).stop 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f0164b00 msgr2=0x7f89f01ea2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f0164b00 0x7f89f01ea2f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f89e800be60 tx=0x7f89e800bf50 comp rx=0 tx=0).stop 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 shutdown_connections 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89dc0744e0 0x7f89dc0769f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89f0164b00 0x7f89f01ea2f0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89f0160890 0x7f89f0158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 --2- 172.21.15.67:0/2193226861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89f015df80 0x7f89f01536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 >> 172.21.15.67:0/2193226861 conn(0x7f89f00d2bc0 msgr2=0x7f89f015f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 shutdown_connections 2024-04-25T15:29:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.026+0000 7f89f5fa8640 1 -- 172.21.15.67:0/2193226861 wait complete. 2024-04-25T15:29:13.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:28:54.107977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:28:55.386227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:29:13.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:29:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.246+0000 7fab7889a640 1 Processor -- start 2024-04-25T15:29:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.246+0000 7fab7889a640 1 -- start start 2024-04-25T15:29:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.247+0000 7fab7889a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.247+0000 7fab7889a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab74163f30 0x7fab74164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.247+0000 7fab7889a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab74164880 con 0x7fab740d1f30 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.247+0000 7fab7889a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fab74164ee0 con 0x7fab74163f30 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.247+0000 7fab7889a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fab740b31e0 con 0x7fab74164ad0 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab71d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab74163f30 0x7fab74164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab71d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab74163f30 0x7fab74164340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57814/0 (socket says 172.21.15.67:57814) 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab71d9b640 1 -- 172.21.15.67:0/3287423256 learned_addr learned my addr 172.21.15.67:0/3287423256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab72d9d640 1 --1- 172.21.15.67:0/3287423256 >> v1:172.21.15.67:6789/0 conn(0x7fab740d1f30 0x7fab740d2340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38384/0 (socket says 172.21.15.67:38384) 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab7259c640 1 --2- 172.21.15.67:0/3287423256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab71d9b640 1 -- 172.21.15.67:0/3287423256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 msgr2=0x7fab740b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab71d9b640 1 --2- 172.21.15.67:0/3287423256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab71d9b640 1 -- 172.21.15.67:0/3287423256 >> v1:172.21.15.67:6789/0 conn(0x7fab740d1f30 legacy=0x7fab740d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab7159a640 1 -- 172.21.15.67:0/3287423256 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1226457286 0 0) 0x7fab6c001490 con 0x7fab740d1f30 2024-04-25T15:29:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.248+0000 7fab71d9b640 1 -- 172.21.15.67:0/3287423256 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab740d3170 con 0x7fab74163f30 2024-04-25T15:29:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.249+0000 7fab71d9b640 1 --2- 172.21.15.67:0/3287423256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab74164340 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fab68009870 tx=0x7fab6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b2a56d644c8cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.249+0000 7fab7159a640 1 -- 172.21.15.67:0/3287423256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68037070 con 0x7fab74163f30 2024-04-25T15:29:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.249+0000 7fab7159a640 1 -- 172.21.15.67:0/3287423256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab68032440 con 0x7fab74163f30 2024-04-25T15:29:13.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.249+0000 7fab7159a640 1 -- 172.21.15.67:0/3287423256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68032810 con 0x7fab74163f30 2024-04-25T15:29:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.249+0000 7fab7889a640 1 -- 172.21.15.67:0/3287423256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 msgr2=0x7fab74164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.249+0000 7fab7889a640 1 --2- 172.21.15.67:0/3287423256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab74164340 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fab68009870 tx=0x7fab6802a270 comp rx=0 tx=0).stop 2024-04-25T15:29:13.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.250+0000 7fab7889a640 1 -- 172.21.15.67:0/3287423256 shutdown_connections 2024-04-25T15:29:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.250+0000 7fab7889a640 1 --2- 172.21.15.67:0/3287423256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab74164340 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.250+0000 7fab7889a640 1 --2- 172.21.15.67:0/3287423256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.250+0000 7fab7889a640 1 -- 172.21.15.67:0/3287423256 >> 172.21.15.67:0/3287423256 conn(0x7fab740d3aa0 msgr2=0x7fab740d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.250+0000 7fab7889a640 1 -- 172.21.15.67:0/3287423256 shutdown_connections 2024-04-25T15:29:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.250+0000 7fab7889a640 1 -- 172.21.15.67:0/3287423256 wait complete. 2024-04-25T15:29:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.252+0000 7fab7889a640 1 Processor -- start 2024-04-25T15:29:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.252+0000 7fab7889a640 1 -- start start 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7889a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab740d1f30 0x7fab7415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7889a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab740b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7889a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7889a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab7406d9b0 con 0x7fab74163f30 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7889a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fab7406d6d0 con 0x7fab740d1f30 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7889a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fab740b9430 con 0x7fab74164ad0 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab740b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab740b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57828/0 (socket says 172.21.15.67:57828) 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 -- 172.21.15.67:0/3380714130 learned_addr learned my addr 172.21.15.67:0/3380714130 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7259c640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab740d1f30 0x7fab7415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab72d9d640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 -- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 msgr2=0x7fab740b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 -- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab740d1f30 msgr2=0x7fab7415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab740d1f30 0x7fab7415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 -- 172.21.15.67:0/3380714130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7415e860 con 0x7fab74163f30 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab7259c640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab740d1f30 0x7fab7415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.253+0000 7fab71d9b640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab740b3d80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fab68009870 tx=0x7fab680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.254+0000 7fab62ffd640 1 -- 172.21.15.67:0/3380714130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68041020 con 0x7fab74163f30 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.254+0000 7fab62ffd640 1 -- 172.21.15.67:0/3380714130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab68030640 con 0x7fab74163f30 2024-04-25T15:29:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.254+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7415d800 con 0x7fab74163f30 2024-04-25T15:29:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.254+0000 7fab62ffd640 1 -- 172.21.15.67:0/3380714130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68037040 con 0x7fab74163f30 2024-04-25T15:29:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.254+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7415dfc0 con 0x7fab74163f30 2024-04-25T15:29:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.255+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab740d4770 con 0x7fab74163f30 2024-04-25T15:29:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.255+0000 7fab62ffd640 1 -- 172.21.15.67:0/3380714130 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fab68044020 con 0x7fab74163f30 2024-04-25T15:29:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.256+0000 7fab62ffd640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab540744b0 0x7fab540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.256+0000 7fab62ffd640 1 -- 172.21.15.67:0/3380714130 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fab680b5960 con 0x7fab74163f30 2024-04-25T15:29:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.256+0000 7fab7259c640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab540744b0 0x7fab540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.257+0000 7fab7259c640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab540744b0 0x7fab540769c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fab740746e0 tx=0x7fab64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.261+0000 7fab62ffd640 1 -- 172.21.15.67:0/3380714130 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab6804b030 con 0x7fab74163f30 2024-04-25T15:29:13.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.444+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fab7415da70 con 0x7fab74163f30 2024-04-25T15:29:13.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.444+0000 7fab62ffd640 1 -- 172.21.15.67:0/3380714130 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fab7415d800 con 0x7fab74163f30 2024-04-25T15:29:13.445 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:29:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab540744b0 msgr2=0x7fab540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab540744b0 0x7fab540769c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fab740746e0 tx=0x7fab64008040 comp rx=0 tx=0).stop 2024-04-25T15:29:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 msgr2=0x7fab740b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab740b3d80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fab68009870 tx=0x7fab680046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 shutdown_connections 2024-04-25T15:29:13.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fab540744b0 0x7fab540769c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab74164ad0 0x7fab740b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab74163f30 0x7fab740b3d80 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 --2- 172.21.15.67:0/3380714130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab740d1f30 0x7fab7415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 >> 172.21.15.67:0/3380714130 conn(0x7fab740d3aa0 msgr2=0x7fab740d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 shutdown_connections 2024-04-25T15:29:13.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.446+0000 7fab7889a640 1 -- 172.21.15.67:0/3380714130 wait complete. 2024-04-25T15:29:13.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:29:13.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.571+0000 7f3d1751f640 1 Processor -- start 2024-04-25T15:29:13.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.571+0000 7f3d1751f640 1 -- start start 2024-04-25T15:29:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.572+0000 7f3d1751f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3d1006dce0 0x7f3d100704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d1751f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d1015adf0 0x7f3d100cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d1751f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d1006fa20 con 0x7f3d100b0c90 2024-04-25T15:29:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d1751f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d10070fd0 con 0x7f3d1015adf0 2024-04-25T15:29:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d1751f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3d100712b0 con 0x7f3d1006dce0 2024-04-25T15:29:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d15abc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d1015adf0 0x7f3d100cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d15abc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d1015adf0 0x7f3d100cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57842/0 (socket says 172.21.15.67:57842) 2024-04-25T15:29:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d15abc640 1 -- 172.21.15.67:0/2822268824 learned_addr learned my addr 172.21.15.67:0/2822268824 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d14aba640 1 --2- 172.21.15.67:0/2822268824 >> v2:172.21.15.176:3300/0 conn(0x7f3d1006dce0 0x7f3d100704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d15abc640 1 -- 172.21.15.67:0/2822268824 >> v2:172.21.15.176:3300/0 conn(0x7f3d1006dce0 msgr2=0x7f3d100704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d15abc640 1 --2- 172.21.15.67:0/2822268824 >> v2:172.21.15.176:3300/0 conn(0x7f3d1006dce0 0x7f3d100704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d15abc640 1 -- 172.21.15.67:0/2822268824 >> v1:172.21.15.176:6789/0 conn(0x7f3d100b0c90 legacy=0x7f3d1006d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.573+0000 7f3d15abc640 1 -- 172.21.15.67:0/2822268824 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1006fcc0 con 0x7f3d1015adf0 2024-04-25T15:29:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.574+0000 7f3d15abc640 1 --2- 172.21.15.67:0/2822268824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d1015adf0 0x7f3d100cec50 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3d10074530 tx=0x7f3d0002c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e48e8f3306141f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.574+0000 7f3d0ffff640 1 -- 172.21.15.67:0/2822268824 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000b2e0 con 0x7f3d1015adf0 2024-04-25T15:29:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.574+0000 7f3d0ffff640 1 -- 172.21.15.67:0/2822268824 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d00037070 con 0x7f3d1015adf0 2024-04-25T15:29:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.574+0000 7f3d0ffff640 1 -- 172.21.15.67:0/2822268824 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000b480 con 0x7f3d1015adf0 2024-04-25T15:29:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 -- 172.21.15.67:0/2822268824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d1015adf0 msgr2=0x7f3d100cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2822268824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d1015adf0 0x7f3d100cec50 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3d10074530 tx=0x7f3d0002c940 comp rx=0 tx=0).stop 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 -- 172.21.15.67:0/2822268824 shutdown_connections 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2822268824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d1015adf0 0x7f3d100cec50 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2822268824 >> v2:172.21.15.176:3300/0 conn(0x7f3d1006dce0 0x7f3d100704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 -- 172.21.15.67:0/2822268824 >> 172.21.15.67:0/2822268824 conn(0x7f3d10073090 msgr2=0x7f3d100734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 -- 172.21.15.67:0/2822268824 shutdown_connections 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.575+0000 7f3d1751f640 1 -- 172.21.15.67:0/2822268824 wait complete. 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.576+0000 7f3d1751f640 1 Processor -- start 2024-04-25T15:29:13.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.577+0000 7f3d1751f640 1 -- start start 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d1751f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d100b0c90 0x7f3d10150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d1751f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d1015adf0 0x7f3d1015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d1751f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d100b5180 0x7f3d1015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d1751f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1015d790 con 0x7f3d100b5180 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d1751f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d10070c80 con 0x7f3d100b0c90 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d1751f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1015d910 con 0x7f3d1015adf0 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d100b5180 0x7f3d1015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d100b5180 0x7f3d1015ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57854/0 (socket says 172.21.15.67:57854) 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 -- 172.21.15.67:0/2270096895 learned_addr learned my addr 172.21.15.67:0/2270096895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d14aba640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d1015adf0 0x7f3d1015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d152bb640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d100b0c90 0x7f3d10150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 -- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d1015adf0 msgr2=0x7f3d1015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d1015adf0 0x7f3d1015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 -- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d100b0c90 msgr2=0x7f3d10150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d100b0c90 0x7f3d10150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 -- 172.21.15.67:0/2270096895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1015c260 con 0x7f3d100b5180 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.578+0000 7f3d15abc640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d100b5180 0x7f3d1015ef70 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002cf30 tx=0x7f3d00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.579+0000 7f3d0dffb640 1 -- 172.21.15.67:0/2270096895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000b720 con 0x7f3d100b5180 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.579+0000 7f3d0dffb640 1 -- 172.21.15.67:0/2270096895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d00032860 con 0x7f3d100b5180 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.579+0000 7f3d0dffb640 1 -- 172.21.15.67:0/2270096895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00032c50 con 0x7f3d100b5180 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.579+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1015c540 con 0x7f3d100b5180 2024-04-25T15:29:13.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.579+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1015cb00 con 0x7f3d100b5180 2024-04-25T15:29:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.580+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d10050740 con 0x7f3d100b5180 2024-04-25T15:29:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.581+0000 7f3d0dffb640 1 -- 172.21.15.67:0/2270096895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d0003c400 con 0x7f3d100b5180 2024-04-25T15:29:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.581+0000 7f3d0dffb640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3cfc0744e0 0x7f3cfc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.581+0000 7f3d0dffb640 1 -- 172.21.15.67:0/2270096895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f3d000b8610 con 0x7f3d100b5180 2024-04-25T15:29:13.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.581+0000 7f3d152bb640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3cfc0744e0 0x7f3cfc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.582+0000 7f3d152bb640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3cfc0744e0 0x7f3cfc0769f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3d08000f60 tx=0x7f3d08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.586+0000 7f3d0dffb640 1 -- 172.21.15.67:0/2270096895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d000827f0 con 0x7f3d100b5180 2024-04-25T15:29:13.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.770+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3d100b4d90 con 0x7f3d100b5180 2024-04-25T15:29:13.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.770+0000 7f3d0dffb640 1 -- 172.21.15.67:0/2270096895 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f3d00086af0 con 0x7f3d100b5180 2024-04-25T15:29:13.771 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:29:13.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3cfc0744e0 msgr2=0x7f3cfc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3cfc0744e0 0x7f3cfc0769f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3d08000f60 tx=0x7f3d08009040 comp rx=0 tx=0).stop 2024-04-25T15:29:13.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d100b5180 msgr2=0x7f3d1015ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d100b5180 0x7f3d1015ef70 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3d0002cf30 tx=0x7f3d00004a40 comp rx=0 tx=0).stop 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 shutdown_connections 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f3cfc0744e0 0x7f3cfc0769f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d100b5180 0x7f3d1015ef70 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d1015adf0 0x7f3d1015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 --2- 172.21.15.67:0/2270096895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d100b0c90 0x7f3d10150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 >> 172.21.15.67:0/2270096895 conn(0x7f3d10073090 msgr2=0x7f3d1006d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 shutdown_connections 2024-04-25T15:29:13.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.772+0000 7f3d1751f640 1 -- 172.21.15.67:0/2270096895 wait complete. 2024-04-25T15:29:13.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:29:13.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.898+0000 7f061ec77640 1 Processor -- start 2024-04-25T15:29:13.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.898+0000 7f061ec77640 1 -- start start 2024-04-25T15:29:13.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.899+0000 7f061ec77640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06180d11f0 0x7f06180d5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.899+0000 7f061ec77640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06180d1830 con 0x7f061806ccd0 2024-04-25T15:29:13.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.899+0000 7f061ec77640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0618160dc0 con 0x7f06180d0de0 2024-04-25T15:29:13.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.899+0000 7f061ec77640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f06180d1600 con 0x7f06180d11f0 2024-04-25T15:29:13.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.899+0000 7f061d214640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06180d11f0 0x7f06180d5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.899+0000 7f0617fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f061806ccd0 0x7f06180d4a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38398/0 (socket says 172.21.15.67:38398) 2024-04-25T15:29:13.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.899+0000 7f0617fff640 1 -- 172.21.15.67:0/2854668903 learned_addr learned my addr 172.21.15.67:0/2854668903 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:13.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.900+0000 7f061d214640 1 -- 172.21.15.67:0/2854668903 >> v1:172.21.15.176:6789/0 conn(0x7f06180d0de0 legacy=0x7f061806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:13.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.900+0000 7f061d214640 1 -- 172.21.15.67:0/2854668903 >> v1:172.21.15.67:6789/0 conn(0x7f061806ccd0 legacy=0x7f06180d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.900+0000 7f061d214640 1 -- 172.21.15.67:0/2854668903 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06181610a0 con 0x7f06180d11f0 2024-04-25T15:29:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.900+0000 7f061d214640 1 --2- 172.21.15.67:0/2854668903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180d11f0 0x7f06180d5140 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f0610009d90 tx=0x7f061002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c593f9fdd66a9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.900+0000 7f06177fe640 1 -- 172.21.15.67:0/2854668903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610031600 con 0x7f06180d11f0 2024-04-25T15:29:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f06177fe640 1 -- 172.21.15.67:0/2854668903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06100317a0 con 0x7f06180d11f0 2024-04-25T15:29:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f06177fe640 1 -- 172.21.15.67:0/2854668903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610031b70 con 0x7f06180d11f0 2024-04-25T15:29:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f061ec77640 1 -- 172.21.15.67:0/2854668903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180d11f0 msgr2=0x7f06180d5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f061ec77640 1 --2- 172.21.15.67:0/2854668903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180d11f0 0x7f06180d5140 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f0610009d90 tx=0x7f061002a610 comp rx=0 tx=0).stop 2024-04-25T15:29:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f061ec77640 1 -- 172.21.15.67:0/2854668903 shutdown_connections 2024-04-25T15:29:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f061ec77640 1 --2- 172.21.15.67:0/2854668903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06180d11f0 0x7f06180d5140 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f061ec77640 1 -- 172.21.15.67:0/2854668903 >> 172.21.15.67:0/2854668903 conn(0x7f06180d5d70 msgr2=0x7f06180d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f061ec77640 1 -- 172.21.15.67:0/2854668903 shutdown_connections 2024-04-25T15:29:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.901+0000 7f061ec77640 1 -- 172.21.15.67:0/2854668903 wait complete. 2024-04-25T15:29:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.903+0000 7f061ec77640 1 Processor -- start 2024-04-25T15:29:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.903+0000 7f061ec77640 1 -- start start 2024-04-25T15:29:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.904+0000 7f061ec77640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180d0de0 0x7f0618158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.904+0000 7f061ec77640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06180d11f0 0x7f0618154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.904+0000 7f061ec77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618156100 0x7f061815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.904+0000 7f061ec77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06180b5300 con 0x7f0618156100 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.904+0000 7f061ec77640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0618160ab0 con 0x7f06180d11f0 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.904+0000 7f061ec77640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0618164de0 con 0x7f06180d0de0 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618156100 0x7f061815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618156100 0x7f061815ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57882/0 (socket says 172.21.15.67:57882) 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 -- 172.21.15.67:0/2553434878 learned_addr learned my addr 172.21.15.67:0/2553434878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061ca13640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180d0de0 0x7f0618158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f0617fff640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06180d11f0 0x7f0618154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 -- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180d0de0 msgr2=0x7f0618158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180d0de0 0x7f0618158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 -- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06180d11f0 msgr2=0x7f0618154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06180d11f0 0x7f0618154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 -- 172.21.15.67:0/2553434878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06180cd390 con 0x7f0618156100 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061ca13640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180d0de0 0x7f0618158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f0617fff640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06180d11f0 0x7f0618154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061d214640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618156100 0x7f061815ec50 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0610009eb0 tx=0x7f0610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f06157fa640 1 -- 172.21.15.67:0/2553434878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061003f070 con 0x7f0618156100 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f06157fa640 1 -- 172.21.15.67:0/2553434878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0610030be0 con 0x7f0618156100 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f06157fa640 1 -- 172.21.15.67:0/2553434878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610009450 con 0x7f0618156100 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06180cd670 con 0x7f0618156100 2024-04-25T15:29:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.905+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06180cdc30 con 0x7f0618156100 2024-04-25T15:29:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.906+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06180cd100 con 0x7f0618156100 2024-04-25T15:29:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.907+0000 7f06157fa640 1 -- 172.21.15.67:0/2553434878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f061003b410 con 0x7f0618156100 2024-04-25T15:29:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.907+0000 7f06157fa640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.908+0000 7f06157fa640 1 -- 172.21.15.67:0/2553434878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f06100b6680 con 0x7f0618156100 2024-04-25T15:29:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.908+0000 7f061ca13640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.908+0000 7f061ca13640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f060c00a830 tx=0x7f060c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:13.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:13.912+0000 7f06157fa640 1 -- 172.21.15.67:0/2553434878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0610080e10 con 0x7f0618156100 2024-04-25T15:29:14.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.097+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f06180d4300 con 0x7f0618156100 2024-04-25T15:29:14.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.097+0000 7f06157fa640 1 -- 172.21.15.67:0/2553434878 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f0610080710 con 0x7f0618156100 2024-04-25T15:29:14.098 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:29:14.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 msgr2=0x7f05fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f060c00a830 tx=0x7f060c008040 comp rx=0 tx=0).stop 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618156100 msgr2=0x7f061815ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618156100 0x7f061815ec50 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f0610009eb0 tx=0x7f0610004a40 comp rx=0 tx=0).stop 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 shutdown_connections 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f05fc0744b0 0x7f05fc0769c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0618156100 0x7f061815ec50 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06180d11f0 0x7f0618154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 --2- 172.21.15.67:0/2553434878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180d0de0 0x7f0618158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 >> 172.21.15.67:0/2553434878 conn(0x7f06180d5d70 msgr2=0x7f0618162080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 shutdown_connections 2024-04-25T15:29:14.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.099+0000 7f061ec77640 1 -- 172.21.15.67:0/2553434878 wait complete. 2024-04-25T15:29:14.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:29:14.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.223+0000 7f955c7a8640 1 Processor -- start 2024-04-25T15:29:14.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.223+0000 7f955c7a8640 1 -- start start 2024-04-25T15:29:14.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955c7a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95540d0ed0 0x7f95540d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955c7a8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9554160be0 0x7f9554160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955c7a8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9554161530 con 0x7f95541434d0 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955c7a8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f95541438e0 con 0x7f9554160be0 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955c7a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f955406f3e0 con 0x7f95540d0ed0 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955a544640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f95541434d0 0x7f9554070480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38408/0 (socket says 172.21.15.67:38408) 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955a544640 1 -- 172.21.15.67:0/2917523484 learned_addr learned my addr 172.21.15.67:0/2917523484 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955ad45640 1 --2- 172.21.15.67:0/2917523484 >> v2:172.21.15.176:3300/0 conn(0x7f9554160be0 0x7f9554160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f9559d43640 1 --2- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95540d0ed0 0x7f95540d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f9559542640 1 -- 172.21.15.67:0/2917523484 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3580454313 0 0) 0x7f954c001110 con 0x7f95541434d0 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f9559542640 1 -- 172.21.15.67:0/2917523484 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9540003520 con 0x7f95541434d0 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955ad45640 1 -- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95540d0ed0 msgr2=0x7f95540d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955ad45640 1 --2- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95540d0ed0 0x7f95540d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955ad45640 1 -- 172.21.15.67:0/2917523484 >> v1:172.21.15.67:6789/0 conn(0x7f95541434d0 legacy=0x7f9554070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.225+0000 7f955ad45640 1 -- 172.21.15.67:0/2917523484 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955406f6c0 con 0x7f9554160be0 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.226+0000 7f955ad45640 1 --2- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9554160be0 0x7f9554160ff0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9550009d90 tx=0x7f955002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c08180e76a68cdb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f9559542640 1 -- 172.21.15.67:0/2917523484 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550007740 con 0x7f9554160be0 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f9559542640 1 -- 172.21.15.67:0/2917523484 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95500078e0 con 0x7f9554160be0 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f9559542640 1 -- 172.21.15.67:0/2917523484 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550007cb0 con 0x7f9554160be0 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 -- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9554160be0 msgr2=0x7f9554160ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 --2- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9554160be0 0x7f9554160ff0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9550009d90 tx=0x7f955002a660 comp rx=0 tx=0).stop 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 -- 172.21.15.67:0/2917523484 shutdown_connections 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 --2- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9554160be0 0x7f9554160ff0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 --2- 172.21.15.67:0/2917523484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95540d0ed0 0x7f95540d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 -- 172.21.15.67:0/2917523484 >> 172.21.15.67:0/2917523484 conn(0x7f95540d1ae0 msgr2=0x7f95540d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 -- 172.21.15.67:0/2917523484 shutdown_connections 2024-04-25T15:29:14.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.227+0000 7f955c7a8640 1 -- 172.21.15.67:0/2917523484 wait complete. 2024-04-25T15:29:14.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.229+0000 7f955c7a8640 1 Processor -- start 2024-04-25T15:29:14.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.229+0000 7f955c7a8640 1 -- start start 2024-04-25T15:29:14.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.230+0000 7f955c7a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540d0ed0 0x7f95541573f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.230+0000 7f955c7a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95541434d0 0x7f9554159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.230+0000 7f955c7a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9554160be0 0x7f955406ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.230+0000 7f955c7a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f955406f090 con 0x7f95541434d0 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.230+0000 7f955c7a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f955406fc80 con 0x7f95540d0ed0 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.230+0000 7f955c7a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f95540b51a0 con 0x7f9554160be0 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95541434d0 0x7f9554159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95541434d0 0x7f9554159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57898/0 (socket says 172.21.15.67:57898) 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 -- 172.21.15.67:0/1729403028 learned_addr learned my addr 172.21.15.67:0/1729403028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f955a544640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540d0ed0 0x7f95541573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f955ad45640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9554160be0 0x7f955406ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 -- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9554160be0 msgr2=0x7f955406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9554160be0 0x7f955406ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 -- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540d0ed0 msgr2=0x7f95541573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540d0ed0 0x7f95541573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 -- 172.21.15.67:0/1729403028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9554157930 con 0x7f95541434d0 2024-04-25T15:29:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f9559d43640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95541434d0 0x7f9554159250 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f954400e870 tx=0x7f954400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:14.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f954affd640 1 -- 172.21.15.67:0/1729403028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544017070 con 0x7f95541434d0 2024-04-25T15:29:14.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f954affd640 1 -- 172.21.15.67:0/1729403028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f954400c940 con 0x7f95541434d0 2024-04-25T15:29:14.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955415eb70 con 0x7f95541434d0 2024-04-25T15:29:14.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f954affd640 1 -- 172.21.15.67:0/1729403028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954400cd10 con 0x7f95541434d0 2024-04-25T15:29:14.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.231+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f955415f130 con 0x7f95541434d0 2024-04-25T15:29:14.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.232+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9554050740 con 0x7f95541434d0 2024-04-25T15:29:14.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.233+0000 7f954affd640 1 -- 172.21.15.67:0/1729403028 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f954401a070 con 0x7f95541434d0 2024-04-25T15:29:14.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.233+0000 7f954affd640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f95380744e0 0x7f95380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.233+0000 7f954affd640 1 -- 172.21.15.67:0/1729403028 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f95440954a0 con 0x7f95541434d0 2024-04-25T15:29:14.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.234+0000 7f955a544640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f95380744e0 0x7f95380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.234+0000 7f955a544640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f95380744e0 0x7f95380769f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f954c0012b0 tx=0x7f954c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:14.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.238+0000 7f954affd640 1 -- 172.21.15.67:0/1729403028 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f954405fbb0 con 0x7f95541434d0 2024-04-25T15:29:14.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.433+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f95540d1f90 con 0x7f95541434d0 2024-04-25T15:29:14.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.434+0000 7f954affd640 1 -- 172.21.15.67:0/1729403028 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v92) ==== 80+0+0 (secure 0 0 0) 0x7f954405f4b0 con 0x7f95541434d0 2024-04-25T15:29:14.435 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f95380744e0 msgr2=0x7f95380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f95380744e0 0x7f95380769f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f954c0012b0 tx=0x7f954c009040 comp rx=0 tx=0).stop 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95541434d0 msgr2=0x7f9554159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95541434d0 0x7f9554159250 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f954400e870 tx=0x7f954400ed30 comp rx=0 tx=0).stop 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 shutdown_connections 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f95380744e0 0x7f95380769f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9554160be0 0x7f955406ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95541434d0 0x7f9554159250 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 --2- 172.21.15.67:0/1729403028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95540d0ed0 0x7f95541573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 >> 172.21.15.67:0/1729403028 conn(0x7f95540d1ae0 msgr2=0x7f95541425e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:14.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 shutdown_connections 2024-04-25T15:29:14.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.436+0000 7f955c7a8640 1 -- 172.21.15.67:0/1729403028 wait complete. 2024-04-25T15:29:14.448 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T15:29:14.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:14.448 DEBUG:teuthology.orchestra.run.smithi176:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-04-25T15:29:14.546 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:29:14.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:14.547 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T15:29:14.566 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:29:14.566 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:29:14.566 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:14.583 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:29:14.584 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T15:29:14.584 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:14.584 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:14.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:29:14.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:29:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.734+0000 7f838cd84640 1 Processor -- start 2024-04-25T15:29:14.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.734+0000 7f838cd84640 1 -- start start 2024-04-25T15:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838cd84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83880d4130 0x7f83880d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838cd84640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83880b3c10 con 0x7f838806d8e0 2024-04-25T15:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838cd84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83880b3ef0 con 0x7f838806dcf0 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838cd84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f83880b9520 con 0x7f83880d4130 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f838806d8e0 0x7f83880d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38412/0 (socket says 172.21.15.67:38412) 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838659c640 1 -- 172.21.15.67:0/1763824224 learned_addr learned my addr 172.21.15.67:0/1763824224 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f8385d9b640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83880d4130 0x7f83880d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163242373 0 0) 0x7f83880b3c10 con 0x7f838806d8e0 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8374003520 con 0x7f838806d8e0 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724231532 0 0) 0x7f83880b3ef0 con 0x7f838806dcf0 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83880b3c10 con 0x7f838806dcf0 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3100936812 0 0) 0x7f8374003520 con 0x7f838806d8e0 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83880d4130 msgr2=0x7f83880d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83880d4130 0x7f83880d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 >> v1:172.21.15.176:6789/0 conn(0x7f838806dcf0 legacy=0x7f83880d22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83880b41d0 con 0x7f838806d8e0 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f8380001670 con 0x7f838806d8e0 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 >> v1:172.21.15.67:6789/0 conn(0x7f838806d8e0 legacy=0x7f83880d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8374003b70 0x7f8374003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.736+0000 7f838559a640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8374004800 0x7f8374004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838559a640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374005260 0x7f837400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83880b9520 con 0x7f8374003b70 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f837400bfe0 con 0x7f8374004800 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f837400c1e0 con 0x7f8374005260 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838659c640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8374003b70 0x7f8374003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f8385d9b640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8374004800 0x7f8374004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f8386d9d640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374005260 0x7f837400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838659c640 1 -- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374005260 msgr2=0x7f837400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838659c640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374005260 0x7f837400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838659c640 1 -- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8374004800 msgr2=0x7f8374004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838659c640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8374004800 0x7f8374004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.737+0000 7f838659c640 1 -- 172.21.15.67:0/1763824224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8374003520 con 0x7f8374003b70 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.738+0000 7f838659c640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8374003b70 0x7f8374003fc0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8380007f90 tx=0x7f8374005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eed49d9dd634efc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.738+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380007be0 con 0x7f8374003b70 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.738+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8380005af0 con 0x7f8374003b70 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.738+0000 7f838559a640 1 -- 172.21.15.67:0/1763824224 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380007070 con 0x7f8374003b70 2024-04-25T15:29:14.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.738+0000 7f838cd84640 1 -- 172.21.15.67:0/1763824224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8374003b70 msgr2=0x7f8374003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.738+0000 7f838cd84640 1 --2- 172.21.15.67:0/1763824224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8374003b70 0x7f8374003fc0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8380007f90 tx=0x7f8374005830 comp rx=0 tx=0).stop 2024-04-25T15:29:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.738+0000 7f8386d9d640 1 -- 172.21.15.67:0/1763824224 reap_dead start 2024-04-25T15:29:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.739+0000 7f838cd84640 1 -- 172.21.15.67:0/1763824224 shutdown_connections 2024-04-25T15:29:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.739+0000 7f838cd84640 1 -- 172.21.15.67:0/1763824224 >> 172.21.15.67:0/1763824224 conn(0x7f838808d110 msgr2=0x7f838808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.739+0000 7f838cd84640 1 -- 172.21.15.67:0/1763824224 shutdown_connections 2024-04-25T15:29:14.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.739+0000 7f838cd84640 1 -- 172.21.15.67:0/1763824224 wait complete. 2024-04-25T15:29:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.741+0000 7f838cd84640 1 Processor -- start 2024-04-25T15:29:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.741+0000 7f838cd84640 1 -- start start 2024-04-25T15:29:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f838cd84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374003b70 0x7f83881f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f838cd84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838815c380 0x7f838815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f838cd84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83881f55f0 0x7f83880b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f838cd84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83880b38e0 con 0x7f838815c380 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f838cd84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f83880b4820 con 0x7f83881f55f0 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f838cd84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f83880d4130 con 0x7f8374003b70 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838815c380 0x7f838815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838815c380 0x7f838815ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57916/0 (socket says 172.21.15.67:57916) 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 -- 172.21.15.67:0/1153826914 learned_addr learned my addr 172.21.15.67:0/1153826914 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f838659c640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374003b70 0x7f83881f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8386d9d640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83881f55f0 0x7f83880b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 -- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374003b70 msgr2=0x7f83881f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374003b70 0x7f83881f9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 -- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83881f55f0 msgr2=0x7f83880b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83881f55f0 0x7f83880b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 -- 172.21.15.67:0/1153826914 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838815caf0 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.742+0000 7f8385d9b640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838815c380 0x7f838815ffd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8370011910 tx=0x7f8370011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.743+0000 7f837effd640 1 -- 172.21.15.67:0/1153826914 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837000fa90 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.743+0000 7f837effd640 1 -- 172.21.15.67:0/1153826914 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f837000fc30 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.743+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83881f9380 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.743+0000 7f837effd640 1 -- 172.21.15.67:0/1153826914 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370014440 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.743+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f838815cdd0 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.744+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8388050740 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.744+0000 7f837effd640 1 -- 172.21.15.67:0/1153826914 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8370012040 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.745+0000 7f837effd640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83680744e0 0x7f83680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.745+0000 7f837effd640 1 -- 172.21.15.67:0/1153826914 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f83700956b0 con 0x7f838815c380 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.745+0000 7f838659c640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83680744e0 0x7f83680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:14.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.746+0000 7f838659c640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83680744e0 0x7f83680769f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8380032000 tx=0x7f8380034040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:14.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.750+0000 7f837effd640 1 -- 172.21.15.67:0/1153826914 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8370019030 con 0x7f838815c380 2024-04-25T15:29:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.973+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f838815c880 con 0x7f838815c380 2024-04-25T15:29:14.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.973+0000 7f837effd640 1 -- 172.21.15.67:0/1153826914 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5189 (secure 0 0 0) 0x7f837005f7d0 con 0x7f838815c380 2024-04-25T15:29:14.975 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:14.975 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2709900765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2709900765},{"type":"v1","addr":"172.21.15.67:6834","nonce":2709900765}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4104906051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4104906051},{"type":"v1","addr":"172.21.15.176:6836","nonce":4104906051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6655,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2981673460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2981673460},{"type":"v1","addr":"172.21.15.176:6837","nonce":2981673460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:56.417898+0000","modified":"2024-04-25T15:28:59.469250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7128":{"gid":7128,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3429198629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3429198629},{"type":"v1","addr":"172.21.15.67:6835","nonce":3429198629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7128,"qdb_cluster":[7128]},"id":7}]} 2024-04-25T15:29:14.975 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 57 2024-04-25T15:29:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83680744e0 msgr2=0x7f83680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83680744e0 0x7f83680769f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8380032000 tx=0x7f8380034040 comp rx=0 tx=0).stop 2024-04-25T15:29:14.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838815c380 msgr2=0x7f838815ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838815c380 0x7f838815ffd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8370011910 tx=0x7f8370011dd0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 shutdown_connections 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f83680744e0 0x7f83680769f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83881f55f0 0x7f83880b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f838815c380 0x7f838815ffd0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 --2- 172.21.15.67:0/1153826914 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8374003b70 0x7f83881f9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 >> 172.21.15.67:0/1153826914 conn(0x7f838808d110 msgr2=0x7f83880b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 shutdown_connections 2024-04-25T15:29:14.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:14.975+0000 7f838cd84640 1 -- 172.21.15.67:0/1153826914 wait complete. 2024-04-25T15:29:14.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:29:15.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.098+0000 7f5b0f334640 1 Processor -- start 2024-04-25T15:29:15.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.099+0000 7f5b0f334640 1 -- start start 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5b0f334640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5b0806dcf0 0x7f5b080d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5b0f334640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b080b3c10 con 0x7f5b0806d8e0 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5b0f334640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b080b3ef0 con 0x7f5b080d4130 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5b0f334640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5b080b9520 con 0x7f5b0806dcf0 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5b0d0d0640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b0806d8e0 0x7f5b080d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38416/0 (socket says 172.21.15.67:38416) 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5b0d0d0640 1 -- 172.21.15.67:0/3815095064 learned_addr learned my addr 172.21.15.67:0/3815095064 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773220612 0 0) 0x7f5b080b3c10 con 0x7f5b0806d8e0 2024-04-25T15:29:15.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5af4003520 con 0x7f5b0806d8e0 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5b0d8d1640 1 --2- 172.21.15.67:0/3815095064 >> v2:172.21.15.176:3300/0 conn(0x7f5b0806dcf0 0x7f5b080d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.100+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3424667218 0 0) 0x7f5af4003520 con 0x7f5b0806d8e0 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 >> v2:172.21.15.176:3300/0 conn(0x7f5b0806dcf0 msgr2=0x7f5b080d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 --2- 172.21.15.67:0/3815095064 >> v2:172.21.15.176:3300/0 conn(0x7f5b0806dcf0 0x7f5b080d2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 >> v1:172.21.15.176:6789/0 conn(0x7f5b080d4130 legacy=0x7f5b080d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b080b41d0 con 0x7f5b0806d8e0 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f5b00001670 con 0x7f5b0806d8e0 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 >> v1:172.21.15.67:6789/0 conn(0x7f5b0806d8e0 legacy=0x7f5b080d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af40039b0 0x7f5af4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af4004800 0x7f5af4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5af4005250 0x7f5af400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080b9520 con 0x7f5af40039b0 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5af400c1e0 con 0x7f5af4004800 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5af400c3e0 con 0x7f5af4005250 2024-04-25T15:29:15.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d0d0640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af40039b0 0x7f5af4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0c8cf640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af4004800 0x7f5af4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d8d1640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5af4005250 0x7f5af400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d0d0640 1 -- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5af4005250 msgr2=0x7f5af400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d0d0640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5af4005250 0x7f5af400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d0d0640 1 -- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af4004800 msgr2=0x7f5af4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d0d0640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af4004800 0x7f5af4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d0d0640 1 -- 172.21.15.67:0/3815095064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af4003520 con 0x7f5af40039b0 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.101+0000 7f5b0d8d1640 1 -- 172.21.15.67:0/3815095064 reap_dead start 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.102+0000 7f5b0d0d0640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af40039b0 0x7f5af4003e20 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c210 tx=0x7f5af4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0ec5d70985ec16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.102+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00007cf0 con 0x7f5af40039b0 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.102+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00005af0 con 0x7f5af40039b0 2024-04-25T15:29:15.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.102+0000 7f5afffff640 1 -- 172.21.15.67:0/3815095064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00007070 con 0x7f5af40039b0 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.103+0000 7f5b0f334640 1 -- 172.21.15.67:0/3815095064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af40039b0 msgr2=0x7f5af4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.103+0000 7f5b0f334640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af40039b0 0x7f5af4003e20 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c210 tx=0x7f5af4005830 comp rx=0 tx=0).stop 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.103+0000 7f5b0f334640 1 -- 172.21.15.67:0/3815095064 shutdown_connections 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.103+0000 7f5b0f334640 1 --2- 172.21.15.67:0/3815095064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af40039b0 0x7f5af4003e20 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.103+0000 7f5b0f334640 1 -- 172.21.15.67:0/3815095064 >> 172.21.15.67:0/3815095064 conn(0x7f5b0808d110 msgr2=0x7f5b0808d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.103+0000 7f5b0f334640 1 -- 172.21.15.67:0/3815095064 shutdown_connections 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.103+0000 7f5b0f334640 1 -- 172.21.15.67:0/3815095064 wait complete. 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.105+0000 7f5b0f334640 1 Processor -- start 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.105+0000 7f5b0f334640 1 -- start start 2024-04-25T15:29:15.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0f334640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af40039b0 0x7f5b081f56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0f334640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b0815cfc0 0x7f5b08158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0f334640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b0815ab10 0x7f5b0815af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0f334640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080b38e0 con 0x7f5b0815ab10 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0f334640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080b4820 con 0x7f5af40039b0 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0f334640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5b080d3a20 con 0x7f5b0815cfc0 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b0815ab10 0x7f5b0815af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b0815ab10 0x7f5b0815af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57922/0 (socket says 172.21.15.67:57922) 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 -- 172.21.15.67:0/2391826952 learned_addr learned my addr 172.21.15.67:0/2391826952 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d0d0640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af40039b0 0x7f5b081f56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0c8cf640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b0815cfc0 0x7f5b08158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 -- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b0815cfc0 msgr2=0x7f5b08158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b0815cfc0 0x7f5b08158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 -- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af40039b0 msgr2=0x7f5b081f56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af40039b0 0x7f5b081f56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 -- 172.21.15.67:0/2391826952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b08159ae0 con 0x7f5b0815ab10 2024-04-25T15:29:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.106+0000 7f5b0d8d1640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b0815ab10 0x7f5b0815af20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5b080d38e0 tx=0x7f5b0400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.107+0000 7f5afdffb640 1 -- 172.21.15.67:0/2391826952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04011e90 con 0x7f5b0815ab10 2024-04-25T15:29:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.107+0000 7f5afdffb640 1 -- 172.21.15.67:0/2391826952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0401c070 con 0x7f5b0815ab10 2024-04-25T15:29:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.107+0000 7f5afdffb640 1 -- 172.21.15.67:0/2391826952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b04003bd0 con 0x7f5b0815ab10 2024-04-25T15:29:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.107+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0815b460 con 0x7f5b0815ab10 2024-04-25T15:29:15.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.107+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b08159d50 con 0x7f5b0815ab10 2024-04-25T15:29:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.108+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b08050740 con 0x7f5b0815ab10 2024-04-25T15:29:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.109+0000 7f5afdffb640 1 -- 172.21.15.67:0/2391826952 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5b04004030 con 0x7f5b0815ab10 2024-04-25T15:29:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.109+0000 7f5afdffb640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5aec074650 0x7f5aec076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.109+0000 7f5afdffb640 1 -- 172.21.15.67:0/2391826952 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f5b04095bf0 con 0x7f5b0815ab10 2024-04-25T15:29:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.109+0000 7f5b0d0d0640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5aec074650 0x7f5aec076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.110+0000 7f5b0d0d0640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5aec074650 0x7f5aec076b60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5b080744c0 tx=0x7f5b00009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.114+0000 7f5afdffb640 1 -- 172.21.15.67:0/2391826952 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b04010040 con 0x7f5b0815ab10 2024-04-25T15:29:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.301+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5b080d32b0 con 0x7f5b0815ab10 2024-04-25T15:29:15.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.302+0000 7f5afdffb640 1 -- 172.21.15.67:0/2391826952 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16572 (secure 0 0 0) 0x7f5b04060380 con 0x7f5b0815ab10 2024-04-25T15:29:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:15.304 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:58.058412+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5aec074650 msgr2=0x7f5aec076b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5aec074650 0x7f5aec076b60 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5b080744c0 tx=0x7f5b00009030 comp rx=0 tx=0).stop 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b0815ab10 msgr2=0x7f5b0815af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b0815ab10 0x7f5b0815af20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f5b080d38e0 tx=0x7f5b0400f580 comp rx=0 tx=0).stop 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 shutdown_connections 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5aec074650 0x7f5aec076b60 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b0815ab10 0x7f5b0815af20 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b0815cfc0 0x7f5b08158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 --2- 172.21.15.67:0/2391826952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af40039b0 0x7f5b081f56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 >> 172.21.15.67:0/2391826952 conn(0x7f5b0808d110 msgr2=0x7f5b08071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 shutdown_connections 2024-04-25T15:29:15.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.305+0000 7f5b0f334640 1 -- 172.21.15.67:0/2391826952 wait complete. 2024-04-25T15:29:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:25:16.812821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:28:54.107977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:28:55.386227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:29:15.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:29:15.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.431+0000 7f2f4f09f640 1 Processor -- start 2024-04-25T15:29:15.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.431+0000 7f2f4f09f640 1 -- start start 2024-04-25T15:29:15.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4f09f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f480ce250 0x7f2f4815cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4f09f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f481458c0 0x7f2f480b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4f09f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2f4806d5d0 0x7f2f4806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4f09f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2f4806df20 con 0x7f2f481458c0 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4f09f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2f48145cd0 con 0x7f2f4806d5d0 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4f09f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4815d5c0 con 0x7f2f480ce250 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f47fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f481458c0 0x7f2f480b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f47fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f481458c0 0x7f2f480b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57936/0 (socket says 172.21.15.67:57936) 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f47fff640 1 -- 172.21.15.67:0/2140517887 learned_addr learned my addr 172.21.15.67:0/2140517887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4d63c640 1 --2- 172.21.15.67:0/2140517887 >> v2:172.21.15.176:3300/0 conn(0x7f2f4806d5d0 0x7f2f4806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.432+0000 7f2f4ce3b640 1 --2- 172.21.15.67:0/2140517887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f480ce250 0x7f2f4815cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.433+0000 7f2f47fff640 1 -- 172.21.15.67:0/2140517887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f480ce250 msgr2=0x7f2f4815cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.433+0000 7f2f47fff640 1 --2- 172.21.15.67:0/2140517887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f480ce250 0x7f2f4815cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.433+0000 7f2f47fff640 1 -- 172.21.15.67:0/2140517887 >> v2:172.21.15.176:3300/0 conn(0x7f2f4806d5d0 msgr2=0x7f2f4806d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.433+0000 7f2f47fff640 1 --2- 172.21.15.67:0/2140517887 >> v2:172.21.15.176:3300/0 conn(0x7f2f4806d5d0 0x7f2f4806d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.433+0000 7f2f47fff640 1 -- 172.21.15.67:0/2140517887 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4815d8a0 con 0x7f2f481458c0 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.433+0000 7f2f47fff640 1 --2- 172.21.15.67:0/2140517887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f480b47b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f2f480cf1e0 tx=0x7f2f3402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db25378be43223bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f477fe640 1 -- 172.21.15.67:0/2140517887 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34037070 con 0x7f2f481458c0 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f477fe640 1 -- 172.21.15.67:0/2140517887 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f34032440 con 0x7f2f481458c0 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f477fe640 1 -- 172.21.15.67:0/2140517887 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34032810 con 0x7f2f481458c0 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2140517887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 msgr2=0x7f2f480b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2140517887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f480b47b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f2f480cf1e0 tx=0x7f2f3402a140 comp rx=0 tx=0).stop 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2140517887 shutdown_connections 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2140517887 >> v2:172.21.15.176:3300/0 conn(0x7f2f4806d5d0 0x7f2f4806d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2140517887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f480b47b0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2140517887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f480ce250 0x7f2f4815cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.434+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2140517887 >> 172.21.15.67:0/2140517887 conn(0x7f2f48073090 msgr2=0x7f2f480734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:15.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.435+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2140517887 shutdown_connections 2024-04-25T15:29:15.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.435+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2140517887 wait complete. 2024-04-25T15:29:15.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.436+0000 7f2f4f09f640 1 Processor -- start 2024-04-25T15:29:15.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.436+0000 7f2f4f09f640 1 -- start start 2024-04-25T15:29:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4f09f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f4806d5d0 0x7f2f4815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4f09f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f480ce250 0x7f2f4815f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4f09f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f4815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4f09f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4815de60 con 0x7f2f481458c0 2024-04-25T15:29:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4f09f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4806f640 con 0x7f2f480ce250 2024-04-25T15:29:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4f09f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f48071310 con 0x7f2f4806d5d0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f4815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f4815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57946/0 (socket says 172.21.15.67:57946) 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 -- 172.21.15.67:0/2490342484 learned_addr learned my addr 172.21.15.67:0/2490342484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4ce3b640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f4806d5d0 0x7f2f4815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f47fff640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f480ce250 0x7f2f4815f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 -- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f4806d5d0 msgr2=0x7f2f4815bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f4806d5d0 0x7f2f4815bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 -- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f480ce250 msgr2=0x7f2f4815f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f480ce250 0x7f2f4815f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.438+0000 7f2f4d63c640 1 -- 172.21.15.67:0/2490342484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4815c150 con 0x7f2f481458c0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.439+0000 7f2f4d63c640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f4815f810 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000eed0 tx=0x7f2f4000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.439+0000 7f2f457fa640 1 -- 172.21.15.67:0/2490342484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40018020 con 0x7f2f481458c0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.439+0000 7f2f457fa640 1 -- 172.21.15.67:0/2490342484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4000bd20 con 0x7f2f481458c0 2024-04-25T15:29:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.439+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f480b0b70 con 0x7f2f481458c0 2024-04-25T15:29:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.439+0000 7f2f457fa640 1 -- 172.21.15.67:0/2490342484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40014640 con 0x7f2f481458c0 2024-04-25T15:29:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.439+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4813c9e0 con 0x7f2f481458c0 2024-04-25T15:29:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.440+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f480af4c0 con 0x7f2f481458c0 2024-04-25T15:29:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.441+0000 7f2f457fa640 1 -- 172.21.15.67:0/2490342484 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2f40005d10 con 0x7f2f481458c0 2024-04-25T15:29:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.441+0000 7f2f457fa640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f2c0744b0 0x7f2f2c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.441+0000 7f2f457fa640 1 -- 172.21.15.67:0/2490342484 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f2f40095670 con 0x7f2f481458c0 2024-04-25T15:29:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.441+0000 7f2f4ce3b640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f2c0744b0 0x7f2f2c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.442+0000 7f2f4ce3b640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f2c0744b0 0x7f2f2c0769c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2f480b6c80 tx=0x7f2f3c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.446+0000 7f2f457fa640 1 -- 172.21.15.67:0/2490342484 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f40022070 con 0x7f2f481458c0 2024-04-25T15:29:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.629+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2f480b1090 con 0x7f2f481458c0 2024-04-25T15:29:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.629+0000 7f2f457fa640 1 -- 172.21.15.67:0/2490342484 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f2f480b1090 con 0x7f2f481458c0 2024-04-25T15:29:15.630 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:29:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f2c0744b0 msgr2=0x7f2f2c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f2c0744b0 0x7f2f2c0769c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2f480b6c80 tx=0x7f2f3c009190 comp rx=0 tx=0).stop 2024-04-25T15:29:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 msgr2=0x7f2f4815f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f4815f810 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000eed0 tx=0x7f2f4000b480 comp rx=0 tx=0).stop 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 shutdown_connections 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2f2c0744b0 0x7f2f2c0769c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f481458c0 0x7f2f4815f810 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f480ce250 0x7f2f4815f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 --2- 172.21.15.67:0/2490342484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f4806d5d0 0x7f2f4815bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 >> 172.21.15.67:0/2490342484 conn(0x7f2f48073090 msgr2=0x7f2f480b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 shutdown_connections 2024-04-25T15:29:15.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.631+0000 7f2f4f09f640 1 -- 172.21.15.67:0/2490342484 wait complete. 2024-04-25T15:29:15.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:29:15.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.757+0000 7fb6ab39d640 1 Processor -- start 2024-04-25T15:29:15.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.757+0000 7fb6ab39d640 1 -- start start 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.758+0000 7fb6ab39d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb6a40d56c0 0x7fb6a40b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.758+0000 7fb6ab39d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a4160440 0x7fb6a406db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.758+0000 7fb6ab39d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6a406e0a0 0x7fb6a40d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.758+0000 7fb6ab39d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a40d43f0 con 0x7fb6a406e0a0 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.758+0000 7fb6ab39d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a406e4b0 con 0x7fb6a40d56c0 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.758+0000 7fb6ab39d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a40d37e0 con 0x7fb6a4160440 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6a406e0a0 0x7fb6a40d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6a406e0a0 0x7fb6a40d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57948/0 (socket says 172.21.15.67:57948) 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 -- 172.21.15.67:0/137002243 learned_addr learned my addr 172.21.15.67:0/137002243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:15.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aa39b640 1 --2- 172.21.15.67:0/137002243 >> v2:172.21.15.176:3300/0 conn(0x7fb6a40d56c0 0x7fb6a40b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6a9b9a640 1 --2- 172.21.15.67:0/137002243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a4160440 0x7fb6a406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 -- 172.21.15.67:0/137002243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a4160440 msgr2=0x7fb6a406db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 --2- 172.21.15.67:0/137002243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a4160440 0x7fb6a406db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 -- 172.21.15.67:0/137002243 >> v2:172.21.15.176:3300/0 conn(0x7fb6a40d56c0 msgr2=0x7fb6a40b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 --2- 172.21.15.67:0/137002243 >> v2:172.21.15.176:3300/0 conn(0x7fb6a40d56c0 0x7fb6a40b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.759+0000 7fb6aab9c640 1 -- 172.21.15.67:0/137002243 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a40d3ac0 con 0x7fb6a406e0a0 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.760+0000 7fb6aab9c640 1 --2- 172.21.15.67:0/137002243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a406e0a0 0x7fb6a40d0900 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb6a408e390 tx=0x7fb69c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99225268d44cfe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.760+0000 7fb6a9399640 1 -- 172.21.15.67:0/137002243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c037070 con 0x7fb6a406e0a0 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.760+0000 7fb6a9399640 1 -- 172.21.15.67:0/137002243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb69c032440 con 0x7fb6a406e0a0 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.760+0000 7fb6a9399640 1 -- 172.21.15.67:0/137002243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c032810 con 0x7fb6a406e0a0 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.760+0000 7fb6ab39d640 1 -- 172.21.15.67:0/137002243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a406e0a0 msgr2=0x7fb6a40d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.760+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/137002243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a406e0a0 0x7fb6a40d0900 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb6a408e390 tx=0x7fb69c02a1e0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.761+0000 7fb6ab39d640 1 -- 172.21.15.67:0/137002243 shutdown_connections 2024-04-25T15:29:15.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.761+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/137002243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a406e0a0 0x7fb6a40d0900 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.761+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/137002243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a4160440 0x7fb6a406db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.761+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/137002243 >> v2:172.21.15.176:3300/0 conn(0x7fb6a40d56c0 0x7fb6a40b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.761+0000 7fb6ab39d640 1 -- 172.21.15.67:0/137002243 >> 172.21.15.67:0/137002243 conn(0x7fb6a40d2c20 msgr2=0x7fb6a40d5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.761+0000 7fb6ab39d640 1 -- 172.21.15.67:0/137002243 shutdown_connections 2024-04-25T15:29:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.761+0000 7fb6ab39d640 1 -- 172.21.15.67:0/137002243 wait complete. 2024-04-25T15:29:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.762+0000 7fb6ab39d640 1 Processor -- start 2024-04-25T15:29:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.762+0000 7fb6ab39d640 1 -- start start 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6ab39d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6a406e0a0 0x7fb6a4163700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6ab39d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a4160440 0x7fb6a4151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6ab39d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a41f5f90 0x7fb6a4153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6ab39d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a40d4880 con 0x7fb6a4160440 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6ab39d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a40d4080 con 0x7fb6a406e0a0 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6ab39d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a40d4a00 con 0x7fb6a41f5f90 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a4160440 0x7fb6a4151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a4160440 0x7fb6a4151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57956/0 (socket says 172.21.15.67:57956) 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 -- 172.21.15.67:0/923042814 learned_addr learned my addr 172.21.15.67:0/923042814 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6aa39b640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6a406e0a0 0x7fb6a4163700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6aab9c640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a41f5f90 0x7fb6a4153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 -- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a41f5f90 msgr2=0x7fb6a4153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a41f5f90 0x7fb6a4153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 -- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6a406e0a0 msgr2=0x7fb6a4163700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6a406e0a0 0x7fb6a4163700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 -- 172.21.15.67:0/923042814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a41524d0 con 0x7fb6a4160440 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.764+0000 7fb6a9b9a640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a4160440 0x7fb6a4151f90 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb69800e870 tx=0x7fb69800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.765+0000 7fb6a2ffd640 1 -- 172.21.15.67:0/923042814 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698017070 con 0x7fb6a4160440 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.765+0000 7fb6a2ffd640 1 -- 172.21.15.67:0/923042814 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb69800c940 con 0x7fb6a4160440 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.765+0000 7fb6a2ffd640 1 -- 172.21.15.67:0/923042814 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69800cd30 con 0x7fb6a4160440 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.765+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a4152e90 con 0x7fb6a4160440 2024-04-25T15:29:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.765+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a4153450 con 0x7fb6a4160440 2024-04-25T15:29:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.766+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a4076c60 con 0x7fb6a4160440 2024-04-25T15:29:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.766+0000 7fb6a2ffd640 1 -- 172.21.15.67:0/923042814 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb69801a070 con 0x7fb6a4160440 2024-04-25T15:29:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.767+0000 7fb6a2ffd640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6900744b0 0x7fb6900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.767+0000 7fb6a2ffd640 1 -- 172.21.15.67:0/923042814 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fb698095020 con 0x7fb6a4160440 2024-04-25T15:29:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.767+0000 7fb6aa39b640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6900744b0 0x7fb6900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:15.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.768+0000 7fb6aa39b640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6900744b0 0x7fb6900769c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40b8b30 tx=0x7fb69400d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:15.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.772+0000 7fb6a2ffd640 1 -- 172.21.15.67:0/923042814 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb69805f1f0 con 0x7fb6a4160440 2024-04-25T15:29:15.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.955+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb6a415f300 con 0x7fb6a4160440 2024-04-25T15:29:15.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.956+0000 7fb6a2ffd640 1 -- 172.21.15.67:0/923042814 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7fb6980634f0 con 0x7fb6a4160440 2024-04-25T15:29:15.956 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:29:15.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6900744b0 msgr2=0x7fb6900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6900744b0 0x7fb6900769c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40b8b30 tx=0x7fb69400d040 comp rx=0 tx=0).stop 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a4160440 msgr2=0x7fb6a4151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a4160440 0x7fb6a4151f90 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb69800e870 tx=0x7fb69800ed30 comp rx=0 tx=0).stop 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 shutdown_connections 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb6900744b0 0x7fb6900769c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb6a41f5f90 0x7fb6a4153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a4160440 0x7fb6a4151f90 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 --2- 172.21.15.67:0/923042814 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6a406e0a0 0x7fb6a4163700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 >> 172.21.15.67:0/923042814 conn(0x7fb6a40d2c20 msgr2=0x7fb6a40bac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.957+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 shutdown_connections 2024-04-25T15:29:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:15.958+0000 7fb6ab39d640 1 -- 172.21.15.67:0/923042814 wait complete. 2024-04-25T15:29:15.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:29:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.083+0000 7ff937e84640 1 Processor -- start 2024-04-25T15:29:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.083+0000 7ff937e84640 1 -- start start 2024-04-25T15:29:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff937e84640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff9300d0e00 0x7ff9300d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff937e84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 0x7ff9300d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff937e84640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9300d1850 con 0x7ff93006c5c0 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff937e84640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff930160de0 con 0x7ff9300d0e00 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff937e84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff9301610c0 con 0x7ff9300d4a50 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff93541f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff93006c5c0 0x7ff93006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38424/0 (socket says 172.21.15.67:38424) 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff93541f640 1 -- 172.21.15.67:0/3670934035 learned_addr learned my addr 172.21.15.67:0/3670934035 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff936421640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 0x7ff9300d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff935c20640 1 --2- 172.21.15.67:0/3670934035 >> v2:172.21.15.176:3300/0 conn(0x7ff9300d0e00 0x7ff9300d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956885997 0 0) 0x7ff9300d1850 con 0x7ff93006c5c0 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff918003520 con 0x7ff93006c5c0 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2728109537 0 0) 0x7ff918003520 con 0x7ff93006c5c0 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 msgr2=0x7ff9300d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff934c1e640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 0x7ff9300d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 >> v2:172.21.15.176:3300/0 conn(0x7ff9300d0e00 msgr2=0x7ff9300d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:16.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.085+0000 7ff934c1e640 1 --2- 172.21.15.67:0/3670934035 >> v2:172.21.15.176:3300/0 conn(0x7ff9300d0e00 0x7ff9300d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9301613a0 con 0x7ff93006c5c0 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7ff924001670 con 0x7ff93006c5c0 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 >> v1:172.21.15.67:6789/0 conn(0x7ff93006c5c0 legacy=0x7ff93006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9180039b0 0x7ff918003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff918004800 0x7ff918004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 0x7ff918005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff930160de0 con 0x7ff918004800 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff9301610c0 con 0x7ff9180039b0 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff91800b9c0 con 0x7ff9300d4a50 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff93541f640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff918004800 0x7ff918004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff935c20640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9180039b0 0x7ff918003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff936421640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 0x7ff918005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff93541f640 1 -- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 msgr2=0x7ff918005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:16.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff93541f640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 0x7ff918005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff93541f640 1 -- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9180039b0 msgr2=0x7ff918003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff93541f640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9180039b0 0x7ff918003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff93541f640 1 -- 172.21.15.67:0/3670934035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff918003520 con 0x7ff918004800 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.086+0000 7ff936421640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9300d4a50 0x7ff918005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.087+0000 7ff93541f640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff918004800 0x7ff918004c10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7ff92400c240 tx=0x7ff918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639812571b59808d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.087+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92402e070 con 0x7ff918004800 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.087+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff924005af0 con 0x7ff918004800 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.087+0000 7ff934c1e640 1 -- 172.21.15.67:0/3670934035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924007d30 con 0x7ff918004800 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.088+0000 7ff937e84640 1 -- 172.21.15.67:0/3670934035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff918004800 msgr2=0x7ff918004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.088+0000 7ff937e84640 1 --2- 172.21.15.67:0/3670934035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff918004800 0x7ff918004c10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7ff92400c240 tx=0x7ff918005830 comp rx=0 tx=0).stop 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.088+0000 7ff936421640 1 -- 172.21.15.67:0/3670934035 reap_dead start 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.088+0000 7ff937e84640 1 -- 172.21.15.67:0/3670934035 shutdown_connections 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.088+0000 7ff937e84640 1 -- 172.21.15.67:0/3670934035 >> 172.21.15.67:0/3670934035 conn(0x7ff9300d5d90 msgr2=0x7ff9300d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:16.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.088+0000 7ff937e84640 1 -- 172.21.15.67:0/3670934035 shutdown_connections 2024-04-25T15:29:16.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.088+0000 7ff937e84640 1 -- 172.21.15.67:0/3670934035 wait complete. 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.089+0000 7ff937e84640 1 Processor -- start 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.090+0000 7ff937e84640 1 -- start start 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff937e84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff918004800 0x7ff9301560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff937e84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9300cb1c0 0x7ff9300ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff937e84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9301f6740 0x7ff9301f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff937e84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff930164e00 con 0x7ff9300cb1c0 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff937e84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff9300b5320 con 0x7ff9301f6740 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff937e84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff930164f80 con 0x7ff918004800 2024-04-25T15:29:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9300cb1c0 0x7ff9300ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9300cb1c0 0x7ff9300ce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57970/0 (socket says 172.21.15.67:57970) 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 -- 172.21.15.67:0/1539391797 learned_addr learned my addr 172.21.15.67:0/1539391797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff935c20640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff918004800 0x7ff9301560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff936421640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9301f6740 0x7ff9301f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 -- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff918004800 msgr2=0x7ff9301560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff918004800 0x7ff9301560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 -- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9301f6740 msgr2=0x7ff9301f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9301f6740 0x7ff9301f6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 -- 172.21.15.67:0/1539391797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9300ca120 con 0x7ff9300cb1c0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff936421640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9301f6740 0x7ff9301f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.091+0000 7ff93541f640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9300cb1c0 0x7ff9300ce8f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff92400c360 tx=0x7ff930161570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.092+0000 7ff92e7fc640 1 -- 172.21.15.67:0/1539391797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924042070 con 0x7ff9300cb1c0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.092+0000 7ff92e7fc640 1 -- 172.21.15.67:0/1539391797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff924007480 con 0x7ff9300cb1c0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.092+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9300ca400 con 0x7ff9300cb1c0 2024-04-25T15:29:16.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.092+0000 7ff92e7fc640 1 -- 172.21.15.67:0/1539391797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924007d30 con 0x7ff9300cb1c0 2024-04-25T15:29:16.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.092+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9300ca9c0 con 0x7ff9300cb1c0 2024-04-25T15:29:16.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.093+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9300c9e90 con 0x7ff9300cb1c0 2024-04-25T15:29:16.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.094+0000 7ff92e7fc640 1 -- 172.21.15.67:0/1539391797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff924045070 con 0x7ff9300cb1c0 2024-04-25T15:29:16.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.094+0000 7ff92e7fc640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff91c0744b0 0x7ff91c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.094+0000 7ff92e7fc640 1 -- 172.21.15.67:0/1539391797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7ff9240b5b60 con 0x7ff9300cb1c0 2024-04-25T15:29:16.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.094+0000 7ff935c20640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff91c0744b0 0x7ff91c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.095+0000 7ff935c20640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff91c0744b0 0x7ff91c0769c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff928009b40 tx=0x7ff928009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.099+0000 7ff92e7fc640 1 -- 172.21.15.67:0/1539391797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff924005ac0 con 0x7ff9300cb1c0 2024-04-25T15:29:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.284+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff930050740 con 0x7ff9300cb1c0 2024-04-25T15:29:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.285+0000 7ff92e7fc640 1 -- 172.21.15.67:0/1539391797 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7ff924080380 con 0x7ff9300cb1c0 2024-04-25T15:29:16.286 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:29:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff91c0744b0 msgr2=0x7ff91c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff91c0744b0 0x7ff91c0769c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff928009b40 tx=0x7ff928009190 comp rx=0 tx=0).stop 2024-04-25T15:29:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9300cb1c0 msgr2=0x7ff9300ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9300cb1c0 0x7ff9300ce8f0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff92400c360 tx=0x7ff930161570 comp rx=0 tx=0).stop 2024-04-25T15:29:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 shutdown_connections 2024-04-25T15:29:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff91c0744b0 0x7ff91c0769c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9301f6740 0x7ff9301f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9300cb1c0 0x7ff9300ce8f0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 --2- 172.21.15.67:0/1539391797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff918004800 0x7ff9301560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 >> 172.21.15.67:0/1539391797 conn(0x7ff9300d5d90 msgr2=0x7ff9300b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 shutdown_connections 2024-04-25T15:29:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.287+0000 7ff937e84640 1 -- 172.21.15.67:0/1539391797 wait complete. 2024-04-25T15:29:16.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:29:16.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.410+0000 7fa206fb2640 1 Processor -- start 2024-04-25T15:29:16.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.411+0000 7fa206fb2640 1 -- start start 2024-04-25T15:29:16.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa206fb2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2000d0ed0 0x7fa2000d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa206fb2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa20006f3e0 con 0x7fa2001434d0 2024-04-25T15:29:16.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa206fb2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa20006f6c0 con 0x7fa200160be0 2024-04-25T15:29:16.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa206fb2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa2001438e0 con 0x7fa2000d0ed0 2024-04-25T15:29:16.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa1fffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2000d0ed0 0x7fa2000d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa204d4e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa2001434d0 0x7fa200070480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38438/0 (socket says 172.21.15.67:38438) 2024-04-25T15:29:16.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa204d4e640 1 -- 172.21.15.67:0/456180284 learned_addr learned my addr 172.21.15.67:0/456180284 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:16.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa1fffff640 1 -- 172.21.15.67:0/456180284 >> v1:172.21.15.176:6789/0 conn(0x7fa200160be0 legacy=0x7fa200160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:16.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa1fffff640 1 -- 172.21.15.67:0/456180284 >> v1:172.21.15.67:6789/0 conn(0x7fa2001434d0 legacy=0x7fa200070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa1ff7fe640 1 -- 172.21.15.67:0/456180284 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047852430 0 0) 0x7fa20006f3e0 con 0x7fa2001434d0 2024-04-25T15:29:16.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.412+0000 7fa1fffff640 1 -- 172.21.15.67:0/456180284 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20006f9a0 con 0x7fa2000d0ed0 2024-04-25T15:29:16.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.413+0000 7fa1fffff640 1 --2- 172.21.15.67:0/456180284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2000d0ed0 0x7fa2000d12e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00097d0 tx=0x7fa1f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43aa8aeb5463b4ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.413+0000 7fa1ff7fe640 1 -- 172.21.15.67:0/456180284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0037070 con 0x7fa2000d0ed0 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.413+0000 7fa1ff7fe640 1 -- 172.21.15.67:0/456180284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f0032440 con 0x7fa2000d0ed0 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.413+0000 7fa1ff7fe640 1 -- 172.21.15.67:0/456180284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0032810 con 0x7fa2000d0ed0 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.414+0000 7fa206fb2640 1 -- 172.21.15.67:0/456180284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2000d0ed0 msgr2=0x7fa2000d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.414+0000 7fa206fb2640 1 --2- 172.21.15.67:0/456180284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2000d0ed0 0x7fa2000d12e0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00097d0 tx=0x7fa1f002a270 comp rx=0 tx=0).stop 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.414+0000 7fa206fb2640 1 -- 172.21.15.67:0/456180284 shutdown_connections 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.414+0000 7fa206fb2640 1 --2- 172.21.15.67:0/456180284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2000d0ed0 0x7fa2000d12e0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.414+0000 7fa206fb2640 1 -- 172.21.15.67:0/456180284 >> 172.21.15.67:0/456180284 conn(0x7fa2000d1ae0 msgr2=0x7fa2000d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.414+0000 7fa206fb2640 1 -- 172.21.15.67:0/456180284 shutdown_connections 2024-04-25T15:29:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.414+0000 7fa206fb2640 1 -- 172.21.15.67:0/456180284 wait complete. 2024-04-25T15:29:16.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.416+0000 7fa206fb2640 1 Processor -- start 2024-04-25T15:29:16.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.416+0000 7fa206fb2640 1 -- start start 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa206fb2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2000d0ed0 0x7fa200144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa206fb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2001434d0 0x7fa2000af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa206fb2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa200160be0 0x7fa2001fa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa206fb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2000b51a0 con 0x7fa2001434d0 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa206fb2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa20006f090 con 0x7fa200160be0 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa206fb2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2000b5320 con 0x7fa2000d0ed0 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2001434d0 0x7fa2000af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2001434d0 0x7fa2000af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57996/0 (socket says 172.21.15.67:57996) 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.417+0000 7fa1fffff640 1 -- 172.21.15.67:0/1964493433 learned_addr learned my addr 172.21.15.67:0/1964493433 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa204d4e640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2000d0ed0 0x7fa200144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa20554f640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa200160be0 0x7fa2001fa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fffff640 1 -- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2000d0ed0 msgr2=0x7fa200144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fffff640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2000d0ed0 0x7fa200144470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fffff640 1 -- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa200160be0 msgr2=0x7fa2001fa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fffff640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa200160be0 0x7fa2001fa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fffff640 1 -- 172.21.15.67:0/1964493433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa200155d30 con 0x7fa2001434d0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa20554f640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa200160be0 0x7fa2001fa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fffff640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2001434d0 0x7fa2000af1b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00098f0 tx=0x7fa1f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fd7fa640 1 -- 172.21.15.67:0/1964493433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0037040 con 0x7fa2001434d0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fd7fa640 1 -- 172.21.15.67:0/1964493433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f0042070 con 0x7fa2001434d0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa200155770 con 0x7fa2001434d0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa1fd7fa640 1 -- 172.21.15.67:0/1964493433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0030bf0 con 0x7fa2001434d0 2024-04-25T15:29:16.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.418+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2001f9cf0 con 0x7fa2001434d0 2024-04-25T15:29:16.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.419+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa20015f680 con 0x7fa2001434d0 2024-04-25T15:29:16.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.420+0000 7fa1fd7fa640 1 -- 172.21.15.67:0/1964493433 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa1f0047460 con 0x7fa2001434d0 2024-04-25T15:29:16.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.420+0000 7fa1fd7fa640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1e00744e0 0x7fa1e00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.420+0000 7fa1fd7fa640 1 -- 172.21.15.67:0/1964493433 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fa1f00b5d80 con 0x7fa2001434d0 2024-04-25T15:29:16.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.421+0000 7fa204d4e640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1e00744e0 0x7fa1e00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.421+0000 7fa204d4e640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1e00744e0 0x7fa1e00769f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80012e0 tx=0x7fa1f8004d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.425+0000 7fa1fd7fa640 1 -- 172.21.15.67:0/1964493433 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa1f0032e20 con 0x7fa2001434d0 2024-04-25T15:29:16.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.651+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2000b5ab0 con 0x7fa2001434d0 2024-04-25T15:29:16.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.652+0000 7fa1fd7fa640 1 -- 172.21.15.67:0/1964493433 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5189 (secure 0 0 0) 0x7fa1f00805a0 con 0x7fa2001434d0 2024-04-25T15:29:16.653 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:16.653 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2709900765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2709900765},{"type":"v1","addr":"172.21.15.67:6834","nonce":2709900765}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4104906051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4104906051},{"type":"v1","addr":"172.21.15.176:6836","nonce":4104906051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6655,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2981673460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2981673460},{"type":"v1","addr":"172.21.15.176:6837","nonce":2981673460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:56.417898+0000","modified":"2024-04-25T15:28:59.469250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7128":{"gid":7128,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3429198629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3429198629},{"type":"v1","addr":"172.21.15.67:6835","nonce":3429198629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7128,"qdb_cluster":[7128]},"id":7}]} 2024-04-25T15:29:16.653 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 57 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1e00744e0 msgr2=0x7fa1e00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1e00744e0 0x7fa1e00769f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80012e0 tx=0x7fa1f8004d40 comp rx=0 tx=0).stop 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2001434d0 msgr2=0x7fa2000af1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2001434d0 0x7fa2000af1b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00098f0 tx=0x7fa1f00046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 shutdown_connections 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa1e00744e0 0x7fa1e00769f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa200160be0 0x7fa2001fa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2001434d0 0x7fa2000af1b0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 --2- 172.21.15.67:0/1964493433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2000d0ed0 0x7fa200144470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 >> 172.21.15.67:0/1964493433 conn(0x7fa2000d1ae0 msgr2=0x7fa2001426e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:16.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 shutdown_connections 2024-04-25T15:29:16.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.654+0000 7fa206fb2640 1 -- 172.21.15.67:0/1964493433 wait complete. 2024-04-25T15:29:16.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:29:16.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.778+0000 7f25abb1f640 1 Processor -- start 2024-04-25T15:29:16.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.778+0000 7f25abb1f640 1 -- start start 2024-04-25T15:29:16.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25abb1f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a40b91d0 0x7f25a4161e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25abb1f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25a41623c0 0x7f25a41627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25abb1f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f25a40ce760 0x7f25a40ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25abb1f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f25a40d1ff0 con 0x7f25a41623c0 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25abb1f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f25a40d2890 con 0x7f25a40ce760 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25abb1f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f25a40cfc30 con 0x7f25a40b91d0 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25a90ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25a41623c0 0x7f25a41627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25a90ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25a41623c0 0x7f25a41627d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58004/0 (socket says 172.21.15.67:58004) 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.779+0000 7f25a90ba640 1 -- 172.21.15.67:0/3798848443 learned_addr learned my addr 172.21.15.67:0/3798848443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25a98bb640 1 --2- 172.21.15.67:0/3798848443 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a40b91d0 0x7f25a4161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25aa0bc640 1 --2- 172.21.15.67:0/3798848443 >> v2:172.21.15.176:3300/0 conn(0x7f25a40ce760 0x7f25a40ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25a90ba640 1 -- 172.21.15.67:0/3798848443 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a40b91d0 msgr2=0x7f25a4161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25a90ba640 1 --2- 172.21.15.67:0/3798848443 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a40b91d0 0x7f25a4161e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25a90ba640 1 -- 172.21.15.67:0/3798848443 >> v2:172.21.15.176:3300/0 conn(0x7f25a40ce760 msgr2=0x7f25a40ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25a90ba640 1 --2- 172.21.15.67:0/3798848443 >> v2:172.21.15.176:3300/0 conn(0x7f25a40ce760 0x7f25a40ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25a90ba640 1 -- 172.21.15.67:0/3798848443 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a40d2240 con 0x7f25a41623c0 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.780+0000 7f25a90ba640 1 --2- 172.21.15.67:0/3798848443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a41623c0 0x7f25a41627d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f25a40caa70 tx=0x7f259402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d1a9ee2c93984f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25a88b9640 1 -- 172.21.15.67:0/3798848443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2594037070 con 0x7f25a41623c0 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25a88b9640 1 -- 172.21.15.67:0/3798848443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2594032440 con 0x7f25a41623c0 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25a88b9640 1 -- 172.21.15.67:0/3798848443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2594032810 con 0x7f25a41623c0 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 -- 172.21.15.67:0/3798848443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a41623c0 msgr2=0x7f25a41627d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3798848443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a41623c0 0x7f25a41627d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f25a40caa70 tx=0x7f259402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 -- 172.21.15.67:0/3798848443 shutdown_connections 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3798848443 >> v2:172.21.15.176:3300/0 conn(0x7f25a40ce760 0x7f25a40ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3798848443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a41623c0 0x7f25a41627d0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3798848443 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a40b91d0 0x7f25a4161e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 -- 172.21.15.67:0/3798848443 >> 172.21.15.67:0/3798848443 conn(0x7f25a40d3aa0 msgr2=0x7f25a40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 -- 172.21.15.67:0/3798848443 shutdown_connections 2024-04-25T15:29:16.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.781+0000 7f25abb1f640 1 -- 172.21.15.67:0/3798848443 wait complete. 2024-04-25T15:29:16.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.783+0000 7f25abb1f640 1 Processor -- start 2024-04-25T15:29:16.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.783+0000 7f25abb1f640 1 -- start start 2024-04-25T15:29:16.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.784+0000 7f25abb1f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a40b91d0 0x7f25a41e8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.784+0000 7f25abb1f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a40ce760 0x7f25a414ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.784+0000 7f25abb1f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a41623c0 0x7f25a415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.784+0000 7f25abb1f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25a40b2320 con 0x7f25a40b91d0 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.784+0000 7f25abb1f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4162d10 con 0x7f25a40ce760 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.784+0000 7f25abb1f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f25a40b24a0 con 0x7f25a41623c0 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a40b91d0 0x7f25a41e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a40b91d0 0x7f25a41e8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58014/0 (socket says 172.21.15.67:58014) 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 -- 172.21.15.67:0/3647012953 learned_addr learned my addr 172.21.15.67:0/3647012953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a90ba640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a40ce760 0x7f25a414ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25aa0bc640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a41623c0 0x7f25a415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 -- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a41623c0 msgr2=0x7f25a415ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a41623c0 0x7f25a415ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 -- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a40ce760 msgr2=0x7f25a414ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a40ce760 0x7f25a414ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 -- 172.21.15.67:0/3647012953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a4150440 con 0x7f25a40b91d0 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25aa0bc640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a41623c0 0x7f25a415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25a98bb640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a40b91d0 0x7f25a41e8640 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c5a0 tx=0x7f25a000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f259a7fc640 1 -- 172.21.15.67:0/3647012953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0015070 con 0x7f25a40b91d0 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f259a7fc640 1 -- 172.21.15.67:0/3647012953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a0004220 con 0x7f25a40b91d0 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a40cac60 con 0x7f25a40b91d0 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f259a7fc640 1 -- 172.21.15.67:0/3647012953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0003610 con 0x7f25a40b91d0 2024-04-25T15:29:16.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.785+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a40cb220 con 0x7f25a40b91d0 2024-04-25T15:29:16.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.786+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a4050740 con 0x7f25a40b91d0 2024-04-25T15:29:16.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.787+0000 7f259a7fc640 1 -- 172.21.15.67:0/3647012953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f25a00037b0 con 0x7f25a40b91d0 2024-04-25T15:29:16.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.788+0000 7f259a7fc640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f25880744e0 0x7f25880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:16.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.788+0000 7f259a7fc640 1 -- 172.21.15.67:0/3647012953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f25a0094fa0 con 0x7f25a40b91d0 2024-04-25T15:29:16.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.788+0000 7f25a90ba640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f25880744e0 0x7f25880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:16.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.789+0000 7f25a90ba640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f25880744e0 0x7f25880769f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2594034000 tx=0x7f259402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:16.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.793+0000 7f259a7fc640 1 -- 172.21.15.67:0/3647012953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f25a4150440 con 0x7f25a40b91d0 2024-04-25T15:29:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.979+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25a40d4700 con 0x7f25a40b91d0 2024-04-25T15:29:16.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.980+0000 7f259a7fc640 1 -- 172.21.15.67:0/3647012953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16572 (secure 0 0 0) 0x7f25a005f040 con 0x7f25a40b91d0 2024-04-25T15:29:16.981 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:16.981 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:58.058412+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f25880744e0 msgr2=0x7f25880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f25880744e0 0x7f25880769f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2594034000 tx=0x7f259402e000 comp rx=0 tx=0).stop 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a40b91d0 msgr2=0x7f25a41e8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a40b91d0 0x7f25a41e8640 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f25a000c5a0 tx=0x7f25a000ca60 comp rx=0 tx=0).stop 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 shutdown_connections 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f25880744e0 0x7f25880769f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a41623c0 0x7f25a415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a40ce760 0x7f25a414ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 --2- 172.21.15.67:0/3647012953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25a40b91d0 0x7f25a41e8640 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 >> 172.21.15.67:0/3647012953 conn(0x7f25a40d3aa0 msgr2=0x7f25a40b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 shutdown_connections 2024-04-25T15:29:16.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:16.982+0000 7f25abb1f640 1 -- 172.21.15.67:0/3647012953 wait complete. 2024-04-25T15:29:16.994 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:29:16.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:29:17.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.105+0000 7f1db0757640 1 Processor -- start 2024-04-25T15:29:17.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.105+0000 7f1db0757640 1 -- start start 2024-04-25T15:29:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1db0757640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1da80b7750 0x7f1da80d1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1db0757640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 0x7f1da8164800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1db0757640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da80b3af0 con 0x7f1da8164d40 2024-04-25T15:29:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1db0757640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1da8164320 con 0x7f1da80b7750 2024-04-25T15:29:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1db0757640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1da80b2f70 con 0x7f1da80cf610 2024-04-25T15:29:17.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1daecf4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1da8164d40 0x7f1da806d7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38448/0 (socket says 172.21.15.67:38448) 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1daecf4640 1 -- 172.21.15.67:0/1470741918 learned_addr learned my addr 172.21.15.67:0/1470741918 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502403417 0 0) 0x7f1da80b3af0 con 0x7f1da8164d40 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1dae4f3640 1 --2- 172.21.15.67:0/1470741918 >> v2:172.21.15.176:3300/0 conn(0x7f1da80b7750 0x7f1da80d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d90003520 con 0x7f1da8164d40 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.107+0000 7f1dadcf2640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 0x7f1da8164800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1606633038 0 0) 0x7f1d90003520 con 0x7f1da8164d40 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 msgr2=0x7f1da8164800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 0x7f1da8164800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 >> v2:172.21.15.176:3300/0 conn(0x7f1da80b7750 msgr2=0x7f1da80d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 --2- 172.21.15.67:0/1470741918 >> v2:172.21.15.176:3300/0 conn(0x7f1da80b7750 0x7f1da80d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da80b3250 con 0x7f1da8164d40 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f1d98001670 con 0x7f1da8164d40 2024-04-25T15:29:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 >> v1:172.21.15.67:6789/0 conn(0x7f1da8164d40 legacy=0x7f1da806d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900039b0 0x7f1d90003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 0x7f1d90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d90004d40 0x7f1d90005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1da8164320 con 0x7f1d90004d40 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1da80b2f70 con 0x7f1d900039b0 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9000bac0 con 0x7f1da80cf610 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1daecf4640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d90004d40 0x7f1d90005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dadcf2640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 0x7f1d90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1dae4f3640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900039b0 0x7f1d90003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1daecf4640 1 -- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 msgr2=0x7f1d90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1daecf4640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da80cf610 0x7f1d90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1daecf4640 1 -- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900039b0 msgr2=0x7f1d90003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1daecf4640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900039b0 0x7f1d90003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.108+0000 7f1daecf4640 1 -- 172.21.15.67:0/1470741918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d90003520 con 0x7f1d90004d40 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.109+0000 7f1daecf4640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d90004d40 0x7f1d90005150 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f1d98001050 tx=0x7f1d90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c87cb77dc60f042 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.109+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98007d70 con 0x7f1d90004d40 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.109+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d98005af0 con 0x7f1d90004d40 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.109+0000 7f1dad4f1640 1 -- 172.21.15.67:0/1470741918 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98007070 con 0x7f1d90004d40 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.110+0000 7f1db0757640 1 -- 172.21.15.67:0/1470741918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d90004d40 msgr2=0x7f1d90005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.110+0000 7f1db0757640 1 --2- 172.21.15.67:0/1470741918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d90004d40 0x7f1d90005150 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f1d98001050 tx=0x7f1d90005830 comp rx=0 tx=0).stop 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.110+0000 7f1daecf4640 1 -- 172.21.15.67:0/1470741918 reap_dead start 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.110+0000 7f1db0757640 1 -- 172.21.15.67:0/1470741918 shutdown_connections 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.110+0000 7f1db0757640 1 -- 172.21.15.67:0/1470741918 >> 172.21.15.67:0/1470741918 conn(0x7f1da809b7d0 msgr2=0x7f1da80b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.110+0000 7f1db0757640 1 -- 172.21.15.67:0/1470741918 shutdown_connections 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.110+0000 7f1db0757640 1 -- 172.21.15.67:0/1470741918 wait complete. 2024-04-25T15:29:17.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.112+0000 7f1db0757640 1 Processor -- start 2024-04-25T15:29:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.112+0000 7f1db0757640 1 -- start start 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1db0757640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d90004d40 0x7f1da809fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1db0757640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80ce050 0x7f1da81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1db0757640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da81548e0 0x7f1da81f9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1db0757640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1da80cfa20 con 0x7f1da80ce050 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1db0757640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1da80b3810 con 0x7f1d90004d40 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1db0757640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1da806def0 con 0x7f1da81548e0 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80ce050 0x7f1da81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80ce050 0x7f1da81543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58042/0 (socket says 172.21.15.67:58042) 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 -- 172.21.15.67:0/2802295437 learned_addr learned my addr 172.21.15.67:0/2802295437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dae4f3640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d90004d40 0x7f1da809fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1daecf4640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da81548e0 0x7f1da81f9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 -- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da81548e0 msgr2=0x7f1da81f9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da81548e0 0x7f1da81f9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 -- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d90004d40 msgr2=0x7f1da809fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d90004d40 0x7f1da809fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dadcf2640 1 -- 172.21.15.67:0/2802295437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da809d550 con 0x7f1da80ce050 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.113+0000 7f1dae4f3640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d90004d40 0x7f1da809fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.114+0000 7f1dadcf2640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80ce050 0x7f1da81543a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0119d0 tx=0x7f1d9c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.114+0000 7f1da6ffd640 1 -- 172.21.15.67:0/2802295437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00fb50 con 0x7f1da80ce050 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.114+0000 7f1da6ffd640 1 -- 172.21.15.67:0/2802295437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9c00fcf0 con 0x7f1da80ce050 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.114+0000 7f1da6ffd640 1 -- 172.21.15.67:0/2802295437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00fea0 con 0x7f1da80ce050 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.114+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da809c4b0 con 0x7f1da80ce050 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.114+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da809ca70 con 0x7f1da80ce050 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.115+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da8050740 con 0x7f1da80ce050 2024-04-25T15:29:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.115+0000 7f1da6ffd640 1 -- 172.21.15.67:0/2802295437 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1d9c012040 con 0x7f1da80ce050 2024-04-25T15:29:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.116+0000 7f1da6ffd640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d940744b0 0x7f1d940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.116+0000 7f1da6ffd640 1 -- 172.21.15.67:0/2802295437 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f1d9c095900 con 0x7f1da80ce050 2024-04-25T15:29:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.116+0000 7f1dae4f3640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d940744b0 0x7f1d940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.117+0000 7f1dae4f3640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d940744b0 0x7f1d940769c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1da80b4dc0 tx=0x7f1da000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.121+0000 7f1da6ffd640 1 -- 172.21.15.67:0/2802295437 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d9c019030 con 0x7f1da80ce050 2024-04-25T15:29:17.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.343+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1da80d2f00 con 0x7f1da80ce050 2024-04-25T15:29:17.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.344+0000 7f1da6ffd640 1 -- 172.21.15.67:0/2802295437 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5189 (secure 0 0 0) 0x7f1d9c060090 con 0x7f1da80ce050 2024-04-25T15:29:17.345 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:17.345 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2709900765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2709900765},{"type":"v1","addr":"172.21.15.67:6834","nonce":2709900765}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4104906051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4104906051},{"type":"v1","addr":"172.21.15.176:6836","nonce":4104906051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6655,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2981673460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2981673460},{"type":"v1","addr":"172.21.15.176:6837","nonce":2981673460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:56.417898+0000","modified":"2024-04-25T15:28:59.469250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7128":{"gid":7128,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3429198629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3429198629},{"type":"v1","addr":"172.21.15.67:6835","nonce":3429198629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7128,"qdb_cluster":[7128]},"id":7}]} 2024-04-25T15:29:17.345 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 57 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d940744b0 msgr2=0x7f1d940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d940744b0 0x7f1d940769c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1da80b4dc0 tx=0x7f1da000b040 comp rx=0 tx=0).stop 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80ce050 msgr2=0x7f1da81543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80ce050 0x7f1da81543a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0119d0 tx=0x7f1d9c011e90 comp rx=0 tx=0).stop 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 shutdown_connections 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1d940744b0 0x7f1d940769c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1da81548e0 0x7f1da81f9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80ce050 0x7f1da81543a0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 --2- 172.21.15.67:0/2802295437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d90004d40 0x7f1da809fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 >> 172.21.15.67:0/2802295437 conn(0x7f1da809b7d0 msgr2=0x7f1da80cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:17.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 shutdown_connections 2024-04-25T15:29:17.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.346+0000 7f1db0757640 1 -- 172.21.15.67:0/2802295437 wait complete. 2024-04-25T15:29:17.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:29:17.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.470+0000 7f55b5714640 1 Processor -- start 2024-04-25T15:29:17.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.470+0000 7f55b5714640 1 -- start start 2024-04-25T15:29:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55b5714640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55b01443c0 0x7f55b006fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55b5714640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f55b0070370 0x7f55b00cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55b5714640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55b00cff30 con 0x7f55b00cfb20 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55b5714640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f55b00708d0 con 0x7f55b01443c0 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55b5714640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f55b015f160 con 0x7f55b0070370 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55aeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55b01443c0 0x7f55b006fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55aeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f55b01443c0 0x7f55b006fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58046/0 (socket says 172.21.15.67:58046) 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.471+0000 7f55aeffd640 1 -- 172.21.15.67:0/2435643886 learned_addr learned my addr 172.21.15.67:0/2435643886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.472+0000 7f55a7fff640 1 --2- 172.21.15.67:0/2435643886 >> v2:172.21.15.176:3300/0 conn(0x7f55b0070370 0x7f55b00cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.472+0000 7f55aeffd640 1 -- 172.21.15.67:0/2435643886 >> v2:172.21.15.176:3300/0 conn(0x7f55b0070370 msgr2=0x7f55b00cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.472+0000 7f55aeffd640 1 --2- 172.21.15.67:0/2435643886 >> v2:172.21.15.176:3300/0 conn(0x7f55b0070370 0x7f55b00cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.472+0000 7f55aeffd640 1 -- 172.21.15.67:0/2435643886 >> v1:172.21.15.176:6789/0 conn(0x7f55b00cfb20 legacy=0x7f55b00ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.472+0000 7f55aeffd640 1 -- 172.21.15.67:0/2435643886 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b015f5f0 con 0x7f55b01443c0 2024-04-25T15:29:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55aeffd640 1 --2- 172.21.15.67:0/2435643886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b01443c0 0x7f55b006fe30 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f55b006f920 tx=0x7f55a002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2082e4f460ffaf2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55ae7fc640 1 -- 172.21.15.67:0/2435643886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0037070 con 0x7f55b01443c0 2024-04-25T15:29:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55ae7fc640 1 -- 172.21.15.67:0/2435643886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a0032440 con 0x7f55b01443c0 2024-04-25T15:29:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55ae7fc640 1 -- 172.21.15.67:0/2435643886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0032810 con 0x7f55b01443c0 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55b5714640 1 -- 172.21.15.67:0/2435643886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b01443c0 msgr2=0x7f55b006fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55b5714640 1 --2- 172.21.15.67:0/2435643886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b01443c0 0x7f55b006fe30 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f55b006f920 tx=0x7f55a002a230 comp rx=0 tx=0).stop 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55b5714640 1 -- 172.21.15.67:0/2435643886 shutdown_connections 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55b5714640 1 --2- 172.21.15.67:0/2435643886 >> v2:172.21.15.176:3300/0 conn(0x7f55b0070370 0x7f55b00cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55b5714640 1 --2- 172.21.15.67:0/2435643886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b01443c0 0x7f55b006fe30 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.473+0000 7f55b5714640 1 -- 172.21.15.67:0/2435643886 >> 172.21.15.67:0/2435643886 conn(0x7f55b00d1ae0 msgr2=0x7f55b00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.474+0000 7f55b5714640 1 -- 172.21.15.67:0/2435643886 shutdown_connections 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.474+0000 7f55b5714640 1 -- 172.21.15.67:0/2435643886 wait complete. 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.475+0000 7f55b5714640 1 Processor -- start 2024-04-25T15:29:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.475+0000 7f55b5714640 1 -- start start 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55b5714640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b0070370 0x7f55b013c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55b5714640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55b00cfb20 0x7f55b00af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55b5714640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55b01443c0 0x7f55b0142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55b5714640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55b015edf0 con 0x7f55b0070370 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55b5714640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f55b015eb10 con 0x7f55b00cfb20 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55b5714640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f55b006d5d0 con 0x7f55b01443c0 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b0070370 0x7f55b013c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b0070370 0x7f55b013c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58062/0 (socket says 172.21.15.67:58062) 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 -- 172.21.15.67:0/1979409211 learned_addr learned my addr 172.21.15.67:0/1979409211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55a7fff640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55b00cfb20 0x7f55b00af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55af7fe640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55b01443c0 0x7f55b0142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 -- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55b01443c0 msgr2=0x7f55b0142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55b01443c0 0x7f55b0142d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 -- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55b00cfb20 msgr2=0x7f55b00af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55b00cfb20 0x7f55b00af000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 -- 172.21.15.67:0/1979409211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b00b2260 con 0x7f55b0070370 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55a7fff640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55b00cfb20 0x7f55b00af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55aeffd640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b0070370 0x7f55b013c5f0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55a77fe640 1 -- 172.21.15.67:0/1979409211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0041070 con 0x7f55b0070370 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.477+0000 7f55a77fe640 1 -- 172.21.15.67:0/1979409211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a003c870 con 0x7f55b0070370 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.478+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b00b4c50 con 0x7f55b0070370 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.478+0000 7f55a77fe640 1 -- 172.21.15.67:0/1979409211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0037040 con 0x7f55b0070370 2024-04-25T15:29:17.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.478+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b00af540 con 0x7f55b0070370 2024-04-25T15:29:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.479+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b00b4a70 con 0x7f55b0070370 2024-04-25T15:29:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.479+0000 7f55a77fe640 1 -- 172.21.15.67:0/1979409211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f55a0044070 con 0x7f55b0070370 2024-04-25T15:29:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.480+0000 7f55a77fe640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55900744b0 0x7f55900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.480+0000 7f55a77fe640 1 -- 172.21.15.67:0/1979409211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f55a00b5ce0 con 0x7f55b0070370 2024-04-25T15:29:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.480+0000 7f55a7fff640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55900744b0 0x7f55900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.481+0000 7f55a7fff640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55900744b0 0x7f55900769c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f55b0161300 tx=0x7f5598008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.484+0000 7f55a77fe640 1 -- 172.21.15.67:0/1979409211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55b00b4a70 con 0x7f55b0070370 2024-04-25T15:29:17.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.718+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55b0050740 con 0x7f55b0070370 2024-04-25T15:29:17.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.718+0000 7f55a77fe640 1 -- 172.21.15.67:0/1979409211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5189 (secure 0 0 0) 0x7f55b00cff30 con 0x7f55b0070370 2024-04-25T15:29:17.719 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:17.720 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2709900765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2709900765},{"type":"v1","addr":"172.21.15.67:6834","nonce":2709900765}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4104906051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4104906051},{"type":"v1","addr":"172.21.15.176:6836","nonce":4104906051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6655,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2981673460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2981673460},{"type":"v1","addr":"172.21.15.176:6837","nonce":2981673460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:28:56.417898+0000","modified":"2024-04-25T15:28:59.469250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7128":{"gid":7128,"name":"a","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3429198629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3429198629},{"type":"v1","addr":"172.21.15.67:6835","nonce":3429198629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7128,"qdb_cluster":[7128]},"id":7}]} 2024-04-25T15:29:17.720 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 57 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.720+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55900744b0 msgr2=0x7f55900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.720+0000 7f55b5714640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55900744b0 0x7f55900769c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f55b0161300 tx=0x7f5598008040 comp rx=0 tx=0).stop 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.720+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b0070370 msgr2=0x7f55b013c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.720+0000 7f55b5714640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b0070370 0x7f55b013c5f0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a00046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 shutdown_connections 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f55900744b0 0x7f55900769c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55b01443c0 0x7f55b0142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55b00cfb20 0x7f55b00af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 --2- 172.21.15.67:0/1979409211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55b0070370 0x7f55b013c5f0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 >> 172.21.15.67:0/1979409211 conn(0x7f55b00d1ae0 msgr2=0x7f55b0073050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 shutdown_connections 2024-04-25T15:29:17.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.721+0000 7f55b5714640 1 -- 172.21.15.67:0/1979409211 wait complete. 2024-04-25T15:29:17.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:29:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.844+0000 7f91baae0640 1 Processor -- start 2024-04-25T15:29:17.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.844+0000 7f91baae0640 1 -- start start 2024-04-25T15:29:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.845+0000 7f91baae0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b40d4130 0x7f91b40d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91baae0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91b406dcf0 0x7f91b40d22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91baae0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91b40b3c10 con 0x7f91b406d8e0 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91baae0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f91b40b9520 con 0x7f91b406dcf0 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91baae0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f91b40b4820 con 0x7f91b40d4130 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b907d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91b406dcf0 0x7f91b40d22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b887c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f91b406d8e0 0x7f91b40d3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38988/0 (socket says 172.21.15.67:38988) 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b887c640 1 -- 172.21.15.67:0/2820726630 learned_addr learned my addr 172.21.15.67:0/2820726630 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b907d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91b406dcf0 0x7f91b40d22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35704/0 (socket says 172.21.15.67:35704) 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b3fff640 1 --2- 172.21.15.67:0/2820726630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b40d4130 0x7f91b40d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b907d640 1 -- 172.21.15.67:0/2820726630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b40d4130 msgr2=0x7f91b40d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b907d640 1 --2- 172.21.15.67:0/2820726630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b40d4130 0x7f91b40d1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b907d640 1 -- 172.21.15.67:0/2820726630 >> v1:172.21.15.67:6789/0 conn(0x7f91b406d8e0 legacy=0x7f91b40d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.846+0000 7f91b907d640 1 -- 172.21.15.67:0/2820726630 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b40b3ef0 con 0x7f91b406dcf0 2024-04-25T15:29:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.847+0000 7f91b907d640 1 --2- 172.21.15.67:0/2820726630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b406dcf0 0x7f91b40d22a0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009d90 tx=0x7f91ac02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aad14acc79eeb36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.847+0000 7f91b37fe640 1 -- 172.21.15.67:0/2820726630 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac031600 con 0x7f91b406dcf0 2024-04-25T15:29:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.847+0000 7f91b37fe640 1 -- 172.21.15.67:0/2820726630 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ac0317a0 con 0x7f91b406dcf0 2024-04-25T15:29:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.847+0000 7f91b37fe640 1 -- 172.21.15.67:0/2820726630 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac031b70 con 0x7f91b406dcf0 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 -- 172.21.15.67:0/2820726630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b406dcf0 msgr2=0x7f91b40d22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 --2- 172.21.15.67:0/2820726630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b406dcf0 0x7f91b40d22a0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009d90 tx=0x7f91ac02a5d0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 -- 172.21.15.67:0/2820726630 shutdown_connections 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 --2- 172.21.15.67:0/2820726630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b406dcf0 0x7f91b40d22a0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 --2- 172.21.15.67:0/2820726630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b40d4130 0x7f91b40d1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 -- 172.21.15.67:0/2820726630 >> 172.21.15.67:0/2820726630 conn(0x7f91b408d110 msgr2=0x7f91b408d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 -- 172.21.15.67:0/2820726630 shutdown_connections 2024-04-25T15:29:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.848+0000 7f91baae0640 1 -- 172.21.15.67:0/2820726630 wait complete. 2024-04-25T15:29:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.849+0000 7f91baae0640 1 Processor -- start 2024-04-25T15:29:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.850+0000 7f91baae0640 1 -- start start 2024-04-25T15:29:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91baae0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b406d8e0 0x7f91b40a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91baae0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91b406dcf0 0x7f91b40cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91baae0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b41fa730 0x7f91b41e9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91baae0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91b40b38e0 con 0x7f91b41fa730 2024-04-25T15:29:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91baae0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f91b40b44b0 con 0x7f91b406dcf0 2024-04-25T15:29:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91baae0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f91b40d27e0 con 0x7f91b406d8e0 2024-04-25T15:29:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b41fa730 0x7f91b41e9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b41fa730 0x7f91b41e9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35716/0 (socket says 172.21.15.67:35716) 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 -- 172.21.15.67:0/3934975239 learned_addr learned my addr 172.21.15.67:0/3934975239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b887c640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b406d8e0 0x7f91b40a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b3fff640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91b406dcf0 0x7f91b40cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 -- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b406d8e0 msgr2=0x7f91b40a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b406d8e0 0x7f91b40a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 -- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91b406dcf0 msgr2=0x7f91b40cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91b406dcf0 0x7f91b40cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.851+0000 7f91b907d640 1 -- 172.21.15.67:0/3934975239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b4150890 con 0x7f91b41fa730 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.852+0000 7f91b907d640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b41fa730 0x7f91b41e9b50 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009eb0 tx=0x7f91ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.852+0000 7f91b17fa640 1 -- 172.21.15.67:0/3934975239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac03f070 con 0x7f91b41fa730 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.852+0000 7f91b17fa640 1 -- 172.21.15.67:0/3934975239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ac030b60 con 0x7f91b41fa730 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.852+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b4150b70 con 0x7f91b41fa730 2024-04-25T15:29:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.852+0000 7f91b17fa640 1 -- 172.21.15.67:0/3934975239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ac009450 con 0x7f91b41fa730 2024-04-25T15:29:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.852+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b4151130 con 0x7f91b41fa730 2024-04-25T15:29:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.853+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b4050740 con 0x7f91b41fa730 2024-04-25T15:29:17.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.853+0000 7f91b17fa640 1 -- 172.21.15.67:0/3934975239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f91ac03b410 con 0x7f91b41fa730 2024-04-25T15:29:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.854+0000 7f91b17fa640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91980744e0 0x7f91980769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.854+0000 7f91b17fa640 1 -- 172.21.15.67:0/3934975239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f91ac0b66c0 con 0x7f91b41fa730 2024-04-25T15:29:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.854+0000 7f91b887c640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91980744e0 0x7f91980769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:17.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.855+0000 7f91b887c640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91980744e0 0x7f91980769f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f91a8007fd0 tx=0x7f91a8007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:17.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:17.859+0000 7f91b17fa640 1 -- 172.21.15.67:0/3934975239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91ac080800 con 0x7f91b41fa730 2024-04-25T15:29:18.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.049+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91b40717d0 con 0x7f91b41fa730 2024-04-25T15:29:18.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.050+0000 7f91b17fa640 1 -- 172.21.15.67:0/3934975239 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16572 (secure 0 0 0) 0x7f91ac084b00 con 0x7f91b41fa730 2024-04-25T15:29:18.051 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:18.051 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:28:58.058412+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:28:54.107977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-25T15:28:55.386227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:29:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91980744e0 msgr2=0x7f91980769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91980744e0 0x7f91980769f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f91a8007fd0 tx=0x7f91a8007e80 comp rx=0 tx=0).stop 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b41fa730 msgr2=0x7f91b41e9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b41fa730 0x7f91b41e9b50 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009eb0 tx=0x7f91ac004a40 comp rx=0 tx=0).stop 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 shutdown_connections 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f91980744e0 0x7f91980769f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91b41fa730 0x7f91b41e9b50 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f91b406dcf0 0x7f91b40cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 --2- 172.21.15.67:0/3934975239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91b406d8e0 0x7f91b40a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 >> 172.21.15.67:0/3934975239 conn(0x7f91b408d110 msgr2=0x7f91b40b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 shutdown_connections 2024-04-25T15:29:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.052+0000 7f91baae0640 1 -- 172.21.15.67:0/3934975239 wait complete. 2024-04-25T15:29:18.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:29:18.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.177+0000 7f38241ce640 1 Processor -- start 2024-04-25T15:29:18.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.177+0000 7f38241ce640 1 -- start start 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.178+0000 7f38241ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0d4570 0x7f381c0b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.178+0000 7f38241ce640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381c06d870 con 0x7f381c0b2dc0 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.178+0000 7f38241ce640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381c06db10 con 0x7f381c06e8f0 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.178+0000 7f38241ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f381c06e460 con 0x7f381c0d4570 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.178+0000 7f382276b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f381c0b2dc0 0x7f381c0b31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39004/0 (socket says 172.21.15.67:39004) 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.178+0000 7f382276b640 1 -- 172.21.15.67:0/3089038298 learned_addr learned my addr 172.21.15.67:0/3089038298 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.178+0000 7f3821f6a640 1 --2- 172.21.15.67:0/3089038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0d4570 0x7f381c0b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.179+0000 7f3820f68640 1 -- 172.21.15.67:0/3089038298 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2947279951 0 0) 0x7f381c06d870 con 0x7f381c0b2dc0 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.179+0000 7f3820f68640 1 -- 172.21.15.67:0/3089038298 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3808003520 con 0x7f381c0b2dc0 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.179+0000 7f3820f68640 1 -- 172.21.15.67:0/3089038298 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969129674 0 0) 0x7f381c06db10 con 0x7f381c06e8f0 2024-04-25T15:29:18.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.179+0000 7f3820f68640 1 -- 172.21.15.67:0/3089038298 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f381c06d870 con 0x7f381c06e8f0 2024-04-25T15:29:18.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.179+0000 7f3821f6a640 1 -- 172.21.15.67:0/3089038298 >> v1:172.21.15.176:6789/0 conn(0x7f381c06e8f0 legacy=0x7f381c0d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.179+0000 7f3821f6a640 1 -- 172.21.15.67:0/3089038298 >> v1:172.21.15.67:6789/0 conn(0x7f381c0b2dc0 legacy=0x7f381c0b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.179+0000 7f3821f6a640 1 -- 172.21.15.67:0/3089038298 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c06e0f0 con 0x7f381c0d4570 2024-04-25T15:29:18.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.180+0000 7f3821f6a640 1 --2- 172.21.15.67:0/3089038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0d4570 0x7f381c0b4830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38180097c0 tx=0x7f381802a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d40fead8f1ca57bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:18.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.180+0000 7f3820f68640 1 -- 172.21.15.67:0/3089038298 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818037070 con 0x7f381c0d4570 2024-04-25T15:29:18.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.180+0000 7f3820f68640 1 -- 172.21.15.67:0/3089038298 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3818032440 con 0x7f381c0d4570 2024-04-25T15:29:18.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.180+0000 7f3820f68640 1 -- 172.21.15.67:0/3089038298 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38180327f0 con 0x7f381c0d4570 2024-04-25T15:29:18.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.181+0000 7f38241ce640 1 -- 172.21.15.67:0/3089038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0d4570 msgr2=0x7f381c0b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.181+0000 7f38241ce640 1 --2- 172.21.15.67:0/3089038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0d4570 0x7f381c0b4830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38180097c0 tx=0x7f381802a160 comp rx=0 tx=0).stop 2024-04-25T15:29:18.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.181+0000 7f38241ce640 1 -- 172.21.15.67:0/3089038298 shutdown_connections 2024-04-25T15:29:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.181+0000 7f38241ce640 1 --2- 172.21.15.67:0/3089038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0d4570 0x7f381c0b4830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.181+0000 7f38241ce640 1 -- 172.21.15.67:0/3089038298 >> 172.21.15.67:0/3089038298 conn(0x7f381c0d3a50 msgr2=0x7f381c0d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.181+0000 7f38241ce640 1 -- 172.21.15.67:0/3089038298 shutdown_connections 2024-04-25T15:29:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.181+0000 7f38241ce640 1 -- 172.21.15.67:0/3089038298 wait complete. 2024-04-25T15:29:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.182+0000 7f38241ce640 1 Processor -- start 2024-04-25T15:29:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.183+0000 7f38241ce640 1 -- start start 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f38241ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c06e8f0 0x7f381c0b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f38241ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0b2dc0 0x7f381c0b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f38241ce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f381c0d4570 0x7f381c1f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f38241ce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f381c06ed00 con 0x7f381c06e8f0 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f38241ce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f381c0d0fb0 con 0x7f381c0d4570 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f38241ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f381c0b38e0 con 0x7f381c0b2dc0 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c06e8f0 0x7f381c0b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c06e8f0 0x7f381c0b88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35722/0 (socket says 172.21.15.67:35722) 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 -- 172.21.15.67:0/2360100072 learned_addr learned my addr 172.21.15.67:0/2360100072 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f382276b640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f381c0d4570 0x7f381c1f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821769640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0b2dc0 0x7f381c0b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 -- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0b2dc0 msgr2=0x7f381c0b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0b2dc0 0x7f381c0b8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 -- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f381c0d4570 msgr2=0x7f381c1f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f381c0d4570 0x7f381c1f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f3821f6a640 1 -- 172.21.15.67:0/2360100072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c1fa210 con 0x7f381c06e8f0 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.184+0000 7f382276b640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f381c0d4570 0x7f381c1f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.185+0000 7f3821f6a640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c06e8f0 0x7f381c0b88b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f38180098e0 tx=0x7f38180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.185+0000 7f38127fc640 1 -- 172.21.15.67:0/2360100072 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818037040 con 0x7f381c06e8f0 2024-04-25T15:29:18.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.185+0000 7f38127fc640 1 -- 172.21.15.67:0/2360100072 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3818042070 con 0x7f381c06e8f0 2024-04-25T15:29:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.185+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c1fa4f0 con 0x7f381c06e8f0 2024-04-25T15:29:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.185+0000 7f38127fc640 1 -- 172.21.15.67:0/2360100072 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818032810 con 0x7f381c06e8f0 2024-04-25T15:29:18.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.185+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c1f77f0 con 0x7f381c06e8f0 2024-04-25T15:29:18.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.186+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c1f75e0 con 0x7f381c06e8f0 2024-04-25T15:29:18.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.186+0000 7f38127fc640 1 -- 172.21.15.67:0/2360100072 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f38180329b0 con 0x7f381c06e8f0 2024-04-25T15:29:18.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.187+0000 7f38127fc640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f38000744b0 0x7f38000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.187+0000 7f38127fc640 1 -- 172.21.15.67:0/2360100072 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f38180b5af0 con 0x7f381c06e8f0 2024-04-25T15:29:18.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.187+0000 7f3821769640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f38000744b0 0x7f38000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.187+0000 7f3821769640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f38000744b0 0x7f38000769c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f381c0a1830 tx=0x7f380c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:18.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.191+0000 7f38127fc640 1 -- 172.21.15.67:0/2360100072 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3818080280 con 0x7f381c06e8f0 2024-04-25T15:29:18.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.416+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f381c050740 con 0x7f381c06e8f0 2024-04-25T15:29:18.417 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:29:18.416+0000 7f3e02b50640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T15:29:18.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.418+0000 7f38127fc640 1 -- 172.21.15.67:0/2360100072 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) ==== 161+0+0 (secure 0 0 0) 0x7f381807fb80 con 0x7f381c06e8f0 2024-04-25T15:29:18.419 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:29:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.420+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f38000744b0 msgr2=0x7f38000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.420+0000 7f38241ce640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f38000744b0 0x7f38000769c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f381c0a1830 tx=0x7f380c009040 comp rx=0 tx=0).stop 2024-04-25T15:29:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.420+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c06e8f0 msgr2=0x7f381c0b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.420+0000 7f38241ce640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c06e8f0 0x7f381c0b88b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f38180098e0 tx=0x7f38180046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.421+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 shutdown_connections 2024-04-25T15:29:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.421+0000 7f38241ce640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f38000744b0 0x7f38000769c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.421+0000 7f38241ce640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f381c0d4570 0x7f381c1f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.421+0000 7f38241ce640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f381c0b2dc0 0x7f381c0b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.421+0000 7f38241ce640 1 --2- 172.21.15.67:0/2360100072 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f381c06e8f0 0x7f381c0b88b0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.421+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 >> 172.21.15.67:0/2360100072 conn(0x7f381c0d3a50 msgr2=0x7f381c0b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.422+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 shutdown_connections 2024-04-25T15:29:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.422+0000 7f38241ce640 1 -- 172.21.15.67:0/2360100072 wait complete. 2024-04-25T15:29:18.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:29:18.487 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:29:18.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.550+0000 7f1a12d46640 1 Processor -- start 2024-04-25T15:29:18.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.550+0000 7f1a12d46640 1 -- start start 2024-04-25T15:29:18.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a12d46640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1a0c06ccd0 0x7f1a0c0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a12d46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d4a50 0x7f1a0c0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a12d46640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a0c0d1850 con 0x7f1a0c0d0e00 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a12d46640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1a0c160de0 con 0x7f1a0c06ccd0 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a12d46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a0c1610c0 con 0x7f1a0c0d4a50 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a112e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d4a50 0x7f1a0c0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a0bfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1a0c06ccd0 0x7f1a0c0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a10ae2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f1a0c0d0e00 0x7f1a0c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:43110/0 (socket says 172.21.15.67:43110) 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a112e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d4a50 0x7f1a0c0d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:55412/0 (socket says 172.21.15.67:55412) 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a0bfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1a0c06ccd0 0x7f1a0c0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:37968/0 (socket says 172.21.15.67:37968) 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a10ae2640 1 -- 172.21.15.67:0/1441474019 learned_addr learned my addr 172.21.15.67:0/1441474019 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:18.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a0b7fe640 1 -- 172.21.15.67:0/1441474019 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650887165 0 0) 0x7f1a0c0d1850 con 0x7f1a0c0d0e00 2024-04-25T15:29:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a0b7fe640 1 -- 172.21.15.67:0/1441474019 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19f8003520 con 0x7f1a0c0d0e00 2024-04-25T15:29:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a0bfff640 1 -- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d4a50 msgr2=0x7f1a0c0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a0bfff640 1 --2- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d4a50 0x7f1a0c0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.552+0000 7f1a0bfff640 1 -- 172.21.15.67:0/1441474019 >> v1:172.21.15.176:6789/0 conn(0x7f1a0c0d0e00 legacy=0x7f1a0c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.553+0000 7f1a0bfff640 1 -- 172.21.15.67:0/1441474019 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0c1613a0 con 0x7f1a0c06ccd0 2024-04-25T15:29:18.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.553+0000 7f1a112e3640 1 --2- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d4a50 0x7f1a0c0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.553+0000 7f1a0bfff640 1 --2- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c06ccd0 0x7f1a0c0d1210 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f19f40097a0 tx=0x7f19f402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69d8c17b03ab935f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:18.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.553+0000 7f1a0b7fe640 1 -- 172.21.15.67:0/1441474019 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4037070 con 0x7f1a0c06ccd0 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.553+0000 7f1a0b7fe640 1 -- 172.21.15.67:0/1441474019 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f4032440 con 0x7f1a0c06ccd0 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.553+0000 7f1a0b7fe640 1 -- 172.21.15.67:0/1441474019 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f4032810 con 0x7f1a0c06ccd0 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 -- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c06ccd0 msgr2=0x7f1a0c0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 --2- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c06ccd0 0x7f1a0c0d1210 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f19f40097a0 tx=0x7f19f402a180 comp rx=0 tx=0).stop 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 -- 172.21.15.67:0/1441474019 shutdown_connections 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 --2- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d4a50 0x7f1a0c0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 --2- 172.21.15.67:0/1441474019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c06ccd0 0x7f1a0c0d1210 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 -- 172.21.15.67:0/1441474019 >> 172.21.15.67:0/1441474019 conn(0x7f1a0c0d5d90 msgr2=0x7f1a0c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 -- 172.21.15.67:0/1441474019 shutdown_connections 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.554+0000 7f1a12d46640 1 -- 172.21.15.67:0/1441474019 wait complete. 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.556+0000 7f1a12d46640 1 Processor -- start 2024-04-25T15:29:18.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.556+0000 7f1a12d46640 1 -- start start 2024-04-25T15:29:18.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.557+0000 7f1a12d46640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d0e00 0x7f1a0c154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.557+0000 7f1a12d46640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c0d4a50 0x7f1a0c154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.557+0000 7f1a12d46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0ce650 0x7f1a0c0cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.557+0000 7f1a12d46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c164e00 con 0x7f1a0c0ce650 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.557+0000 7f1a12d46640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c0b5320 con 0x7f1a0c0d4a50 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.557+0000 7f1a12d46640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a0c164f80 con 0x7f1a0c0d0e00 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0ce650 0x7f1a0c0cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0ce650 0x7f1a0c0cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35744/0 (socket says 172.21.15.67:35744) 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 -- 172.21.15.67:0/586321445 learned_addr learned my addr 172.21.15.67:0/586321445 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a0bfff640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c0d4a50 0x7f1a0c154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a10ae2640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d0e00 0x7f1a0c154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 -- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d0e00 msgr2=0x7f1a0c154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d0e00 0x7f1a0c154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 -- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c0d4a50 msgr2=0x7f1a0c154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c0d4a50 0x7f1a0c154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:18.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 -- 172.21.15.67:0/586321445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0c0cd5b0 con 0x7f1a0c0ce650 2024-04-25T15:29:18.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a112e3640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0ce650 0x7f1a0c0cb200 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009e60 tx=0x7f1a0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:18.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a097fa640 1 -- 172.21.15.67:0/586321445 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04018030 con 0x7f1a0c0ce650 2024-04-25T15:29:18.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a097fa640 1 -- 172.21.15.67:0/586321445 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0400cdf0 con 0x7f1a0c0ce650 2024-04-25T15:29:18.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a0c0cd890 con 0x7f1a0c0ce650 2024-04-25T15:29:18.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a097fa640 1 -- 172.21.15.67:0/586321445 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04003bb0 con 0x7f1a0c0ce650 2024-04-25T15:29:18.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.558+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a0c0cde50 con 0x7f1a0c0ce650 2024-04-25T15:29:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.559+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a0c0cd320 con 0x7f1a0c0ce650 2024-04-25T15:29:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.560+0000 7f1a097fa640 1 -- 172.21.15.67:0/586321445 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1a0401b070 con 0x7f1a0c0ce650 2024-04-25T15:29:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.560+0000 7f1a097fa640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19ec0744e0 0x7f19ec0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.560+0000 7f1a097fa640 1 -- 172.21.15.67:0/586321445 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5795+0+0 (secure 0 0 0) 0x7f1a04095a60 con 0x7f1a0c0ce650 2024-04-25T15:29:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.561+0000 7f1a10ae2640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19ec0744e0 0x7f19ec0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.561+0000 7f1a10ae2640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19ec0744e0 0x7f19ec0769f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f19fc005880 tx=0x7f19fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:18.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.565+0000 7f1a097fa640 1 -- 172.21.15.67:0/586321445 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a04010040 con 0x7f1a0c0ce650 2024-04-25T15:29:18.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:18.787+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1a0c050740 con 0x7f1a0c0ce650 2024-04-25T15:29:19.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.448+0000 7f1a097fa640 1 -- 172.21.15.67:0/586321445 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) ==== 104+0+0 (secure 0 0 0) 0x7f1a0405fbd0 con 0x7f1a0c0ce650 2024-04-25T15:29:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.450+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19ec0744e0 msgr2=0x7f19ec0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.450+0000 7f1a12d46640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19ec0744e0 0x7f19ec0769f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f19fc005880 tx=0x7f19fc009040 comp rx=0 tx=0).stop 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.450+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0ce650 msgr2=0x7f1a0c0cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.450+0000 7f1a12d46640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0ce650 0x7f1a0c0cb200 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009e60 tx=0x7f1a0400c490 comp rx=0 tx=0).stop 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 shutdown_connections 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19ec0744e0 0x7f19ec0769f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0ce650 0x7f1a0c0cb200 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c0d4a50 0x7f1a0c154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 --2- 172.21.15.67:0/586321445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c0d0e00 0x7f1a0c154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 >> 172.21.15.67:0/586321445 conn(0x7f1a0c0d5d90 msgr2=0x7f1a0c161d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 shutdown_connections 2024-04-25T15:29:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.451+0000 7f1a12d46640 1 -- 172.21.15.67:0/586321445 wait complete. 2024-04-25T15:29:19.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:29:19.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.584+0000 7ff820292640 1 Processor -- start 2024-04-25T15:29:19.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.584+0000 7ff820292640 1 -- start start 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff820292640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8180ce040 0x7ff8180b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff820292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8181649f0 0x7ff8180b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff820292640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff81806d770 con 0x7ff81806e350 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff820292640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff81806da50 con 0x7ff8180ce040 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff820292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8180d2200 con 0x7ff8181649f0 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81e02e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8180ce040 0x7ff8180b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81e02e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8180ce040 0x7ff8180b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35756/0 (socket says 172.21.15.67:35756) 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81e02e640 1 -- 172.21.15.67:0/1775222446 learned_addr learned my addr 172.21.15.67:0/1775222446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81d82d640 1 --2- 172.21.15.67:0/1775222446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8181649f0 0x7ff8180b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81e02e640 1 -- 172.21.15.67:0/1775222446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8181649f0 msgr2=0x7ff8180b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:19.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81e02e640 1 --2- 172.21.15.67:0/1775222446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8181649f0 0x7ff8180b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81e02e640 1 -- 172.21.15.67:0/1775222446 >> v1:172.21.15.176:6789/0 conn(0x7ff81806e350 legacy=0x7ff81806e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.586+0000 7ff81e02e640 1 -- 172.21.15.67:0/1775222446 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8180d24e0 con 0x7ff8180ce040 2024-04-25T15:29:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.587+0000 7ff81e02e640 1 --2- 172.21.15.67:0/1775222446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8180ce040 0x7ff8180b2d60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7ff810009e50 tx=0x7ff81002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28e561ef339864e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.587+0000 7ff81d02c640 1 -- 172.21.15.67:0/1775222446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810036070 con 0x7ff8180ce040 2024-04-25T15:29:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.587+0000 7ff81d02c640 1 -- 172.21.15.67:0/1775222446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff810031600 con 0x7ff8180ce040 2024-04-25T15:29:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.587+0000 7ff81d02c640 1 -- 172.21.15.67:0/1775222446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100319d0 con 0x7ff8180ce040 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 -- 172.21.15.67:0/1775222446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8180ce040 msgr2=0x7ff8180b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 --2- 172.21.15.67:0/1775222446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8180ce040 0x7ff8180b2d60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7ff810009e50 tx=0x7ff81002a680 comp rx=0 tx=0).stop 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 -- 172.21.15.67:0/1775222446 shutdown_connections 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 --2- 172.21.15.67:0/1775222446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8181649f0 0x7ff8180b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 --2- 172.21.15.67:0/1775222446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8180ce040 0x7ff8180b2d60 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 -- 172.21.15.67:0/1775222446 >> 172.21.15.67:0/1775222446 conn(0x7ff8180d3a80 msgr2=0x7ff8180d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 -- 172.21.15.67:0/1775222446 shutdown_connections 2024-04-25T15:29:19.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.588+0000 7ff820292640 1 -- 172.21.15.67:0/1775222446 wait complete. 2024-04-25T15:29:19.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.590+0000 7ff820292640 1 Processor -- start 2024-04-25T15:29:19.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.590+0000 7ff820292640 1 -- start start 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff820292640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff81806e350 0x7ff8181fad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff820292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8180ce040 0x7ff818162b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff820292640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8181649f0 0x7ff81815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff820292640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8180b5400 con 0x7ff8181649f0 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff820292640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff818164e00 con 0x7ff81806e350 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff820292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8180b5580 con 0x7ff8180ce040 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff81e82f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8181649f0 0x7ff81815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff81e82f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8181649f0 0x7ff81815bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35766/0 (socket says 172.21.15.67:35766) 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff81e82f640 1 -- 172.21.15.67:0/1501543039 learned_addr learned my addr 172.21.15.67:0/1501543039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff81d82d640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8180ce040 0x7ff818162b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.591+0000 7ff81e02e640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff81806e350 0x7ff8181fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff81e82f640 1 -- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8180ce040 msgr2=0x7ff818162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff81e82f640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8180ce040 0x7ff818162b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff81e82f640 1 -- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff81806e350 msgr2=0x7ff8181fad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff81e82f640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff81806e350 0x7ff8181fad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff81e82f640 1 -- 172.21.15.67:0/1501543039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8181fa2f0 con 0x7ff8181649f0 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff81e82f640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8181649f0 0x7ff81815bce0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff80800c3c0 tx=0x7ff80800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff816ffd640 1 -- 172.21.15.67:0/1501543039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808017070 con 0x7ff8181649f0 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff816ffd640 1 -- 172.21.15.67:0/1501543039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8080052d0 con 0x7ff8181649f0 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff818163570 con 0x7ff8181649f0 2024-04-25T15:29:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff816ffd640 1 -- 172.21.15.67:0/1501543039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808003810 con 0x7ff8181649f0 2024-04-25T15:29:19.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.592+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8181fa870 con 0x7ff8181649f0 2024-04-25T15:29:19.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.593+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff818050740 con 0x7ff8181649f0 2024-04-25T15:29:19.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.594+0000 7ff816ffd640 1 -- 172.21.15.67:0/1501543039 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff80801a070 con 0x7ff8181649f0 2024-04-25T15:29:19.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.594+0000 7ff816ffd640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff8040744e0 0x7ff8040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:19.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.594+0000 7ff816ffd640 1 -- 172.21.15.67:0/1501543039 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5795+0+0 (secure 0 0 0) 0x7ff808094580 con 0x7ff8181649f0 2024-04-25T15:29:19.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.594+0000 7ff81e02e640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff8040744e0 0x7ff8040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:19.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.595+0000 7ff81e02e640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff8040744e0 0x7ff8040769f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff810033000 tx=0x7ff810009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:19.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.599+0000 7ff816ffd640 1 -- 172.21.15.67:0/1501543039 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff80805e5e0 con 0x7ff8181649f0 2024-04-25T15:29:19.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:19.786+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff81806ef80 con 0x7ff8181649f0 2024-04-25T15:29:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.454+0000 7ff816ffd640 1 -- 172.21.15.67:0/1501543039 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) ==== 181+0+0 (secure 0 0 0) 0x7ff8080628e0 con 0x7ff8181649f0 2024-04-25T15:29:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:29:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff8040744e0 msgr2=0x7ff8040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff8040744e0 0x7ff8040769f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff810033000 tx=0x7ff810009000 comp rx=0 tx=0).stop 2024-04-25T15:29:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8181649f0 msgr2=0x7ff81815bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8181649f0 0x7ff81815bce0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ff80800c3c0 tx=0x7ff80800c880 comp rx=0 tx=0).stop 2024-04-25T15:29:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 shutdown_connections 2024-04-25T15:29:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7ff8040744e0 0x7ff8040769f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8181649f0 0x7ff81815bce0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8180ce040 0x7ff818162b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 --2- 172.21.15.67:0/1501543039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff81806e350 0x7ff8181fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.460+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 >> 172.21.15.67:0/1501543039 conn(0x7ff8180d3a80 msgr2=0x7ff8180d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.461+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 shutdown_connections 2024-04-25T15:29:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.461+0000 7ff820292640 1 -- 172.21.15.67:0/1501543039 wait complete. 2024-04-25T15:29:20.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:29:20.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.599+0000 7f8ed3734640 1 Processor -- start 2024-04-25T15:29:20.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.599+0000 7f8ed3734640 1 -- start start 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed3734640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ecc0d0e00 0x7f8ecc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed3734640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d4a50 0x7f8ecc0d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed3734640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ecc0d1850 con 0x7f8ecc06c5c0 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed3734640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8ecc160de0 con 0x7f8ecc0d0e00 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed3734640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ecc1610c0 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed14d0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ecc0d0e00 0x7f8ecc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed14d0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ecc0d0e00 0x7f8ecc0d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35768/0 (socket says 172.21.15.67:35768) 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.600+0000 7f8ed14d0640 1 -- 172.21.15.67:0/564827673 learned_addr learned my addr 172.21.15.67:0/564827673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.601+0000 7f8ed1cd1640 1 --2- 172.21.15.67:0/564827673 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d4a50 0x7f8ecc0d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.601+0000 7f8ed14d0640 1 -- 172.21.15.67:0/564827673 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d4a50 msgr2=0x7f8ecc0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.601+0000 7f8ed14d0640 1 --2- 172.21.15.67:0/564827673 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d4a50 0x7f8ecc0d4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.601+0000 7f8ed14d0640 1 -- 172.21.15.67:0/564827673 >> v1:172.21.15.176:6789/0 conn(0x7f8ecc06c5c0 legacy=0x7f8ecc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.601+0000 7f8ed14d0640 1 -- 172.21.15.67:0/564827673 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ecc1613a0 con 0x7f8ecc0d0e00 2024-04-25T15:29:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.601+0000 7f8ed1cd1640 1 --2- 172.21.15.67:0/564827673 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d4a50 0x7f8ecc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.601+0000 7f8ed14d0640 1 --2- 172.21.15.67:0/564827673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d0e00 0x7f8ecc0d1210 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009870 tx=0x7f8ec802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f02c76f02dbaf9de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ec3fff640 1 -- 172.21.15.67:0/564827673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8037070 con 0x7f8ecc0d0e00 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ec3fff640 1 -- 172.21.15.67:0/564827673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec8032440 con 0x7f8ecc0d0e00 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ec3fff640 1 -- 172.21.15.67:0/564827673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8032810 con 0x7f8ecc0d0e00 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ed3734640 1 -- 172.21.15.67:0/564827673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d0e00 msgr2=0x7f8ecc0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ed3734640 1 --2- 172.21.15.67:0/564827673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d0e00 0x7f8ecc0d1210 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009870 tx=0x7f8ec802a250 comp rx=0 tx=0).stop 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ed3734640 1 -- 172.21.15.67:0/564827673 shutdown_connections 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ed3734640 1 --2- 172.21.15.67:0/564827673 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d4a50 0x7f8ecc0d4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ed3734640 1 --2- 172.21.15.67:0/564827673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d0e00 0x7f8ecc0d1210 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.602+0000 7f8ed3734640 1 -- 172.21.15.67:0/564827673 >> 172.21.15.67:0/564827673 conn(0x7f8ecc0d5d90 msgr2=0x7f8ecc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.603+0000 7f8ed3734640 1 -- 172.21.15.67:0/564827673 shutdown_connections 2024-04-25T15:29:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.603+0000 7f8ed3734640 1 -- 172.21.15.67:0/564827673 wait complete. 2024-04-25T15:29:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.604+0000 7f8ed3734640 1 Processor -- start 2024-04-25T15:29:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.604+0000 7f8ed3734640 1 -- start start 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed3734640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc06c5c0 0x7f8ecc0ce070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed3734640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d0e00 0x7f8ecc0ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed3734640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d4a50 0x7f8ecc0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed3734640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ecc164e00 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed3734640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ecc0b5320 con 0x7f8ecc06c5c0 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed3734640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ecc164f80 con 0x7f8ecc0d0e00 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d4a50 0x7f8ecc0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d4a50 0x7f8ecc0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35782/0 (socket says 172.21.15.67:35782) 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 -- 172.21.15.67:0/2236363642 learned_addr learned my addr 172.21.15.67:0/2236363642 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed0ccf640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d0e00 0x7f8ecc0ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed14d0640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc06c5c0 0x7f8ecc0ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 -- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d0e00 msgr2=0x7f8ecc0ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d0e00 0x7f8ecc0ce5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 -- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc06c5c0 msgr2=0x7f8ecc0ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc06c5c0 0x7f8ecc0ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 -- 172.21.15.67:0/2236363642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ecc0b6600 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed14d0640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc06c5c0 0x7f8ecc0ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ed1cd1640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d4a50 0x7f8ecc0b76a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400cd50 tx=0x7f8ec400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ec1ffb640 1 -- 172.21.15.67:0/2236363642 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec400f680 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ec1ffb640 1 -- 172.21.15.67:0/2236363642 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec400f840 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.606+0000 7f8ec1ffb640 1 -- 172.21.15.67:0/2236363642 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec400fc30 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.607+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ecc0b68e0 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.607+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ecc0b6ea0 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.608+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ecc050740 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.608+0000 7f8ec1ffb640 1 -- 172.21.15.67:0/2236363642 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8ec40213f0 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.609+0000 7f8ec1ffb640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8eb00744b0 0x7f8eb00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.609+0000 7f8ec1ffb640 1 -- 172.21.15.67:0/2236363642 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5357+0+0 (secure 0 0 0) 0x7f8ec4095d40 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.609+0000 7f8ed14d0640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8eb00744b0 0x7f8eb00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.610+0000 7f8ed14d0640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8eb00744b0 0x7f8eb00769c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8034000 tx=0x7f8ec802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.614+0000 7f8ec1ffb640 1 -- 172.21.15.67:0/2236363642 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ec4014030 con 0x7f8ecc0d4a50 2024-04-25T15:29:20.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:20.798+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8ecc15c990 con 0x7f8ecc0d4a50 2024-04-25T15:29:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.459+0000 7f8ec1ffb640 1 -- 172.21.15.67:0/2236363642 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) ==== 169+0+0 (secure 0 0 0) 0x7f8ec40606c0 con 0x7f8ecc0d4a50 2024-04-25T15:29:21.460 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.466+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8eb00744b0 msgr2=0x7f8eb00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8eb00744b0 0x7f8eb00769c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8034000 tx=0x7f8ec802e000 comp rx=0 tx=0).stop 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d4a50 msgr2=0x7f8ecc0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d4a50 0x7f8ecc0b76a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400cd50 tx=0x7f8ec400a650 comp rx=0 tx=0).stop 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 shutdown_connections 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8eb00744b0 0x7f8eb00769c0 secure :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8034000 tx=0x7f8ec802e000 comp rx=0 tx=0).stop 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ecc0d4a50 0x7f8ecc0b76a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0d0e00 0x7f8ecc0ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 --2- 172.21.15.67:0/2236363642 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc06c5c0 0x7f8ecc0ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 >> 172.21.15.67:0/2236363642 conn(0x7f8ecc0d5d90 msgr2=0x7f8ecc06c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:21.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 shutdown_connections 2024-04-25T15:29:21.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.467+0000 7f8ed3734640 1 -- 172.21.15.67:0/2236363642 wait complete. 2024-04-25T15:29:21.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T15:29:21.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.597+0000 7f017939d640 1 Processor -- start 2024-04-25T15:29:21.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.597+0000 7f017939d640 1 -- start start 2024-04-25T15:29:21.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.598+0000 7f017939d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f017409ccc0 0x7f017406d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.598+0000 7f017939d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0174164dd0 0x7f01740ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.598+0000 7f017939d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01740d2f30 con 0x7f017406ddf0 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.598+0000 7f017939d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f01740d31d0 con 0x7f0174164dd0 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.598+0000 7f017939d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f01740d34b0 con 0x7f017409ccc0 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0178b9c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0174164dd0 0x7f01740ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0178b9c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0174164dd0 0x7f01740ce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35796/0 (socket says 172.21.15.67:35796) 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0178b9c640 1 -- 172.21.15.67:0/1682819333 learned_addr learned my addr 172.21.15.67:0/1682819333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0173fff640 1 --2- 172.21.15.67:0/1682819333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f017409ccc0 0x7f017406d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0178b9c640 1 -- 172.21.15.67:0/1682819333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f017409ccc0 msgr2=0x7f017406d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0178b9c640 1 --2- 172.21.15.67:0/1682819333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f017409ccc0 0x7f017406d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0178b9c640 1 -- 172.21.15.67:0/1682819333 >> v1:172.21.15.176:6789/0 conn(0x7f017406ddf0 legacy=0x7f01741646c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.599+0000 7f0178b9c640 1 -- 172.21.15.67:0/1682819333 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01740d3790 con 0x7f0174164dd0 2024-04-25T15:29:21.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.600+0000 7f0178b9c640 1 --2- 172.21.15.67:0/1682819333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0174164dd0 0x7f01740ce8f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e50 tx=0x7f016c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9717680226ffbec0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:21.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.600+0000 7f0172ffd640 1 -- 172.21.15.67:0/1682819333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c036070 con 0x7f0174164dd0 2024-04-25T15:29:21.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.600+0000 7f0172ffd640 1 -- 172.21.15.67:0/1682819333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016c031600 con 0x7f0174164dd0 2024-04-25T15:29:21.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.600+0000 7f0172ffd640 1 -- 172.21.15.67:0/1682819333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c0319d0 con 0x7f0174164dd0 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.600+0000 7f017939d640 1 -- 172.21.15.67:0/1682819333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0174164dd0 msgr2=0x7f01740ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.600+0000 7f017939d640 1 --2- 172.21.15.67:0/1682819333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0174164dd0 0x7f01740ce8f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e50 tx=0x7f016c02a680 comp rx=0 tx=0).stop 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.601+0000 7f017939d640 1 -- 172.21.15.67:0/1682819333 shutdown_connections 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.601+0000 7f017939d640 1 --2- 172.21.15.67:0/1682819333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0174164dd0 0x7f01740ce8f0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.601+0000 7f017939d640 1 --2- 172.21.15.67:0/1682819333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f017409ccc0 0x7f017406d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.601+0000 7f017939d640 1 -- 172.21.15.67:0/1682819333 >> 172.21.15.67:0/1682819333 conn(0x7f01740d0480 msgr2=0x7f01740d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.601+0000 7f017939d640 1 -- 172.21.15.67:0/1682819333 shutdown_connections 2024-04-25T15:29:21.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.601+0000 7f017939d640 1 -- 172.21.15.67:0/1682819333 wait complete. 2024-04-25T15:29:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.602+0000 7f017939d640 1 Processor -- start 2024-04-25T15:29:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.603+0000 7f017939d640 1 -- start start 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f017939d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f017406ddf0 0x7f017415a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f017939d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017409ccc0 0x7f01740cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f017939d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0174164dd0 0x7f01740cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f017939d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f01740d1010 con 0x7f017409ccc0 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f017939d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f01740b5220 con 0x7f017406ddf0 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f017939d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f01740d1190 con 0x7f0174164dd0 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017409ccc0 0x7f01740cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017409ccc0 0x7f01740cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35800/0 (socket says 172.21.15.67:35800) 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 -- 172.21.15.67:0/1040727963 learned_addr learned my addr 172.21.15.67:0/1040727963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f0173fff640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f017406ddf0 0x7f017415a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f0178b9c640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0174164dd0 0x7f01740cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 -- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0174164dd0 msgr2=0x7f01740cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0174164dd0 0x7f01740cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 -- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f017406ddf0 msgr2=0x7f017415a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f017406ddf0 0x7f017415a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 -- 172.21.15.67:0/1040727963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174160e40 con 0x7f017409ccc0 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.604+0000 7f01737fe640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017409ccc0 0x7f01740cb940 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f016400c470 tx=0x7f016400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.605+0000 7f0170ff9640 1 -- 172.21.15.67:0/1040727963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01640052d0 con 0x7f017409ccc0 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.605+0000 7f0170ff9640 1 -- 172.21.15.67:0/1040727963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0164005490 con 0x7f017409ccc0 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.605+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0174161120 con 0x7f017409ccc0 2024-04-25T15:29:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.605+0000 7f0170ff9640 1 -- 172.21.15.67:0/1040727963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0164007260 con 0x7f017409ccc0 2024-04-25T15:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.605+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f017415ab90 con 0x7f017409ccc0 2024-04-25T15:29:21.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.606+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01740cb210 con 0x7f017409ccc0 2024-04-25T15:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.606+0000 7f0170ff9640 1 -- 172.21.15.67:0/1040727963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0164007400 con 0x7f017409ccc0 2024-04-25T15:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.607+0000 7f0170ff9640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f01540744e0 0x7f01540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.607+0000 7f0170ff9640 1 -- 172.21.15.67:0/1040727963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f0164094300 con 0x7f017409ccc0 2024-04-25T15:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.607+0000 7f0173fff640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f01540744e0 0x7f01540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.607+0000 7f0173fff640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f01540744e0 0x7f01540769f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f015c005f80 tx=0x7f015c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.611+0000 7f0170ff9640 1 -- 172.21.15.67:0/1040727963 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f016405e740 con 0x7f017409ccc0 2024-04-25T15:29:21.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.799+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f0174076c60 con 0x7f017409ccc0 2024-04-25T15:29:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.803+0000 7f0170ff9640 1 -- 172.21.15.67:0/1040727963 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f0164062a40 con 0x7f017409ccc0 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f01540744e0 msgr2=0x7f01540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f01540744e0 0x7f01540769f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f015c005f80 tx=0x7f015c008040 comp rx=0 tx=0).stop 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017409ccc0 msgr2=0x7f01740cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017409ccc0 0x7f01740cb940 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f016400c470 tx=0x7f016400c930 comp rx=0 tx=0).stop 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 shutdown_connections 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f01540744e0 0x7f01540769f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0174164dd0 0x7f01740cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f017409ccc0 0x7f01740cb940 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 --2- 172.21.15.67:0/1040727963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f017406ddf0 0x7f017415a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 >> 172.21.15.67:0/1040727963 conn(0x7f01740d0480 msgr2=0x7f01740b6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 shutdown_connections 2024-04-25T15:29:21.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.805+0000 7f017939d640 1 -- 172.21.15.67:0/1040727963 wait complete. 2024-04-25T15:29:21.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write' 2024-04-25T15:29:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.928+0000 7f449d892640 1 Processor -- start 2024-04-25T15:29:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.928+0000 7f449d892640 1 -- start start 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.929+0000 7f449d892640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f44980d0de0 0x7f44980d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.929+0000 7f449d892640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44980d4a30 0x7f44980d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.929+0000 7f449d892640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44980d1830 con 0x7f449806c5c0 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.929+0000 7f449d892640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4498160dc0 con 0x7f44980d4a30 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.929+0000 7f449d892640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f44981610a0 con 0x7f44980d0de0 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.929+0000 7f44977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44980d4a30 0x7f44980d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.930+0000 7f44977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44980d4a30 0x7f44980d4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35814/0 (socket says 172.21.15.67:35814) 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.930+0000 7f44977fe640 1 -- 172.21.15.67:0/3767564396 learned_addr learned my addr 172.21.15.67:0/3767564396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.930+0000 7f4496ffd640 1 --2- 172.21.15.67:0/3767564396 >> v2:172.21.15.176:3300/0 conn(0x7f44980d0de0 0x7f44980d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.930+0000 7f44977fe640 1 -- 172.21.15.67:0/3767564396 >> v2:172.21.15.176:3300/0 conn(0x7f44980d0de0 msgr2=0x7f44980d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:21.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.930+0000 7f44977fe640 1 --2- 172.21.15.67:0/3767564396 >> v2:172.21.15.176:3300/0 conn(0x7f44980d0de0 0x7f44980d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.930+0000 7f44977fe640 1 -- 172.21.15.67:0/3767564396 >> v1:172.21.15.176:6789/0 conn(0x7f449806c5c0 legacy=0x7f449806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.930+0000 7f44977fe640 1 -- 172.21.15.67:0/3767564396 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4498161380 con 0x7f44980d4a30 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f44977fe640 1 --2- 172.21.15.67:0/3767564396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d4a30 0x7f44980d4e40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f44980d3e40 tx=0x7f449002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76bd83ab4d57b64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f44967fc640 1 -- 172.21.15.67:0/3767564396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490036070 con 0x7f44980d4a30 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f44967fc640 1 -- 172.21.15.67:0/3767564396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4490031600 con 0x7f44980d4a30 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f44967fc640 1 -- 172.21.15.67:0/3767564396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44900319d0 con 0x7f44980d4a30 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f449d892640 1 -- 172.21.15.67:0/3767564396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d4a30 msgr2=0x7f44980d4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f449d892640 1 --2- 172.21.15.67:0/3767564396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d4a30 0x7f44980d4e40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f44980d3e40 tx=0x7f449002a660 comp rx=0 tx=0).stop 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f449d892640 1 -- 172.21.15.67:0/3767564396 shutdown_connections 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f449d892640 1 --2- 172.21.15.67:0/3767564396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d4a30 0x7f44980d4e40 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.931+0000 7f449d892640 1 --2- 172.21.15.67:0/3767564396 >> v2:172.21.15.176:3300/0 conn(0x7f44980d0de0 0x7f44980d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.932+0000 7f449d892640 1 -- 172.21.15.67:0/3767564396 >> 172.21.15.67:0/3767564396 conn(0x7f44980d5d70 msgr2=0x7f44980d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.932+0000 7f449d892640 1 -- 172.21.15.67:0/3767564396 shutdown_connections 2024-04-25T15:29:21.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.932+0000 7f449d892640 1 -- 172.21.15.67:0/3767564396 wait complete. 2024-04-25T15:29:21.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.933+0000 7f449d892640 1 Processor -- start 2024-04-25T15:29:21.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.933+0000 7f449d892640 1 -- start start 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f449806c5c0 0x7f4498149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d0de0 0x7f449814ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44980d4a30 0x7f449814b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4498164de0 con 0x7f44980d0de0 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f44980b5300 con 0x7f44980d4a30 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4498164f60 con 0x7f449806c5c0 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d0de0 0x7f449814ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d0de0 0x7f449814ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35816/0 (socket says 172.21.15.67:35816) 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 -- 172.21.15.67:0/87785919 learned_addr learned my addr 172.21.15.67:0/87785919 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f44977fe640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44980d4a30 0x7f449814b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 -- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f449806c5c0 msgr2=0x7f4498149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f449806c5c0 0x7f4498149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 -- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44980d4a30 msgr2=0x7f449814b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44980d4a30 0x7f449814b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 -- 172.21.15.67:0/87785919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449814b290 con 0x7f44980d0de0 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f44977fe640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44980d4a30 0x7f449814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f448ffff640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d0de0 0x7f449814ad50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f448000b420 tx=0x7f448000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f4494ff9640 1 -- 172.21.15.67:0/87785919 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480011830 con 0x7f44980d0de0 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f4494ff9640 1 -- 172.21.15.67:0/87785919 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4480016070 con 0x7f44980d0de0 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4498149cb0 con 0x7f44980d0de0 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f4494ff9640 1 -- 172.21.15.67:0/87785919 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480011c10 con 0x7f44980d0de0 2024-04-25T15:29:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.935+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f449814a270 con 0x7f44980d0de0 2024-04-25T15:29:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.936+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44980cf3a0 con 0x7f44980d0de0 2024-04-25T15:29:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.937+0000 7f4494ff9640 1 -- 172.21.15.67:0/87785919 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f448001b460 con 0x7f44980d0de0 2024-04-25T15:29:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.938+0000 7f4494ff9640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44780744b0 0x7f44780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.938+0000 7f4494ff9640 1 -- 172.21.15.67:0/87785919 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f4480094a90 con 0x7f44980d0de0 2024-04-25T15:29:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.938+0000 7f4496ffd640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44780744b0 0x7f44780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.939+0000 7f4496ffd640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44780744b0 0x7f44780769c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4488006fd0 tx=0x7f4488008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:21.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:21.943+0000 7f4494ff9640 1 -- 172.21.15.67:0/87785919 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4480020020 con 0x7f44980d0de0 2024-04-25T15:29:22.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.203+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]} v 0) -- 0x7f4498149a40 con 0x7f44980d0de0 2024-04-25T15:29:22.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.461+0000 7f4494ff9640 1 -- 172.21.15.67:0/87785919 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]}]=0 v245) ==== 137+0+0 (secure 0 0 0) 0x7f448005f4a0 con 0x7f44980d0de0 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44780744b0 msgr2=0x7f44780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44780744b0 0x7f44780769c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4488006fd0 tx=0x7f4488008040 comp rx=0 tx=0).stop 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d0de0 msgr2=0x7f449814ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d0de0 0x7f449814ad50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f448000b420 tx=0x7f448000b8e0 comp rx=0 tx=0).stop 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 shutdown_connections 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f44780744b0 0x7f44780769c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44980d4a30 0x7f449814b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44980d0de0 0x7f449814ad50 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 --2- 172.21.15.67:0/87785919 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f449806c5c0 0x7f4498149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 >> 172.21.15.67:0/87785919 conn(0x7f44980d5d70 msgr2=0x7f44980b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 shutdown_connections 2024-04-25T15:29:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:22.463+0000 7f449d892640 1 -- 172.21.15.67:0/87785919 wait complete. 2024-04-25T15:29:22.475 INFO:tasks.cephfs_test_runner:That when a client modifies the quota on a directory used ... ok 2024-04-25T15:29:22.476 INFO:tasks.cephfs_test_runner: 2024-04-25T15:29:22.476 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T15:29:22.476 INFO:tasks.cephfs_test_runner:Ran 6 tests in 205.892s 2024-04-25T15:29:22.476 INFO:tasks.cephfs_test_runner: 2024-04-25T15:29:22.476 INFO:tasks.cephfs_test_runner:OK 2024-04-25T15:29:22.476 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-25T15:29:22.486 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T15:29:22.507 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T15:29:22.507 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:29:22.507 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:22.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:29:22.523 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:29:22.523 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:22.579 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056113,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T15:29:22.579 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:22.580 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:22.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:29:22.599 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:29:22.599 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:29:22.657 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056112,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20060 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T15:29:22.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.657 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:29:22.672 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T15:29:22.673 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:29:22.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T15:29:22.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:29:22.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-25T15:29:22.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.894 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:29:22.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-25T15:29:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-25T15:29:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:22.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.979 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:29:22.998 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T15:29:22.998 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T15:29:22.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:22.998 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T15:29:23.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:23.066 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T15:29:23.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:23.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link delete ceph-brx 2024-04-25T15:29:23.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:23.242 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-04-25T15:29:23.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:29:23.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link delete ceph-brx 2024-04-25T15:29:23.325 INFO:teuthology.orchestra.run.smithi176.stderr:Cannot find device "ceph-brx" 2024-04-25T15:29:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:23.326 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T15:29:23.336 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T15:29:23.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:23.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.449+0000 7f5d0e4ee640 1 Processor -- start 2024-04-25T15:29:23.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.449+0000 7f5d0e4ee640 1 -- start start 2024-04-25T15:29:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d0e4ee640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d081434d0 0x7f5d08070480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d0e4ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d081612f0 0x7f5d080d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d0e4ee640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d080709c0 con 0x7f5d080d0ed0 2024-04-25T15:29:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d0e4ee640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5d081438e0 con 0x7f5d081434d0 2024-04-25T15:29:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d0e4ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0806f3e0 con 0x7f5d081612f0 2024-04-25T15:29:23.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d07fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d081434d0 0x7f5d08070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d07fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d081434d0 0x7f5d08070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35824/0 (socket says 172.21.15.67:35824) 2024-04-25T15:29:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d07fff640 1 -- 172.21.15.67:0/2441312331 learned_addr learned my addr 172.21.15.67:0/2441312331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d0ca8b640 1 --2- 172.21.15.67:0/2441312331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d081612f0 0x7f5d080d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d07fff640 1 -- 172.21.15.67:0/2441312331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d081612f0 msgr2=0x7f5d080d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d07fff640 1 --2- 172.21.15.67:0/2441312331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d081612f0 0x7f5d080d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d07fff640 1 -- 172.21.15.67:0/2441312331 >> v1:172.21.15.176:6789/0 conn(0x7f5d080d0ed0 legacy=0x7f5d08160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.451+0000 7f5d07fff640 1 -- 172.21.15.67:0/2441312331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0806f6c0 con 0x7f5d081434d0 2024-04-25T15:29:23.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.452+0000 7f5d07fff640 1 --2- 172.21.15.67:0/2441312331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d08070480 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009870 tx=0x7f5cf802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae053ef6fcc1e8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:23.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.452+0000 7f5d06ffd640 1 -- 172.21.15.67:0/2441312331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8037070 con 0x7f5d081434d0 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.452+0000 7f5d06ffd640 1 -- 172.21.15.67:0/2441312331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf8032440 con 0x7f5d081434d0 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.452+0000 7f5d06ffd640 1 -- 172.21.15.67:0/2441312331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8032810 con 0x7f5d081434d0 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/2441312331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 msgr2=0x7f5d08070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/2441312331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d08070480 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009870 tx=0x7f5cf802a270 comp rx=0 tx=0).stop 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/2441312331 shutdown_connections 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/2441312331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d081612f0 0x7f5d080d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/2441312331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d08070480 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/2441312331 >> 172.21.15.67:0/2441312331 conn(0x7f5d080d1ae0 msgr2=0x7f5d080d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/2441312331 shutdown_connections 2024-04-25T15:29:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.453+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/2441312331 wait complete. 2024-04-25T15:29:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.455+0000 7f5d0e4ee640 1 Processor -- start 2024-04-25T15:29:23.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.455+0000 7f5d0e4ee640 1 -- start start 2024-04-25T15:29:23.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d0e4ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d080d0ed0 0x7f5d0814cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d0e4ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d0813d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d0e4ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d080b2dc0 0x7f5d0815fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d0e4ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0806f090 con 0x7f5d081434d0 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d0e4ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0806fc80 con 0x7f5d080b2dc0 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d0e4ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d080b51a0 con 0x7f5d080d0ed0 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d0813d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d0813d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35840/0 (socket says 172.21.15.67:35840) 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d077fe640 1 -- 172.21.15.67:0/332270164 learned_addr learned my addr 172.21.15.67:0/332270164 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.456+0000 7f5d07fff640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d080d0ed0 0x7f5d0814cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d0ca8b640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d080b2dc0 0x7f5d0815fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d077fe640 1 -- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d080d0ed0 msgr2=0x7f5d0814cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d077fe640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d080d0ed0 0x7f5d0814cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d077fe640 1 -- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d080b2dc0 msgr2=0x7f5d0815fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d077fe640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d080b2dc0 0x7f5d0815fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d077fe640 1 -- 172.21.15.67:0/332270164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0815e980 con 0x7f5d081434d0 2024-04-25T15:29:23.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d077fe640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d0813d120 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c470 tx=0x7f5cfc00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:23.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d04ff9640 1 -- 172.21.15.67:0/332270164 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc0052d0 con 0x7f5d081434d0 2024-04-25T15:29:23.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d04ff9640 1 -- 172.21.15.67:0/332270164 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cfc005490 con 0x7f5d081434d0 2024-04-25T15:29:23.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d0815ec60 con 0x7f5d081434d0 2024-04-25T15:29:23.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d04ff9640 1 -- 172.21.15.67:0/332270164 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc007260 con 0x7f5d081434d0 2024-04-25T15:29:23.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.457+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0815f220 con 0x7f5d081434d0 2024-04-25T15:29:23.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.458+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d08050740 con 0x7f5d081434d0 2024-04-25T15:29:23.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.459+0000 7f5d04ff9640 1 -- 172.21.15.67:0/332270164 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5cfc007400 con 0x7f5d081434d0 2024-04-25T15:29:23.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.459+0000 7f5d04ff9640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5cf40744b0 0x7f5cf40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.459+0000 7f5d04ff9640 1 -- 172.21.15.67:0/332270164 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f5cfc094300 con 0x7f5d081434d0 2024-04-25T15:29:23.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.460+0000 7f5d07fff640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5cf40744b0 0x7f5cf40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.460+0000 7f5d07fff640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5cf40744b0 0x7f5cf40769c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8034000 tx=0x7f5cf802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:23.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.464+0000 7f5d04ff9640 1 -- 172.21.15.67:0/332270164 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5cfc05e740 con 0x7f5d081434d0 2024-04-25T15:29:23.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.648+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d08142b40 con 0x7f5cf40744b0 2024-04-25T15:29:23.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.649+0000 7f5d04ff9640 1 -- 172.21.15.67:0/332270164 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13487 (secure 0 0 0) 0x7f5cf80454c0 con 0x7f5cf40744b0 2024-04-25T15:29:23.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:23.651 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.651+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5cf40744b0 msgr2=0x7f5cf40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.651+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5cf40744b0 0x7f5cf40769c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8034000 tx=0x7f5cf802e000 comp rx=0 tx=0).stop 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 msgr2=0x7f5d0813d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d0813d120 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c470 tx=0x7f5cfc00c930 comp rx=0 tx=0).stop 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 shutdown_connections 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f5cf40744b0 0x7f5cf40769c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d080b2dc0 0x7f5d0815fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d081434d0 0x7f5d0813d120 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 --2- 172.21.15.67:0/332270164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d080d0ed0 0x7f5d0814cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 >> 172.21.15.67:0/332270164 conn(0x7f5d080d1ae0 msgr2=0x7f5d081416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 shutdown_connections 2024-04-25T15:29:23.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.652+0000 7f5d0e4ee640 1 -- 172.21.15.67:0/332270164 wait complete. 2024-04-25T15:29:23.663 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-04-25T15:29:22.853769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":301084,"kb_used_data":55800,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754673636,"statfs":{"total":773094113280,"available":772785803264,"internally_reserved":0,"allocated":57139200,"data_stored":51606026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19085096,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-163,"num_write_kb":-39217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001790"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":246,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-25T15:29:21.559852+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:29:21.559852+0000","last_peered":"2024-04-25T15:29:21.559852+0000","last_clean":"2024-04-25T15:29:21.559852+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:29:21.559852+0000","last_undegraded":"2024-04-25T15:29:21.559852+0000","last_fullsized":"2024-04-25T15:29:21.559852+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":9932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":10170368,"data_stored":9476455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1175552,"data_stored":497475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40536,"kb_used_data":9356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331304,"statfs":{"total":96636764160,"available":96595255296,"internally_reserved":0,"allocated":9580544,"data_stored":8886090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40520,"kb_used_data":9340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331320,"statfs":{"total":96636764160,"available":96595271680,"internally_reserved":0,"allocated":9564160,"data_stored":8886083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42880,"kb_used_data":11700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328960,"statfs":{"total":96636764160,"available":96592855040,"internally_reserved":0,"allocated":11980800,"data_stored":11291348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34380,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337460,"statfs":{"total":96636764160,"available":96601559040,"internally_reserved":0,"allocated":3276800,"data_stored":2594629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":5268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":5394432,"data_stored":4691788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":5996544,"data_stored":5282158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:23.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:23.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.774+0000 7fe138642640 1 Processor -- start 2024-04-25T15:29:23.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.774+0000 7fe138642640 1 -- start start 2024-04-25T15:29:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe138642640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe1300ce800 0x7fe1300d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe138642640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300d1620 0x7fe130162a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe138642640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1300b0770 con 0x7fe130162f60 2024-04-25T15:29:23.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe138642640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe1300b10a0 con 0x7fe1300ce800 2024-04-25T15:29:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe138642640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe1300d2100 con 0x7fe1300d1620 2024-04-25T15:29:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe1363de640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe1300ce800 0x7fe1300d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe135bdd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300d1620 0x7fe130162a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe136bdf640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe130162f60 0x7fe1300b1530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:43150/0 (socket says 172.21.15.67:43150) 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe136bdf640 1 -- 172.21.15.67:0/1124554256 learned_addr learned my addr 172.21.15.67:0/1124554256 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe1353dc640 1 -- 172.21.15.67:0/1124554256 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094753698 0 0) 0x7fe1300b0770 con 0x7fe130162f60 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe1353dc640 1 -- 172.21.15.67:0/1124554256 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe11c003520 con 0x7fe130162f60 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe135bdd640 1 -- 172.21.15.67:0/1124554256 >> v2:172.21.15.176:3300/0 conn(0x7fe1300ce800 msgr2=0x7fe1300d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe135bdd640 1 --2- 172.21.15.67:0/1124554256 >> v2:172.21.15.176:3300/0 conn(0x7fe1300ce800 0x7fe1300d1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe135bdd640 1 -- 172.21.15.67:0/1124554256 >> v1:172.21.15.176:6789/0 conn(0x7fe130162f60 legacy=0x7fe1300b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe135bdd640 1 -- 172.21.15.67:0/1124554256 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1300b0a50 con 0x7fe1300d1620 2024-04-25T15:29:23.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.776+0000 7fe1363de640 1 --2- 172.21.15.67:0/1124554256 >> v2:172.21.15.176:3300/0 conn(0x7fe1300ce800 0x7fe1300d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.777+0000 7fe135bdd640 1 --2- 172.21.15.67:0/1124554256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300d1620 0x7fe130162a20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe1200097a0 tx=0x7fe12002e470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d9d8d0dceb9a223 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.777+0000 7fe1353dc640 1 -- 172.21.15.67:0/1124554256 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12003a070 con 0x7fe1300d1620 2024-04-25T15:29:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.777+0000 7fe1353dc640 1 -- 172.21.15.67:0/1124554256 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe120033710 con 0x7fe1300d1620 2024-04-25T15:29:23.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.777+0000 7fe1353dc640 1 -- 172.21.15.67:0/1124554256 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120033ae0 con 0x7fe1300d1620 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 -- 172.21.15.67:0/1124554256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300d1620 msgr2=0x7fe130162a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 --2- 172.21.15.67:0/1124554256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300d1620 0x7fe130162a20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe1200097a0 tx=0x7fe12002e470 comp rx=0 tx=0).stop 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 -- 172.21.15.67:0/1124554256 shutdown_connections 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 --2- 172.21.15.67:0/1124554256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300d1620 0x7fe130162a20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 --2- 172.21.15.67:0/1124554256 >> v2:172.21.15.176:3300/0 conn(0x7fe1300ce800 0x7fe1300d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 -- 172.21.15.67:0/1124554256 >> 172.21.15.67:0/1124554256 conn(0x7fe1300d3a50 msgr2=0x7fe1300d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 -- 172.21.15.67:0/1124554256 shutdown_connections 2024-04-25T15:29:23.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.778+0000 7fe138642640 1 -- 172.21.15.67:0/1124554256 wait complete. 2024-04-25T15:29:23.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.780+0000 7fe138642640 1 Processor -- start 2024-04-25T15:29:23.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.780+0000 7fe138642640 1 -- start start 2024-04-25T15:29:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.781+0000 7fe138642640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300ce800 0x7fe1300b8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.781+0000 7fe138642640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1300d1620 0x7fe130159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.781+0000 7fe138642640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe130162f60 0x7fe1300b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.781+0000 7fe138642640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe130163370 con 0x7fe1300d1620 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.781+0000 7fe138642640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300b51c0 con 0x7fe130162f60 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.781+0000 7fe138642640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe1301634f0 con 0x7fe1300ce800 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1300d1620 0x7fe130159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1300d1620 0x7fe130159f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35846/0 (socket says 172.21.15.67:35846) 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 -- 172.21.15.67:0/639324699 learned_addr learned my addr 172.21.15.67:0/639324699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe1363de640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300ce800 0x7fe1300b8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe136bdf640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe130162f60 0x7fe1300b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 -- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300ce800 msgr2=0x7fe1300b8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300ce800 0x7fe1300b8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 -- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe130162f60 msgr2=0x7fe1300b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe130162f60 0x7fe1300b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 -- 172.21.15.67:0/639324699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1301600a0 con 0x7fe1300d1620 2024-04-25T15:29:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe136bdf640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe130162f60 0x7fe1300b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe135bdd640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1300d1620 0x7fe130159f50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fe120038040 tx=0x7fe1200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe126ffd640 1 -- 172.21.15.67:0/639324699 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120044020 con 0x7fe1300d1620 2024-04-25T15:29:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe126ffd640 1 -- 172.21.15.67:0/639324699 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12003f870 con 0x7fe1300d1620 2024-04-25T15:29:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe126ffd640 1 -- 172.21.15.67:0/639324699 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12003a040 con 0x7fe1300d1620 2024-04-25T15:29:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.782+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe130160380 con 0x7fe1300d1620 2024-04-25T15:29:23.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.783+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe130160940 con 0x7fe1300d1620 2024-04-25T15:29:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.784+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe130050740 con 0x7fe1300d1620 2024-04-25T15:29:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.784+0000 7fe126ffd640 1 -- 172.21.15.67:0/639324699 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe120047020 con 0x7fe1300d1620 2024-04-25T15:29:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.785+0000 7fe126ffd640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe1140744e0 0x7fe1140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.785+0000 7fe126ffd640 1 -- 172.21.15.67:0/639324699 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7fe1200b9520 con 0x7fe1300d1620 2024-04-25T15:29:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.785+0000 7fe1363de640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe1140744e0 0x7fe1140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:23.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.785+0000 7fe1363de640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe1140744e0 0x7fe1140769f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009500 tx=0x7fe12c009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.789+0000 7fe126ffd640 1 -- 172.21.15.67:0/639324699 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe12004e030 con 0x7fe1300d1620 2024-04-25T15:29:23.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.971+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe13015fdf0 con 0x7fe1140744e0 2024-04-25T15:29:23.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.973+0000 7fe126ffd640 1 -- 172.21.15.67:0/639324699 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13487 (secure 0 0 0) 0x7fe13015fdf0 con 0x7fe1140744e0 2024-04-25T15:29:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:23.974 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe1140744e0 msgr2=0x7fe1140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe1140744e0 0x7fe1140769f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009500 tx=0x7fe12c009110 comp rx=0 tx=0).stop 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1300d1620 msgr2=0x7fe130159f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1300d1620 0x7fe130159f50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fe120038040 tx=0x7fe1200046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 shutdown_connections 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fe1140744e0 0x7fe1140769f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe130162f60 0x7fe1300b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1300d1620 0x7fe130159f50 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 --2- 172.21.15.67:0/639324699 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1300ce800 0x7fe1300b8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 >> 172.21.15.67:0/639324699 conn(0x7fe1300d3a50 msgr2=0x7fe13015aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 shutdown_connections 2024-04-25T15:29:23.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:23.975+0000 7fe138642640 1 -- 172.21.15.67:0/639324699 wait complete. 2024-04-25T15:29:23.986 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-04-25T15:29:22.853769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":301084,"kb_used_data":55800,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754673636,"statfs":{"total":773094113280,"available":772785803264,"internally_reserved":0,"allocated":57139200,"data_stored":51606026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19085096,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-163,"num_write_kb":-39217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001790"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":246,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-25T15:29:21.559852+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:29:21.559852+0000","last_peered":"2024-04-25T15:29:21.559852+0000","last_clean":"2024-04-25T15:29:21.559852+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:29:21.559852+0000","last_undegraded":"2024-04-25T15:29:21.559852+0000","last_fullsized":"2024-04-25T15:29:21.559852+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":9932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":10170368,"data_stored":9476455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1175552,"data_stored":497475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40536,"kb_used_data":9356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331304,"statfs":{"total":96636764160,"available":96595255296,"internally_reserved":0,"allocated":9580544,"data_stored":8886090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40520,"kb_used_data":9340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331320,"statfs":{"total":96636764160,"available":96595271680,"internally_reserved":0,"allocated":9564160,"data_stored":8886083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42880,"kb_used_data":11700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328960,"statfs":{"total":96636764160,"available":96592855040,"internally_reserved":0,"allocated":11980800,"data_stored":11291348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34380,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337460,"statfs":{"total":96636764160,"available":96601559040,"internally_reserved":0,"allocated":3276800,"data_stored":2594629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":5268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":5394432,"data_stored":4691788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":5996544,"data_stored":5282158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:23.987 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T15:29:23.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.097+0000 7f7c866e2640 1 Processor -- start 2024-04-25T15:29:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.097+0000 7f7c866e2640 1 -- start start 2024-04-25T15:29:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c866e2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7c800d0e00 0x7f7c800d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c866e2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c800d1850 con 0x7f7c8006c5c0 2024-04-25T15:29:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c866e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c80160de0 con 0x7f7c800d4a50 2024-04-25T15:29:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c866e2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7c801610c0 con 0x7f7c800d0e00 2024-04-25T15:29:24.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7c8006c5c0 0x7f7c8006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39016/0 (socket says 172.21.15.67:39016) 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7f7fe640 1 -- 172.21.15.67:0/3540413296 learned_addr learned my addr 172.21.15.67:0/3540413296 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7ffff640 1 --2- 172.21.15.67:0/3540413296 >> v2:172.21.15.176:3300/0 conn(0x7f7c800d0e00 0x7f7c800d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7effd640 1 -- 172.21.15.67:0/3540413296 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556544207 0 0) 0x7f7c800d1850 con 0x7f7c8006c5c0 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7effd640 1 -- 172.21.15.67:0/3540413296 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c68003520 con 0x7f7c8006c5c0 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7effd640 1 -- 172.21.15.67:0/3540413296 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 956857748 0 0) 0x7f7c80160de0 con 0x7f7c800d4a50 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7effd640 1 -- 172.21.15.67:0/3540413296 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c800d1850 con 0x7f7c800d4a50 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7ffff640 1 -- 172.21.15.67:0/3540413296 >> v1:172.21.15.176:6789/0 conn(0x7f7c800d4a50 legacy=0x7f7c800d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7ffff640 1 -- 172.21.15.67:0/3540413296 >> v1:172.21.15.67:6789/0 conn(0x7f7c8006c5c0 legacy=0x7f7c8006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.099+0000 7f7c7ffff640 1 -- 172.21.15.67:0/3540413296 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c801613a0 con 0x7f7c800d0e00 2024-04-25T15:29:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.100+0000 7f7c7ffff640 1 --2- 172.21.15.67:0/3540413296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 0x7f7c800d1210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c740097a0 tx=0x7f7c7402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de51449bd99d1da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.100+0000 7f7c7effd640 1 -- 172.21.15.67:0/3540413296 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74037070 con 0x7f7c800d0e00 2024-04-25T15:29:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.100+0000 7f7c7effd640 1 -- 172.21.15.67:0/3540413296 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c74032440 con 0x7f7c800d0e00 2024-04-25T15:29:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.100+0000 7f7c7effd640 1 -- 172.21.15.67:0/3540413296 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74032810 con 0x7f7c800d0e00 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.101+0000 7f7c866e2640 1 -- 172.21.15.67:0/3540413296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 msgr2=0x7f7c800d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.101+0000 7f7c866e2640 1 --2- 172.21.15.67:0/3540413296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 0x7f7c800d1210 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c740097a0 tx=0x7f7c7402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.101+0000 7f7c866e2640 1 -- 172.21.15.67:0/3540413296 shutdown_connections 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.101+0000 7f7c866e2640 1 --2- 172.21.15.67:0/3540413296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 0x7f7c800d1210 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.101+0000 7f7c866e2640 1 -- 172.21.15.67:0/3540413296 >> 172.21.15.67:0/3540413296 conn(0x7f7c800d5d90 msgr2=0x7f7c800d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.101+0000 7f7c866e2640 1 -- 172.21.15.67:0/3540413296 shutdown_connections 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.101+0000 7f7c866e2640 1 -- 172.21.15.67:0/3540413296 wait complete. 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.103+0000 7f7c866e2640 1 Processor -- start 2024-04-25T15:29:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.103+0000 7f7c866e2640 1 -- start start 2024-04-25T15:29:24.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c866e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c8006c5c0 0x7f7c8014e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c866e2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 0x7f7c800b6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c866e2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c800d4a50 0x7f7c801fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c866e2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7c80164e00 con 0x7f7c8006c5c0 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c866e2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7c800b5320 con 0x7f7c800d0e00 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c866e2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7c80164f80 con 0x7f7c800d4a50 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c8006c5c0 0x7f7c8014e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c7ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c8006c5c0 0x7f7c8014e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35850/0 (socket says 172.21.15.67:35850) 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.104+0000 7f7c7ffff640 1 -- 172.21.15.67:0/1037141712 learned_addr learned my addr 172.21.15.67:0/1037141712 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c84c7f640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c800d4a50 0x7f7c801fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7f7fe640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 0x7f7c800b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7ffff640 1 -- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c800d4a50 msgr2=0x7f7c801fa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7ffff640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c800d4a50 0x7f7c801fa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7ffff640 1 -- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 msgr2=0x7f7c800b6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7ffff640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 0x7f7c800b6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7ffff640 1 -- 172.21.15.67:0/1037141712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c800c9550 con 0x7f7c8006c5c0 2024-04-25T15:29:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c84c7f640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c800d4a50 0x7f7c801fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7ffff640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c8006c5c0 0x7f7c8014e2b0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7c7402a790 tx=0x7f7c740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7cff9640 1 -- 172.21.15.67:0/1037141712 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74041030 con 0x7f7c8006c5c0 2024-04-25T15:29:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7cff9640 1 -- 172.21.15.67:0/1037141712 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c7403d4e0 con 0x7f7c8006c5c0 2024-04-25T15:29:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c800c9830 con 0x7f7c8006c5c0 2024-04-25T15:29:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c7cff9640 1 -- 172.21.15.67:0/1037141712 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74037040 con 0x7f7c8006c5c0 2024-04-25T15:29:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.105+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c800b6cb0 con 0x7f7c8006c5c0 2024-04-25T15:29:24.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.106+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c80050740 con 0x7f7c8006c5c0 2024-04-25T15:29:24.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.107+0000 7f7c7cff9640 1 -- 172.21.15.67:0/1037141712 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7c74044070 con 0x7f7c8006c5c0 2024-04-25T15:29:24.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.107+0000 7f7c7cff9640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7c6c0744e0 0x7f7c6c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.107+0000 7f7c7cff9640 1 -- 172.21.15.67:0/1037141712 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f7c740b4e90 con 0x7f7c8006c5c0 2024-04-25T15:29:24.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.108+0000 7f7c7f7fe640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7c6c0744e0 0x7f7c6c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.108+0000 7f7c7f7fe640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7c6c0744e0 0x7f7c6c0769f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7c800baaa0 tx=0x7f7c70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:24.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.112+0000 7f7c7cff9640 1 -- 172.21.15.67:0/1037141712 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c7404b020 con 0x7f7c8006c5c0 2024-04-25T15:29:24.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.294+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c800d6530 con 0x7f7c6c0744e0 2024-04-25T15:29:24.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.295+0000 7f7c7cff9640 1 -- 172.21.15.67:0/1037141712 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13487 (secure 0 0 0) 0x7f7c800d6530 con 0x7f7c6c0744e0 2024-04-25T15:29:24.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:24.296 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:24.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7c6c0744e0 msgr2=0x7f7c6c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7c6c0744e0 0x7f7c6c0769f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7c800baaa0 tx=0x7f7c70009040 comp rx=0 tx=0).stop 2024-04-25T15:29:24.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c8006c5c0 msgr2=0x7f7c8014e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c8006c5c0 0x7f7c8014e2b0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7c7402a790 tx=0x7f7c740046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 shutdown_connections 2024-04-25T15:29:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f7c6c0744e0 0x7f7c6c0769f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c800d4a50 0x7f7c801fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c800d0e00 0x7f7c800b6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 --2- 172.21.15.67:0/1037141712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c8006c5c0 0x7f7c8014e2b0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 >> 172.21.15.67:0/1037141712 conn(0x7f7c800d5d90 msgr2=0x7f7c80163960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 shutdown_connections 2024-04-25T15:29:24.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.297+0000 7f7c866e2640 1 -- 172.21.15.67:0/1037141712 wait complete. 2024-04-25T15:29:24.309 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-04-25T15:29:22.853769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":171,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":301084,"kb_used_data":55800,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754673636,"statfs":{"total":773094113280,"available":772785803264,"internally_reserved":0,"allocated":57139200,"data_stored":51606026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19085096,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-163,"num_write_kb":-39217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001790"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":246,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-25T15:29:21.559852+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:29:21.559852+0000","last_peered":"2024-04-25T15:29:21.559852+0000","last_clean":"2024-04-25T15:29:21.559852+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:29:21.559852+0000","last_undegraded":"2024-04-25T15:29:21.559852+0000","last_fullsized":"2024-04-25T15:29:21.559852+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":9932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":10170368,"data_stored":9476455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339512,"statfs":{"total":96636764160,"available":96603660288,"internally_reserved":0,"allocated":1175552,"data_stored":497475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40536,"kb_used_data":9356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331304,"statfs":{"total":96636764160,"available":96595255296,"internally_reserved":0,"allocated":9580544,"data_stored":8886090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40520,"kb_used_data":9340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331320,"statfs":{"total":96636764160,"available":96595271680,"internally_reserved":0,"allocated":9564160,"data_stored":8886083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42880,"kb_used_data":11700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328960,"statfs":{"total":96636764160,"available":96592855040,"internally_reserved":0,"allocated":11980800,"data_stored":11291348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574902,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34380,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337460,"statfs":{"total":96636764160,"available":96601559040,"internally_reserved":0,"allocated":3276800,"data_stored":2594629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":5268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":5394432,"data_stored":4691788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":5856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":5996544,"data_stored":5282158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:24.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:29:24.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.420+0000 7f2cc67d7640 1 Processor -- start 2024-04-25T15:29:24.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.420+0000 7f2cc67d7640 1 -- start start 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc67d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc67d7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cc00d2410 0x7f2cc00b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc67d7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cc00cfb30 con 0x7f2cc00cf300 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc67d7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2cc00d00f0 con 0x7f2cc00d2410 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc67d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc00b2470 con 0x7f2cc0160870 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc4d74640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cc00d2410 0x7f2cc00b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc4d74640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2cc00d2410 0x7f2cc00b2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35864/0 (socket says 172.21.15.67:35864) 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.421+0000 7f2cc4d74640 1 -- 172.21.15.67:0/2356662421 learned_addr learned my addr 172.21.15.67:0/2356662421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.422+0000 7f2cbffff640 1 --2- 172.21.15.67:0/2356662421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.422+0000 7f2cc4d74640 1 -- 172.21.15.67:0/2356662421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 msgr2=0x7f2cc00b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:24.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.422+0000 7f2cc4d74640 1 --2- 172.21.15.67:0/2356662421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc00b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.422+0000 7f2cc4d74640 1 -- 172.21.15.67:0/2356662421 >> v1:172.21.15.176:6789/0 conn(0x7f2cc00cf300 legacy=0x7f2cc00d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.422+0000 7f2cc4d74640 1 -- 172.21.15.67:0/2356662421 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc00cfe10 con 0x7f2cc00d2410 2024-04-25T15:29:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.422+0000 7f2cc4d74640 1 --2- 172.21.15.67:0/2356662421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00d2410 0x7f2cc00b2900 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009e50 tx=0x7f2cb802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a78c82cb56b137 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cbeffd640 1 -- 172.21.15.67:0/2356662421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8036070 con 0x7f2cc00d2410 2024-04-25T15:29:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cbeffd640 1 -- 172.21.15.67:0/2356662421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cb8031600 con 0x7f2cc00d2410 2024-04-25T15:29:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cbeffd640 1 -- 172.21.15.67:0/2356662421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80319d0 con 0x7f2cc00d2410 2024-04-25T15:29:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 -- 172.21.15.67:0/2356662421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00d2410 msgr2=0x7f2cc00b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/2356662421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00d2410 0x7f2cc00b2900 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009e50 tx=0x7f2cb802a710 comp rx=0 tx=0).stop 2024-04-25T15:29:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 -- 172.21.15.67:0/2356662421 shutdown_connections 2024-04-25T15:29:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/2356662421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00d2410 0x7f2cc00b2900 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/2356662421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc00b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 -- 172.21.15.67:0/2356662421 >> 172.21.15.67:0/2356662421 conn(0x7f2cc009b7d0 msgr2=0x7f2cc009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:24.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 -- 172.21.15.67:0/2356662421 shutdown_connections 2024-04-25T15:29:24.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.423+0000 7f2cc67d7640 1 -- 172.21.15.67:0/2356662421 wait complete. 2024-04-25T15:29:24.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.425+0000 7f2cc67d7640 1 Processor -- start 2024-04-25T15:29:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.425+0000 7f2cc67d7640 1 -- start start 2024-04-25T15:29:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cc67d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00cf300 0x7f2cc01fa590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cc67d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc00d2410 0x7f2cc0151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cc67d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc0071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cc67d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc00af9d0 con 0x7f2cc00cf300 2024-04-25T15:29:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cc67d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc00af850 con 0x7f2cc00d2410 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cc67d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc00cf710 con 0x7f2cc0160870 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00cf300 0x7f2cc01fa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00cf300 0x7f2cc01fa590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35868/0 (socket says 172.21.15.67:35868) 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 -- 172.21.15.67:0/757290925 learned_addr learned my addr 172.21.15.67:0/757290925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbf7fe640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc00d2410 0x7f2cc0151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cc4d74640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc0071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 -- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 msgr2=0x7f2cc0071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc0071370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 -- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc00d2410 msgr2=0x7f2cc0151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc00d2410 0x7f2cc0151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 -- 172.21.15.67:0/757290925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc01f6b30 con 0x7f2cc00cf300 2024-04-25T15:29:24.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbffff640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00cf300 0x7f2cc01fa590 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800d5d0 tx=0x7f2ca800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.427+0000 7f2cbcff9640 1 -- 172.21.15.67:0/757290925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8004240 con 0x7f2cc00cf300 2024-04-25T15:29:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.428+0000 7f2cbcff9640 1 -- 172.21.15.67:0/757290925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca80043e0 con 0x7f2cc00cf300 2024-04-25T15:29:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.428+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc01f6e10 con 0x7f2cc00cf300 2024-04-25T15:29:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.428+0000 7f2cbcff9640 1 -- 172.21.15.67:0/757290925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8003830 con 0x7f2cc00cf300 2024-04-25T15:29:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.428+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc01f73d0 con 0x7f2cc00cf300 2024-04-25T15:29:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.429+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc0050740 con 0x7f2cc00cf300 2024-04-25T15:29:24.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.429+0000 7f2cbcff9640 1 -- 172.21.15.67:0/757290925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2ca80039d0 con 0x7f2cc00cf300 2024-04-25T15:29:24.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.430+0000 7f2cbcff9640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ca00744b0 0x7f2ca00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:24.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.430+0000 7f2cbcff9640 1 -- 172.21.15.67:0/757290925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f2ca8095110 con 0x7f2cc00cf300 2024-04-25T15:29:24.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.430+0000 7f2cbf7fe640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ca00744b0 0x7f2ca00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:24.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.431+0000 7f2cbf7fe640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ca00744b0 0x7f2ca00769c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0158de0 tx=0x7f2cb0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.435+0000 7f2cbcff9640 1 -- 172.21.15.67:0/757290925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ca805f550 con 0x7f2cc00cf300 2024-04-25T15:29:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.623+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2cc00d2170 con 0x7f2cc00cf300 2024-04-25T15:29:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.623+0000 7f2cbcff9640 1 -- 172.21.15.67:0/757290925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+12985 (secure 0 0 0) 0x7f2ca8063850 con 0x7f2cc00cf300 2024-04-25T15:29:24.624 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:24.624 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":95,"fsid":"b43d0acb-252a-4f7a-bf14-34d789b94b95","created":"2024-04-25T15:25:04.588266+0000","modified":"2024-04-25T15:29:21.455726+0000","last_up_change":"2024-04-25T15:25:15.873131+0000","last_in_change":"2024-04-25T15:25:06.951778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:25:16.812821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfdf9fe-7025-4530-b11c-fd7ee84b9fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6809","nonce":3594090151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6811","nonce":3594090151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6815","nonce":3594090151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3594090151},{"type":"v1","addr":"172.21.15.67:6813","nonce":3594090151}]},"public_addr":"172.21.15.67:6809/3594090151","cluster_addr":"172.21.15.67:6811/3594090151","heartbeat_back_addr":"172.21.15.67:6815/3594090151","heartbeat_front_addr":"172.21.15.67:6813/3594090151","state":["exists","up"]},{"osd":1,"uuid":"bc9efa57-f337-42d2-b403-f6c8d5e2a4c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6801","nonce":1421304389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6803","nonce":1421304389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6807","nonce":1421304389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1421304389},{"type":"v1","addr":"172.21.15.67:6805","nonce":1421304389}]},"public_addr":"172.21.15.67:6801/1421304389","cluster_addr":"172.21.15.67:6803/1421304389","heartbeat_back_addr":"172.21.15.67:6807/1421304389","heartbeat_front_addr":"172.21.15.67:6805/1421304389","state":["exists","up"]},{"osd":2,"uuid":"fff867ff-c0c4-4cc4-a8b5-8806f6f6e34c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6819","nonce":2881232327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6823","nonce":2881232327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6829","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6831","nonce":2881232327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2881232327},{"type":"v1","addr":"172.21.15.67:6827","nonce":2881232327}]},"public_addr":"172.21.15.67:6819/2881232327","cluster_addr":"172.21.15.67:6823/2881232327","heartbeat_back_addr":"172.21.15.67:6831/2881232327","heartbeat_front_addr":"172.21.15.67:6827/2881232327","state":["exists","up"]},{"osd":3,"uuid":"42f8d3ce-92d8-4b80-90ab-f847ee917452","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6818","nonce":1127217835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6822","nonce":1127217835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6830","nonce":1127217835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":1127217835},{"type":"v1","addr":"172.21.15.67:6826","nonce":1127217835}]},"public_addr":"172.21.15.67:6818/1127217835","cluster_addr":"172.21.15.67:6822/1127217835","heartbeat_back_addr":"172.21.15.67:6830/1127217835","heartbeat_front_addr":"172.21.15.67:6826/1127217835","state":["exists","up"]},{"osd":4,"uuid":"a422a859-f7f2-4ade-ae44-67a905058938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6817","nonce":614570349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6819","nonce":614570349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6823","nonce":614570349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":614570349},{"type":"v1","addr":"172.21.15.176:6821","nonce":614570349}]},"public_addr":"172.21.15.176:6817/614570349","cluster_addr":"172.21.15.176:6819/614570349","heartbeat_back_addr":"172.21.15.176:6823/614570349","heartbeat_front_addr":"172.21.15.176:6821/614570349","state":["exists","up"]},{"osd":5,"uuid":"da442e17-9ae8-4b0a-939f-64578aa41f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6825","nonce":960414228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6827","nonce":960414228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6831","nonce":960414228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":960414228},{"type":"v1","addr":"172.21.15.176:6829","nonce":960414228}]},"public_addr":"172.21.15.176:6825/960414228","cluster_addr":"172.21.15.176:6827/960414228","heartbeat_back_addr":"172.21.15.176:6831/960414228","heartbeat_front_addr":"172.21.15.176:6829/960414228","state":["exists","up"]},{"osd":6,"uuid":"c6374d82-7a32-4b64-b4ba-e33737cb15f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6801","nonce":744880207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6803","nonce":744880207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6807","nonce":744880207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":744880207},{"type":"v1","addr":"172.21.15.176:6805","nonce":744880207}]},"public_addr":"172.21.15.176:6801/744880207","cluster_addr":"172.21.15.176:6803/744880207","heartbeat_back_addr":"172.21.15.176:6807/744880207","heartbeat_front_addr":"172.21.15.176:6805/744880207","state":["exists","up"]},{"osd":7,"uuid":"6546bb86-09ac-4dc9-b996-84004d2444d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6809","nonce":592477191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6811","nonce":592477191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6815","nonce":592477191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":592477191},{"type":"v1","addr":"172.21.15.176:6813","nonce":592477191}]},"public_addr":"172.21.15.176:6809/592477191","cluster_addr":"172.21.15.176:6811/592477191","heartbeat_back_addr":"172.21.15.176:6815/592477191","heartbeat_front_addr":"172.21.15.176:6813/592477191","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.062208+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.061820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.162735+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.105978+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.138220+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.180356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.121204+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:25:12.137950+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/3429198629":"2024-04-26T15:29:18.417267+0000","172.21.15.67:6833/3429198629":"2024-04-26T15:29:18.417267+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:29:24.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ca00744b0 msgr2=0x7f2ca00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ca00744b0 0x7f2ca00769c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0158de0 tx=0x7f2cb0009040 comp rx=0 tx=0).stop 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00cf300 msgr2=0x7f2cc01fa590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00cf300 0x7f2cc01fa590 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800d5d0 tx=0x7f2ca800da90 comp rx=0 tx=0).stop 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 shutdown_connections 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f2ca00744b0 0x7f2ca00769c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc0160870 0x7f2cc0071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc00d2410 0x7f2cc0151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 --2- 172.21.15.67:0/757290925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cc00cf300 0x7f2cc01fa590 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.625+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 >> 172.21.15.67:0/757290925 conn(0x7f2cc009b7d0 msgr2=0x7f2cc00afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.626+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 shutdown_connections 2024-04-25T15:29:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:24.626+0000 7f2cc67d7640 1 -- 172.21.15.67:0/757290925 wait complete. 2024-04-25T15:29:25.639 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T15:29:25.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:25.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.768+0000 7fd186ea8640 1 Processor -- start 2024-04-25T15:29:25.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.768+0000 7fd186ea8640 1 -- start start 2024-04-25T15:29:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd186ea8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd18015cfa0 0x7fd1801458c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd186ea8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 0x7fd18006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd186ea8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18006df20 con 0x7fd1800ce250 2024-04-25T15:29:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd186ea8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd1800b4ec0 con 0x7fd18015cfa0 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd186ea8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd18015d5c0 con 0x7fd18006d5d0 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd184c44640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd1800ce250 0x7fd1800b47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39018/0 (socket says 172.21.15.67:39018) 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd184c44640 1 -- 172.21.15.67:0/800527579 learned_addr learned my addr 172.21.15.67:0/800527579 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd185445640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 0x7fd18006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.769+0000 7fd17ffff640 1 --2- 172.21.15.67:0/800527579 >> v2:172.21.15.176:3300/0 conn(0x7fd18015cfa0 0x7fd1801458c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116579321 0 0) 0x7fd178001490 con 0x7fd1800ce250 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd16c003520 con 0x7fd1800ce250 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4124517307 0 0) 0x7fd16c003520 con 0x7fd1800ce250 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 msgr2=0x7fd18006d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 0x7fd18006d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 >> v2:172.21.15.176:3300/0 conn(0x7fd18015cfa0 msgr2=0x7fd1801458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 --2- 172.21.15.67:0/800527579 >> v2:172.21.15.176:3300/0 conn(0x7fd18015cfa0 0x7fd1801458c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18015d8a0 con 0x7fd1800ce250 2024-04-25T15:29:25.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fd178001bc0 con 0x7fd1800ce250 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17ffff640 1 --2- 172.21.15.67:0/800527579 >> v2:172.21.15.176:3300/0 conn(0x7fd18015cfa0 0x7fd1801458c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd185445640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 0x7fd18006d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 >> v1:172.21.15.67:6789/0 conn(0x7fd1800ce250 legacy=0x7fd1800b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 0x7fd16c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd16c003f10 0x7fd16c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd16c004d40 0x7fd16c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd1800b4ec0 con 0x7fd16c003f10 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd18015d5c0 con 0x7fd16c004d40 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c00baa0 con 0x7fd18006d5d0 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd17ffff640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd16c003f10 0x7fd16c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd185445640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd16c004d40 0x7fd16c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.770+0000 7fd184c44640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 0x7fd16c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.771+0000 7fd17ffff640 1 -- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 msgr2=0x7fd16c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.771+0000 7fd17ffff640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd18006d5d0 0x7fd16c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.771+0000 7fd17ffff640 1 -- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd16c004d40 msgr2=0x7fd16c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.771+0000 7fd17ffff640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd16c004d40 0x7fd16c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.771+0000 7fd17ffff640 1 -- 172.21.15.67:0/800527579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c003520 con 0x7fd16c003f10 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.771+0000 7fd17ffff640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd16c003f10 0x7fd16c004800 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd168009c60 tx=0x7fd16c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd16a4fadefb846a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.771+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168036070 con 0x7fd16c003f10 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd168031400 con 0x7fd16c003f10 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd17f7fe640 1 -- 172.21.15.67:0/800527579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168009860 con 0x7fd16c003f10 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd186ea8640 1 -- 172.21.15.67:0/800527579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd16c003f10 msgr2=0x7fd16c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd186ea8640 1 --2- 172.21.15.67:0/800527579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd16c003f10 0x7fd16c004800 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd168009c60 tx=0x7fd16c005830 comp rx=0 tx=0).stop 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd185445640 1 -- 172.21.15.67:0/800527579 reap_dead start 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd186ea8640 1 -- 172.21.15.67:0/800527579 shutdown_connections 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd186ea8640 1 -- 172.21.15.67:0/800527579 >> 172.21.15.67:0/800527579 conn(0x7fd180073090 msgr2=0x7fd1800734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd186ea8640 1 -- 172.21.15.67:0/800527579 shutdown_connections 2024-04-25T15:29:25.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.772+0000 7fd186ea8640 1 -- 172.21.15.67:0/800527579 wait complete. 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.774+0000 7fd186ea8640 1 Processor -- start 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.774+0000 7fd186ea8640 1 -- start start 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd186ea8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd16c003f10 0x7fd18015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd186ea8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd180133500 0x7fd1801f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd186ea8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd180138e00 0x7fd180131020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd186ea8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd18015de60 con 0x7fd180133500 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd186ea8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd18006f640 con 0x7fd180138e00 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd186ea8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd180071310 con 0x7fd16c003f10 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd180133500 0x7fd1801f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd180133500 0x7fd1801f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35886/0 (socket says 172.21.15.67:35886) 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 -- 172.21.15.67:0/3699412061 learned_addr learned my addr 172.21.15.67:0/3699412061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd184c44640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd16c003f10 0x7fd18015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 -- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd16c003f10 msgr2=0x7fd18015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd16c003f10 0x7fd18015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd185445640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd180138e00 0x7fd180131020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 -- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd180138e00 msgr2=0x7fd180131020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd180138e00 0x7fd180131020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 -- 172.21.15.67:0/3699412061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd180137d60 con 0x7fd180133500 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd184c44640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd16c003f10 0x7fd18015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17ffff640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd180133500 0x7fd1801f4a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd168009d80 tx=0x7fd168004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168040070 con 0x7fd180133500 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd16803b660 con 0x7fd180133500 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.776+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd168036040 con 0x7fd180133500 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.777+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd180138040 con 0x7fd180133500 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.777+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd180138600 con 0x7fd180133500 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.777+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd180050740 con 0x7fd180133500 2024-04-25T15:29:25.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.778+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd168043070 con 0x7fd180133500 2024-04-25T15:29:25.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd17d7fa640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd1600744b0 0x7fd1600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7fd1680b5740 con 0x7fd180133500 2024-04-25T15:29:25.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd17d7fa640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fd16007a3c0 0x7fd16007c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 --> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd168040e70 con 0x7fd16007a3c0 2024-04-25T15:29:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd184c44640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd1600744b0 0x7fd1600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fd1680bc050 con 0x7fd180133500 2024-04-25T15:29:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd185445640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fd16007a3c0 0x7fd16007c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd185445640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fd16007a3c0 0x7fd16007c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.779+0000 7fd184c44640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd1600744b0 0x7fd1600769c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd1800b6c80 tx=0x7fd178006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:25.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.780+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== osd.0 v2:172.21.15.67:6808/3594090151 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd168040e70 con 0x7fd16007a3c0 2024-04-25T15:29:25.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 --> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd1800b6600 con 0x7fd16007a3c0 2024-04-25T15:29:25.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd17d7fa640 1 -- 172.21.15.67:0/3699412061 <== osd.0 v2:172.21.15.67:6808/3594090151 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd1800b6600 con 0x7fd16007a3c0 2024-04-25T15:29:25.795 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:25.795 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:25.795 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fd16007a3c0 msgr2=0x7fd16007c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:6808/3594090151,v1:172.21.15.67:6809/3594090151] conn(0x7fd16007a3c0 0x7fd16007c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd1600744b0 msgr2=0x7fd1600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fd1600744b0 0x7fd1600769c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd1800b6c80 tx=0x7fd178006040 comp rx=0 tx=0).stop 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd180133500 msgr2=0x7fd1801f4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 --2- 172.21.15.67:0/3699412061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd180133500 0x7fd1801f4a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd168009d80 tx=0x7fd168004730 comp rx=0 tx=0).stop 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd185445640 1 -- 172.21.15.67:0/3699412061 reap_dead start 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 shutdown_connections 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 >> 172.21.15.67:0/3699412061 conn(0x7fd180073090 msgr2=0x7fd1800b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 shutdown_connections 2024-04-25T15:29:25.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.794+0000 7fd186ea8640 1 -- 172.21.15.67:0/3699412061 wait complete. 2024-04-25T15:29:25.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T15:29:25.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.917+0000 7f19f0230640 1 Processor -- start 2024-04-25T15:29:25.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.917+0000 7f19f0230640 1 -- start start 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19f0230640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80d1210 0x7f19e80d5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19f0230640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19e80d1850 con 0x7f19e80d0e00 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19f0230640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19e8160de0 con 0x7f19e806ccd0 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19f0230640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f19e80d1620 con 0x7f19e80d1210 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19edfcc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f19e80d0e00 0x7f19e806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39024/0 (socket says 172.21.15.67:39024) 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19edfcc640 1 -- 172.21.15.67:0/912418663 learned_addr learned my addr 172.21.15.67:0/912418663 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19ee7cd640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80d1210 0x7f19e80d5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3433969186 0 0) 0x7f19e80d1850 con 0x7f19e80d0e00 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19d0003520 con 0x7f19e80d0e00 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2492272609 0 0) 0x7f19d0003520 con 0x7f19e80d0e00 2024-04-25T15:29:25.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80d1210 msgr2=0x7f19e80d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19ecfca640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80d1210 0x7f19e80d5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.919+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 >> v1:172.21.15.176:6789/0 conn(0x7f19e806ccd0 legacy=0x7f19e80d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e81610c0 con 0x7f19e80d0e00 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f19e0001670 con 0x7f19e80d0e00 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 >> v1:172.21.15.67:6789/0 conn(0x7f19e80d0e00 legacy=0x7f19e806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19d00039e0 0x7f19d0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d0004390 0x7f19d00048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d00050c0 0x7f19d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19d000c000 con 0x7f19d00050c0 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f19d000c1b0 con 0x7f19d0004390 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f19d000c3b0 con 0x7f19d00039e0 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ee7cd640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80d1210 0x7f19e80d5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:25.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ee7cd640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d00050c0 0x7f19d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19edfcc640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19d00039e0 0x7f19d0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ed7cb640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d0004390 0x7f19d00048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ee7cd640 1 -- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19d00039e0 msgr2=0x7f19d0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ee7cd640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19d00039e0 0x7f19d0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ee7cd640 1 -- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d0004390 msgr2=0x7f19d00048c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ee7cd640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d0004390 0x7f19d00048c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.920+0000 7f19ee7cd640 1 -- 172.21.15.67:0/912418663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d0003520 con 0x7f19d00050c0 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.921+0000 7f19ee7cd640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d00050c0 0x7f19d000b9c0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f19e80bbeb0 tx=0x7f19d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f6d88b482cc5be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.921+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8036070 con 0x7f19d00050c0 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.921+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d8003c50 con 0x7f19d00050c0 2024-04-25T15:29:25.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.921+0000 7f19ecfca640 1 -- 172.21.15.67:0/912418663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8009d40 con 0x7f19d00050c0 2024-04-25T15:29:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.922+0000 7f19f0230640 1 -- 172.21.15.67:0/912418663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d00050c0 msgr2=0x7f19d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.922+0000 7f19f0230640 1 --2- 172.21.15.67:0/912418663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d00050c0 0x7f19d000b9c0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f19e80bbeb0 tx=0x7f19d0005830 comp rx=0 tx=0).stop 2024-04-25T15:29:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.922+0000 7f19ee7cd640 1 -- 172.21.15.67:0/912418663 reap_dead start 2024-04-25T15:29:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.922+0000 7f19f0230640 1 -- 172.21.15.67:0/912418663 shutdown_connections 2024-04-25T15:29:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.922+0000 7f19f0230640 1 -- 172.21.15.67:0/912418663 >> 172.21.15.67:0/912418663 conn(0x7f19e80d5d90 msgr2=0x7f19e80d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.922+0000 7f19f0230640 1 -- 172.21.15.67:0/912418663 shutdown_connections 2024-04-25T15:29:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.922+0000 7f19f0230640 1 -- 172.21.15.67:0/912418663 wait complete. 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.924+0000 7f19f0230640 1 Processor -- start 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.924+0000 7f19f0230640 1 -- start start 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19f0230640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d00050c0 0x7f19e80ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19f0230640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19e80cc0a0 0x7f19e81f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19f0230640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80cc4b0 0x7f19e80cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19f0230640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19e80b5320 con 0x7f19e80cc0a0 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19f0230640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f19e8160ad0 con 0x7f19d00050c0 2024-04-25T15:29:25.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19f0230640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f19e8164e00 con 0x7f19e80cc4b0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19e80cc0a0 0x7f19e81f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19e80cc0a0 0x7f19e81f6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35900/0 (socket says 172.21.15.67:35900) 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 -- 172.21.15.67:0/690772051 learned_addr learned my addr 172.21.15.67:0/690772051 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19edfcc640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d00050c0 0x7f19e80ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ee7cd640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80cc4b0 0x7f19e80cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 -- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80cc4b0 msgr2=0x7f19e80cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80cc4b0 0x7f19e80cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 -- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d00050c0 msgr2=0x7f19e80ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d00050c0 0x7f19e80ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 -- 172.21.15.67:0/690772051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e80ca810 con 0x7f19e80cc0a0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.925+0000 7f19ed7cb640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19e80cc0a0 0x7f19e81f6100 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00f7c0 tx=0x7f19dc00fbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.926+0000 7f19e67fc640 1 -- 172.21.15.67:0/690772051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0050d0 con 0x7f19e80cc0a0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.926+0000 7f19e67fc640 1 -- 172.21.15.67:0/690772051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19dc018070 con 0x7f19e80cc0a0 2024-04-25T15:29:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.926+0000 7f19e67fc640 1 -- 172.21.15.67:0/690772051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc00d730 con 0x7f19e80cc0a0 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.926+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e80cfb90 con 0x7f19e80cc0a0 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.926+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e81f6fc0 con 0x7f19e80cc0a0 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.927+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e8050740 con 0x7f19e80cc0a0 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.927+0000 7f19e67fc640 1 -- 172.21.15.67:0/690772051 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f19dc00d8d0 con 0x7f19e80cc0a0 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.928+0000 7f19e67fc640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19d40744b0 0x7f19d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.928+0000 7f19e67fc640 1 -- 172.21.15.67:0/690772051 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f19dc094d90 con 0x7f19e80cc0a0 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.928+0000 7f19edfcc640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19d40744b0 0x7f19d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.928+0000 7f19edfcc640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19d40744b0 0x7f19d40769c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f19e0007d90 tx=0x7f19e00078f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:25.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:25.932+0000 7f19e67fc640 1 -- 172.21.15.67:0/690772051 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19dc05f820 con 0x7f19e80cc0a0 2024-04-25T15:29:26.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.117+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f19e81f6a70 con 0x7f19d40744b0 2024-04-25T15:29:26.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.118+0000 7f19e67fc640 1 -- 172.21.15.67:0/690772051 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f19e81f6a70 con 0x7f19d40744b0 2024-04-25T15:29:26.119 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T15:29:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19d40744b0 msgr2=0x7f19d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19d40744b0 0x7f19d40769c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f19e0007d90 tx=0x7f19e00078f0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19e80cc0a0 msgr2=0x7f19e81f6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19e80cc0a0 0x7f19e81f6100 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f19dc00f7c0 tx=0x7f19dc00fbc0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 shutdown_connections 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f19d40744b0 0x7f19d40769c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e80cc4b0 0x7f19e80cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19e80cc0a0 0x7f19e81f6100 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 --2- 172.21.15.67:0/690772051 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19d00050c0 0x7f19e80ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.120+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 >> 172.21.15.67:0/690772051 conn(0x7f19e80d5d90 msgr2=0x7f19e80b7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.121+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 shutdown_connections 2024-04-25T15:29:26.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.121+0000 7f19f0230640 1 -- 172.21.15.67:0/690772051 wait complete. 2024-04-25T15:29:26.132 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T15:29:26.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.247+0000 7f83a65b2640 1 Processor -- start 2024-04-25T15:29:26.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.247+0000 7f83a65b2640 1 -- start start 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f83a65b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f83a0161710 0x7f83a015e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f83a65b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83a006c5c0 0x7f83a006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f83a65b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a01643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f83a65b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f83a01609b0 con 0x7f83a006c5c0 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f83a65b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f83a006cf10 con 0x7f83a0161710 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f83a65b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f83a0164cd0 con 0x7f83a0163fa0 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f839f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83a006c5c0 0x7f83a006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f839f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83a006c5c0 0x7f83a006c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35902/0 (socket says 172.21.15.67:35902) 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.248+0000 7f839f7fe640 1 -- 172.21.15.67:0/2340450472 learned_addr learned my addr 172.21.15.67:0/2340450472 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f83a4b4f640 1 --2- 172.21.15.67:0/2340450472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a01643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f839ffff640 1 --2- 172.21.15.67:0/2340450472 >> v2:172.21.15.176:3300/0 conn(0x7f83a0161710 0x7f83a015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f839f7fe640 1 -- 172.21.15.67:0/2340450472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 msgr2=0x7f83a01643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f839f7fe640 1 --2- 172.21.15.67:0/2340450472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a01643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f839f7fe640 1 -- 172.21.15.67:0/2340450472 >> v2:172.21.15.176:3300/0 conn(0x7f83a0161710 msgr2=0x7f83a015e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f839f7fe640 1 --2- 172.21.15.67:0/2340450472 >> v2:172.21.15.176:3300/0 conn(0x7f83a0161710 0x7f83a015e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f839f7fe640 1 -- 172.21.15.67:0/2340450472 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a0160c30 con 0x7f83a006c5c0 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.249+0000 7f839f7fe640 1 --2- 172.21.15.67:0/2340450472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a006c5c0 0x7f83a006c9d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f83a00d5cf0 tx=0x7f839402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dc5f05f453e4e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f839effd640 1 -- 172.21.15.67:0/2340450472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394037070 con 0x7f83a006c5c0 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f839effd640 1 -- 172.21.15.67:0/2340450472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8394032440 con 0x7f83a006c5c0 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f839effd640 1 -- 172.21.15.67:0/2340450472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394032810 con 0x7f83a006c5c0 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 -- 172.21.15.67:0/2340450472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a006c5c0 msgr2=0x7f83a006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 --2- 172.21.15.67:0/2340450472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a006c5c0 0x7f83a006c9d0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f83a00d5cf0 tx=0x7f839402a140 comp rx=0 tx=0).stop 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 -- 172.21.15.67:0/2340450472 shutdown_connections 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 --2- 172.21.15.67:0/2340450472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a01643b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 --2- 172.21.15.67:0/2340450472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a006c5c0 0x7f83a006c9d0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 --2- 172.21.15.67:0/2340450472 >> v2:172.21.15.176:3300/0 conn(0x7f83a0161710 0x7f83a015e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 -- 172.21.15.67:0/2340450472 >> 172.21.15.67:0/2340450472 conn(0x7f83a00d5d90 msgr2=0x7f83a00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 -- 172.21.15.67:0/2340450472 shutdown_connections 2024-04-25T15:29:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.250+0000 7f83a65b2640 1 -- 172.21.15.67:0/2340450472 wait complete. 2024-04-25T15:29:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.252+0000 7f83a65b2640 1 Processor -- start 2024-04-25T15:29:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.252+0000 7f83a65b2640 1 -- start start 2024-04-25T15:29:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a65b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83a006c5c0 0x7f83a00ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a65b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a0161710 0x7f83a00ba3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a65b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a00ba930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a65b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83a01648f0 con 0x7f83a0161710 2024-04-25T15:29:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a65b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f83a01611f0 con 0x7f83a006c5c0 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a65b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f83a0164a70 con 0x7f83a0163fa0 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a0161710 0x7f83a00ba3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a0161710 0x7f83a00ba3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35904/0 (socket says 172.21.15.67:35904) 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 -- 172.21.15.67:0/1921870299 learned_addr learned my addr 172.21.15.67:0/1921870299 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a4b4f640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a00ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839ffff640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83a006c5c0 0x7f83a00ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 -- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 msgr2=0x7f83a00ba930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a00ba930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 -- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83a006c5c0 msgr2=0x7f83a00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83a006c5c0 0x7f83a00ca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 -- 172.21.15.67:0/1921870299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a00b6070 con 0x7f83a0161710 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f83a4b4f640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83a0163fa0 0x7f83a00ba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839ffff640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83a006c5c0 0x7f83a00ca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:26.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839f7fe640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a0161710 0x7f83a00ba3f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f839402a180 tx=0x7f83940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.254+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394041020 con 0x7f83a0161710 2024-04-25T15:29:26.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.255+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83940306c0 con 0x7f83a0161710 2024-04-25T15:29:26.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.255+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394037040 con 0x7f83a0161710 2024-04-25T15:29:26.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.255+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a00b6350 con 0x7f83a0161710 2024-04-25T15:29:26.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.255+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a00b6910 con 0x7f83a0161710 2024-04-25T15:29:26.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.255+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83a0050740 con 0x7f83a0161710 2024-04-25T15:29:26.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.256+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8394044020 con 0x7f83a0161710 2024-04-25T15:29:26.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f839cff9640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f838c0744b0 0x7f838c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f83940b5c50 con 0x7f83a0161710 2024-04-25T15:29:26.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f839cff9640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f838c07a3c0 0x7f838c07c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 --> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8394008f00 con 0x7f838c07a3c0 2024-04-25T15:29:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f83940b60e0 con 0x7f83a0161710 2024-04-25T15:29:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f839ffff640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f838c0744b0 0x7f838c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f83a4b4f640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f838c07a3c0 0x7f838c07c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.257+0000 7f83a4b4f640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f838c07a3c0 0x7f838c07c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.258+0000 7f839ffff640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f838c0744b0 0x7f838c0769c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8390006fd0 tx=0x7f8390008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.258+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== osd.1 v2:172.21.15.67:6800/1421304389 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8394008f00 con 0x7f838c07a3c0 2024-04-25T15:29:26.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.271+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 --> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f83a00b5e20 con 0x7f838c07a3c0 2024-04-25T15:29:26.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f839cff9640 1 -- 172.21.15.67:0/1921870299 <== osd.1 v2:172.21.15.67:6800/1421304389 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f83a00b5e20 con 0x7f838c07a3c0 2024-04-25T15:29:26.272 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:26.273 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f838c07a3c0 msgr2=0x7f838c07c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:6800/1421304389,v1:172.21.15.67:6801/1421304389] conn(0x7f838c07a3c0 0x7f838c07c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f838c0744b0 msgr2=0x7f838c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f838c0744b0 0x7f838c0769c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8390006fd0 tx=0x7f8390008040 comp rx=0 tx=0).stop 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a0161710 msgr2=0x7f83a00ba3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 --2- 172.21.15.67:0/1921870299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a0161710 0x7f83a00ba3f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f839402a180 tx=0x7f83940046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a4b4f640 1 -- 172.21.15.67:0/1921870299 reap_dead start 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 shutdown_connections 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 >> 172.21.15.67:0/1921870299 conn(0x7f83a00d5d90 msgr2=0x7f83a00d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 shutdown_connections 2024-04-25T15:29:26.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.272+0000 7f83a65b2640 1 -- 172.21.15.67:0/1921870299 wait complete. 2024-04-25T15:29:26.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T15:29:26.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.395+0000 7fcc79506640 1 Processor -- start 2024-04-25T15:29:26.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.395+0000 7fcc79506640 1 -- start start 2024-04-25T15:29:26.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.396+0000 7fcc79506640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc74164b00 0x7fcc740b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.396+0000 7fcc79506640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc741615f0 con 0x7fcc7415df80 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.396+0000 7fcc79506640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc74164020 con 0x7fcc74160fa0 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.396+0000 7fcc79506640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcc741613b0 con 0x7fcc74164b00 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.396+0000 7fcc72ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc74164b00 0x7fcc740b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.396+0000 7fcc6bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc7415df80 0x7fcc74160890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39030/0 (socket says 172.21.15.67:39030) 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.396+0000 7fcc6bfff640 1 -- 172.21.15.67:0/1756621730 learned_addr learned my addr 172.21.15.67:0/1756621730 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.397+0000 7fcc72ffd640 1 -- 172.21.15.67:0/1756621730 >> v1:172.21.15.176:6789/0 conn(0x7fcc74160fa0 legacy=0x7fcc740b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.397+0000 7fcc72ffd640 1 -- 172.21.15.67:0/1756621730 >> v1:172.21.15.67:6789/0 conn(0x7fcc7415df80 legacy=0x7fcc74160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.397+0000 7fcc727fc640 1 -- 172.21.15.67:0/1756621730 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 116470980 0 0) 0x7fcc741615f0 con 0x7fcc7415df80 2024-04-25T15:29:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.397+0000 7fcc72ffd640 1 -- 172.21.15.67:0/1756621730 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc74164300 con 0x7fcc74164b00 2024-04-25T15:29:26.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/1756621730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74164b00 0x7fcc740b7b80 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fcc640097a0 tx=0x7fcc6402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb420752ead86443 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc727fc640 1 -- 172.21.15.67:0/1756621730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64037070 con 0x7fcc74164b00 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc727fc640 1 -- 172.21.15.67:0/1756621730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc64032440 con 0x7fcc74164b00 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc727fc640 1 -- 172.21.15.67:0/1756621730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64032810 con 0x7fcc74164b00 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc79506640 1 -- 172.21.15.67:0/1756621730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74164b00 msgr2=0x7fcc740b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc79506640 1 --2- 172.21.15.67:0/1756621730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74164b00 0x7fcc740b7b80 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fcc640097a0 tx=0x7fcc6402a1e0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc79506640 1 -- 172.21.15.67:0/1756621730 shutdown_connections 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc79506640 1 --2- 172.21.15.67:0/1756621730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74164b00 0x7fcc740b7b80 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc79506640 1 -- 172.21.15.67:0/1756621730 >> 172.21.15.67:0/1756621730 conn(0x7fcc740d2bc0 msgr2=0x7fcc740d2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc79506640 1 -- 172.21.15.67:0/1756621730 shutdown_connections 2024-04-25T15:29:26.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.398+0000 7fcc79506640 1 -- 172.21.15.67:0/1756621730 wait complete. 2024-04-25T15:29:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.400+0000 7fcc79506640 1 Processor -- start 2024-04-25T15:29:26.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.400+0000 7fcc79506640 1 -- start start 2024-04-25T15:29:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.401+0000 7fcc79506640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7415df80 0x7fcc740ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.401+0000 7fcc79506640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74160fa0 0x7fcc740cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc79506640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc74164b00 0x7fcc740cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc79506640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc740b84d0 con 0x7fcc74160fa0 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc79506640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcc740b8350 con 0x7fcc7415df80 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc79506640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcc74164670 con 0x7fcc74164b00 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74160fa0 0x7fcc740cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74160fa0 0x7fcc740cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35932/0 (socket says 172.21.15.67:35932) 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 -- 172.21.15.67:0/4208612945 learned_addr learned my addr 172.21.15.67:0/4208612945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7415df80 0x7fcc740ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc737fe640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc74164b00 0x7fcc740cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 -- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc74164b00 msgr2=0x7fcc740cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc74164b00 0x7fcc740cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 -- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7415df80 msgr2=0x7fcc740ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7415df80 0x7fcc740ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 -- 172.21.15.67:0/4208612945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc740ac8a0 con 0x7fcc74160fa0 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6bfff640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74160fa0 0x7fcc740cdb70 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00c430 tx=0x7fcc5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6b7fe640 1 -- 172.21.15.67:0/4208612945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c0074e0 con 0x7fcc74160fa0 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6b7fe640 1 -- 172.21.15.67:0/4208612945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc5c0047e0 con 0x7fcc74160fa0 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc740acb80 con 0x7fcc74160fa0 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc6b7fe640 1 -- 172.21.15.67:0/4208612945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c004bd0 con 0x7fcc74160fa0 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.402+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc740ad140 con 0x7fcc74160fa0 2024-04-25T15:29:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.403+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc74050740 con 0x7fcc74160fa0 2024-04-25T15:29:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.404+0000 7fcc6b7fe640 1 -- 172.21.15.67:0/4208612945 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcc5c011070 con 0x7fcc74160fa0 2024-04-25T15:29:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.405+0000 7fcc6b7fe640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc540744b0 0x7fcc540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.405+0000 7fcc6b7fe640 1 -- 172.21.15.67:0/4208612945 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7fcc5c094910 con 0x7fcc74160fa0 2024-04-25T15:29:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.405+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc540744b0 0x7fcc540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.405+0000 7fcc72ffd640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc540744b0 0x7fcc540769c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fcc64034000 tx=0x7fcc6402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.409+0000 7fcc6b7fe640 1 -- 172.21.15.67:0/4208612945 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc5c05ee70 con 0x7fcc74160fa0 2024-04-25T15:29:26.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.595+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fcc740d4170 con 0x7fcc540744b0 2024-04-25T15:29:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.596+0000 7fcc6b7fe640 1 -- 172.21.15.67:0/4208612945 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcc740d4170 con 0x7fcc540744b0 2024-04-25T15:29:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc540744b0 msgr2=0x7fcc540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc540744b0 0x7fcc540769c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fcc64034000 tx=0x7fcc6402e000 comp rx=0 tx=0).stop 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74160fa0 msgr2=0x7fcc740cdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74160fa0 0x7fcc740cdb70 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00c430 tx=0x7fcc5c00c8f0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 shutdown_connections 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fcc540744b0 0x7fcc540769c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc74164b00 0x7fcc740cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc74160fa0 0x7fcc740cdb70 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 --2- 172.21.15.67:0/4208612945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7415df80 0x7fcc740ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 >> 172.21.15.67:0/4208612945 conn(0x7fcc740d2bc0 msgr2=0x7fcc740d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 shutdown_connections 2024-04-25T15:29:26.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.598+0000 7fcc79506640 1 -- 172.21.15.67:0/4208612945 wait complete. 2024-04-25T15:29:26.610 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T15:29:26.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:26.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.725+0000 7fc1ad615640 1 Processor -- start 2024-04-25T15:29:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.725+0000 7fc1ad615640 1 -- start start 2024-04-25T15:29:26.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1ad615640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8161710 0x7fc1a815e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1ad615640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1a8163fa0 0x7fc1a81643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1ad615640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a81609b0 con 0x7fc1a806c5c0 2024-04-25T15:29:26.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1ad615640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc1a8164cd0 con 0x7fc1a8163fa0 2024-04-25T15:29:26.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1ad615640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a8160c30 con 0x7fc1a8161710 2024-04-25T15:29:26.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1a8163fa0 0x7fc1a81643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1a8163fa0 0x7fc1a81643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35934/0 (socket says 172.21.15.67:35934) 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a77fe640 1 -- 172.21.15.67:0/426248 learned_addr learned my addr 172.21.15.67:0/426248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a67fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1a806c5c0 0x7fc1a806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39038/0 (socket says 172.21.15.67:39038) 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a6ffd640 1 --2- 172.21.15.67:0/426248 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8161710 0x7fc1a815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a77fe640 1 -- 172.21.15.67:0/426248 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8161710 msgr2=0x7fc1a815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a77fe640 1 --2- 172.21.15.67:0/426248 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8161710 0x7fc1a815e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a77fe640 1 -- 172.21.15.67:0/426248 >> v1:172.21.15.67:6789/0 conn(0x7fc1a806c5c0 legacy=0x7fc1a806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a5ffb640 1 -- 172.21.15.67:0/426248 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 622118304 0 0) 0x7fc190001110 con 0x7fc1a806c5c0 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.727+0000 7fc1a77fe640 1 -- 172.21.15.67:0/426248 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a8160f10 con 0x7fc1a8163fa0 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.728+0000 7fc1a77fe640 1 --2- 172.21.15.67:0/426248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8163fa0 0x7fc1a81643b0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fc198009e50 tx=0x7fc19802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffe169da5078145 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.728+0000 7fc1a5ffb640 1 -- 172.21.15.67:0/426248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc198036070 con 0x7fc1a8163fa0 2024-04-25T15:29:26.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.728+0000 7fc1a5ffb640 1 -- 172.21.15.67:0/426248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc198031600 con 0x7fc1a8163fa0 2024-04-25T15:29:26.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.728+0000 7fc1a5ffb640 1 -- 172.21.15.67:0/426248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1980319d0 con 0x7fc1a8163fa0 2024-04-25T15:29:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 -- 172.21.15.67:0/426248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8163fa0 msgr2=0x7fc1a81643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 --2- 172.21.15.67:0/426248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8163fa0 0x7fc1a81643b0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fc198009e50 tx=0x7fc19802a680 comp rx=0 tx=0).stop 2024-04-25T15:29:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 -- 172.21.15.67:0/426248 shutdown_connections 2024-04-25T15:29:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 --2- 172.21.15.67:0/426248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8163fa0 0x7fc1a81643b0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 --2- 172.21.15.67:0/426248 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8161710 0x7fc1a815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 -- 172.21.15.67:0/426248 >> 172.21.15.67:0/426248 conn(0x7fc1a80d5d90 msgr2=0x7fc1a80d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 -- 172.21.15.67:0/426248 shutdown_connections 2024-04-25T15:29:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.729+0000 7fc1ad615640 1 -- 172.21.15.67:0/426248 wait complete. 2024-04-25T15:29:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.731+0000 7fc1ad615640 1 Processor -- start 2024-04-25T15:29:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.731+0000 7fc1ad615640 1 -- start start 2024-04-25T15:29:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1ad615640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a806c5c0 0x7fc1a8155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1ad615640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8161710 0x7fc1a8159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1ad615640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8163fa0 0x7fc1a815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1ad615640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a8164a70 con 0x7fc1a8161710 2024-04-25T15:29:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1ad615640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a81648f0 con 0x7fc1a806c5c0 2024-04-25T15:29:26.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1ad615640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a80d49f0 con 0x7fc1a8163fa0 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1a67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8161710 0x7fc1a8159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1a67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8161710 0x7fc1a8159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35938/0 (socket says 172.21.15.67:35938) 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.732+0000 7fc1a67fc640 1 -- 172.21.15.67:0/4088546856 learned_addr learned my addr 172.21.15.67:0/4088546856 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a77fe640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8163fa0 0x7fc1a815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a6ffd640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a806c5c0 0x7fc1a8155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a67fc640 1 -- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8163fa0 msgr2=0x7fc1a815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a67fc640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8163fa0 0x7fc1a815a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a67fc640 1 -- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a806c5c0 msgr2=0x7fc1a8155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a67fc640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1a806c5c0 0x7fc1a8155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a67fc640 1 -- 172.21.15.67:0/4088546856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a80cf160 con 0x7fc1a8161710 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a77fe640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1a8163fa0 0x7fc1a815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1a67fc640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8161710 0x7fc1a8159da0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fc19000c3e0 tx=0x7fc19000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1900074e0 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1900047e0 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a80cf440 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190004bd0 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.733+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a80cfa00 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.734+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc1a80b55e0 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc190011070 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc19f7fe640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc1880744b0 0x7fc1880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7fc190094ae0 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc19f7fe640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7fc18807a3c0 0x7fc18807c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 --> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc18807cee0 con 0x7fc18807a3c0 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fc1900992c0 con 0x7fc1a8161710 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc1a6ffd640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc1880744b0 0x7fc1880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.735+0000 7fc1a77fe640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7fc18807a3c0 0x7fc18807c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.736+0000 7fc1a77fe640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7fc18807a3c0 0x7fc18807c810 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.736+0000 7fc1a6ffd640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc1880744b0 0x7fc1880769c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0006fd0 tx=0x7fc1a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.736+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== osd.2 v2:172.21.15.67:6817/2881232327 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc18807cee0 con 0x7fc18807a3c0 2024-04-25T15:29:26.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.750+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 --> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc1a8050740 con 0x7fc18807a3c0 2024-04-25T15:29:26.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.750+0000 7fc19f7fe640 1 -- 172.21.15.67:0/4088546856 <== osd.2 v2:172.21.15.67:6817/2881232327 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc1a8050740 con 0x7fc18807a3c0 2024-04-25T15:29:26.751 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:26.751 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:26.751 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.750+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7fc18807a3c0 msgr2=0x7fc18807c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.750+0000 7fc1ad615640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:6817/2881232327,v1:172.21.15.67:6819/2881232327] conn(0x7fc18807a3c0 0x7fc18807c810 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc1880744b0 msgr2=0x7fc1880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fc1880744b0 0x7fc1880769c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0006fd0 tx=0x7fc1a0008040 comp rx=0 tx=0).stop 2024-04-25T15:29:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8161710 msgr2=0x7fc1a8159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 --2- 172.21.15.67:0/4088546856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1a8161710 0x7fc1a8159da0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fc19000c3e0 tx=0x7fc19000c8a0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1a77fe640 1 -- 172.21.15.67:0/4088546856 reap_dead start 2024-04-25T15:29:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 shutdown_connections 2024-04-25T15:29:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 >> 172.21.15.67:0/4088546856 conn(0x7fc1a80d5d90 msgr2=0x7fc1a815e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 shutdown_connections 2024-04-25T15:29:26.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.751+0000 7fc1ad615640 1 -- 172.21.15.67:0/4088546856 wait complete. 2024-04-25T15:29:26.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T15:29:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.874+0000 7f76e1d9d640 1 Processor -- start 2024-04-25T15:29:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.874+0000 7f76e1d9d640 1 -- start start 2024-04-25T15:29:26.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.875+0000 7f76e1d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 0x7f76dc1600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.875+0000 7f76e1d9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f76dc0b7bc0 0x7f76dc0b7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.875+0000 7f76e1d9d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76dc0b8510 con 0x7f76dc0b5260 2024-04-25T15:29:26.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.875+0000 7f76e1d9d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f76dc160620 con 0x7f76dc0b7bc0 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.875+0000 7f76e1d9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f76dc0d0b30 con 0x7f76dc0b4da0 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.875+0000 7f76dbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76dc0b5260 0x7f76dc0b9170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39050/0 (socket says 172.21.15.67:39050) 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76dbfff640 1 -- 172.21.15.67:0/2208328978 learned_addr learned my addr 172.21.15.67:0/2208328978 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76e159c640 1 --2- 172.21.15.67:0/2208328978 >> v2:172.21.15.176:3300/0 conn(0x7f76dc0b7bc0 0x7f76dc0b7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76e0d9b640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 0x7f76dc1600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775925989 0 0) 0x7f76c8001110 con 0x7f76dc0b5260 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76cc003520 con 0x7f76dc0b5260 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1640809160 0 0) 0x7f76cc003520 con 0x7f76dc0b5260 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 msgr2=0x7f76dc1600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 0x7f76dc1600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 >> v2:172.21.15.176:3300/0 conn(0x7f76dc0b7bc0 msgr2=0x7f76dc0b7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 --2- 172.21.15.67:0/2208328978 >> v2:172.21.15.176:3300/0 conn(0x7f76dc0b7bc0 0x7f76dc0b7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc0d4420 con 0x7f76dc0b5260 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f76c8001840 con 0x7f76dc0b5260 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76e159c640 1 --2- 172.21.15.67:0/2208328978 >> v2:172.21.15.176:3300/0 conn(0x7f76dc0b7bc0 0x7f76dc0b7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76e0d9b640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 0x7f76dc1600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 >> v1:172.21.15.67:6789/0 conn(0x7f76dc0b5260 legacy=0x7f76dc0b9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc0039b0 0x7f76cc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc004800 0x7f76cc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 0x7f76cc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc160620 con 0x7f76cc004800 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc0d0b30 con 0x7f76cc0039b0 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f76cc00ba70 con 0x7f76dc0b4da0 2024-04-25T15:29:26.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76dbfff640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc004800 0x7f76cc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76e0d9b640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc0039b0 0x7f76cc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.876+0000 7f76e159c640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 0x7f76cc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.877+0000 7f76dbfff640 1 -- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 msgr2=0x7f76cc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.877+0000 7f76dbfff640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0b4da0 0x7f76cc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.877+0000 7f76dbfff640 1 -- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc0039b0 msgr2=0x7f76cc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.877+0000 7f76dbfff640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc0039b0 0x7f76cc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.877+0000 7f76dbfff640 1 -- 172.21.15.67:0/2208328978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc003520 con 0x7f76cc004800 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.877+0000 7f76dbfff640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc004800 0x7f76cc004c10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c3e0 tx=0x7f76cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5c290d39a62839 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8032020 con 0x7f76cc004800 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76c8005af0 con 0x7f76cc004800 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76db7fe640 1 -- 172.21.15.67:0/2208328978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c800c0e0 con 0x7f76cc004800 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76e1d9d640 1 -- 172.21.15.67:0/2208328978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc004800 msgr2=0x7f76cc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76e1d9d640 1 --2- 172.21.15.67:0/2208328978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76cc004800 0x7f76cc004c10 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f76c800c3e0 tx=0x7f76cc005830 comp rx=0 tx=0).stop 2024-04-25T15:29:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76e159c640 1 -- 172.21.15.67:0/2208328978 reap_dead start 2024-04-25T15:29:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76e1d9d640 1 -- 172.21.15.67:0/2208328978 shutdown_connections 2024-04-25T15:29:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76e1d9d640 1 -- 172.21.15.67:0/2208328978 >> 172.21.15.67:0/2208328978 conn(0x7f76dc0d2bd0 msgr2=0x7f76dc0d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76e1d9d640 1 -- 172.21.15.67:0/2208328978 shutdown_connections 2024-04-25T15:29:26.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.878+0000 7f76e1d9d640 1 -- 172.21.15.67:0/2208328978 wait complete. 2024-04-25T15:29:26.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.880+0000 7f76e1d9d640 1 Processor -- start 2024-04-25T15:29:26.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.880+0000 7f76e1d9d640 1 -- start start 2024-04-25T15:29:26.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.881+0000 7f76e1d9d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc004800 0x7f76dc1fa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.881+0000 7f76e1d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0cba30 0x7f76dc14b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e1d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0cdb40 0x7f76dc0cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e1d9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc0d4cc0 con 0x7f76dc0cba30 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e1d9d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc0d49e0 con 0x7f76cc004800 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e1d9d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f76dc0d0750 con 0x7f76dc0cdb40 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0cba30 0x7f76dc14b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0cba30 0x7f76dc14b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35946/0 (socket says 172.21.15.67:35946) 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 -- 172.21.15.67:0/3436486255 learned_addr learned my addr 172.21.15.67:0/3436486255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e0d9b640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc004800 0x7f76dc1fa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e159c640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0cdb40 0x7f76dc0cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 -- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0cdb40 msgr2=0x7f76dc0cdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0cdb40 0x7f76dc0cdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 -- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc004800 msgr2=0x7f76dc1fa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc004800 0x7f76dc1fa380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 -- 172.21.15.67:0/3436486255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc14c4f0 con 0x7f76dc0cba30 2024-04-25T15:29:26.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e159c640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0cdb40 0x7f76dc0cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e0d9b640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc004800 0x7f76dc1fa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76dbfff640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0cba30 0x7f76dc14b8e0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f76c8031040 tx=0x7f76dc0d45f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76d97fa640 1 -- 172.21.15.67:0/3436486255 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8032030 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76d97fa640 1 -- 172.21.15.67:0/3436486255 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76c8004cb0 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76d97fa640 1 -- 172.21.15.67:0/3436486255 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8004e60 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.882+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76dc14c7d0 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.883+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76dc1f9b80 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.884+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76dc14c210 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.884+0000 7f76d97fa640 1 -- 172.21.15.67:0/3436486255 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f76c8035020 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.884+0000 7f76d97fa640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76c00744b0 0x7f76c00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.885+0000 7f76d97fa640 1 -- 172.21.15.67:0/3436486255 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f76c80b54a0 con 0x7f76dc0cba30 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.885+0000 7f76e0d9b640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76c00744b0 0x7f76c00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:26.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.885+0000 7f76e0d9b640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76c00744b0 0x7f76c00769c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009160 tx=0x7f76d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:26.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:26.889+0000 7f76d97fa640 1 -- 172.21.15.67:0/3436486255 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76dc0b8510 con 0x7f76dc0cba30 2024-04-25T15:29:27.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.073+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f76dc0d2320 con 0x7f76c00744b0 2024-04-25T15:29:27.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.074+0000 7f76d97fa640 1 -- 172.21.15.67:0/3436486255 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f76dc0d2320 con 0x7f76c00744b0 2024-04-25T15:29:27.075 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T15:29:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76c00744b0 msgr2=0x7f76c00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76c00744b0 0x7f76c00769c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009160 tx=0x7f76d0008040 comp rx=0 tx=0).stop 2024-04-25T15:29:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0cba30 msgr2=0x7f76dc14b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0cba30 0x7f76dc14b8e0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f76c8031040 tx=0x7f76dc0d45f0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 shutdown_connections 2024-04-25T15:29:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f76c00744b0 0x7f76c00769c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76dc0cdb40 0x7f76dc0cdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0cba30 0x7f76dc14b8e0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 --2- 172.21.15.67:0/3436486255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f76cc004800 0x7f76dc1fa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 >> 172.21.15.67:0/3436486255 conn(0x7f76dc0d2bd0 msgr2=0x7f76dc0b91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 shutdown_connections 2024-04-25T15:29:27.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.076+0000 7f76e1d9d640 1 -- 172.21.15.67:0/3436486255 wait complete. 2024-04-25T15:29:27.088 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T15:29:27.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:27.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.205+0000 7f136a364640 1 Processor -- start 2024-04-25T15:29:27.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.205+0000 7f136a364640 1 -- start start 2024-04-25T15:29:27.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f136a364640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f13640b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f136a364640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13640d3170 con 0x7f1364163f30 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f136a364640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13640d3450 con 0x7f13640b0a70 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f136a364640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f13640d2f40 con 0x7f1364164ad0 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f13637fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1364163f30 0x7f1364164340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39062/0 (socket says 172.21.15.67:39062) 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f13637fe640 1 -- 172.21.15.67:0/483883031 learned_addr learned my addr 172.21.15.67:0/483883031 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f1363fff640 1 --2- 172.21.15.67:0/483883031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f13640b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077167101 0 0) 0x7f13640d3170 con 0x7f1364163f30 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f134c003520 con 0x7f1364163f30 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.206+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821207017 0 0) 0x7f13640d3450 con 0x7f13640b0a70 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.207+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13640d3170 con 0x7f13640b0a70 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.207+0000 7f1363fff640 1 -- 172.21.15.67:0/483883031 >> v1:172.21.15.176:6789/0 conn(0x7f13640b0a70 legacy=0x7f13640b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.207+0000 7f1363fff640 1 -- 172.21.15.67:0/483883031 >> v1:172.21.15.67:6789/0 conn(0x7f1364163f30 legacy=0x7f1364164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.207+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3630508770 0 0) 0x7f134c003520 con 0x7f1364163f30 2024-04-25T15:29:27.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.207+0000 7f1363fff640 1 -- 172.21.15.67:0/483883031 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136406d6d0 con 0x7f1364164ad0 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.207+0000 7f1363fff640 1 --2- 172.21.15.67:0/483883031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f13640b2ca0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f13540097d0 tx=0x7f135402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bbb74bb511482e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1354037070 con 0x7f1364164ad0 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f135402ac30 con 0x7f1364164ad0 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f1362ffd640 1 -- 172.21.15.67:0/483883031 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358001670 con 0x7f1364164ad0 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f136a364640 1 -- 172.21.15.67:0/483883031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 msgr2=0x7f13640b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f136a364640 1 --2- 172.21.15.67:0/483883031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f13640b2ca0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f13540097d0 tx=0x7f135402a1a0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f136a364640 1 -- 172.21.15.67:0/483883031 shutdown_connections 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f136a364640 1 --2- 172.21.15.67:0/483883031 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f13640b2ca0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f136a364640 1 -- 172.21.15.67:0/483883031 >> 172.21.15.67:0/483883031 conn(0x7f13640d3aa0 msgr2=0x7f13640d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f136a364640 1 -- 172.21.15.67:0/483883031 shutdown_connections 2024-04-25T15:29:27.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.208+0000 7f136a364640 1 -- 172.21.15.67:0/483883031 wait complete. 2024-04-25T15:29:27.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.210+0000 7f136a364640 1 Processor -- start 2024-04-25T15:29:27.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.210+0000 7f136a364640 1 -- start start 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.211+0000 7f136a364640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13640b0a70 0x7f136414c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.211+0000 7f136a364640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1364163f30 0x7f136414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f136a364640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f1364162650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f136a364640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13640b31e0 con 0x7f1364163f30 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f136a364640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1364164ee0 con 0x7f13640b0a70 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f136a364640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f13640b9430 con 0x7f1364164ad0 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1364163f30 0x7f136414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1364163f30 0x7f136414eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35950/0 (socket says 172.21.15.67:35950) 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 -- 172.21.15.67:0/2120449041 learned_addr learned my addr 172.21.15.67:0/2120449041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f1363fff640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13640b0a70 0x7f136414c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f1368901640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f1364162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 -- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 msgr2=0x7f1364162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f1364162650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 -- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13640b0a70 msgr2=0x7f136414c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13640b0a70 0x7f136414c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 -- 172.21.15.67:0/2120449041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136414cb20 con 0x7f1364163f30 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f1368901640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1364164ad0 0x7f1364162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f13637fe640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1364163f30 0x7f136414eea0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f135800c4f0 tx=0x7f135800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358017070 con 0x7f1364163f30 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1358005ce0 con 0x7f1364163f30 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136414fb40 con 0x7f1364163f30 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.212+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358007290 con 0x7f1364163f30 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.213+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13641f6d30 con 0x7f1364163f30 2024-04-25T15:29:27.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.213+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f136414fe20 con 0x7f1364163f30 2024-04-25T15:29:27.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.214+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f135801a070 con 0x7f1364163f30 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.214+0000 7f1360ff9640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f13500744b0 0x7f13500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f1358094320 con 0x7f1364163f30 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1363fff640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f13500744b0 0x7f13500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1360ff9640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7f135007a3c0 0x7f135007c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 --> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1358010e50 con 0x7f135007a3c0 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f135809b050 con 0x7f1364163f30 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1368901640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7f135007a3c0 0x7f135007c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1368901640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7f135007a3c0 0x7f135007c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.215+0000 7f1363fff640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f13500744b0 0x7f13500769c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1354034000 tx=0x7f135402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.216+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== osd.3 v2:172.21.15.67:6816/1127217835 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1358010e50 con 0x7f135007a3c0 2024-04-25T15:29:27.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.229+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 --> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f13640d4770 con 0x7f135007a3c0 2024-04-25T15:29:27.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.229+0000 7f1360ff9640 1 -- 172.21.15.67:0/2120449041 <== osd.3 v2:172.21.15.67:6816/1127217835 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f13640d4770 con 0x7f135007a3c0 2024-04-25T15:29:27.230 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:27.230 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:27.230 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7f135007a3c0 msgr2=0x7f135007c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:6816/1127217835,v1:172.21.15.67:6818/1127217835] conn(0x7f135007a3c0 0x7f135007c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f13500744b0 msgr2=0x7f13500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f13500744b0 0x7f13500769c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1354034000 tx=0x7f135402e000 comp rx=0 tx=0).stop 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1364163f30 msgr2=0x7f136414eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 --2- 172.21.15.67:0/2120449041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1364163f30 0x7f136414eea0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f135800c4f0 tx=0x7f135800c9b0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f1368901640 1 -- 172.21.15.67:0/2120449041 reap_dead start 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 shutdown_connections 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 >> 172.21.15.67:0/2120449041 conn(0x7f13640d3aa0 msgr2=0x7f1364072560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 shutdown_connections 2024-04-25T15:29:27.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.230+0000 7f136a364640 1 -- 172.21.15.67:0/2120449041 wait complete. 2024-04-25T15:29:27.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T15:29:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.353+0000 7f99d1ebd640 1 Processor -- start 2024-04-25T15:29:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.353+0000 7f99d1ebd640 1 -- start start 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.354+0000 7f99d1ebd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1646d0 0x7f99cc15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.354+0000 7f99d1ebd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc164c80 con 0x7f99cc1616a0 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.354+0000 7f99d1ebd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99cc160bc0 con 0x7f99cc06e740 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.354+0000 7f99d1ebd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f99cc160980 con 0x7f99cc1646d0 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.354+0000 7f99cb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f99cc1616a0 0x7f99cc06e030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39068/0 (socket says 172.21.15.67:39068) 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.354+0000 7f99cb7fe640 1 -- 172.21.15.67:0/1950310751 learned_addr learned my addr 172.21.15.67:0/1950310751 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99cbfff640 1 --2- 172.21.15.67:0/1950310751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1646d0 0x7f99cc15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1376462622 0 0) 0x7f99cc164c80 con 0x7f99cc1616a0 2024-04-25T15:29:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b4003520 con 0x7f99cc1616a0 2024-04-25T15:29:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099382377 0 0) 0x7f99cc160bc0 con 0x7f99cc06e740 2024-04-25T15:29:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99cc164c80 con 0x7f99cc06e740 2024-04-25T15:29:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1806382771 0 0) 0x7f99b4003520 con 0x7f99cc1616a0 2024-04-25T15:29:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99cbfff640 1 -- 172.21.15.67:0/1950310751 >> v1:172.21.15.176:6789/0 conn(0x7f99cc06e740 legacy=0x7f99cc163fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99cbfff640 1 -- 172.21.15.67:0/1950310751 >> v1:172.21.15.67:6789/0 conn(0x7f99cc1616a0 legacy=0x7f99cc06e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.355+0000 7f99cbfff640 1 -- 172.21.15.67:0/1950310751 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc160ea0 con 0x7f99cc1646d0 2024-04-25T15:29:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.356+0000 7f99cbfff640 1 --2- 172.21.15.67:0/1950310751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1646d0 0x7f99cc15ded0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009dc0 tx=0x7f99c402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae19f14d9a2778b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.356+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc001670 con 0x7f99cc1646d0 2024-04-25T15:29:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.356+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99c4007540 con 0x7f99cc1646d0 2024-04-25T15:29:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.356+0000 7f99ca7fc640 1 -- 172.21.15.67:0/1950310751 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c4007910 con 0x7f99cc1646d0 2024-04-25T15:29:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.357+0000 7f99d1ebd640 1 -- 172.21.15.67:0/1950310751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1646d0 msgr2=0x7f99cc15ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.357+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/1950310751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1646d0 0x7f99cc15ded0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009dc0 tx=0x7f99c402a660 comp rx=0 tx=0).stop 2024-04-25T15:29:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.357+0000 7f99d1ebd640 1 -- 172.21.15.67:0/1950310751 shutdown_connections 2024-04-25T15:29:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.357+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/1950310751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1646d0 0x7f99cc15ded0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.357+0000 7f99d1ebd640 1 -- 172.21.15.67:0/1950310751 >> 172.21.15.67:0/1950310751 conn(0x7f99cc0d27c0 msgr2=0x7f99cc0d2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.357+0000 7f99d1ebd640 1 -- 172.21.15.67:0/1950310751 shutdown_connections 2024-04-25T15:29:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.357+0000 7f99d1ebd640 1 -- 172.21.15.67:0/1950310751 wait complete. 2024-04-25T15:29:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.358+0000 7f99d1ebd640 1 Processor -- start 2024-04-25T15:29:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.359+0000 7f99d1ebd640 1 -- start start 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99d1ebd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1616a0 0x7f99cc159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99d1ebd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99cc1646d0 0x7f99cc154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99d1ebd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99cc154970 0x7f99cc0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99d1ebd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc16b280 con 0x7f99cc1646d0 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99d1ebd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc16b100 con 0x7f99cc154970 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99d1ebd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f99cc16b400 con 0x7f99cc1616a0 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99cc1646d0 0x7f99cc154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99cc1646d0 0x7f99cc154210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35956/0 (socket says 172.21.15.67:35956) 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 -- 172.21.15.67:0/3937924527 learned_addr learned my addr 172.21.15.67:0/3937924527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99cb7fe640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1616a0 0x7f99cc159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99cbfff640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99cc154970 0x7f99cc0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 -- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1616a0 msgr2=0x7f99cc159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1616a0 0x7f99cc159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 -- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99cc154970 msgr2=0x7f99cc0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99cc154970 0x7f99cc0ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.360+0000 7f99caffd640 1 -- 172.21.15.67:0/3937924527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc154750 con 0x7f99cc1646d0 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.361+0000 7f99caffd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99cc1646d0 0x7f99cc154210 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f99c000c3c0 tx=0x7f99c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.361+0000 7f99b3fff640 1 -- 172.21.15.67:0/3937924527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0017070 con 0x7f99cc1646d0 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.361+0000 7f99b3fff640 1 -- 172.21.15.67:0/3937924527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99c00052f0 con 0x7f99cc1646d0 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.361+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc155150 con 0x7f99cc1646d0 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.361+0000 7f99b3fff640 1 -- 172.21.15.67:0/3937924527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0003830 con 0x7f99cc1646d0 2024-04-25T15:29:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.361+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99cc155710 con 0x7f99cc1646d0 2024-04-25T15:29:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.362+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99cc154ea0 con 0x7f99cc1646d0 2024-04-25T15:29:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.363+0000 7f99b3fff640 1 -- 172.21.15.67:0/3937924527 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f99c001a070 con 0x7f99cc1646d0 2024-04-25T15:29:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.363+0000 7f99b3fff640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f99b80744b0 0x7f99b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.363+0000 7f99b3fff640 1 -- 172.21.15.67:0/3937924527 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f99c0094240 con 0x7f99cc1646d0 2024-04-25T15:29:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.363+0000 7f99cb7fe640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f99b80744b0 0x7f99b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.364+0000 7f99cb7fe640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f99b80744b0 0x7f99b80769c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0014a0 tx=0x7f99bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.368+0000 7f99b3fff640 1 -- 172.21.15.67:0/3937924527 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99c005ecd0 con 0x7f99cc1646d0 2024-04-25T15:29:27.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.552+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f99cc050740 con 0x7f99b80744b0 2024-04-25T15:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.552+0000 7f99b3fff640 1 -- 172.21.15.67:0/3937924527 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f99cc050740 con 0x7f99b80744b0 2024-04-25T15:29:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T15:29:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f99b80744b0 msgr2=0x7f99b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f99b80744b0 0x7f99b80769c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f99bc0014a0 tx=0x7f99bc009040 comp rx=0 tx=0).stop 2024-04-25T15:29:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99cc1646d0 msgr2=0x7f99cc154210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99cc1646d0 0x7f99cc154210 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f99c000c3c0 tx=0x7f99c000c880 comp rx=0 tx=0).stop 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 shutdown_connections 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f99b80744b0 0x7f99b80769c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99cc154970 0x7f99cc0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99cc1646d0 0x7f99cc154210 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 --2- 172.21.15.67:0/3937924527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99cc1616a0 0x7f99cc159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 >> 172.21.15.67:0/3937924527 conn(0x7f99cc0d27c0 msgr2=0x7f99cc15eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 shutdown_connections 2024-04-25T15:29:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.554+0000 7f99d1ebd640 1 -- 172.21.15.67:0/3937924527 wait complete. 2024-04-25T15:29:27.566 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T15:29:27.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:27.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.682+0000 7fef6099d640 1 Processor -- start 2024-04-25T15:29:27.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.682+0000 7fef6099d640 1 -- start start 2024-04-25T15:29:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.683+0000 7fef6099d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef5c0d0ed0 0x7fef5c0d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.683+0000 7fef6099d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef5c0b29c0 con 0x7fef5c0d2eb0 2024-04-25T15:29:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.683+0000 7fef6099d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef5c0b2140 con 0x7fef5c06eb80 2024-04-25T15:29:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.683+0000 7fef6099d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fef5c0b1f90 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.683+0000 7fef5bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef5c0d0ed0 0x7fef5c0d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.683+0000 7fef5affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef5c0d2eb0 0x7fef5c0d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35314/0 (socket says 172.21.15.67:35314) 2024-04-25T15:29:27.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.683+0000 7fef5affd640 1 -- 172.21.15.67:0/2351338287 learned_addr learned my addr 172.21.15.67:0/2351338287 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:27.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.684+0000 7fef5bfff640 1 -- 172.21.15.67:0/2351338287 >> v1:172.21.15.176:6789/0 conn(0x7fef5c06eb80 legacy=0x7fef5c0d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:27.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.684+0000 7fef5bfff640 1 -- 172.21.15.67:0/2351338287 >> v1:172.21.15.67:6789/0 conn(0x7fef5c0d2eb0 legacy=0x7fef5c0d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.684+0000 7fef5bfff640 1 -- 172.21.15.67:0/2351338287 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c161840 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.684+0000 7fef5a7fc640 1 -- 172.21.15.67:0/2351338287 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344332869 0 0) 0x7fef50001110 con 0x7fef5c0d2eb0 2024-04-25T15:29:27.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.684+0000 7fef5bfff640 1 --2- 172.21.15.67:0/2351338287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c0d12e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fef5400bfa0 tx=0x7fef5402c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363551d6f44b680d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.684+0000 7fef5a7fc640 1 -- 172.21.15.67:0/2351338287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54009540 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.684+0000 7fef5a7fc640 1 -- 172.21.15.67:0/2351338287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef540096e0 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef5a7fc640 1 -- 172.21.15.67:0/2351338287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54009ab0 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef6099d640 1 -- 172.21.15.67:0/2351338287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 msgr2=0x7fef5c0d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef6099d640 1 --2- 172.21.15.67:0/2351338287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c0d12e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fef5400bfa0 tx=0x7fef5402c830 comp rx=0 tx=0).stop 2024-04-25T15:29:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef6099d640 1 -- 172.21.15.67:0/2351338287 shutdown_connections 2024-04-25T15:29:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef6099d640 1 --2- 172.21.15.67:0/2351338287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c0d12e0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef6099d640 1 -- 172.21.15.67:0/2351338287 >> 172.21.15.67:0/2351338287 conn(0x7fef5c09bae0 msgr2=0x7fef5c0b7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef6099d640 1 -- 172.21.15.67:0/2351338287 shutdown_connections 2024-04-25T15:29:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.685+0000 7fef6099d640 1 -- 172.21.15.67:0/2351338287 wait complete. 2024-04-25T15:29:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.687+0000 7fef6099d640 1 Processor -- start 2024-04-25T15:29:27.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.687+0000 7fef6099d640 1 -- start start 2024-04-25T15:29:27.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.688+0000 7fef6099d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c06eb80 0x7fef5c1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.688+0000 7fef6099d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c15b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.688+0000 7fef6099d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c98c0 0x7fef5c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.688+0000 7fef6099d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c06ef90 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.688+0000 7fef6099d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c0b26e0 con 0x7fef5c06eb80 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.688+0000 7fef6099d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fef5c161460 con 0x7fef5c0c98c0 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c15b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35462/0 (socket says 172.21.15.67:35462) 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 -- 172.21.15.67:0/772402623 learned_addr learned my addr 172.21.15.67:0/772402623 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c06eb80 0x7fef5c1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5bfff640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c98c0 0x7fef5c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 -- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c98c0 msgr2=0x7fef5c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c98c0 0x7fef5c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 -- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c06eb80 msgr2=0x7fef5c1585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c06eb80 0x7fef5c1585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 -- 172.21.15.67:0/772402623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c15a310 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5bfff640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c98c0 0x7fef5c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef5affd640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c15b310 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fef5000c410 tx=0x7fef5000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50017070 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef50005ce0 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c15a5f0 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef500072f0 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.689+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c0ccf40 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.690+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fef5c076c60 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.691+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fef5001a070 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.691+0000 7fef43fff640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef480744b0 0x7fef480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.691+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7fef50094180 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.692+0000 7fef43fff640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7fef4807a3c0 0x7fef4807c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.692+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 --> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef4807cee0 con 0x7fef4807a3c0 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.692+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fef5009b050 con 0x7fef5c0d0ed0 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.692+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef480744b0 0x7fef480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.692+0000 7fef5bfff640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7fef4807a3c0 0x7fef4807c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.692+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef480744b0 0x7fef480769c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0098c0 tx=0x7fef4c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.692+0000 7fef5bfff640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7fef4807a3c0 0x7fef4807c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.693+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== osd.4 v2:172.21.15.176:6816/614570349 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fef4807cee0 con 0x7fef4807a3c0 2024-04-25T15:29:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.706+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 --> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fef5c15ba60 con 0x7fef4807a3c0 2024-04-25T15:29:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef43fff640 1 -- 172.21.15.67:0/772402623 <== osd.4 v2:172.21.15.176:6816/614570349 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fef5c15ba60 con 0x7fef4807a3c0 2024-04-25T15:29:27.708 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:27.708 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:27.708 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7fef4807a3c0 msgr2=0x7fef4807c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6816/614570349,v1:172.21.15.176:6817/614570349] conn(0x7fef4807a3c0 0x7fef4807c810 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef480744b0 msgr2=0x7fef480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fef480744b0 0x7fef480769c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0098c0 tx=0x7fef4c009060 comp rx=0 tx=0).stop 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 msgr2=0x7fef5c15b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 --2- 172.21.15.67:0/772402623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c0d0ed0 0x7fef5c15b310 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fef5000c410 tx=0x7fef5000c8d0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef5bfff640 1 -- 172.21.15.67:0/772402623 reap_dead start 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 shutdown_connections 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 >> 172.21.15.67:0/772402623 conn(0x7fef5c09bae0 msgr2=0x7fef5c0d28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 shutdown_connections 2024-04-25T15:29:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.707+0000 7fef6099d640 1 -- 172.21.15.67:0/772402623 wait complete. 2024-04-25T15:29:27.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T15:29:27.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.830+0000 7f9d710a7640 1 Processor -- start 2024-04-25T15:29:27.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.830+0000 7f9d710a7640 1 -- start start 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d710a7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 0x7f9d6c0d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d710a7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0d12c0 con 0x7f9d6c0d2d70 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d710a7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0b3c10 con 0x7f9d6c06dcf0 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d710a7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6c0b9520 con 0x7f9d6c06d8e0 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d6b59e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9d6c0d2d70 0x7f9d6c0d0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35328/0 (socket says 172.21.15.67:35328) 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d6b59e640 1 -- 172.21.15.67:0/2284530149 learned_addr learned my addr 172.21.15.67:0/2284530149 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d6ad9d640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 0x7f9d6c0d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 209423357 0 0) 0x7f9d6c0d12c0 con 0x7f9d6c0d2d70 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d58003520 con 0x7f9d6c0d2d70 2024-04-25T15:29:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2447877382 0 0) 0x7f9d58003520 con 0x7f9d6c0d2d70 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 msgr2=0x7f9d6c0d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 0x7f9d6c0d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 >> v1:172.21.15.176:6789/0 conn(0x7f9d6c06dcf0 legacy=0x7f9d6c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c0b3ef0 con 0x7f9d6c0d2d70 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f9d5c001670 con 0x7f9d6c0d2d70 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d6ad9d640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 0x7f9d6c0d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.832+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 >> v1:172.21.15.67:6789/0 conn(0x7f9d6c0d2d70 legacy=0x7f9d6c0d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d69d9b640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 0x7f9d580039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d69d9b640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d58003f40 0x7f9d58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d69d9b640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58004d40 0x7f9d58005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0b9520 con 0x7f9d58003f40 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5800bb80 con 0x7f9d58004d40 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5800bd80 con 0x7f9d6c06d8e0 2024-04-25T15:29:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6a59c640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d58003f40 0x7f9d58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6ad9d640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 0x7f9d580039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6b59e640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58004d40 0x7f9d58005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6a59c640 1 -- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 msgr2=0x7f9d580039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6a59c640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d6c06d8e0 0x7f9d580039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6a59c640 1 -- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58004d40 msgr2=0x7f9d58005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6a59c640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58004d40 0x7f9d58005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.833+0000 7f9d6a59c640 1 -- 172.21.15.67:0/2284530149 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d58003520 con 0x7f9d58003f40 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.834+0000 7f9d6a59c640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d58003f40 0x7f9d58004800 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c06f790 tx=0x7f9d58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6fec7e485f77ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.834+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54037070 con 0x7f9d58003f40 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.834+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d5402abe0 con 0x7f9d58003f40 2024-04-25T15:29:27.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.834+0000 7f9d69d9b640 1 -- 172.21.15.67:0/2284530149 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d540014e0 con 0x7f9d58003f40 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.835+0000 7f9d710a7640 1 -- 172.21.15.67:0/2284530149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d58003f40 msgr2=0x7f9d58004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.835+0000 7f9d710a7640 1 --2- 172.21.15.67:0/2284530149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d58003f40 0x7f9d58004800 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c06f790 tx=0x7f9d58005830 comp rx=0 tx=0).stop 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.835+0000 7f9d6b59e640 1 -- 172.21.15.67:0/2284530149 reap_dead start 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.835+0000 7f9d710a7640 1 -- 172.21.15.67:0/2284530149 shutdown_connections 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.835+0000 7f9d710a7640 1 -- 172.21.15.67:0/2284530149 >> 172.21.15.67:0/2284530149 conn(0x7f9d6c08d110 msgr2=0x7f9d6c08d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.835+0000 7f9d710a7640 1 -- 172.21.15.67:0/2284530149 shutdown_connections 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.835+0000 7f9d710a7640 1 -- 172.21.15.67:0/2284530149 wait complete. 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.837+0000 7f9d710a7640 1 Processor -- start 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.837+0000 7f9d710a7640 1 -- start start 2024-04-25T15:29:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d710a7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d58003f40 0x7f9d6c15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d710a7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d6c15cdb0 0x7f9d6c14eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d710a7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d6c15f1f0 0x7f9d6c15f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d710a7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0b44b0 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d710a7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0b4820 con 0x7f9d6c15f1f0 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d710a7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6c0b38e0 con 0x7f9d58003f40 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d6c15cdb0 0x7f9d6c14eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d6c15cdb0 0x7f9d6c14eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35476/0 (socket says 172.21.15.67:35476) 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6a59c640 1 -- 172.21.15.67:0/2169334317 learned_addr learned my addr 172.21.15.67:0/2169334317 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6ad9d640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d58003f40 0x7f9d6c15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6b59e640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d6c15f1f0 0x7f9d6c15f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6a59c640 1 -- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d58003f40 msgr2=0x7f9d6c15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6a59c640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d58003f40 0x7f9d6c15a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6a59c640 1 -- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d6c15f1f0 msgr2=0x7f9d6c15f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.838+0000 7f9d6a59c640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d6c15f1f0 0x7f9d6c15f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.839+0000 7f9d6a59c640 1 -- 172.21.15.67:0/2169334317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c15fb40 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.839+0000 7f9d6a59c640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d6c15cdb0 0x7f9d6c14eff0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f9d5402a6d0 tx=0x7f9d540012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.839+0000 7f9d637fe640 1 -- 172.21.15.67:0/2169334317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54037040 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.839+0000 7f9d637fe640 1 -- 172.21.15.67:0/2169334317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d540052e0 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.839+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c15d2b0 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.839+0000 7f9d637fe640 1 -- 172.21.15.67:0/2169334317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54007240 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.839+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6c15b290 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.840+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c0b5040 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.841+0000 7f9d637fe640 1 -- 172.21.15.67:0/2169334317 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9d5403a070 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.841+0000 7f9d637fe640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9d4c0744e0 0x7f9d4c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.841+0000 7f9d637fe640 1 -- 172.21.15.67:0/2169334317 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f9d540b4eb0 con 0x7f9d6c15cdb0 2024-04-25T15:29:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.842+0000 7f9d6ad9d640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9d4c0744e0 0x7f9d4c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.842+0000 7f9d6ad9d640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9d4c0744e0 0x7f9d4c0769f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0744c0 tx=0x7f9d6400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:27.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:27.846+0000 7f9d637fe640 1 -- 172.21.15.67:0/2169334317 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d6c0b5040 con 0x7f9d6c15cdb0 2024-04-25T15:29:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.031+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f9d6c050740 con 0x7f9d4c0744e0 2024-04-25T15:29:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.031+0000 7f9d637fe640 1 -- 172.21.15.67:0/2169334317 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9d6c050740 con 0x7f9d4c0744e0 2024-04-25T15:29:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.033+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9d4c0744e0 msgr2=0x7f9d4c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.033+0000 7f9d710a7640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9d4c0744e0 0x7f9d4c0769f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0744c0 tx=0x7f9d6400b040 comp rx=0 tx=0).stop 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.033+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d6c15cdb0 msgr2=0x7f9d6c14eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.033+0000 7f9d710a7640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d6c15cdb0 0x7f9d6c14eff0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f9d5402a6d0 tx=0x7f9d540012b0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.033+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 shutdown_connections 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.034+0000 7f9d710a7640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f9d4c0744e0 0x7f9d4c0769f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.034+0000 7f9d710a7640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d6c15f1f0 0x7f9d6c15f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.034+0000 7f9d710a7640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d6c15cdb0 0x7f9d6c14eff0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.034+0000 7f9d710a7640 1 --2- 172.21.15.67:0/2169334317 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d58003f40 0x7f9d6c15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.034+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 >> 172.21.15.67:0/2169334317 conn(0x7f9d6c08d110 msgr2=0x7f9d6c06e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.034+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 shutdown_connections 2024-04-25T15:29:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.034+0000 7f9d710a7640 1 -- 172.21.15.67:0/2169334317 wait complete. 2024-04-25T15:29:28.045 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T15:29:28.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.160+0000 7fb358cee640 1 Processor -- start 2024-04-25T15:29:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.160+0000 7fb358cee640 1 -- start start 2024-04-25T15:29:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb358cee640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb354164ad0 0x7fb3540b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb358cee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 0x7fb3540b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb358cee640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3540d3170 con 0x7fb354163f30 2024-04-25T15:29:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb358cee640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb3540d2f40 con 0x7fb354164ad0 2024-04-25T15:29:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb358cee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb354164ee0 con 0x7fb3540b0a70 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb35259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb354164ad0 0x7fb3540b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb35259c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb354164ad0 0x7fb3540b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35490/0 (socket says 172.21.15.67:35490) 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb35259c640 1 -- 172.21.15.67:0/2928725888 learned_addr learned my addr 172.21.15.67:0/2928725888 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb352d9d640 1 --2- 172.21.15.67:0/2928725888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 0x7fb3540b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb35259c640 1 -- 172.21.15.67:0/2928725888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 msgr2=0x7fb3540b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb35259c640 1 --2- 172.21.15.67:0/2928725888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 0x7fb3540b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb35259c640 1 -- 172.21.15.67:0/2928725888 >> v1:172.21.15.67:6789/0 conn(0x7fb354163f30 legacy=0x7fb354164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.162+0000 7fb35259c640 1 -- 172.21.15.67:0/2928725888 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3540d3450 con 0x7fb354164ad0 2024-04-25T15:29:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.163+0000 7fb35259c640 1 --2- 172.21.15.67:0/2928725888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354164ad0 0x7fb3540b2ca0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb344009870 tx=0x7fb34402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c37c84b939fdd1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.163+0000 7fb35159a640 1 -- 172.21.15.67:0/2928725888 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344037070 con 0x7fb354164ad0 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.163+0000 7fb35159a640 1 -- 172.21.15.67:0/2928725888 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb344032440 con 0x7fb354164ad0 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.163+0000 7fb35159a640 1 -- 172.21.15.67:0/2928725888 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb344032810 con 0x7fb354164ad0 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 -- 172.21.15.67:0/2928725888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354164ad0 msgr2=0x7fb3540b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 --2- 172.21.15.67:0/2928725888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354164ad0 0x7fb3540b2ca0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb344009870 tx=0x7fb34402a270 comp rx=0 tx=0).stop 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 -- 172.21.15.67:0/2928725888 shutdown_connections 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 --2- 172.21.15.67:0/2928725888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 0x7fb3540b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 --2- 172.21.15.67:0/2928725888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354164ad0 0x7fb3540b2ca0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 -- 172.21.15.67:0/2928725888 >> 172.21.15.67:0/2928725888 conn(0x7fb3540d3aa0 msgr2=0x7fb3540d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 -- 172.21.15.67:0/2928725888 shutdown_connections 2024-04-25T15:29:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.164+0000 7fb358cee640 1 -- 172.21.15.67:0/2928725888 wait complete. 2024-04-25T15:29:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.166+0000 7fb358cee640 1 Processor -- start 2024-04-25T15:29:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.166+0000 7fb358cee640 1 -- start start 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.167+0000 7fb358cee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 0x7fb354163510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.167+0000 7fb358cee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354163f30 0x7fb3541f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.167+0000 7fb358cee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb354164ad0 0x7fb3541f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.167+0000 7fb358cee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb35406d9b0 con 0x7fb354163f30 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.167+0000 7fb358cee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb3540b31e0 con 0x7fb354164ad0 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.167+0000 7fb358cee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb3540b9430 con 0x7fb3540b0a70 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354163f30 0x7fb3541f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354163f30 0x7fb3541f6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35500/0 (socket says 172.21.15.67:35500) 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 -- 172.21.15.67:0/3628265686 learned_addr learned my addr 172.21.15.67:0/3628265686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:28.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb35259c640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 0x7fb354163510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb352d9d640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb354164ad0 0x7fb3541f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 -- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 msgr2=0x7fb354163510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3540b0a70 0x7fb354163510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 -- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb354164ad0 msgr2=0x7fb3541f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb354164ad0 0x7fb3541f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 -- 172.21.15.67:0/3628265686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3541fa000 con 0x7fb354163f30 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb351d9b640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354163f30 0x7fb3541f6df0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c420 tx=0x7fb34800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3480052d0 con 0x7fb354163f30 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb348005490 con 0x7fb354163f30 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3541fa2e0 con 0x7fb354163f30 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348007260 con 0x7fb354163f30 2024-04-25T15:29:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.168+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3541fa8a0 con 0x7fb354163f30 2024-04-25T15:29:28.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.169+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3540d4770 con 0x7fb354163f30 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.170+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb348007400 con 0x7fb354163f30 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.170+0000 7fb342ffd640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3340744e0 0x7fb3340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7fb348094d80 con 0x7fb354163f30 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb342ffd640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7fb33407a3f0 0x7fb33407c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb35259c640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3340744e0 0x7fb3340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 --> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb34800ee50 con 0x7fb33407a3f0 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fb348095210 con 0x7fb354163f30 2024-04-25T15:29:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb352d9d640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7fb33407a3f0 0x7fb33407c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb35259c640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3340744e0 0x7fb3340769f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb344034000 tx=0x7fb34402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.171+0000 7fb352d9d640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7fb33407a3f0 0x7fb33407c840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.172+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== osd.5 v2:172.21.15.176:6824/960414228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb34800ee50 con 0x7fb33407a3f0 2024-04-25T15:29:28.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 --> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb3541f7960 con 0x7fb33407a3f0 2024-04-25T15:29:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb342ffd640 1 -- 172.21.15.67:0/3628265686 <== osd.5 v2:172.21.15.176:6824/960414228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb3541f7960 con 0x7fb33407a3f0 2024-04-25T15:29:28.187 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:28.187 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:28.187 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7fb33407a3f0 msgr2=0x7fb33407c840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb358cee640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6824/960414228,v1:172.21.15.176:6825/960414228] conn(0x7fb33407a3f0 0x7fb33407c840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3340744e0 msgr2=0x7fb3340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb358cee640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fb3340744e0 0x7fb3340769f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb344034000 tx=0x7fb34402e000 comp rx=0 tx=0).stop 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354163f30 msgr2=0x7fb3541f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.186+0000 7fb358cee640 1 --2- 172.21.15.67:0/3628265686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354163f30 0x7fb3541f6df0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb34800c420 tx=0x7fb34800c8e0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.187+0000 7fb352d9d640 1 -- 172.21.15.67:0/3628265686 reap_dead start 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.187+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 shutdown_connections 2024-04-25T15:29:28.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.187+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 >> 172.21.15.67:0/3628265686 conn(0x7fb3540d3aa0 msgr2=0x7fb3540723d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.187+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 shutdown_connections 2024-04-25T15:29:28.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.187+0000 7fb358cee640 1 -- 172.21.15.67:0/3628265686 wait complete. 2024-04-25T15:29:28.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T15:29:28.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.310+0000 7f258579d640 1 Processor -- start 2024-04-25T15:29:28.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.310+0000 7f258579d640 1 -- start start 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.311+0000 7f258579d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f25800d1000 0x7f25800d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.311+0000 7f258579d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800d3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.311+0000 7f258579d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25800b0740 con 0x7f25800d3cd0 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.311+0000 7f258579d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f25800d2b30 con 0x7f25800d1000 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.311+0000 7f258579d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f25800b0a20 con 0x7f25800d25b0 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f25800d3cd0 0x7f25800b38e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35360/0 (socket says 172.21.15.67:35360) 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257f7fe640 1 -- 172.21.15.67:0/4147512389 learned_addr learned my addr 172.21.15.67:0/4147512389 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257e7fc640 1 --2- 172.21.15.67:0/4147512389 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800d3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257effd640 1 --2- 172.21.15.67:0/4147512389 >> v2:172.21.15.176:3300/0 conn(0x7f25800d1000 0x7f25800d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257dffb640 1 -- 172.21.15.67:0/4147512389 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823968000 0 0) 0x7f25800b0740 con 0x7f25800d3cd0 2024-04-25T15:29:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257dffb640 1 -- 172.21.15.67:0/4147512389 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2568003520 con 0x7f25800d3cd0 2024-04-25T15:29:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257e7fc640 1 -- 172.21.15.67:0/4147512389 >> v2:172.21.15.176:3300/0 conn(0x7f25800d1000 msgr2=0x7f25800d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257e7fc640 1 --2- 172.21.15.67:0/4147512389 >> v2:172.21.15.176:3300/0 conn(0x7f25800d1000 0x7f25800d2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257e7fc640 1 -- 172.21.15.67:0/4147512389 >> v1:172.21.15.67:6789/0 conn(0x7f25800d3cd0 legacy=0x7f25800b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257e7fc640 1 -- 172.21.15.67:0/4147512389 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25800b0d00 con 0x7f25800d25b0 2024-04-25T15:29:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.312+0000 7f257effd640 1 --2- 172.21.15.67:0/4147512389 >> v2:172.21.15.176:3300/0 conn(0x7f25800d1000 0x7f25800d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:28.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.313+0000 7f257e7fc640 1 --2- 172.21.15.67:0/4147512389 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800d3790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f25700097a0 tx=0x7f257002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3b0ddddf2ac8846 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.313+0000 7f257dffb640 1 -- 172.21.15.67:0/4147512389 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2570037070 con 0x7f25800d25b0 2024-04-25T15:29:28.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.313+0000 7f257dffb640 1 -- 172.21.15.67:0/4147512389 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2570032440 con 0x7f25800d25b0 2024-04-25T15:29:28.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.313+0000 7f257dffb640 1 -- 172.21.15.67:0/4147512389 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25700327f0 con 0x7f25800d25b0 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 -- 172.21.15.67:0/4147512389 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 msgr2=0x7f25800d3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 --2- 172.21.15.67:0/4147512389 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800d3790 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f25700097a0 tx=0x7f257002a180 comp rx=0 tx=0).stop 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 -- 172.21.15.67:0/4147512389 shutdown_connections 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 --2- 172.21.15.67:0/4147512389 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800d3790 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 --2- 172.21.15.67:0/4147512389 >> v2:172.21.15.176:3300/0 conn(0x7f25800d1000 0x7f25800d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 -- 172.21.15.67:0/4147512389 >> 172.21.15.67:0/4147512389 conn(0x7f258008d110 msgr2=0x7f258008d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 -- 172.21.15.67:0/4147512389 shutdown_connections 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.314+0000 7f258579d640 1 -- 172.21.15.67:0/4147512389 wait complete. 2024-04-25T15:29:28.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.316+0000 7f258579d640 1 Processor -- start 2024-04-25T15:29:28.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.316+0000 7f258579d640 1 -- start start 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f258579d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25800d1000 0x7f258014f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f258579d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f258579d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25800d3cd0 0x7f258014f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f258579d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25800b2ad0 con 0x7f25800d1000 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f258579d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f25800d40e0 con 0x7f25800d3cd0 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f258579d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f25800b2510 con 0x7f25800d25b0 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25800d1000 0x7f258014f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25800d1000 0x7f258014f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35502/0 (socket says 172.21.15.67:35502) 2024-04-25T15:29:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257effd640 1 -- 172.21.15.67:0/4243031903 learned_addr learned my addr 172.21.15.67:0/4243031903 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257f7fe640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25800d3cd0 0x7f258014f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257e7fc640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257effd640 1 -- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 msgr2=0x7f25800c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257effd640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257effd640 1 -- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25800d3cd0 msgr2=0x7f258014f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.317+0000 7f257effd640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25800d3cd0 0x7f258014f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f257effd640 1 -- 172.21.15.67:0/4243031903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258014f610 con 0x7f25800d1000 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f257e7fc640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f257effd640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25800d1000 0x7f258014f0d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f257400e990 tx=0x7f257400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f25677fe640 1 -- 172.21.15.67:0/4243031903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2574017070 con 0x7f25800d1000 2024-04-25T15:29:28.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f25677fe640 1 -- 172.21.15.67:0/4243031903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257400c9e0 con 0x7f25800d1000 2024-04-25T15:29:28.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25800c7b00 con 0x7f25800d1000 2024-04-25T15:29:28.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f25677fe640 1 -- 172.21.15.67:0/4243031903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257400cdb0 con 0x7f25800d1000 2024-04-25T15:29:28.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.318+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25800cc420 con 0x7f25800d1000 2024-04-25T15:29:28.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.319+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2580050740 con 0x7f25800d1000 2024-04-25T15:29:28.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.320+0000 7f25677fe640 1 -- 172.21.15.67:0/4243031903 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f257401a070 con 0x7f25800d1000 2024-04-25T15:29:28.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.320+0000 7f25677fe640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f256c0744e0 0x7f256c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.321+0000 7f25677fe640 1 -- 172.21.15.67:0/4243031903 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f2574094de0 con 0x7f25800d1000 2024-04-25T15:29:28.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.321+0000 7f257e7fc640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f256c0744e0 0x7f256c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.321+0000 7f257e7fc640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f256c0744e0 0x7f256c0769f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2570034000 tx=0x7f257002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.325+0000 7f25677fe640 1 -- 172.21.15.67:0/4243031903 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f257405f1a0 con 0x7f25800d1000 2024-04-25T15:29:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.510+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f2580076240 con 0x7f256c0744e0 2024-04-25T15:29:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.511+0000 7f25677fe640 1 -- 172.21.15.67:0/4243031903 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2580076240 con 0x7f256c0744e0 2024-04-25T15:29:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T15:29:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f256c0744e0 msgr2=0x7f256c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f256c0744e0 0x7f256c0769f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2570034000 tx=0x7f257002e000 comp rx=0 tx=0).stop 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25800d1000 msgr2=0x7f258014f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25800d1000 0x7f258014f0d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f257400e990 tx=0x7f257400ee50 comp rx=0 tx=0).stop 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 shutdown_connections 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f256c0744e0 0x7f256c0769f0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25800d3cd0 0x7f258014f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25800d25b0 0x7f25800c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 --2- 172.21.15.67:0/4243031903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25800d1000 0x7f258014f0d0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 >> 172.21.15.67:0/4243031903 conn(0x7f258008d110 msgr2=0x7f2580071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 shutdown_connections 2024-04-25T15:29:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.513+0000 7f258579d640 1 -- 172.21.15.67:0/4243031903 wait complete. 2024-04-25T15:29:28.526 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T15:29:28.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:28.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.647+0000 7f4aa3764640 1 Processor -- start 2024-04-25T15:29:28.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.647+0000 7f4aa3764640 1 -- start start 2024-04-25T15:29:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa3764640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a9c06ccd0 0x7f4a9c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa3764640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a9c1609b0 con 0x7f4a9c161710 2024-04-25T15:29:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa3764640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a9c164cd0 con 0x7f4a9c163fa0 2024-04-25T15:29:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa3764640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4a9c160c30 con 0x7f4a9c06ccd0 2024-04-25T15:29:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa0cff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a9c06ccd0 0x7f4a9c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa1500640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4a9c161710 0x7f4a9c06c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35370/0 (socket says 172.21.15.67:35370) 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa1500640 1 -- 172.21.15.67:0/1875682793 learned_addr learned my addr 172.21.15.67:0/1875682793 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.648+0000 7f4aa0cff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a9c06ccd0 0x7f4a9c15e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35506/0 (socket says 172.21.15.67:35506) 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.649+0000 7f4aa0cff640 1 -- 172.21.15.67:0/1875682793 >> v1:172.21.15.176:6789/0 conn(0x7f4a9c163fa0 legacy=0x7f4a9c1643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.649+0000 7f4aa0cff640 1 -- 172.21.15.67:0/1875682793 >> v1:172.21.15.67:6789/0 conn(0x7f4a9c161710 legacy=0x7f4a9c06c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.649+0000 7f4aa0cff640 1 -- 172.21.15.67:0/1875682793 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c160f10 con 0x7f4a9c06ccd0 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.649+0000 7f4aa0cff640 1 --2- 172.21.15.67:0/1875682793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c06ccd0 0x7f4a9c15e080 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0097a0 tx=0x7f4a8c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f85f402a898c2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4a93fff640 1 -- 172.21.15.67:0/1875682793 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c037070 con 0x7f4a9c06ccd0 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4a93fff640 1 -- 172.21.15.67:0/1875682793 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8c032440 con 0x7f4a9c06ccd0 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4a93fff640 1 -- 172.21.15.67:0/1875682793 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c032810 con 0x7f4a9c06ccd0 2024-04-25T15:29:28.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4aa3764640 1 -- 172.21.15.67:0/1875682793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c06ccd0 msgr2=0x7f4a9c15e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4aa3764640 1 --2- 172.21.15.67:0/1875682793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c06ccd0 0x7f4a9c15e080 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0097a0 tx=0x7f4a8c02a1a0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4aa3764640 1 -- 172.21.15.67:0/1875682793 shutdown_connections 2024-04-25T15:29:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4aa3764640 1 --2- 172.21.15.67:0/1875682793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c06ccd0 0x7f4a9c15e080 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4aa3764640 1 -- 172.21.15.67:0/1875682793 >> 172.21.15.67:0/1875682793 conn(0x7f4a9c0d5d90 msgr2=0x7f4a9c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4aa3764640 1 -- 172.21.15.67:0/1875682793 shutdown_connections 2024-04-25T15:29:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.650+0000 7f4aa3764640 1 -- 172.21.15.67:0/1875682793 wait complete. 2024-04-25T15:29:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.652+0000 7f4aa3764640 1 Processor -- start 2024-04-25T15:29:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.652+0000 7f4aa3764640 1 -- start start 2024-04-25T15:29:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.653+0000 7f4aa3764640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c161710 0x7f4a9c0cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.653+0000 7f4aa3764640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a9c163fa0 0x7f4a9c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.653+0000 7f4aa3764640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a9c142620 0x7f4a9c13fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.653+0000 7f4aa3764640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c16b280 con 0x7f4a9c161710 2024-04-25T15:29:28.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.653+0000 7f4aa3764640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c16b100 con 0x7f4a9c142620 2024-04-25T15:29:28.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.653+0000 7f4aa3764640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4a9c16b400 con 0x7f4a9c163fa0 2024-04-25T15:29:28.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c161710 0x7f4a9c0cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c161710 0x7f4a9c0cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35516/0 (socket says 172.21.15.67:35516) 2024-04-25T15:29:28.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 -- 172.21.15.67:0/1779804964 learned_addr learned my addr 172.21.15.67:0/1779804964 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:28.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa0cff640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a9c163fa0 0x7f4a9c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1d01640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a9c142620 0x7f4a9c13fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 -- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a9c163fa0 msgr2=0x7f4a9c155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a9c163fa0 0x7f4a9c155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 -- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a9c142620 msgr2=0x7f4a9c13fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a9c142620 0x7f4a9c13fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 -- 172.21.15.67:0/1779804964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c0cb300 con 0x7f4a9c161710 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa1500640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c161710 0x7f4a9c0cc050 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800c3f0 tx=0x7f4a9800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a98017070 con 0x7f4a9c161710 2024-04-25T15:29:28.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a98005ce0 con 0x7f4a9c161710 2024-04-25T15:29:28.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9c15a010 con 0x7f4a9c161710 2024-04-25T15:29:28.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.654+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a980072d0 con 0x7f4a9c161710 2024-04-25T15:29:28.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.655+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9c0cb850 con 0x7f4a9c161710 2024-04-25T15:29:28.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.655+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a9c0c96d0 con 0x7f4a9c161710 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.656+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4a9801a070 con 0x7f4a9c161710 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.656+0000 7f4a91ffb640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a800744b0 0x7f4a800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.656+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f4a98094140 con 0x7f4a9c161710 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.657+0000 7f4a91ffb640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f4a8007a3c0 0x7f4a8007c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.657+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 --> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a98020e50 con 0x7f4a8007a3c0 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.657+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f4a9809b050 con 0x7f4a9c161710 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.657+0000 7f4aa0cff640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a800744b0 0x7f4a800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.657+0000 7f4aa1d01640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f4a8007a3c0 0x7f4a8007c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.657+0000 7f4aa0cff640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a800744b0 0x7f4a800769c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c034000 tx=0x7f4a8c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.657+0000 7f4aa1d01640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f4a8007a3c0 0x7f4a8007c810 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.658+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== osd.6 v2:172.21.15.176:6800/744880207 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a98020e50 con 0x7f4a8007a3c0 2024-04-25T15:29:28.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.672+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 --> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4a9c050740 con 0x7f4a8007a3c0 2024-04-25T15:29:28.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.672+0000 7f4a91ffb640 1 -- 172.21.15.67:0/1779804964 <== osd.6 v2:172.21.15.176:6800/744880207 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4a9c050740 con 0x7f4a8007a3c0 2024-04-25T15:29:28.674 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:28.674 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:28.674 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:28.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f4a8007a3c0 msgr2=0x7f4a8007c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6800/744880207,v1:172.21.15.176:6801/744880207] conn(0x7f4a8007a3c0 0x7f4a8007c810 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a800744b0 msgr2=0x7f4a800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f4a800744b0 0x7f4a800769c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c034000 tx=0x7f4a8c02e000 comp rx=0 tx=0).stop 2024-04-25T15:29:28.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c161710 msgr2=0x7f4a9c0cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 --2- 172.21.15.67:0/1779804964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a9c161710 0x7f4a9c0cc050 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800c3f0 tx=0x7f4a9800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa1d01640 1 -- 172.21.15.67:0/1779804964 reap_dead start 2024-04-25T15:29:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 shutdown_connections 2024-04-25T15:29:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 >> 172.21.15.67:0/1779804964 conn(0x7f4a9c0d5d90 msgr2=0x7f4a9c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 shutdown_connections 2024-04-25T15:29:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.673+0000 7f4aa3764640 1 -- 172.21.15.67:0/1779804964 wait complete. 2024-04-25T15:29:28.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T15:29:28.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.797+0000 7f240a364640 1 Processor -- start 2024-04-25T15:29:28.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.797+0000 7f240a364640 1 -- start start 2024-04-25T15:29:28.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.798+0000 7f240a364640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f24040d0e00 0x7f24040d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.798+0000 7f240a364640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f240406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.798+0000 7f240a364640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240406cf10 con 0x7f24040b7810 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.798+0000 7f240a364640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f24040d5a60 con 0x7f24040d0e00 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.798+0000 7f240a364640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f24040d1850 con 0x7f240406c5c0 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.798+0000 7f2408901640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24040b7810 0x7f24040b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35376/0 (socket says 172.21.15.67:35376) 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.798+0000 7f2408901640 1 -- 172.21.15.67:0/3731142005 learned_addr learned my addr 172.21.15.67:0/3731142005 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2403fff640 1 --2- 172.21.15.67:0/3731142005 >> v2:172.21.15.176:3300/0 conn(0x7f24040d0e00 0x7f24040d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f24037fe640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f240406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 111781268 0 0) 0x7f23fc001490 con 0x7f24040b7810 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23ec003520 con 0x7f24040b7810 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3985454472 0 0) 0x7f23ec003520 con 0x7f24040b7810 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 msgr2=0x7f240406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f240406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 >> v2:172.21.15.176:3300/0 conn(0x7f24040d0e00 msgr2=0x7f24040d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 --2- 172.21.15.67:0/3731142005 >> v2:172.21.15.176:3300/0 conn(0x7f24040d0e00 0x7f24040d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2404160de0 con 0x7f24040b7810 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f24037fe640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f240406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7f23fc001bc0 con 0x7f24040b7810 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 >> v1:172.21.15.67:6789/0 conn(0x7f24040b7810 legacy=0x7f24040b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f23ec003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23ec004800 0x7f23ec004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f23ec005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f24040d5a60 con 0x7f23ec0039b0 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f24040d1850 con 0x7f23ec004800 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f23ec00b9c0 con 0x7f240406c5c0 2024-04-25T15:29:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2403fff640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f23ec003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f24037fe640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23ec004800 0x7f23ec004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2408901640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f23ec005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2403fff640 1 -- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 msgr2=0x7f23ec005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2403fff640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f23ec005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2403fff640 1 -- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23ec004800 msgr2=0x7f23ec004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2403fff640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23ec004800 0x7f23ec004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2403fff640 1 -- 172.21.15.67:0/3731142005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec003520 con 0x7f23ec0039b0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.799+0000 7f2408901640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f240406c5c0 0x7f23ec005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.800+0000 7f2403fff640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f23ec003e20 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009ac0 tx=0x7f23ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5856f03986103e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.800+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4036070 con 0x7f23ec0039b0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.800+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f4031400 con 0x7f23ec0039b0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.800+0000 7f2402ffd640 1 -- 172.21.15.67:0/3731142005 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4009690 con 0x7f23ec0039b0 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.801+0000 7f240a364640 1 -- 172.21.15.67:0/3731142005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 msgr2=0x7f23ec003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.801+0000 7f240a364640 1 --2- 172.21.15.67:0/3731142005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f23ec003e20 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009ac0 tx=0x7f23ec005830 comp rx=0 tx=0).stop 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.801+0000 7f2408901640 1 -- 172.21.15.67:0/3731142005 reap_dead start 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.801+0000 7f240a364640 1 -- 172.21.15.67:0/3731142005 shutdown_connections 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.801+0000 7f240a364640 1 -- 172.21.15.67:0/3731142005 >> 172.21.15.67:0/3731142005 conn(0x7f24040d5d90 msgr2=0x7f24040d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.801+0000 7f240a364640 1 -- 172.21.15.67:0/3731142005 shutdown_connections 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.801+0000 7f240a364640 1 -- 172.21.15.67:0/3731142005 wait complete. 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.803+0000 7f240a364640 1 Processor -- start 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.803+0000 7f240a364640 1 -- start start 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f240a364640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f24041643a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f240a364640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24040cbd00 0x7f24040ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f240a364640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24040caa40 0x7f24040cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f240a364640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2404160ad0 con 0x7f23ec0039b0 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f240a364640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f24041613a0 con 0x7f24040caa40 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f240a364640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f24040b5320 con 0x7f24040cbd00 2024-04-25T15:29:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f2403fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f24041643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f2403fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f24041643a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35526/0 (socket says 172.21.15.67:35526) 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.804+0000 7f2403fff640 1 -- 172.21.15.67:0/1238306310 learned_addr learned my addr 172.21.15.67:0/1238306310 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f24037fe640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24040cbd00 0x7f24040ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2403fff640 1 -- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24040cbd00 msgr2=0x7f24040ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2408901640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24040caa40 0x7f24040cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2403fff640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24040cbd00 0x7f24040ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2403fff640 1 -- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24040caa40 msgr2=0x7f24040cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2403fff640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24040caa40 0x7f24040cae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2403fff640 1 -- 172.21.15.67:0/1238306310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2404158d70 con 0x7f23ec0039b0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2403fff640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f24041643a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009ac0 tx=0x7f23f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2400ff9640 1 -- 172.21.15.67:0/1238306310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4041020 con 0x7f23ec0039b0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2400ff9640 1 -- 172.21.15.67:0/1238306310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f403c470 con 0x7f23ec0039b0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2404159050 con 0x7f23ec0039b0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f2400ff9640 1 -- 172.21.15.67:0/1238306310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4036040 con 0x7f23ec0039b0 2024-04-25T15:29:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.805+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2404163420 con 0x7f23ec0039b0 2024-04-25T15:29:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.806+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2404163680 con 0x7f23ec0039b0 2024-04-25T15:29:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.807+0000 7f2400ff9640 1 -- 172.21.15.67:0/1238306310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f23f4044020 con 0x7f23ec0039b0 2024-04-25T15:29:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.807+0000 7f2400ff9640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f23f00744e0 0x7f23f00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.807+0000 7f2400ff9640 1 -- 172.21.15.67:0/1238306310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f23f40b5790 con 0x7f23ec0039b0 2024-04-25T15:29:28.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.808+0000 7f24037fe640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f23f00744e0 0x7f23f00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.808+0000 7f24037fe640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f23f00744e0 0x7f23f00769f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f23f80099f0 tx=0x7f23f8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.812+0000 7f2400ff9640 1 -- 172.21.15.67:0/1238306310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f23f404b030 con 0x7f23ec0039b0 2024-04-25T15:29:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.995+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f2404050740 con 0x7f23f00744e0 2024-04-25T15:29:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.995+0000 7f2400ff9640 1 -- 172.21.15.67:0/1238306310 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2404050740 con 0x7f23f00744e0 2024-04-25T15:29:28.996 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f23f00744e0 msgr2=0x7f23f00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f23f00744e0 0x7f23f00769f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f23f80099f0 tx=0x7f23f8008040 comp rx=0 tx=0).stop 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 msgr2=0x7f24041643a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f24041643a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009ac0 tx=0x7f23f4004730 comp rx=0 tx=0).stop 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 shutdown_connections 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f23f00744e0 0x7f23f00769f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24040caa40 0x7f24040cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:28.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24040cbd00 0x7f24040ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 --2- 172.21.15.67:0/1238306310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec0039b0 0x7f24041643a0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 >> 172.21.15.67:0/1238306310 conn(0x7f24040d5d90 msgr2=0x7f24040b9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:29.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 shutdown_connections 2024-04-25T15:29:29.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:28.998+0000 7f240a364640 1 -- 172.21.15.67:0/1238306310 wait complete. 2024-04-25T15:29:29.009 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T15:29:29.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:29:29.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.125+0000 7f1fd0eaa640 1 Processor -- start 2024-04-25T15:29:29.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.125+0000 7f1fd0eaa640 1 -- start start 2024-04-25T15:29:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fd0eaa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc0d0e00 0x7f1fcc0d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fd0eaa640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fcc0d1850 con 0x7f1fcc06c5c0 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fd0eaa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fcc160de0 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fd0eaa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1fcc1610c0 con 0x7f1fcc0d0e00 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fc9d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1fcc06c5c0 0x7f1fcc06c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35388/0 (socket says 172.21.15.67:35388) 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fc9d9b640 1 -- 172.21.15.67:0/1427794864 learned_addr learned my addr 172.21.15.67:0/1427794864 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fca59c640 1 --2- 172.21.15.67:0/1427794864 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc0d0e00 0x7f1fcc0d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fc959a640 1 -- 172.21.15.67:0/1427794864 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3830345345 0 0) 0x7f1fcc0d1850 con 0x7f1fcc06c5c0 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.127+0000 7f1fc959a640 1 -- 172.21.15.67:0/1427794864 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb8003520 con 0x7f1fcc06c5c0 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.128+0000 7f1fc959a640 1 -- 172.21.15.67:0/1427794864 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079758013 0 0) 0x7f1fcc160de0 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.128+0000 7f1fc959a640 1 -- 172.21.15.67:0/1427794864 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fcc0d1850 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.128+0000 7f1fca59c640 1 -- 172.21.15.67:0/1427794864 >> v1:172.21.15.176:6789/0 conn(0x7f1fcc0d4a50 legacy=0x7f1fcc0d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.128+0000 7f1fca59c640 1 -- 172.21.15.67:0/1427794864 >> v1:172.21.15.67:6789/0 conn(0x7f1fcc06c5c0 legacy=0x7f1fcc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.128+0000 7f1fca59c640 1 -- 172.21.15.67:0/1427794864 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc1613a0 con 0x7f1fcc0d0e00 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.128+0000 7f1fca59c640 1 --2- 172.21.15.67:0/1427794864 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc0d0e00 0x7f1fcc0d1210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009dc0 tx=0x7f1fbc02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48f663222bf45f44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fc959a640 1 -- 172.21.15.67:0/1427794864 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc007740 con 0x7f1fcc0d0e00 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fc959a640 1 -- 172.21.15.67:0/1427794864 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fbc0078e0 con 0x7f1fcc0d0e00 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fc959a640 1 -- 172.21.15.67:0/1427794864 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc007c90 con 0x7f1fcc0d0e00 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/1427794864 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc0d0e00 msgr2=0x7f1fcc0d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fd0eaa640 1 --2- 172.21.15.67:0/1427794864 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc0d0e00 0x7f1fcc0d1210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009dc0 tx=0x7f1fbc02a6d0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/1427794864 shutdown_connections 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fd0eaa640 1 --2- 172.21.15.67:0/1427794864 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc0d0e00 0x7f1fcc0d1210 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/1427794864 >> 172.21.15.67:0/1427794864 conn(0x7f1fcc0d5d90 msgr2=0x7f1fcc0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/1427794864 shutdown_connections 2024-04-25T15:29:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.129+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/1427794864 wait complete. 2024-04-25T15:29:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.131+0000 7f1fd0eaa640 1 Processor -- start 2024-04-25T15:29:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.131+0000 7f1fd0eaa640 1 -- start start 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fd0eaa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc06c5c0 0x7f1fcc1eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fd0eaa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1fcc0d0e00 0x7f1fcc0ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fd0eaa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fcc0d4a50 0x7f1fcc1eac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fd0eaa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc164e00 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fd0eaa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc0b5320 con 0x7f1fcc0d0e00 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fd0eaa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1fcc164f80 con 0x7f1fcc06c5c0 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fcc0d4a50 0x7f1fcc1eac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fcc0d4a50 0x7f1fcc1eac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35528/0 (socket says 172.21.15.67:35528) 2024-04-25T15:29:29.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 -- 172.21.15.67:0/3920924056 learned_addr learned my addr 172.21.15.67:0/3920924056 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fca59c640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc06c5c0 0x7f1fcc1eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fc9d9b640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1fcc0d0e00 0x7f1fcc0ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 -- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc06c5c0 msgr2=0x7f1fcc1eb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1fcc06c5c0 0x7f1fcc1eb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 -- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1fcc0d0e00 msgr2=0x7f1fcc0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1fcc0d0e00 0x7f1fcc0ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 -- 172.21.15.67:0/3920924056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc1e9ee0 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.133+0000 7f1fcad9d640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fcc0d4a50 0x7f1fcc1eac60 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400c3f0 tx=0x7f1fc400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.134+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4017070 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.134+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc40052d0 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.134+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fcc1ea180 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.134+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4003830 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.134+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fcc1ea740 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.134+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1fcc050740 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.135+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1fc401a070 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.136+0000 7f1fc2ffd640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1fac0744b0 0x7f1fac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.136+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f1fc40942a0 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.136+0000 7f1fc2ffd640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f1fac07a3c0 0x7f1fac07c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.136+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 --> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fc401ee50 con 0x7f1fac07a3c0 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.136+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f1fc409b050 con 0x7f1fcc0d4a50 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.136+0000 7f1fca59c640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1fac0744b0 0x7f1fac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.136+0000 7f1fc9d9b640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f1fac07a3c0 0x7f1fac07c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.137+0000 7f1fca59c640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1fac0744b0 0x7f1fac0769c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc034000 tx=0x7f1fbc033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.137+0000 7f1fc9d9b640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f1fac07a3c0 0x7f1fac07c810 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.138+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== osd.7 v2:172.21.15.176:6808/592477191 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1fc401ee50 con 0x7f1fac07a3c0 2024-04-25T15:29:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.151+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 --> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1fcc1eba50 con 0x7f1fac07a3c0 2024-04-25T15:29:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.151+0000 7f1fc2ffd640 1 -- 172.21.15.67:0/3920924056 <== osd.7 v2:172.21.15.176:6808/592477191 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1fcc1eba50 con 0x7f1fac07a3c0 2024-04-25T15:29:29.152 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:29:29.152 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:29:29.152 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:29:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f1fac07a3c0 msgr2=0x7f1fac07c810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6808/592477191,v1:172.21.15.176:6809/592477191] conn(0x7f1fac07a3c0 0x7f1fac07c810 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1fac0744b0 msgr2=0x7f1fac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f1fac0744b0 0x7f1fac0769c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc034000 tx=0x7f1fbc033040 comp rx=0 tx=0).stop 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fcc0d4a50 msgr2=0x7f1fcc1eac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 --2- 172.21.15.67:0/3920924056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1fcc0d4a50 0x7f1fcc1eac60 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400c3f0 tx=0x7f1fc400c8b0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fcad9d640 1 -- 172.21.15.67:0/3920924056 reap_dead start 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 shutdown_connections 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 >> 172.21.15.67:0/3920924056 conn(0x7f1fcc0d5d90 msgr2=0x7f1fcc0b7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 shutdown_connections 2024-04-25T15:29:29.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.152+0000 7f1fd0eaa640 1 -- 172.21.15.67:0/3920924056 wait complete. 2024-04-25T15:29:29.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T15:29:29.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.275+0000 7f805d292640 1 Processor -- start 2024-04-25T15:29:29.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.275+0000 7f805d292640 1 -- start start 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.276+0000 7f805d292640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f80580d0de0 0x7f80580d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.276+0000 7f805d292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f805806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.276+0000 7f805d292640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f805806cf10 con 0x7f80580b77f0 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.276+0000 7f805d292640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f80580d5a40 con 0x7f80580d0de0 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.276+0000 7f805d292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f80580d1830 con 0x7f805806c5c0 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.276+0000 7f80577fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f80580b77f0 0x7f80580b7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35398/0 (socket says 172.21.15.67:35398) 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.276+0000 7f80577fe640 1 -- 172.21.15.67:0/1596684264 learned_addr learned my addr 172.21.15.67:0/1596684264 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f804e7fc640 1 --2- 172.21.15.67:0/1596684264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f805806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f8056ffd640 1 --2- 172.21.15.67:0/1596684264 >> v2:172.21.15.176:3300/0 conn(0x7f80580d0de0 0x7f80580d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f80567fc640 1 -- 172.21.15.67:0/1596684264 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68052665 0 0) 0x7f8050001490 con 0x7f80580b77f0 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f80567fc640 1 -- 172.21.15.67:0/1596684264 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8044003520 con 0x7f80580b77f0 2024-04-25T15:29:29.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f804e7fc640 1 -- 172.21.15.67:0/1596684264 >> v2:172.21.15.176:3300/0 conn(0x7f80580d0de0 msgr2=0x7f80580d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:29.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f804e7fc640 1 --2- 172.21.15.67:0/1596684264 >> v2:172.21.15.176:3300/0 conn(0x7f80580d0de0 0x7f80580d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f804e7fc640 1 -- 172.21.15.67:0/1596684264 >> v1:172.21.15.67:6789/0 conn(0x7f80580b77f0 legacy=0x7f80580b7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.277+0000 7f804e7fc640 1 -- 172.21.15.67:0/1596684264 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8058160dc0 con 0x7f805806c5c0 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.278+0000 7f804e7fc640 1 --2- 172.21.15.67:0/1596684264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f805806c9d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80400097a0 tx=0x7f804002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=938cb982a99dca1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.278+0000 7f80567fc640 1 -- 172.21.15.67:0/1596684264 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040037070 con 0x7f805806c5c0 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.278+0000 7f80567fc640 1 -- 172.21.15.67:0/1596684264 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8040032440 con 0x7f805806c5c0 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.278+0000 7f80567fc640 1 -- 172.21.15.67:0/1596684264 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80400327f0 con 0x7f805806c5c0 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.278+0000 7f805d292640 1 -- 172.21.15.67:0/1596684264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 msgr2=0x7f805806c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.278+0000 7f805d292640 1 --2- 172.21.15.67:0/1596684264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f805806c9d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f80400097a0 tx=0x7f804002a180 comp rx=0 tx=0).stop 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.279+0000 7f805d292640 1 -- 172.21.15.67:0/1596684264 shutdown_connections 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.279+0000 7f805d292640 1 --2- 172.21.15.67:0/1596684264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f805806c9d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.279+0000 7f805d292640 1 --2- 172.21.15.67:0/1596684264 >> v2:172.21.15.176:3300/0 conn(0x7f80580d0de0 0x7f80580d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.279+0000 7f805d292640 1 -- 172.21.15.67:0/1596684264 >> 172.21.15.67:0/1596684264 conn(0x7f80580d5d70 msgr2=0x7f80580d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:29.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.279+0000 7f805d292640 1 -- 172.21.15.67:0/1596684264 shutdown_connections 2024-04-25T15:29:29.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.279+0000 7f805d292640 1 -- 172.21.15.67:0/1596684264 wait complete. 2024-04-25T15:29:29.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.280+0000 7f805d292640 1 Processor -- start 2024-04-25T15:29:29.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.281+0000 7f805d292640 1 -- start start 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f805d292640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f80580cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f805d292640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80580b77f0 0x7f805815f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f805d292640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80580d0de0 0x7f805815f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f805d292640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8058160ab0 con 0x7f80580b77f0 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f805d292640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8058161380 con 0x7f80580d0de0 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f805d292640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f80580b5300 con 0x7f805806c5c0 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80580b77f0 0x7f805815f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80580b77f0 0x7f805815f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35534/0 (socket says 172.21.15.67:35534) 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 -- 172.21.15.67:0/3171067832 learned_addr learned my addr 172.21.15.67:0/3171067832 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f8056ffd640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f80580cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f80577fe640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80580d0de0 0x7f805815f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 -- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 msgr2=0x7f80580cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f80580cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 -- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80580d0de0 msgr2=0x7f805815f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80580d0de0 0x7f805815f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 -- 172.21.15.67:0/3171067832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805814ccf0 con 0x7f80580b77f0 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.282+0000 7f804e7fc640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80580b77f0 0x7f805815f000 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8040035040 tx=0x7f80400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.283+0000 7f804ffff640 1 -- 172.21.15.67:0/3171067832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040037040 con 0x7f80580b77f0 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.283+0000 7f804ffff640 1 -- 172.21.15.67:0/3171067832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8040042070 con 0x7f80580b77f0 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.283+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805814cfd0 con 0x7f80580b77f0 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.283+0000 7f804ffff640 1 -- 172.21.15.67:0/3171067832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804003b930 con 0x7f80580b77f0 2024-04-25T15:29:29.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.283+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f805814d590 con 0x7f80580b77f0 2024-04-25T15:29:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.284+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8058050740 con 0x7f80580b77f0 2024-04-25T15:29:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.284+0000 7f804ffff640 1 -- 172.21.15.67:0/3171067832 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f804003bad0 con 0x7f80580b77f0 2024-04-25T15:29:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.285+0000 7f804ffff640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f80380744e0 0x7f80380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.285+0000 7f804ffff640 1 -- 172.21.15.67:0/3171067832 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f80400b5770 con 0x7f80580b77f0 2024-04-25T15:29:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.285+0000 7f8056ffd640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f80380744e0 0x7f80380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.286+0000 7f8056ffd640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f80380744e0 0x7f80380769f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8048009aa0 tx=0x7f8048009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.289+0000 7f804ffff640 1 -- 172.21.15.67:0/3171067832 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f804007fb30 con 0x7f80580b77f0 2024-04-25T15:29:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.474+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f80580cbc50 con 0x7f80380744e0 2024-04-25T15:29:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.475+0000 7f804ffff640 1 -- 172.21.15.67:0/3171067832 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f80580cbc50 con 0x7f80380744e0 2024-04-25T15:29:29.476 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f80380744e0 msgr2=0x7f80380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f80380744e0 0x7f80380769f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8048009aa0 tx=0x7f8048009190 comp rx=0 tx=0).stop 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80580b77f0 msgr2=0x7f805815f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80580b77f0 0x7f805815f000 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f8040035040 tx=0x7f80400046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 shutdown_connections 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f80380744e0 0x7f80380769f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80580d0de0 0x7f805815f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80580b77f0 0x7f805815f000 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 --2- 172.21.15.67:0/3171067832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f805806c5c0 0x7f80580cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 >> 172.21.15.67:0/3171067832 conn(0x7f80580d5d70 msgr2=0x7f805815e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 shutdown_connections 2024-04-25T15:29:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.477+0000 7f805d292640 1 -- 172.21.15.67:0/3171067832 wait complete. 2024-04-25T15:29:29.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:29.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.601+0000 7f89b5815640 1 Processor -- start 2024-04-25T15:29:29.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.601+0000 7f89b5815640 1 -- start start 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.602+0000 7f89b5815640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b00d0e00 0x7f89b00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.602+0000 7f89b5815640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b006c5c0 0x7f89b006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89b5815640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b006cf10 con 0x7f89b00b7810 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89b5815640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89b00d5a60 con 0x7f89b00d0e00 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89b5815640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89b00d1850 con 0x7f89b006c5c0 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89aeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b00d0e00 0x7f89b00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89aeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b00d0e00 0x7f89b00d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35538/0 (socket says 172.21.15.67:35538) 2024-04-25T15:29:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89aeffd640 1 -- 172.21.15.67:0/319526477 learned_addr learned my addr 172.21.15.67:0/319526477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89ae7fc640 1 --2- 172.21.15.67:0/319526477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b006c5c0 0x7f89b006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89aeffd640 1 -- 172.21.15.67:0/319526477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b006c5c0 msgr2=0x7f89b006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89aeffd640 1 --2- 172.21.15.67:0/319526477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b006c5c0 0x7f89b006c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89aeffd640 1 -- 172.21.15.67:0/319526477 >> v1:172.21.15.176:6789/0 conn(0x7f89b00b7810 legacy=0x7f89b00b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89aeffd640 1 -- 172.21.15.67:0/319526477 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b0160de0 con 0x7f89b00d0e00 2024-04-25T15:29:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.603+0000 7f89ae7fc640 1 --2- 172.21.15.67:0/319526477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b006c5c0 0x7f89b006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:29.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.604+0000 7f89aeffd640 1 --2- 172.21.15.67:0/319526477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b00d1210 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009e50 tx=0x7f89a002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827fcc901ea12307 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.604+0000 7f89adffb640 1 -- 172.21.15.67:0/319526477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0036070 con 0x7f89b00d0e00 2024-04-25T15:29:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.604+0000 7f89adffb640 1 -- 172.21.15.67:0/319526477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a0031600 con 0x7f89b00d0e00 2024-04-25T15:29:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.604+0000 7f89adffb640 1 -- 172.21.15.67:0/319526477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a00319d0 con 0x7f89b00d0e00 2024-04-25T15:29:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 -- 172.21.15.67:0/319526477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 msgr2=0x7f89b00d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 --2- 172.21.15.67:0/319526477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b00d1210 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009e50 tx=0x7f89a002a6c0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 -- 172.21.15.67:0/319526477 shutdown_connections 2024-04-25T15:29:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 --2- 172.21.15.67:0/319526477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b006c5c0 0x7f89b006c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 --2- 172.21.15.67:0/319526477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b00d1210 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 -- 172.21.15.67:0/319526477 >> 172.21.15.67:0/319526477 conn(0x7f89b00d5d90 msgr2=0x7f89b00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 -- 172.21.15.67:0/319526477 shutdown_connections 2024-04-25T15:29:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.605+0000 7f89b5815640 1 -- 172.21.15.67:0/319526477 wait complete. 2024-04-25T15:29:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.606+0000 7f89b5815640 1 Processor -- start 2024-04-25T15:29:29.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.607+0000 7f89b5815640 1 -- start start 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89b5815640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b006c5c0 0x7f89b01fa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89b5815640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b00b7810 0x7f89b014bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89b5815640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b015f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89b5815640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89b0160ad0 con 0x7f89b00d0e00 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89b5815640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89b01613a0 con 0x7f89b006c5c0 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89b5815640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89b00b5320 con 0x7f89b00b7810 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b015f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b015f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35540/0 (socket says 172.21.15.67:35540) 2024-04-25T15:29:29.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 -- 172.21.15.67:0/2297371451 learned_addr learned my addr 172.21.15.67:0/2297371451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89ae7fc640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b00b7810 0x7f89b014bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89aeffd640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b006c5c0 0x7f89b01fa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 -- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b00b7810 msgr2=0x7f89b014bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b00b7810 0x7f89b014bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 -- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b006c5c0 msgr2=0x7f89b01fa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b006c5c0 0x7f89b01fa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 -- 172.21.15.67:0/2297371451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b00cf1d0 con 0x7f89b00d0e00 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.608+0000 7f89af7fe640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b015f9c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f89a800b440 tx=0x7f89a800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.609+0000 7f89a7fff640 1 -- 172.21.15.67:0/2297371451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8011830 con 0x7f89b00d0e00 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.609+0000 7f89a7fff640 1 -- 172.21.15.67:0/2297371451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a8016070 con 0x7f89b00d0e00 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.609+0000 7f89a7fff640 1 -- 172.21.15.67:0/2297371451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8011c10 con 0x7f89b00d0e00 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.609+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b00c9a80 con 0x7f89b00d0e00 2024-04-25T15:29:29.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.609+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b00ca040 con 0x7f89b00d0e00 2024-04-25T15:29:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.610+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b0050740 con 0x7f89b00d0e00 2024-04-25T15:29:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.610+0000 7f89a7fff640 1 -- 172.21.15.67:0/2297371451 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f89a801b460 con 0x7f89b00d0e00 2024-04-25T15:29:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.611+0000 7f89a7fff640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89900744e0 0x7f89900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.611+0000 7f89a7fff640 1 -- 172.21.15.67:0/2297371451 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f89a8094760 con 0x7f89b00d0e00 2024-04-25T15:29:29.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.611+0000 7f89aeffd640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89900744e0 0x7f89900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:29.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.611+0000 7f89aeffd640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89900744e0 0x7f89900769f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f89a0033000 tx=0x7f89a0009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:29.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.616+0000 7f89a7fff640 1 -- 172.21.15.67:0/2297371451 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89a8020020 con 0x7f89b00d0e00 2024-04-25T15:29:29.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:29:29.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:29:29.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.800+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89b00d4320 con 0x7f89900744e0 2024-04-25T15:29:29.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.802+0000 7f89a7fff640 1 -- 172.21.15.67:0/2297371451 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13443 (secure 0 0 0) 0x7f89b00d4320 con 0x7f89900744e0 2024-04-25T15:29:29.803 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:29.803 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89900744e0 msgr2=0x7f89900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89900744e0 0x7f89900769f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f89a0033000 tx=0x7f89a0009000 comp rx=0 tx=0).stop 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 msgr2=0x7f89b015f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b015f9c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f89a800b440 tx=0x7f89a800b900 comp rx=0 tx=0).stop 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 shutdown_connections 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f89900744e0 0x7f89900769f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b00d0e00 0x7f89b015f9c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b00b7810 0x7f89b014bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 --2- 172.21.15.67:0/2297371451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b006c5c0 0x7f89b01fa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:29.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 >> 172.21.15.67:0/2297371451 conn(0x7f89b00d5d90 msgr2=0x7f89b00b7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:29.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 shutdown_connections 2024-04-25T15:29:29.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:29.804+0000 7f89b5815640 1 -- 172.21.15.67:0/2297371451 wait complete. 2024-04-25T15:29:29.816 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-04-25T15:29:28.854893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":259872,"kb_used_data":10408,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714848,"statfs":{"total":773094113280,"available":772828004352,"internally_reserved":0,"allocated":10657792,"data_stored":5205032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23068702,"num_objects":-26,"num_object_clones":0,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-37,"num_write_kb":-43028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.398789"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":246,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-25T15:29:21.559852+0000","last_change":"2024-04-25T15:25:17.888495+0000","last_active":"2024-04-25T15:29:21.559852+0000","last_peered":"2024-04-25T15:29:21.559852+0000","last_clean":"2024-04-25T15:29:21.559852+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:29:21.559852+0000","last_undegraded":"2024-04-25T15:29:21.559852+0000","last_fullsized":"2024-04-25T15:29:21.559852+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:25:16.874887+0000","last_clean_scrub_stamp":"2024-04-25T15:25:16.874887+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:36.209719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1777664,"data_stored":1093405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1777664,"data_stored":1093405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:29.819 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T15:25:16.874887+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=25, tm_sec=16, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=29, tm_sec=24, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:29:29.819 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T15:29:49.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:29:49.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.945+0000 7f8b37bb2640 1 Processor -- start 2024-04-25T15:29:49.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.945+0000 7f8b37bb2640 1 -- start start 2024-04-25T15:29:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b37bb2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b300d4570 0x7f8b300b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b37bb2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8b300b2dc0 0x7f8b300b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b37bb2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b3006d870 con 0x7f8b3006e8f0 2024-04-25T15:29:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b37bb2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8b3006e460 con 0x7f8b300d4570 2024-04-25T15:29:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b37bb2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8b3006db10 con 0x7f8b300b2dc0 2024-04-25T15:29:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3594e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b300d4570 0x7f8b300b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3594e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8b300d4570 0x7f8b300b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41590/0 (socket says 172.21.15.67:41590) 2024-04-25T15:29:49.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3594e640 1 -- 172.21.15.67:0/1180825827 learned_addr learned my addr 172.21.15.67:0/1180825827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:49.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3594e640 1 -- 172.21.15.67:0/1180825827 >> v2:172.21.15.176:3300/0 conn(0x7f8b300b2dc0 msgr2=0x7f8b300b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:49.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3614f640 1 --2- 172.21.15.67:0/1180825827 >> v2:172.21.15.176:3300/0 conn(0x7f8b300b2dc0 0x7f8b300b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:49.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3594e640 1 --2- 172.21.15.67:0/1180825827 >> v2:172.21.15.176:3300/0 conn(0x7f8b300b2dc0 0x7f8b300b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:49.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3594e640 1 -- 172.21.15.67:0/1180825827 >> v1:172.21.15.176:6789/0 conn(0x7f8b3006e8f0 legacy=0x7f8b300d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:49.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.947+0000 7f8b3594e640 1 -- 172.21.15.67:0/1180825827 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3006e0f0 con 0x7f8b300d4570 2024-04-25T15:29:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.948+0000 7f8b3594e640 1 --2- 172.21.15.67:0/1180825827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b300d4570 0x7f8b300b4830 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f8b3006e5e0 tx=0x7f8b2802a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3587b19a691c205e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.948+0000 7f8b3494c640 1 -- 172.21.15.67:0/1180825827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28004030 con 0x7f8b300d4570 2024-04-25T15:29:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.948+0000 7f8b3494c640 1 -- 172.21.15.67:0/1180825827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b280041d0 con 0x7f8b300d4570 2024-04-25T15:29:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.948+0000 7f8b3494c640 1 -- 172.21.15.67:0/1180825827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28003610 con 0x7f8b300d4570 2024-04-25T15:29:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1180825827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b300d4570 msgr2=0x7f8b300b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1180825827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b300d4570 0x7f8b300b4830 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f8b3006e5e0 tx=0x7f8b2802a160 comp rx=0 tx=0).stop 2024-04-25T15:29:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1180825827 shutdown_connections 2024-04-25T15:29:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1180825827 >> v2:172.21.15.176:3300/0 conn(0x7f8b300b2dc0 0x7f8b300b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1180825827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b300d4570 0x7f8b300b4830 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1180825827 >> 172.21.15.67:0/1180825827 conn(0x7f8b300d3a50 msgr2=0x7f8b300d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1180825827 shutdown_connections 2024-04-25T15:29:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.949+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1180825827 wait complete. 2024-04-25T15:29:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.951+0000 7f8b37bb2640 1 Processor -- start 2024-04-25T15:29:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.951+0000 7f8b37bb2640 1 -- start start 2024-04-25T15:29:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b37bb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b3006e8f0 0x7f8b30159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b37bb2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b300b2dc0 0x7f8b301f9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b37bb2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300d4570 0x7f8b300b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b37bb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3006ed00 con 0x7f8b3006e8f0 2024-04-25T15:29:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b37bb2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8b300d0fb0 con 0x7f8b300d4570 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b37bb2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8b300b3710 con 0x7f8b300b2dc0 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b3594e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b3006e8f0 0x7f8b30159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b3594e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b3006e8f0 0x7f8b30159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41594/0 (socket says 172.21.15.67:41594) 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b3594e640 1 -- 172.21.15.67:0/1478596095 learned_addr learned my addr 172.21.15.67:0/1478596095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b3614f640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300d4570 0x7f8b300b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.952+0000 7f8b3514d640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b300b2dc0 0x7f8b301f9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b3594e640 1 -- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b300b2dc0 msgr2=0x7f8b301f9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b3594e640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b300b2dc0 0x7f8b301f9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b3594e640 1 -- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300d4570 msgr2=0x7f8b300b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b3594e640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300d4570 0x7f8b300b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b3594e640 1 -- 172.21.15.67:0/1478596095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b301faa70 con 0x7f8b3006e8f0 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b3614f640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300d4570 0x7f8b300b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b3594e640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b3006e8f0 0x7f8b30159840 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8b28034040 tx=0x7f8b280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b2e7fc640 1 -- 172.21.15.67:0/1478596095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28041030 con 0x7f8b3006e8f0 2024-04-25T15:29:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b2e7fc640 1 -- 172.21.15.67:0/1478596095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b28003c50 con 0x7f8b3006e8f0 2024-04-25T15:29:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3009f1c0 con 0x7f8b3006e8f0 2024-04-25T15:29:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b2e7fc640 1 -- 172.21.15.67:0/1478596095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28004030 con 0x7f8b3006e8f0 2024-04-25T15:29:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.953+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b3009f780 con 0x7f8b3006e8f0 2024-04-25T15:29:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.954+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b30050740 con 0x7f8b3006e8f0 2024-04-25T15:29:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.955+0000 7f8b2e7fc640 1 -- 172.21.15.67:0/1478596095 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8b28044070 con 0x7f8b3006e8f0 2024-04-25T15:29:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.955+0000 7f8b2e7fc640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b1c0744e0 0x7f8b1c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.955+0000 7f8b2e7fc640 1 -- 172.21.15.67:0/1478596095 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f8b280b5b50 con 0x7f8b3006e8f0 2024-04-25T15:29:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.955+0000 7f8b3514d640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b1c0744e0 0x7f8b1c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.956+0000 7f8b3514d640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b1c0744e0 0x7f8b1c0769f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8b24006fd0 tx=0x7f8b24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:49.960+0000 7f8b2e7fc640 1 -- 172.21.15.67:0/1478596095 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b28080670 con 0x7f8b3006e8f0 2024-04-25T15:29:50.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.141+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 --> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b30050740 con 0x7f8b1c0744e0 2024-04-25T15:29:50.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.142+0000 7f8b2e7fc640 1 -- 172.21.15.67:0/1478596095 <== mgr.4100 v2:172.21.15.176:6832/1591852742 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13360 (secure 0 0 0) 0x7f8b30050740 con 0x7f8b1c0744e0 2024-04-25T15:29:50.143 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:50.143 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b1c0744e0 msgr2=0x7f8b1c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b1c0744e0 0x7f8b1c0769f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8b24006fd0 tx=0x7f8b24008040 comp rx=0 tx=0).stop 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b3006e8f0 msgr2=0x7f8b30159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b3006e8f0 0x7f8b30159840 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8b28034040 tx=0x7f8b280046c0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 shutdown_connections 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f8b1c0744e0 0x7f8b1c0769f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b300d4570 0x7f8b300b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b300b2dc0 0x7f8b301f9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 --2- 172.21.15.67:0/1478596095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b3006e8f0 0x7f8b30159840 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 >> 172.21.15.67:0/1478596095 conn(0x7f8b300d3a50 msgr2=0x7f8b30072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:50.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 shutdown_connections 2024-04-25T15:29:50.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.144+0000 7f8b37bb2640 1 -- 172.21.15.67:0/1478596095 wait complete. 2024-04-25T15:29:50.156 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-04-25T15:29:48.859484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":259856,"kb_used_data":10408,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714864,"statfs":{"total":773094113280,"available":772828020736,"internally_reserved":0,"allocated":10657792,"data_stored":5205032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.0","version":"95'74","reported_seq":257,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-25T15:29:30.486976+0000","last_change":"2024-04-25T15:29:30.487010+0000","last_active":"2024-04-25T15:29:30.486976+0000","last_peered":"2024-04-25T15:29:30.486976+0000","last_clean":"2024-04-25T15:29:30.486976+0000","last_became_active":"2024-04-25T15:25:17.885664+0000","last_became_peered":"2024-04-25T15:25:17.885664+0000","last_unstale":"2024-04-25T15:29:30.486976+0000","last_undegraded":"2024-04-25T15:29:30.486976+0000","last_fullsized":"2024-04-25T15:29:30.486976+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'74","last_scrub_stamp":"2024-04-25T15:29:30.486903+0000","last_deep_scrub":"95'74","last_deep_scrub_stamp":"2024-04-25T15:29:30.486903+0000","last_clean_scrub_stamp":"2024-04-25T15:29:30.486903+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:26.907112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574905,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1777664,"data_stored":1093405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574907,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32336,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339504,"statfs":{"total":96636764160,"available":96603652096,"internally_reserved":0,"allocated":1183744,"data_stored":503037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32912,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338928,"statfs":{"total":96636764160,"available":96603062272,"internally_reserved":0,"allocated":1777664,"data_stored":1093405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:29:50.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T15:29:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.267+0000 7fa67d9b2640 1 Processor -- start 2024-04-25T15:29:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.267+0000 7fa67d9b2640 1 -- start start 2024-04-25T15:29:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa67d9b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa67806ccd0 0x7fa6780d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa67d9b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa6780d4a30 0x7fa6780d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa67d9b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6780d1830 con 0x7fa6780d0de0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa67d9b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa678160dc0 con 0x7fa6780d4a30 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa67d9b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa6781610a0 con 0x7fa67806ccd0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa676ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa6780d0de0 0x7fa67806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56280/0 (socket says 172.21.15.67:56280) 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa676ffd640 1 -- 172.21.15.67:0/1164490013 learned_addr learned my addr 172.21.15.67:0/1164490013 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa6777fe640 1 --2- 172.21.15.67:0/1164490013 >> v2:172.21.15.176:3300/0 conn(0x7fa6780d4a30 0x7fa6780d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa6767fc640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa67806ccd0 0x7fa6780d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2301053480 0 0) 0x7fa6780d1830 con 0x7fa6780d0de0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa664003520 con 0x7fa6780d0de0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2504458041 0 0) 0x7fa664003520 con 0x7fa6780d0de0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa67806ccd0 msgr2=0x7fa6780d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa67806ccd0 0x7fa6780d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 >> v2:172.21.15.176:3300/0 conn(0x7fa6780d4a30 msgr2=0x7fa6780d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 --2- 172.21.15.67:0/1164490013 >> v2:172.21.15.176:3300/0 conn(0x7fa6780d4a30 0x7fa6780d4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa678161380 con 0x7fa6780d0de0 2024-04-25T15:29:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1482936191 0 0) 0x7fa670001670 con 0x7fa6780d0de0 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa6777fe640 1 --2- 172.21.15.67:0/1164490013 >> v2:172.21.15.176:3300/0 conn(0x7fa6780d4a30 0x7fa6780d4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.269+0000 7fa6767fc640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa67806ccd0 0x7fa6780d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 >> v1:172.21.15.67:6789/0 conn(0x7fa6780d0de0 legacy=0x7fa67806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa675ffb640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6640039b0 0x7fa664003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa675ffb640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa664004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa675ffb640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa664004eb0 0x7fa66400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa678160dc0 con 0x7fa664004320 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa6781610a0 con 0x7fa664004eb0 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa66400bf00 con 0x7fa6640039b0 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa6767fc640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa664004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa676ffd640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6640039b0 0x7fa664003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa6777fe640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa664004eb0 0x7fa66400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa6767fc640 1 -- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6640039b0 msgr2=0x7fa664003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:50.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa6767fc640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6640039b0 0x7fa664003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa6767fc640 1 -- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa664004eb0 msgr2=0x7fa66400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa6767fc640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa664004eb0 0x7fa66400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa6767fc640 1 -- 172.21.15.67:0/1164490013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa664003520 con 0x7fa664004320 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.270+0000 7fa676ffd640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6640039b0 0x7fa664003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa6767fc640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa664004840 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa660009c90 tx=0x7fa664005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf661b1313619fa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660036070 con 0x7fa664004320 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa660031400 con 0x7fa664004320 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa675ffb640 1 -- 172.21.15.67:0/1164490013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660009860 con 0x7fa664004320 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa67d9b2640 1 -- 172.21.15.67:0/1164490013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 msgr2=0x7fa664004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa67d9b2640 1 --2- 172.21.15.67:0/1164490013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa664004840 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa660009c90 tx=0x7fa664005830 comp rx=0 tx=0).stop 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa6777fe640 1 -- 172.21.15.67:0/1164490013 reap_dead start 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.271+0000 7fa67d9b2640 1 -- 172.21.15.67:0/1164490013 shutdown_connections 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.272+0000 7fa67d9b2640 1 -- 172.21.15.67:0/1164490013 >> 172.21.15.67:0/1164490013 conn(0x7fa6780d5d70 msgr2=0x7fa6780d6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:50.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.272+0000 7fa67d9b2640 1 -- 172.21.15.67:0/1164490013 shutdown_connections 2024-04-25T15:29:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.272+0000 7fa67d9b2640 1 -- 172.21.15.67:0/1164490013 wait complete. 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.273+0000 7fa67d9b2640 1 Processor -- start 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.273+0000 7fa67d9b2640 1 -- start start 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa67d9b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa6780cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa67d9b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6780cce40 0x7fa6781f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa67d9b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6781fa290 0x7fa6781fa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa67d9b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa678164de0 con 0x7fa664004320 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa67d9b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780b5300 con 0x7fa6781fa290 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa67d9b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa678164f60 con 0x7fa6780cce40 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa6780cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa6780cf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41620/0 (socket says 172.21.15.67:41620) 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 -- 172.21.15.67:0/282788379 learned_addr learned my addr 172.21.15.67:0/282788379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa6777fe640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6781fa290 0x7fa6781fa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa6767fc640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6780cce40 0x7fa6781f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 -- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6780cce40 msgr2=0x7fa6781f6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6780cce40 0x7fa6781f6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 -- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6781fa290 msgr2=0x7fa6781fa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6781fa290 0x7fa6781fa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 -- 172.21.15.67:0/282788379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6781fabe0 con 0x7fa664004320 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.275+0000 7fa676ffd640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa6780cf350 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa67000e820 tx=0x7fa67000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.276+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67000c870 con 0x7fa664004320 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.276+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa670018070 con 0x7fa664004320 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.276+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6780cade0 con 0x7fa664004320 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.276+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67000cc10 con 0x7fa664004320 2024-04-25T15:29:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.276+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6780ca300 con 0x7fa664004320 2024-04-25T15:29:50.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.277+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6780b9c00 con 0x7fa664004320 2024-04-25T15:29:50.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.277+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa67001d460 con 0x7fa664004320 2024-04-25T15:29:50.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.278+0000 7fa66ffff640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6580744b0 0x7fa6580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.278+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7fa670094cd0 con 0x7fa664004320 2024-04-25T15:29:50.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.278+0000 7fa6767fc640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6580744b0 0x7fa6580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.279+0000 7fa6767fc640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6580744b0 0x7fa6580769c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa660033000 tx=0x7fa660008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.283+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6780b9c00 con 0x7fa664004320 2024-04-25T15:29:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.467+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa678050740 con 0x7fa664004320 2024-04-25T15:29:50.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.471+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v22)=0 v22) ==== 123+0+0 (secure 0 0 0) 0x7fa6781fabe0 con 0x7fa664004320 2024-04-25T15:29:50.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.472+0000 7fa66ffff640 1 -- 172.21.15.67:0/282788379 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa67005ec20 con 0x7fa664004320 2024-04-25T15:29:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6580744b0 msgr2=0x7fa6580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6580744b0 0x7fa6580769c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa660033000 tx=0x7fa660008030 comp rx=0 tx=0).stop 2024-04-25T15:29:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 msgr2=0x7fa6780cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa6780cf350 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa67000e820 tx=0x7fa67000ece0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 shutdown_connections 2024-04-25T15:29:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7fa6580744b0 0x7fa6580769c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6781fa290 0x7fa6781fa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6780cce40 0x7fa6781f6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 --2- 172.21.15.67:0/282788379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa664004320 0x7fa6780cf350 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 >> 172.21.15.67:0/282788379 conn(0x7fa6780d5d70 msgr2=0x7fa6781641b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 shutdown_connections 2024-04-25T15:29:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.474+0000 7fa67d9b2640 1 -- 172.21.15.67:0/282788379 wait complete. 2024-04-25T15:29:50.486 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:29:50.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:29:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.598+0000 7f1159297640 1 Processor -- start 2024-04-25T15:29:50.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.598+0000 7f1159297640 1 -- start start 2024-04-25T15:29:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1159297640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f11540d0e00 0x7f11540d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1159297640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f115406c5c0 0x7f115406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1159297640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115406cf10 con 0x7f11540b7810 2024-04-25T15:29:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1159297640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f11540d5a60 con 0x7f115406c5c0 2024-04-25T15:29:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1159297640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f11540d1850 con 0x7f11540d0e00 2024-04-25T15:29:50.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f11527fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f115406c5c0 0x7f115406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f11537fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f11540b7810 0x7f11540b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56288/0 (socket says 172.21.15.67:56288) 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f11537fe640 1 -- 172.21.15.67:0/77096378 learned_addr learned my addr 172.21.15.67:0/77096378 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1152ffd640 1 --2- 172.21.15.67:0/77096378 >> v2:172.21.15.176:3300/0 conn(0x7f11540d0e00 0x7f11540d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f11527fc640 1 -- 172.21.15.67:0/77096378 >> v2:172.21.15.176:3300/0 conn(0x7f11540d0e00 msgr2=0x7f11540d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f11527fc640 1 --2- 172.21.15.67:0/77096378 >> v2:172.21.15.176:3300/0 conn(0x7f11540d0e00 0x7f11540d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f11527fc640 1 -- 172.21.15.67:0/77096378 >> v1:172.21.15.67:6789/0 conn(0x7f11540b7810 legacy=0x7f11540b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1151ffb640 1 -- 172.21.15.67:0/77096378 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040599331 0 0) 0x7f1144001490 con 0x7f11540b7810 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f11527fc640 1 -- 172.21.15.67:0/77096378 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1154160de0 con 0x7f115406c5c0 2024-04-25T15:29:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.600+0000 7f1152ffd640 1 --2- 172.21.15.67:0/77096378 >> v2:172.21.15.176:3300/0 conn(0x7f11540d0e00 0x7f11540d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.601+0000 7f11527fc640 1 --2- 172.21.15.67:0/77096378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f115406c9d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f11540d5be0 tx=0x7f113c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed43a9ee35aad03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.601+0000 7f1151ffb640 1 -- 172.21.15.67:0/77096378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c037070 con 0x7f115406c5c0 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.601+0000 7f1151ffb640 1 -- 172.21.15.67:0/77096378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f113c032440 con 0x7f115406c5c0 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.601+0000 7f1151ffb640 1 -- 172.21.15.67:0/77096378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c02ae60 con 0x7f115406c5c0 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 -- 172.21.15.67:0/77096378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 msgr2=0x7f115406c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 --2- 172.21.15.67:0/77096378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f115406c9d0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f11540d5be0 tx=0x7f113c02a110 comp rx=0 tx=0).stop 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 -- 172.21.15.67:0/77096378 shutdown_connections 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 --2- 172.21.15.67:0/77096378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f115406c9d0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 --2- 172.21.15.67:0/77096378 >> v2:172.21.15.176:3300/0 conn(0x7f11540d0e00 0x7f11540d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 -- 172.21.15.67:0/77096378 >> 172.21.15.67:0/77096378 conn(0x7f11540d5d90 msgr2=0x7f11540d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 -- 172.21.15.67:0/77096378 shutdown_connections 2024-04-25T15:29:50.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.602+0000 7f1159297640 1 -- 172.21.15.67:0/77096378 wait complete. 2024-04-25T15:29:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.604+0000 7f1159297640 1 Processor -- start 2024-04-25T15:29:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.604+0000 7f1159297640 1 -- start start 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1159297640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f11541f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1159297640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11540b7810 0x7f11541f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1159297640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11540d0e00 0x7f11540cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1159297640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1154160ad0 con 0x7f115406c5c0 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1159297640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f11541613a0 con 0x7f11540d0e00 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1159297640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f11540b5320 con 0x7f11540b7810 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f11541f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f11541f77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41626/0 (socket says 172.21.15.67:41626) 2024-04-25T15:29:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 -- 172.21.15.67:0/3732996969 learned_addr learned my addr 172.21.15.67:0/3732996969 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f11537fe640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11540d0e00 0x7f11540cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 -- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11540b7810 msgr2=0x7f11541f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11540b7810 0x7f11541f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 -- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11540d0e00 msgr2=0x7f11540cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11540d0e00 0x7f11540cbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f1152ffd640 1 -- 172.21.15.67:0/3732996969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11541f6c10 con 0x7f115406c5c0 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.605+0000 7f11537fe640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11540d0e00 0x7f11540cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.606+0000 7f1152ffd640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f11541f77d0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f114c00c820 tx=0x7f114c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.606+0000 7f114b7fe640 1 -- 172.21.15.67:0/3732996969 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c015070 con 0x7f115406c5c0 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.606+0000 7f114b7fe640 1 -- 172.21.15.67:0/3732996969 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f114c007a30 con 0x7f115406c5c0 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.606+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11540cd020 con 0x7f115406c5c0 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.606+0000 7f114b7fe640 1 -- 172.21.15.67:0/3732996969 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c007e00 con 0x7f115406c5c0 2024-04-25T15:29:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.606+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11540cd5e0 con 0x7f115406c5c0 2024-04-25T15:29:50.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.607+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1154050740 con 0x7f115406c5c0 2024-04-25T15:29:50.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.608+0000 7f114b7fe640 1 -- 172.21.15.67:0/3732996969 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f114c018070 con 0x7f115406c5c0 2024-04-25T15:29:50.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.608+0000 7f114b7fe640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f11340744e0 0x7f11340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:29:50.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.608+0000 7f114b7fe640 1 -- 172.21.15.67:0/3732996969 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 4943+0+0 (secure 0 0 0) 0x7f114c094e70 con 0x7f115406c5c0 2024-04-25T15:29:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.608+0000 7f11527fc640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f11340744e0 0x7f11340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:29:50.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.609+0000 7f11527fc640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f11340744e0 0x7f11340769f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f113c034000 tx=0x7f113c02c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:29:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.613+0000 7f114b7fe640 1 -- 172.21.15.67:0/3732996969 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f114c05f900 con 0x7f115406c5c0 2024-04-25T15:29:50.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.836+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11540d4320 con 0x7f115406c5c0 2024-04-25T15:29:50.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.837+0000 7f114b7fe640 1 -- 172.21.15.67:0/3732996969 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+3875 (secure 0 0 0) 0x7f114c05f200 con 0x7f115406c5c0 2024-04-25T15:29:50.839 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:29:50.839 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6643,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2709900765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2709900765},{"type":"v1","addr":"172.21.15.67:6834","nonce":2709900765}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6652,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6836/4104906051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":4104906051},{"type":"v1","addr":"172.21.15.176:6836","nonce":4104906051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":6655,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2981673460","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6835","nonce":2981673460},{"type":"v1","addr":"172.21.15.176:6837","nonce":2981673460}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":52},{"gid":7473,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3091998462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3091998462},{"type":"v1","addr":"172.21.15.67:6835","nonce":3091998462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":59}],"filesystems":[]} 2024-04-25T15:29:50.839 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 59 2024-04-25T15:29:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f11340744e0 msgr2=0x7f11340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f11340744e0 0x7f11340769f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f113c034000 tx=0x7f113c02c000 comp rx=0 tx=0).stop 2024-04-25T15:29:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 msgr2=0x7f11541f77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:29:50.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f11541f77d0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f114c00c820 tx=0x7f114c00cce0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 shutdown_connections 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:6832/1591852742,v1:172.21.15.176:6833/1591852742] conn(0x7f11340744e0 0x7f11340769f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11540d0e00 0x7f11540cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11540b7810 0x7f11541f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 --2- 172.21.15.67:0/3732996969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f115406c5c0 0x7f11541f77d0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 >> 172.21.15.67:0/3732996969 conn(0x7f11540d5d90 msgr2=0x7f11541620d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 shutdown_connections 2024-04-25T15:29:50.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:29:50.839+0000 7f1159297640 1 -- 172.21.15.67:0/3732996969 wait complete. 2024-04-25T15:29:50.851 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-25T15:29:50.851 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T15:29:50.852 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:29:50.852 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:50.863 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:29:50.863 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:29:50.863 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:50.913 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:29:50.914 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:29:50.914 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:50.926 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:29:50.926 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:29:50.926 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:50.979 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:29:50.980 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T15:29:50.980 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T15:29:50.980 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.008 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T15:29:51.008 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T15:29:51.009 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.074 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T15:29:51.074 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T15:29:51.075 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.150 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T15:29:51.150 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T15:29:51.150 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.220 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T15:29:51.221 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T15:29:51.221 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.257 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T15:29:51.257 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T15:29:51.257 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.334 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T15:29:51.335 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T15:29:51.335 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.414 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T15:29:51.414 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T15:29:51.414 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.486 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T15:29:51.486 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T15:29:51.486 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T15:29:51.486 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.511 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T15:29:51.512 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T15:29:51.512 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.559 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T15:29:51.560 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T15:29:51.560 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T15:29:51.560 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.572 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T15:29:51.572 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T15:29:51.572 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.612 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T15:29:51.612 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T15:29:51.613 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:29:51.665 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T15:29:51.665 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T15:29:51.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-25T15:29:51.700 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:29:51.700 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T15:29:51.903 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:29:51.904 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T15:29:51.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:29:51.996 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T15:29:52.071 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:29:52.071 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T15:29:52.145 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:29:52.145 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T15:29:52.303 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:29:52.303 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T15:29:52.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:29:52.373 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T15:29:52.423 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:29:52.424 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T15:29:52.473 INFO:tasks.ceph:Archiving mon data... 2024-04-25T15:29:52.474 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/data/mon.a.tgz 2024-04-25T15:29:52.474 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-25T15:29:52.491 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.GhPoJiSpkh 2024-04-25T15:29:52.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GhPoJiSpkh 2024-04-25T15:29:52.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.GhPoJiSpkh 2024-04-25T15:29:52.737 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.GhPoJiSpkh is 463KB 2024-04-25T15:29:52.836 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.GhPoJiSpkh 2024-04-25T15:29:52.851 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/data/mon.b.tgz 2024-04-25T15:29:52.852 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-04-25T15:29:52.870 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.VplWYy7YSo 2024-04-25T15:29:52.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VplWYy7YSo 2024-04-25T15:29:53.070 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.VplWYy7YSo 2024-04-25T15:29:53.152 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.VplWYy7YSo is 599KB 2024-04-25T15:29:53.253 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.VplWYy7YSo 2024-04-25T15:29:53.267 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/data/mon.c.tgz 2024-04-25T15:29:53.267 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-04-25T15:29:53.321 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.YI6VohDHmO 2024-04-25T15:29:53.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.YI6VohDHmO 2024-04-25T15:29:53.499 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.YI6VohDHmO 2024-04-25T15:29:53.580 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.YI6VohDHmO is 599KB 2024-04-25T15:29:53.680 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.YI6VohDHmO 2024-04-25T15:29:53.694 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T15:29:53.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T15:29:53.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T15:29:53.779 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T15:29:53.811 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T15:29:53.811 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T15:29:53.813 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/remote/smithi067/crash 2024-04-25T15:29:53.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T15:29:53.846 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/remote/smithi176/crash 2024-04-25T15:29:53.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T15:29:53.874 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T15:29:53.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:29:53.891 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:29:53.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T15:29:53.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T15:29:54.020 INFO:tasks.ceph:Compressing logs... 2024-04-25T15:29:54.020 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T15:29:54.040 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T15:29:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47466.log 2024-04-25T15:29:54.062 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55694.log 2024-04-25T15:29:54.120 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41011.log 2024-04-25T15:29:54.120 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43459.log 2024-04-25T15:29:54.120 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T15:29:54.120 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.1.41011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41011.log.gz 2024-04-25T15:29:54.120 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.1.43459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43751.log 2024-04-25T15:29:54.120 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43459.log.gz 2024-04-25T15:29:54.120 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34715.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.1.43751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44808.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.43751.log.gz 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34715.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34715.log.gz 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.1.44808.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44808.log.gz 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mds.b.log: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34544.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34675.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34544.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T15:29:54.121 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34675.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34544.log.gz 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34675.log.gz 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose/var/log/ceph/ceph-mgr.x.log: -- /var/log/ceph/ceph-client.3.41408.log 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.3.41408.log: /var/log/ceph/ceph-mon.c.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.41408.log.gz 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42019.log 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42962.log 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.1.42019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42019.log.gz 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44276.log 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.1.42962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42962.log.gz 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34586.log 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.1.44276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44276.log.gz 2024-04-25T15:29:54.122 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34795.log 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34586.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34586.log.gz 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34504.log 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34795.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34795.log.gz 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34635.log 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34504.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34504.log.gz 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34635.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34635.log.gz 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34755.log 2024-04-25T15:29:54.123 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42483.log 2024-04-25T15:29:54.132 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34755.log: /var/log/ceph/ceph-client.1.42483.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34755.log.gz 2024-04-25T15:29:54.139 INFO:teuthology.orchestra.run.smithi176.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T15:29:54.139 INFO:teuthology.orchestra.run.smithi176.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42483.log.gz 2024-04-25T15:29:54.146 INFO:teuthology.orchestra.run.smithi176.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T15:29:54.163 INFO:teuthology.orchestra.run.smithi176.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T15:29:54.263 INFO:teuthology.orchestra.run.smithi176.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49660.log 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46512.log 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49660.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.47466.log.gz 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55694.log.gz 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51075.log 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49660.log.gz 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61645.log 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46512.log: /var/log/ceph/ceph-client.admin.51075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47340.log 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46512.log.gz 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61645.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51075.log.gz 2024-04-25T15:29:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63218.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61645.log.gz 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58167.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63218.log: gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.47340.log.gz --verbose 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.61304.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63218.log.gz 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48031.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58167.log.gz 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64045.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54576.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61304.log.gz 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64045.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.48031.log.gz --verbose 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.44254.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64045.log.gz 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40883.log 2024-04-25T15:29:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58594.log 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54576.log.gz 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44254.log.gz 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46947.log 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40883.log.gz 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51986.log 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58594.log.gz 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46099.log 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51986.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46947.log.gzgzip 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42437.log 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46099.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51986.log.gz 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61010.log 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46099.log.gz 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40357.log 2024-04-25T15:29:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61010.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42437.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51321.log 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40357.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61010.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58717.log 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51321.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40357.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61178.log 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59834.log 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51321.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61178.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58717.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50182.log 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61178.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.59834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48403.log 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.59834.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58932.log 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50182.log.gz 2024-04-25T15:29:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40799.log 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48403.log.gz 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62310.log 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50056.log 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58932.log.gz 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62310.log: gzip -5 --verbose 88.3% -- -- replaced with /var/log/ceph/ceph-client.admin.40799.log.gz /var/log/ceph/ceph-client.admin.56213.log 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62310.log.gz 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42521.log 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56213.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50056.log.gz -5 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47214.log 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42521.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56213.log.gz 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49026.log 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42521.log.gz 2024-04-25T15:29:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46309.log 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47214.log: /var/log/ceph/ceph-client.admin.49026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65261.log 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47214.log.gz 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49236.log 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49026.log.gz 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65261.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46309.log.gz 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57728.log 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61603.log 88.5% 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65261.log.gz 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49236.log.gz 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61441.log 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57728.log.gz 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.61603.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45805.log 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45763.log 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61603.log.gz 2024-04-25T15:29:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45805.log: /var/log/ceph/ceph-client.admin.41798.log 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61441.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63774.log 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45805.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60590.log 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45763.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63774.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41798.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63858.log 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63774.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41790.log 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60590.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52665.log 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41790.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63858.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55919.log 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41790.log.gz 2024-04-25T15:29:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56129.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3%/var/log/ceph/ceph-client.admin.55919.log: -- replaced with /var/log/ceph/ceph-client.admin.52665.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54870.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56129.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55919.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53058.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54870.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56129.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51715.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54870.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64571.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51715.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.53058.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.64487.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64571.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51715.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58251.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64487.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64571.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64445.log 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58251.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64487.log.gz 2024-04-25T15:29:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52453.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64445.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58251.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42785.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64445.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46991.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42785.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52453.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62394.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42785.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60548.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46991.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46597.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52028.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62394.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60548.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64361.log 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46597.log.gz 2024-04-25T15:29:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46057.log 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57896.log 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64361.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52028.log.gz 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46057.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64361.log.gz 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46057.log.gz 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52848.log 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57896.log.gz 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51841.log 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60490.log 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52848.log.gz 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51841.log.gz 2024-04-25T15:29:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56447.log 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60490.log: /var/log/ceph/ceph-client.admin.41140.log: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61521.log 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60490.log.gz 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56447.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48839.log 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56447.log.gz 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43834.log 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48839.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.61521.log.gz -5 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41146.log 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43834.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48839.log.gz 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40631.log 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43834.log.gz 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60322.log 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41146.log.gz 2024-04-25T15:29:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48755.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60322.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40631.log.gz -5 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57602.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48755.log: gzip -5 --verbose 89.0% -- -- replaced with /var/log/ceph/ceph-client.admin.60322.log.gz 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.47298.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48755.log.gz 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60244.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57602.log.gz 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47298.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47298.log.gz 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56171.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60244.log.gz 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63606.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58759.log 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56171.log.gz 2024-04-25T15:29:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45433.log 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63606.log.gz 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51379.log 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42059.log 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58759.log.gz 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51379.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45433.log.gz 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60674.log 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51379.log.gz 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63564.log 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60884.log 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42059.log.gz 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63564.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60674.log.gz 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49703.log 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63564.log.gz 2024-04-25T15:29:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42563.log 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60884.log.gz 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60632.log 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62586.log 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49703.log.gz 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60632.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42563.log.gz 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55650.log 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60632.log.gz 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51799.log 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62586.log.gz 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45349.log 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55650.log.gz 2024-04-25T15:29:54.325 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51799.log.gz 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65159.log 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56816.log 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34405.log 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45349.log.gz 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56816.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65159.log.gz 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59454.log 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56816.log.gz 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50641.log 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45595.log 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34405.log.gz 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50641.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59454.log.gz 2024-04-25T15:29:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44464.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50641.log.gz 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43574.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58675.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45595.log.gz 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44464.log.gz 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.2.43574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.43574.log.gz 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46639.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58675.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49320.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58675.log.gz 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46639.log.gz 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53250.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53184.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53250.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49320.log.gz 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44170.log 2024-04-25T15:29:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55320.log 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53184.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53250.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64781.log 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53184.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58209.log 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54492.log 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55320.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64781.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60758.log 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46141.log 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58209.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54492.log: /var/log/ceph/ceph-client.admin.60758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54492.log.gz 2024-04-25T15:29:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46141.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60758.log.gz 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54182.log 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41148.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46141.log.gz 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63816.log 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40231.log 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44548.log 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54182.log.gz 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40231.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63816.log.gz 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49884.log 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40231.log.gz 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54104.log 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49884.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44548.log.gzgzip 2024-04-25T15:29:54.329 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54104.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49884.log.gz 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58844.log 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54104.log.gz 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58844.log: gzip 87.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.48545.log 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40673.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58844.log.gz 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34751.log 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34751.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.51195.log 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48545.log.gz 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41144.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34751.log.gz 2024-04-25T15:29:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60202.log 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40443.log 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40443.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60202.log.gz 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50562.log 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57770.log 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.50562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42185.log 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50562.log.gz 2024-04-25T15:29:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40146.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42185.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57770.log.gz 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51944.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42185.log.gz 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52495.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51944.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40146.log.gz 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45553.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52495.log: gzip 88.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.51944.log.gz -- 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.55099.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45553.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52495.log.gzgzip 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45553.log.gz 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40188.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55099.log.gz 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48797.log 2024-04-25T15:29:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50308.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48797.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40188.log.gz 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53918.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57544.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48797.log.gz 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50308.log: /var/log/ceph/ceph-client.admin.53918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54408.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53918.log.gz 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2%/var/log/ceph/ceph-client.admin.57544.log: -- replaced with /var/log/ceph/ceph-client.admin.50308.log.gz 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56255.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58335.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54408.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57544.log.gz 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47172.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56255.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54408.log.gz 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53100.log 2024-04-25T15:29:54.333 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56255.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58335.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46723.log 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60448.log 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47172.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46723.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53100.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59913.log 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46723.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55961.log 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60448.log: /var/log/ceph/ceph-client.admin.59913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58293.log 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60448.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59913.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49833.log 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58293.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55961.log.gz 2024-04-25T15:29:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57418.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58293.log.gz 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49833.log: /var/log/ceph/ceph-client.admin.57418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44929.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49833.log.gz 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.c.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57418.log.gz 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42143.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44803.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44929.log: /var/log/ceph/ceph-client.admin.42143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44590.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44929.log.gz 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44803.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42143.log.gz 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61859.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44590.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44803.log.gz 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44590.log.gz 2024-04-25T15:29:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61346.log 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55873.log 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44212.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61859.log.gz 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65075.log 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61346.log.gz 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46807.log 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57130.log 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65075.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55873.log.gz 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54828.log 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46807.log: 89.0%/var/log/ceph/ceph-client.admin.57130.log: -- replaced with /var/log/ceph/ceph-client.admin.65075.log.gz 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61771.log 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46807.log.gz 2024-04-25T15:29:54.336 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57130.log.gz 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40273.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54828.log.gz 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46389.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61771.log: /var/log/ceph/ceph-client.admin.40273.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61771.log.gz 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46389.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40273.log.gz 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46389.log.gz 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52621.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59310.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56774.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43792.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52621.log.gz 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58125.log 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59310.log.gz 2024-04-25T15:29:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61136.log 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56774.log.gz 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52806.log 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43792.log.gz 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45931.log 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58125.log.gz 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61136.log: /var/log/ceph/ceph-client.admin.52806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62226.log 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61136.log.gz 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58802.log 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52806.log.gz 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45931.log: /var/log/ceph/ceph-client.admin.62226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59058.log 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45931.log.gz 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9%/var/log/ceph/ceph-client.admin.58802.log: -- replaced with /var/log/ceph/ceph-client.admin.62226.log.gz 2024-04-25T15:29:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60716.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58802.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60716.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59058.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63338.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60716.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41485.log: /var/log/ceph/ceph-client.admin.63338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63338.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46225.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.0.log: 93.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53438.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50991.log: /var/log/ceph/ceph-client.admin.46225.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56045.log 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.53438.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46225.log.gz 2024-04-25T15:29:54.339 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53438.log.gz 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48965.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61910.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56045.log: /var/log/ceph/ceph-client.admin.48965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44002.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56045.log.gz 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57172.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48965.log.gz 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61729.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61910.log.gz 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46892.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44002.log.gz 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57172.log: /var/log/ceph/ceph-client.admin.61729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62520.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57172.log.gz 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53016.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61729.log.gz 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46892.log: /var/log/ceph/ceph-client.admin.62520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49576.log 2024-04-25T15:29:54.340 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4%/var/log/ceph/ceph-client.admin.53016.log: -- replaced with /var/log/ceph/ceph-client.admin.46892.log.gz 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44887.log 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% 90.8% -- replaced with /var/log/ceph/ceph-client.admin.62520.log.gz 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53016.log.gz 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41681.log 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49576.log: /var/log/ceph/ceph-client.admin.44887.log: gzip -5 --verbose 88.8% -- /var/log/ceph/ceph-client.admin.41468.log 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49576.log.gz 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41681.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44887.log.gz 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41141.log 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41681.log.gz 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.41468.log: -- /var/log/ceph/ceph-client.admin.56900.log 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41141.log: gzip 87.9% -5 --verbose -- /var/log/ceph/ceph-client.admin.65301.log 2024-04-25T15:29:54.341 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41468.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41141.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63464.log 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65301.log: gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56900.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.52070.log 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63464.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65301.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49789.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63464.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51631.log 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49789.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52070.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41145.log 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51631.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49789.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43876.log 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51631.log.gz 2024-04-25T15:29:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42227.log 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43876.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41145.log.gz 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43134.log 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43876.log.gz 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57298.log 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.43134.log: gzip -5 --verbose 87.4% -- /var/log/ceph/ceph-client.admin.60926.log -- replaced with /var/log/ceph/ceph-client.admin.42227.log.gz 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43134.log.gz 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44086.log 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57298.log.gz 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42647.log 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51547.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60926.log.gz 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49495.log 2024-04-25T15:29:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.42647.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44086.log.gz 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54308.log 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49495.log: gzip -5 --verbose -- 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49495.log.gz 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.52537.log 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54308.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42647.log.gz 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54062.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54308.log.gz 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63690.log 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52537.log.gz 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.63690.log: -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54062.log.gz 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44632.log 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42017.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63690.log.gz 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49618.log 2024-04-25T15:29:54.344 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47126.log 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58974.log 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44632.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62142.log 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49618.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51902.log 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47126.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58974.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62142.log: --verbose -- /var/log/ceph/ceph-client.admin.42353.log 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51902.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62142.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52890.log 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51902.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44128.log 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42353.log.gz 2024-04-25T15:29:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55225.log 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44128.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52890.log.gz 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43708.log 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44128.log.gz 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55141.log 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43708.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55225.log.gz 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65343.log 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55141.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43708.log.gz 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62040.log 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55141.log.gz 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.65343.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44380.log 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62040.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65343.log.gz 2024-04-25T15:29:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56297.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62040.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63380.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44380.log: /var/log/ceph/ceph-client.admin.56297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65117.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44380.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63380.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56297.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63380.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.41801.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60406.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65117.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54786.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60406.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48199.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60406.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41800.log 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48199.log: gzip -5 --verbose 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54786.log.gz 2024-04-25T15:29:54.347 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.56381.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41800.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48199.log.gz 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40967.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41800.log.gz 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64171.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40967.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64171.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40967.log.gz 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.56381.log.gz 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48923.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64171.log.gz 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52196.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52974.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52196.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48923.log.gz 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63092.log 2024-04-25T15:29:54.348 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0%/var/log/ceph/ceph-client.admin.52974.log: -- replaced with /var/log/ceph/ceph-client.admin.52196.log.gz 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54224.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52974.log.gz 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41799.log 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63092.log.gz 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64991.log 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54224.log.gz 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55015.log 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41799.log.gz/var/log/ceph/ceph-client.admin.64991.log: 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48115.log 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64991.log.gz 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58064.log 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55015.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55015.log.gz 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48115.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48115.log.gz 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62268.log 2024-04-25T15:29:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52579.log 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62268.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58064.log.gz 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54744.log 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62268.log.gz 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34648.log 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52579.log: /var/log/ceph/ceph-client.admin.54744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47042.log 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34648.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52579.log.gz 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54744.log.gz 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34648.log.gz 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55831.log 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61952.log 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9%/var/log/ceph/ceph-client.admin.55831.log: -- replaced with /var/log/ceph/ceph-client.admin.47042.log.gz 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52932.log 2024-04-25T15:29:54.350 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55831.log.gz 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60800.log 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61952.log: /var/log/ceph/ceph-client.admin.52932.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61952.log.gz 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52932.log.gz 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41147.log 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54350.log 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41147.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60800.log.gz 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48713.log 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41147.log.gz 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60364.log 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54350.log: /var/log/ceph/ceph-client.admin.48713.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54350.log.gz 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48713.log.gz 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45889.log 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54660.log 2024-04-25T15:29:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45889.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60364.log.gz 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55523.log 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45889.log.gz 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56858.log 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54660.log: /var/log/ceph/ceph-client.admin.55523.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55523.log.gz 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54660.log.gz 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51153.log 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59184.log 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51153.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56858.log.gz 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64129.log 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%/var/log/ceph/ceph-client.admin.59184.log: -- replaced with /var/log/ceph/ceph-client.admin.51153.log.gz 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65201.log 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59184.log.gz 2024-04-25T15:29:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53142.log 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64129.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64129.log.gz 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45679.log 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65201.log.gz 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45307.log 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28355.log 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53142.log.gz 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45679.log: /var/log/ceph/ceph-client.admin.45307.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45679.log.gz 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45307.log.gz 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40841.log 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.28355.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.59538.log 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.28355.log.gz 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46554.log 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59538.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40841.log.gz 2024-04-25T15:29:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59100.log 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59538.log.gz 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59142.log 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46554.log: /var/log/ceph/ceph-client.admin.59100.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46554.log.gz 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59100.log.gz 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63422.log 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49415.log 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63422.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59142.log.gz 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46470.log 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63422.log.gz 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57686.log 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49415.log: /var/log/ceph/ceph-client.admin.46470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44506.log 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57686.log: 88.8% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46470.log.gz 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49415.log.gz 2024-04-25T15:29:54.354 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57686.log.gz 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49968.log 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8%/var/log/ceph/ceph-client.admin.49968.log: -- replaced with /var/log/ceph/ceph-client.admin.44506.log.gz 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46849.log 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49968.log.gz 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40547.log: gzip -5 87.8% --verbose -- /var/log/ceph/ceph-client.admin.59352.log 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64697.log 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46849.log.gz/var/log/ceph/ceph-client.admin.59352.log: 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43960.log 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59352.log.gz 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55608.log 2024-04-25T15:29:54.355 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64697.log.gz 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51033.log 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43960.log.gz/var/log/ceph/ceph-client.admin.55608.log: 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60842.log 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55608.log.gz/var/log/ceph/ceph-client.admin.51033.log: 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64739.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51033.log.gz 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60842.log: gzip -5 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60842.log.gz 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49745.log 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41484.log 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64739.log.gz 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52291.log 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2%/var/log/ceph/ceph-client.admin.41484.log: -- replaced with /var/log/ceph/ceph-client.admin.49745.log.gz 2024-04-25T15:29:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58419.log 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41484.log.gz 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44971.log 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52291.log: /var/log/ceph/ceph-client.admin.58419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57644.log 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44971.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52291.log.gz 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58419.log.gz 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose 88.9% -- /var/log/ceph/ceph-client.admin.58377.log 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44971.log.gz 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49278.log 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6%/var/log/ceph/ceph-client.admin.58377.log: -- replaced with /var/log/ceph/ceph-client.admin.57644.log.gz 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60160.log 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58377.log.gz 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48503.log 2024-04-25T15:29:54.357 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49278.log.gz/var/log/ceph/ceph-client.admin.60160.log: 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60160.log.gz 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.56732.log 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48503.log: gzip -5 88.8% --verbose -- /var/log/ceph/ceph-client.admin.62184.log 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48503.log.gz 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39969.log 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56732.log.gz/var/log/ceph/ceph-client.admin.62184.log: 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62184.log.gz 2024-04-25T15:29:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64655.log 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39969.log: gzip -5 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39969.log.gz 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63296.log 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40065.log: /var/log/ceph/ceph-client.admin.64655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6%/var/log/ceph/ceph-client.admin.63296.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64655.log.gz 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64403.log 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63296.log.gz 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64087.log 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44845.log 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64403.log: /var/log/ceph/ceph-client.admin.64087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52371.log 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64403.log.gz 2024-04-25T15:29:54.359 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%/var/log/ceph/ceph-client.admin.44845.log: -- replaced with /var/log/ceph/ceph-client.admin.64087.log.gz 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54002.log 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44845.log.gz 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42605.log 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52371.log: /var/log/ceph/ceph-client.admin.54002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62774.log 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42605.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52371.log.gz 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54002.log.gz 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42605.log.gz 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47382.log 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.62774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.62774.log.gz 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62853.log 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40715.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47382.log.gz 2024-04-25T15:29:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.0.56635.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51673.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.56635.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62853.log.gz 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56635.log.gz 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45973.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56339.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51673.log: /var/log/ceph/ceph-client.admin.45973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45475.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45973.log.gz 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51673.log.gz 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56339.log: 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56339.log.gz 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42395.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45475.log.gz 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56942.log 2024-04-25T15:29:54.361 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46267.log 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42395.log.gz 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52709.log 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7%/var/log/ceph/ceph-client.admin.46267.log: -- replaced with /var/log/ceph/ceph-client.admin.56942.log.gz 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose 88.7% -- /var/log/ceph/ceph-client.admin.64823.log 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46267.log.gz 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41479.log 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52709.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57460.log 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64823.log: /var/log/ceph/ceph-client.admin.41479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44338.log 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57460.log: 87.5% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41479.log.gz 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64823.log.gz 2024-04-25T15:29:54.362 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54973.log 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57460.log.gz 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63176.log 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57018.log 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44338.log.gz 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63176.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54973.log.gz 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58022.log 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63176.log.gz 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51505.log 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40315.log 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57018.log.gz 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58022.log: gzip/var/log/ceph/ceph-client.admin.51505.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48587.log 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40315.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51505.log.gz 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58022.log.gz 2024-04-25T15:29:54.363 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40315.log.gz 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59394.log 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63984.log 88.2% 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48587.log.gz 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63900.log 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59394.log.gz 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46765.log 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57980.log 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2%/var/log/ceph/ceph-client.admin.63900.log: -- replaced with /var/log/ceph/ceph-client.admin.63984.log.gz 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51589.log 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63900.log.gz 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46765.log.gz 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57502.log 89.6% 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57980.log.gz 2024-04-25T15:29:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41821.log 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5%/var/log/ceph/ceph-client.admin.57502.log: -- replaced with /var/log/ceph/ceph-client.admin.51589.log.gz 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63942.log 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57502.log.gz 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63732.log 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41821.log: /var/log/ceph/ceph-client.admin.63942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61094.log 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63732.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41821.log.gz 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63942.log.gz 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63732.log.gz 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61994.log 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60118.log 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61094.log.gz 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55565.log 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61994.log.gz 2024-04-25T15:29:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41483.log 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60118.log: /var/log/ceph/ceph-client.admin.55565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45091.log 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41483.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60118.log.gz 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55565.log.gz 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50266.log 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41483.log.gz 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40925.log 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45091.log.gz 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50266.log: gzip/var/log/ceph/ceph-client.admin.40925.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51463.log 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40757.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40925.log.gz 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50266.log.gz 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-04-25T15:29:54.366 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42479.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47799.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51463.log.gz 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42479.log.gz 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60968.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47799.log.gz 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64529.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55400.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60968.log: /var/log/ceph/ceph-client.admin.64529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48361.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55400.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60968.log.gz 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64529.log.gz 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55400.log.gz 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45175.log 2024-04-25T15:29:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.48361.log: /var/log/ceph/ceph-client.admin.48881.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45175.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48361.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65033.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45175.log.gz 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.48881.log: --verbose -- /var/log/ceph/ceph-client.admin.59226.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65033.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48881.log.gz 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63522.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65033.log.gz 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.59226.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51757.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63522.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59226.log.gz 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51421.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63522.log.gz 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51237.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51757.log: /var/log/ceph/ceph-client.admin.51421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57214.log 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4%/var/log/ceph/ceph-client.admin.51237.log: -- replaced with /var/log/ceph/ceph-client.admin.51757.log.gz 2024-04-25T15:29:54.368 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51421.log.gz 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51237.log.gz 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48157.log 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50098.log 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57214.log.gz 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63648.log 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48157.log.gz/var/log/ceph/ceph-client.admin.50098.log: 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47256.log 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50098.log.gz 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48445.log 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58514.log 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47256.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63648.log.gz 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48445.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47256.log.gz 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52112.log 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2%/var/log/ceph/ceph-client.admin.58514.log: -- replaced with /var/log/ceph/ceph-client.admin.48445.log.gz 2024-04-25T15:29:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55789.log 88.3% 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58514.log.gz 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40487.log 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52112.log.gz 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46681.log 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55789.log: /var/log/ceph/ceph-client.admin.40487.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55738.log 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55789.log.gz 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49926.log 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40487.log.gz 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55738.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46681.log.gz 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55738.log.gz 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50140.log 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41480.log 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49926.log.gz 2024-04-25T15:29:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64949.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50140.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46015.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41480.log: /var/log/ceph/ceph-client.admin.64949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59268.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4%/var/log/ceph/ceph-client.admin.46015.log: -- replaced with /var/log/ceph/ceph-client.admin.41480.log.gz 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64949.log.gz 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46015.log.gz 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45391.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59496.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59268.log.gz 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59580.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45391.log.gz 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50224.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59496.log: /var/log/ceph/ceph-client.admin.59580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59646.log 2024-04-25T15:29:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50224.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59496.log.gz 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62436.log 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50224.log.gz 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59580.log.gz 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52154.log 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47424.log 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59646.log.gz 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52154.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54450.log 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62436.log.gz 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52154.log.gz 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44422.log 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64907.log 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47424.log.gz 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44422.log: gzip -5 89.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.54450.log.gz /var/log/ceph/ceph-client.admin.56003.log 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48277.log 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44422.log.gz 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64907.log: /var/log/ceph/ceph-client.admin.56003.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64907.log.gz 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56003.log.gz 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54618.log 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48277.log.gz 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42101.log 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.54618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61262.log 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42101.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54618.log.gz 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42101.log.gz 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63134.log 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60076.log 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63134.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61262.log.gz 2024-04-25T15:29:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55057.log 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63134.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60076.log: gzip -5 88.2% --verbose -- /var/log/ceph/ceph-client.admin.61052.log 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60076.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42269.log 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61052.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55057.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61052.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52760.log 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43750.log 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52760.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42269.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52760.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48319.log 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48671.log 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48319.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43750.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48319.log.gz 2024-04-25T15:29:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53517.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63050.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53517.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48671.log.gz 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43270.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53517.log.gz 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51279.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63050.log.gz 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62100.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51279.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43270.log.gz 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54931.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51279.log.gz 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50949.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2%/var/log/ceph/ceph-client.admin.54931.log: -- replaced with /var/log/ceph/ceph-client.admin.62100.log.gz 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49110.log 2024-04-25T15:29:54.375 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54931.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50374.log 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49110.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50949.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49110.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49194.log 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40399.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50374.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50889.log 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57854.log 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49194.log: /var/log/ceph/ceph-client.admin.50889.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49194.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50889.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50014.log 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57854.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57854.log.gz 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57812.log 2024-04-25T15:29:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54702.log 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50014.log.gz/var/log/ceph/ceph-client.admin.57812.log: 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54534.log 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54702.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57812.log.gz 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57376.log 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54702.log.gz 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49068.log 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57376.log: 88.2% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57376.log.gz 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54534.log.gz 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44044.log 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49152.log 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49068.log.gz 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48629.log 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49152.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44044.log.gz 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49152.log.gz 2024-04-25T15:29:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44674.log 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55183.log 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48629.log.gz 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56087.log 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55183.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44674.log.gz 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55183.log.gz 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45049.log 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47720.log 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56087.log.gz 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45721.log 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.47720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47720.log.gz 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45049.log.gz 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64613.log 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45721.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45721.log.gz 2024-04-25T15:29:54.378 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47084.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48073.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2%/var/log/ceph/ceph-client.admin.47084.log: -- replaced with /var/log/ceph/ceph-client.admin.64613.log.gz 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53960.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48073.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47084.log.gz 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57256.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48073.log.gz 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54266.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57256.log: 88.0% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57256.log.gz 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53960.log.gz 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43918.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45847.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54266.log.gz 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61687.log 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45847.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43918.log.gz 2024-04-25T15:29:54.379 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45847.log.gz 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58888.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41482.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61687.log.gz 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55481.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41482.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58888.log.gz 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41482.log.gz 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62352.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62352.log: /var/log/ceph/ceph-client.admin.55481.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62352.log.gz 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55481.log.gz 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44296.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57938.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45637.log 2024-04-25T15:29:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57938.log: 89.7% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57938.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44296.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61220.log 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59016.log 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45637.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64865.log 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59016.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61220.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59016.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61815.log 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62478.log 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64865.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46183.log 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62478.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61815.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62478.log.gz 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41797.log 2024-04-25T15:29:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46183.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46183.log.gz 2024-04-25T15:29:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41797.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41797.log.gz 2024-04-25T15:29:54.382 INFO:teuthology.orchestra.run.smithi176.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T15:29:54.382 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T15:29:54.402 INFO:teuthology.orchestra.run.smithi176.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T15:29:54.430 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T15:29:54.438 INFO:teuthology.orchestra.run.smithi067.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T15:29:54.440 INFO:teuthology.orchestra.run.smithi176.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T15:29:54.447 INFO:teuthology.orchestra.run.smithi067.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T15:29:54.464 INFO:teuthology.orchestra.run.smithi176.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T15:29:54.466 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-04-25T15:29:54.466 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m0.392s 2024-04-25T15:29:54.466 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m1.812s 2024-04-25T15:29:54.466 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.099s 2024-04-25T15:29:54.484 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T15:29:54.529 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T15:29:54.621 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T15:29:54.623 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:29:54.623 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m0.571s 2024-04-25T15:29:54.623 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m2.315s 2024-04-25T15:29:54.623 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.379s 2024-04-25T15:29:54.624 INFO:tasks.ceph:Archiving logs... 2024-04-25T15:29:54.624 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/remote/smithi067/log 2024-04-25T15:29:54.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T15:29:55.337 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/remote/smithi176/log 2024-04-25T15:29:55.338 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T15:29:55.607 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T15:29:55.617 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T15:29:55.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T15:29:55.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T15:29:55.704 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T15:29:55.713 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T15:29:55.713 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:29:55.716 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:29:55.730 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:29:55.732 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:29:55.734 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:29:55.734 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-25T15:29:55.734 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +846us[ +845us] +/- 42ms 2024-04-25T15:29:55.734 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1301us[-1301us] +/- 79ms 2024-04-25T15:29:55.735 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -321us[ -321us] +/- 73ms 2024-04-25T15:29:55.735 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:29:55.736 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:29:55.736 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-04-25T15:29:55.736 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 +863us[ +878us] +/- 42ms 2024-04-25T15:29:55.737 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1286us[-1286us] +/- 79ms 2024-04-25T15:29:55.737 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -299us[ -284us] +/- 73ms 2024-04-25T15:29:55.737 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:29:55.737 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T15:29:55.751 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T15:29:55.752 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T15:29:55.761 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T15:29:55.856 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T15:29:55.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:29:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:55.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:29:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:55.983 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T15:29:55.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T15:29:56.002 INFO:teuthology.task.internal:Duration was 639.245269 seconds 2024-04-25T15:29:56.002 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T15:29:56.011 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T15:29:56.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T15:29:56.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T15:29:56.053 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:29:56.061 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:29:56.492 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T15:29:56.492 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:29:56.492 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T15:29:56.515 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:29:56.515 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T15:29:56.556 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T15:29:56.556 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T15:29:56.558 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T15:29:56.620 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T15:29:56.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T15:29:56.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T15:29:56.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T15:29:56.772 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T15:29:56.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T15:29:56.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T15:29:56.819 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T15:29:56.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:29:56.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:29:56.859 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-04-25T15:29:56.881 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-04-25T15:29:56.894 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:29:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:56.926 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:29:56.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:29:56.948 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T15:29:56.958 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T15:29:56.959 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/remote/smithi067 2024-04-25T15:29:56.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T15:29:57.001 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673213/remote/smithi176 2024-04-25T15:29:57.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T15:29:57.033 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T15:29:57.033 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T15:29:57.037 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T15:29:57.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T15:29:57.094 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T15:29:57.094 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T15:29:57.104 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T15:29:57.104 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T15:29:57.106 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T15:29:57.121 INFO:teuthology.orchestra.run.smithi067.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 15:29 /home/ubuntu/cephtest 2024-04-25T15:29:57.137 INFO:teuthology.orchestra.run.smithi176.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 15:29 /home/ubuntu/cephtest 2024-04-25T15:29:57.138 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T15:29:57.148 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T15:29:57.190 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quota} duration: 639.2452690601349 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T15:29:57.190 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T15:29:57.273 INFO:teuthology.run:pass